Reality Check: Reviewing Games Right Is Hard

Why timeliness and quality don't always go hand-in-hand.

by

You probably think that reviewing games is easy, especially on consoles. A decade ago you would have been right, but reviewing these increasingly complex games has become a tricky business in recent years, especially for those of us who want to be timely but refuse to compromise on quality. Back when Lara Croft was the new girl on the block and Crash Bandicoot was widely considered the unofficial mascot for the PlayStation, reviewing games involved little more than opening packages from publishers, playing the games contained within, and then writing about our experiences. Nowadays it's not nearly as straightforward. There are still games that we get to review in this way, but they're fast becoming the exception rather than the rule.

In stores next week. In our hands for review soon, hopefully.

For starters, contrary to what you might think, we don't always get copies of games far enough in advance of their arrival on shelves to facilitate release-date reviews. At the time of this writing, we've just received PlayStation 3 and PC copies of Brink (in stores today), we've yet to receive MX vs. ATV Alive (also in stores today), and it's looking like we might not get our hands on The Witcher 2: Assassins of Kings (in stores next week) this side of the weekend. Historically, receiving a game too late to turn around a review ahead of its release date was often indicative of a publisher's lack of confidence in its product. That's still the case occasionally, but increasingly even great games aren't being sent out for review in a timely fashion. In fairness, the fact that we insist on reviewing finished games on retail hardware probably doesn't help matters, but that's the only way we can be sure that we're reviewing exactly the same game that's going to be sold in stores. The unacceptable alternative is to base our reviews on code that isn't 100 percent final and that, more often than not, arrives in the mail with a list of caveats detailing problems that will really, totally, honestly be fixed before the game ships. Sadly, even our policy of reviewing retail copies of games doesn't always ensure that we have the same experience you will.

In September 2009, we posted our reviews of the Xbox 360 and PlayStation 3 versions of NHL 10 and noted that the latter's online play suffered from noticeable lag. That review was accurate when we posted it the evening before the game's release, but mere hours later, the PS3 servers had been optimized, and the multiplayer was silky smooth. We opted to change our original review on that occasion for reasons that are detailed in this blog post. That situation was a first for us, but the current prevalence of day-one patches (sorry, updates) means that it's not uncommon for games to change significantly after they're pressed and packaged for retail but before they go on sale. In October 2008 we added an update to our review of Fable II after testing the cooperative play that was added via an update on day one, and just this week we decided to delay our Brink review so that we could determine for ourselves the significance of a promised day-one patch.

Despite the fact that Brink's release date was brought forward by a week not so long ago, apparently because production on the game had finished earlier than planned, the arrival of our boxed retail copies was preceded by an e-mail detailing a number of known problems that would be addressed by an "automatically-applied day one update on release." These included "sporadic visual glitches at a distance," "texture pop-up," and "minor networking issues." Sure enough, all of these issues and more were evident when we started playing. Unfortunately for Bethesda and Splash Damage, publisher and developer respectively, the patch didn't hit until after the review embargo was up, so authors of the first reviews to show up online weren't able to review the "finished" version of the game. Indeed, according to Paul Wedgwood, Splash Damage CEO and game director, talking on the GameSpot UK Podcast earlier this week, "Some early US journalists were playing on a really broken, horrible build." He went on to say, "I think it had broken textures, broken network code, and broken AI and stuff," hopefully unaware that reviewers in the US were sent the exact same discs currently being sold in stores more than a week ago.

Significant (and purportedly significant) day-one patches are always a headache, because by choosing to wait for them we end up looking like we're "late" or "slow" (as opposed to "thorough" or "diligent") with our reviews. We face a similar problem anytime we review a game with a significant online component. Most publishers are good about sending multiple copies of games and organizing online sessions for us to participate in with other reviewers, but these sessions aren't always played on retail servers (we had to manually configure the DNS settings on our retail PS3 in order to test SOCOM 4: U.S. Navy SEALs, for example), and even when they are, it's impossible for us to know how well the game's servers will perform when thousands of players are online simultaneously, based on time spent with fewer than a hundred. Furthermore, even waiting until day one to test the game online (which we've done on occasion) is an imperfect solution, because there are generally a lot more players hitting the servers a lot harder during the first couple of days than there are a week later.

These problems are symptomatic of a much larger issue, which is that games are no longer static pieces of entertainment; they're not finished and then forgotten. Rather, many of them evolve over time, and in the process, they sometimes render reviews posted at the time of their release inaccurate. The nature of games has significantly changed, but the way that we review them, for the most part, has not. The After the Fact updates that we introduced on GameSpot in 2008 afford us a way to add updates to reviews that are in need of a refresh, but perhaps what's really needed is an entirely different approach.

Reality Check is GameSpot's recurring editorial column. Each week, members of the GameSpot editorial team sound off on current gaming events as well as various topics that surround the gaming industry.

Discussion

385 comments
Neil_Clancy
Neil_Clancy

@dendrosteve That's what GS used to do, all of the early-2000s video reviews show subscores. Not quite sure why they eliminated that, because I agree that subscores are helpful.

achilles123
achilles123

@bbmax88 Skipping your rude attitude which makes you sound like a horrible person to talk to, but what's with this TheSpoonFeed crap, their latest game review is from November 2010...

xSlider257
xSlider257

I love this editorial, especially considering how much controversy it elicits. Keep up the great work Justin and the rest of the GS staff, some of us actually think you're doing a bang up job.

jedikevin2
jedikevin2

@JusticeCovert This post sounds very good but it comes out a bit hypocritical considering this site has always done a horrible job of adding information after a review. Many games that are given horrible scores because of bugs, gameplay, or performance issues (Serious Sam HD close encounter, IL-2 Sturmovik, Elemental: War and Magic to name a few) get pounded on heavily to the point they are classified as horrible game pieces. Problem is that editors neither want or have the time to go back and use features like After the facts when the bulk of what degraded the review has changed significantly through patching If you added all "after the facts" or "review revisions" on gamespot you will see it is rarely used. A easy solution is to do reviews base on time and state that in the review article. This way, a person reviewing will not just assume a game is in such a sad state because a year ago it got a 4/10 because of bugs and performance issues. Also, do a after the facts statement on patches and issues resolved or just use the After the facts stating just patch notes with date they were released and DLC. I remember a time when this site would keep gamers informed on updates and patchwork on games (specifically PC gaming). As the console centric gameplay and atmosphere took over GS, that went away but as many know, console gaming has evolved now to the same patch works and bugs as Pc games in the past. This is the life of gaming these days and it really needs to be demonstrated and stated for people reading reviews of games.

JusticeCovert
JusticeCovert

@bbmax88: I have no problem dealing with publishers, but applying all of the pressure in the world to get a finished copy of a game doesn't help much if finished copies don't exist yet.

bbmax88
bbmax88

forget about the release date whiny banter -- as a journalist it is your responsibility to maintain the utmost professionalism and dedication to your chosen profession. just because you lack the professional rapport or confidence to put pressure on the publishers doesn't mean you can justify your lack of professionalism with a featured article explaining why you can't do your job to the best of your ability. Gamer Journalism is already viewed by most media outlets as shallow and sophomoric, and this just add fuel to that fire. I am tired of reading reviews by people who have only played a couple games in their life, and that goes for 1Up as well -- I used to listen to their podcast, but after the original crew left it became horrible -- just people who like games, not people who are actually gamers and have played all the classics that have influenced the games of today. Gamespot is no different -- you guys should be leading the pack but instead you are falling flat on your faces. If you want a real site I always go to TheSpoonFeed for my reviews -- they almost never do scores, and they actually give a detailed report on games, which they follow up on months later to figure out if the game is still worth playing. GS and 1Up are a wave of the past, it's no wonder they are underperforming fiscally every single year.

ddd87
ddd87

All reviewers need is just being professional in their job and unbiased. To me that is much more important than doing the review on time. If the review is biased in any way, I wont care how much time was used to write it anyway.

dendrosteve
dendrosteve

I don't purchase any game until the review comes out. I don't rely entirely on the review, but it plays a big part in my purchase. The review should reflect what you are given, but the review and score should change to reflect updates. If a game is realy buggy on release and gets a 6.0, but a month later those bugs are fixed then if the game deserves it the score should move up to an 8.0. I also like the idea of some sort of sub-scores. I agree with some other comments that pretty games tend to get higher reviews than a better playing game without the perfect 10 graphics. I'd like to know more about why the score is what it is.

Arther-la-Blunt
Arther-la-Blunt

I'm just wondering if it's still fun to play games when you're doing it with deadlines and for money. I mean after a long week of playing games do you go home and work?

u1tradt
u1tradt

I see where you're coming from, Justin. From your point of view it's hard to place emphasis on the quality of your interviews given the unpredictable nature of newly released games' intricacies nowadays. Definitely GS needs to approach their reviews at a different angle, I've been thinking this for a while now. The one thing that lets down a lot of the reviews here (particularly over the past 2 or so years) is the emphasis on the technical aspects of a game when judging the overall score. That's not to say that there shouldn't be any focus on this category at all, but I feel many times a little too much emphasis is placed on this aspect which results in the best looking games getting that extra 1-2.5 over games that play better but don't deliver absolute technical perfection; it's not about what you do but how you do it should be more prominent in the philosophy. I think maybe a sub-category for the scores should be reintroduced. I wouldn't want the exact same thing GS had before (I mean, really wtf was Tilt anyway?) but nevertheless maybe under the official score there should be an extra four or five criteria to score a game by that has no significant bearing on the overall scores but gives graphic junkies, music junkies etc. something which is important to them at a glance and really helps highlight the good and bad of the games in a somewhat more accessible way (not that the 'good and bad' feature should go anywhere though). By simply judging Gameplay (strictly how a game plays and feels rather than reading too much into the sum of its parts), Graphics, Music, SFX and possibly Multiplayer or Replay Value (or something of the sort) from a score out of 5 I believe you may greatly enhance the quality of your reviews in a small and relatively hassle-free way. Overall though, I think the reviewers would benefit from holding on to their finished reviews for an extra couple of days after completing them so they can rework it briefly for better retrospect overall. At least this way the reviewer has time to let the experience properly sink in before submitting the final draft.

radioactive1wiz
radioactive1wiz

Does anyone else remember the day when a day one patch would take a week to download? I hated that.

PeterDuck
PeterDuck

Just realized that my comment that was posted days ago on this board was deleted by gamespot, AGAIN. So I will attempt to re-phrase myself: "Ah please. We know that in most if not all cases GS reviews the XBOX version of the game first and then applies the same rating to PS3 and PC. Hell, in most cases the PC version of even the newest games dont get a review"

jim_uk
jim_uk

I think you should review what you are given, updating reviews when patches are released only encourages the "Sell now, (maybe) finish later" mentality that started with PC gaming and has now spread to consoles. By all means add a note to the bottom of a review about any patches but people have the right to know what a game is like out of the box, after all not all consoles are connected to the internet.

Diablo-B
Diablo-B

@Sepewrath - Why wouldn't redoing the review for the game be effective? If a game had good content but was unplayable due to glitches then they patch those glitches why wouldn't it be effective to tell people, "Hey remember those problems game X had, well now they are fixed and game X is great"?

Casual gamers don't purchase games based on reviews but instead they go after the most popular, well marketed game out during the holidays or when they have some extra cash laying around. There are people who will buy the game on release no matter what score it gets because its a game that appeals to them. There are those who will never play a game even if you gave it to them free cause they are prejudice/dislike something about it. The only people who are affected by a bad review are gamers who had doubts about a game and are taking a wait and see attitude. If you tell them that all the issues are fixed, Im sure they will more then likely change their minds.

Tekknight
Tekknight

One major problem is, many don't look to Review Scores to MAKE purchasing decisions. They look to them to validate their choice on a game they've already decided to buy or not buy. Some people feel better knowing the game they ALREADY purchased got high marks. And if they're not a fan they like to see it get low marks, which validates their "taste in games". Truth is the most important thing about a game is how enjoyable it is, and that's VERY HARD to quantify. There are some common factors that are known to contribute; controls, graphics, music (at least music is important to me), etc. Sometimes a game is STILL fun despite bad graphic, sloppy controls and boring music. But should a reviewer not consider that in their score? But the article touches on that very issue. Maybe the way Reviews are done IS becoming outdated. A game experience is more than the sum of its features. Still some people NEED that score to make an overall estimate of a game's quality. And again some need it for validation. But the people who need a review to precisely reflect their expectations, AS WELL AS their personal experience for a game AND ALSO be done AS SOON AS POSSIBLE, while simultaneously predicting how possible updates will affect their experience; are asking for TOO much. Those people will always be disappointed by GameSpot's or ANY scored reviews.

Wormkid_64
Wormkid_64

I thought perhaps he would explain sort of the mental reasons why some games get bad or average reviews,when the people and other reviewers praise it.Like the GS review for Super Scribblenauts. That was one of the worst reviews I've ever seen because Tom did something wrong,and then blamed the game's AI.Elsewhere it scored bunches of reviews in the 8.5 to 9.5 range. But no. This article was about timing and thoroughness. Whoop-de-flippin'-do.

matob27
matob27

[This message was deleted at the request of a moderator or administrator]

DarkTan01
DarkTan01

Knowing that the game you guys play for review are unfinished and buggy why review it in the first place?! personally i would've preferred a proper review and not one that is done to meet a deadline, i used to have high regards for gamespot but your review of BRINK is a bit mediocre because gamers that are actually playing the game loved it (check Youtube videos) The fact is that i usually rely on word of mouth or personal plays to have my own opinion on games.

ChromeGrill212
ChromeGrill212

Yeah i bet its hard for them to have the game reviews make as much sense as possible after when they get paid by game companies to give their games better reviews.

damnstraight003
damnstraight003

I get it, Gamespot. The big game developing and producing companies are paying you to say good things about their awful games. Don't feel bad, I'd do the same thing in your position.

Kratos_Kills
Kratos_Kills

I really think this article is like something "cover-up" operation. They were always reviewing 360 first, then others and now they try to show some excuses. I clearly remember in 2008 Lego Batman reviews, only 360 version was reviewed and even after release date they were yet to write a review for PS2,Wii or others... This is not something new, but obviously many people is complaining about it and now they are defensive. Some old games still dont have reviews... its just sad.

elmeray1124
elmeray1124

@slickvic1990 I agree to you. +9999999999999

method115
method115

@xReikenx You give Brink an 8 and it got a 7. I think that's a pretty close score, you make it seem like they were complete wrong.

D3thN1gh7
D3thN1gh7

@311frn311 Well all the CoD's are legitimately great games. Why people don't see this I don't understand. I loved CoD 4, loved Modern Warfare 2, and also equally love Black Ops. They're all great. Who cares if they play the same, they still add a lot of new features with each one. The gameplay is perfect so why change it? Anyway, as far as this article is concerned, it is quite correct. Games these days can change a lot from the time of release date to being out for a few weeks (or months). Reviews are still good though because if a game is great before release date that means it's going to be great when everyone gets there hands on it. Only to get even better as patches are released. :)

xReikenx
xReikenx

Day one reviews aren't really important except for indecisive people who aren't sold on the game but want to buy it the second it hit store shelves. I'd much rather have accurate reviews. Like Brink getting a 7? Sure it's got bugs but it's at least worth an 8. Hell I play it with an ATI card. I HAVE big, huge, noticeable graphical glitches. But I still love the game so much that I bare with it since I know it'll be fixed with a patch or a driver update in the near future. I've also never had any lag with the game, even prior to the patch. Now it just needs map packs, hopefully free ones but I'm sure those days are long gone by now.

311fran311
311fran311

Activision pays gamespot, they rate all CODs a 9. Other shooters pay nothing, they get 6s and 7s. Simple.

khimeron13
khimeron13

Why should it matter if they are day 1 reviews?? Unless u preorder a really hyped up game and it actually sucks... but otherwise i dont ever do day 1 purchases. I think the only game i purchased day 1 in the past 5 years waas Starcraft 2 and thats explanatory. I will probably preorder Diablo 3 also...

jhonel83
jhonel83

@desauce , to answer your question - I think MMO's should be the only ones to be allowed patches after release because they are expected to be played online. When I pick a game, I want to shove in the cd and enjoy it, like I used to do in the past. I don't want to have a good internet connection required, I don't want to overlook critical errors and I certainly don't want to pay for additional online content to make the game complete! Indie games should also be cut some slack because they are cheap as dirt or even free -you don't look the gift horse in the mouth. Every other games that get released unfinished are just pure crap and should be reviewed as such. They are just greed motivated and proof of unprofessional devs. The publisher consults with the devs for a release date, you know... you can't rush someone to make something faster then it's possible. The fact that most games get released like that nowadays is NOT an excuse!

JusticeCovert
JusticeCovert

@slickvic1990 @BTWilliam04 Your two posts next to each other are too perfect. One moment we're being asked to put quality before timeliness (which is what we do), the next we're being told "We just want a number up as soon as the game is available." You can never please everybody.

brunorr
brunorr

Well, the only constant is change. Since games are different and more complex it's only natural to change the way they are reviewed. I don't care to wait a week or 2 for a review. MMORPGs should have a review on launch and other at 6 moths and a year. Games with heavy online content should get a similar approach.

darkassassin911
darkassassin911

I don't really mind how long GS takes to review the games because I'd rather read a review thats accurate rather than one thats totally inaccurate. Oh, and also because i know that Gamespot's reviews are going to be great :)

Blankgen86
Blankgen86

I'm impressed with how you guys are handling this. I prefer an accurate review after the game's release date, rather than an earlier one that may not represent the final retail product.

bb64x
bb64x

Most of these ratings are just based on the opinion of others. The ratings go different for ourselves as video game players, Gamespot can give a 6.5 to a game, but the majority of the players rate it a 8.2 or higher, it's all varied that is why I am grateful for the Gamer Reviews. Sure, Gamespot can keep rating games on their opinion, but our opinion as players are helpful in everyway as well.

Shardz7
Shardz7

Well, I guess that explains why a lot of reviews are, quite frankly, way off like Mrs. Obama's wardrobe. Every time Brett Todd reviews a game, I simply add a +2 to his score and I have my number. Frankly, I go to Metacritic and read customer comments for the real unbiased truth from those who produced $50 to buy DRM riddled, bugged, uninspired corporate titles. The opinion of just one man is that man's opinion alone, not the Ten Commandments cast in stone. I just wish the sheep would realize that before funding more crappy games to fuel the junk pile.

Q8-KAMIKAZI
Q8-KAMIKAZI

take your time Gamespot i trust ur reviews

slickvic1990
slickvic1990

I hate to say it, but this edition of Reality Check just sounds like whining. It sounds like GS got a little too fed up with readers questioning their reviews, and decided to give a nice little "reality check". well, here's a little reality check for you GS. No matter what, a review should never be half-assed, and should always be a quality review, no matter the time constraints. Even if you have to delay it a bit, you owe it to your consumers to deliver the best possible product. Obviously there are factors like the ones you mentioned, but none of those should be used as an excuse. If you want to be taken seriously as a credible journalism website, you should spend more time to focus on making sure your reviews are of the best possible quality, and less time writing "whiny" pieces like these. anyways, just imo. it came off negative to me.

BTWilliam04
BTWilliam04

Wow, this post actually makes me angry. Excuses? Really? You are the last website to post reviews EVERY SINGLE TIME. I will say that I find your reviews to be the most realistic as far as ratings go, but I don't care what score a game gets three days after it comes out. By then I've already been to IGN or metacritic what have you to see how the game is. I've sent multiple comments that you guys need to review the game and get your scores out ASAP. You should be able to tell how good a game is in the first five hours. If the next day, after you are thirty hours in, you have some different opinions, update the review. No one will mind. You can call the original review your warmup. We just want a number up as soon as the game is available. You are a game reviewing website, stop whining about how hard your job is. YOU GET TO PLAY GAMES ALL DAY, SUCK IT UP!

jchristenberry
jchristenberry

I agree with yoda101280 I always look to gamespot to aid me in my decision to buy a game. Though my standards a little different. As with my experience in buying crap even after you guys call it crap. So I generally stick to 8.0 or higher before I am willing to fork over the cash. There has been the rare occasion of a game you guys rated well that I did not like or vice-versa, but it is VERY rare. So keep up the good work as I have no problem waiting a few days after the release of a game to see if its any good. Lastly for the pending attacks, its not to say that I dont form my own opinion. I have just found though several thousands of dollars spent on games and hardware that gamestops opinion of a game is nearly always the same as my thoughts of it. Im just not as nice when I put it into words. lol

yoda101280
yoda101280

Well I have to say I come to gamespot frequently and this is the first place I check for my game reviews, this page is bookmarked and is one of my hompage tabs on my browser. Not saying I always agree 100% with the reviews of some games but generally the reviews here are pretty good and I honestly have decided on purchases based on review scores given here. Basically any game that doesn't score at least a 7 I generally will not try, depending on the game I might let it slide with a 6.5. I've bought a couple of games that scored lower than that on a Gamespot review just b/c they were titles I was really looking forward to, so I bought them anyway and ended up not liking them. Most of the games that have scored at least a 7, while maybe they weren't ground breakers, I at least had fun with them. So that's why I make my cut off a 7. Not saying I'll never buy a game that scores lower but I do indeed always look for the review first and if I was on the fence about getting the game, if the score/review doesn't meet my expectations, I won't get the game.

akhorahill
akhorahill

@JusticeCovert : I'm glad that GS also thinks it doesn't have to apologize for reviewing the retail version of Brink and not a better, possible, not yet available version. I also saw the same reference of a day one patch on Game Informer's review and as the developer had sent you an e-mail acknowledging the game's rough edges, I thought that this feature was meant as an apology (to the developers) for sticking up to your system. I'm glad to be wrong. The review is well written and doesn't forgive the game's flaws, which I think it's the best and correct way to review a game. There's no way to envision a better version of the game. If the developer had such big plans to correct it, it should not have released the game on the first place.

RJJONES
RJJONES

The reviews are the main reason I come to gamespot on the daily. I appreciate this article and the effort it takes to render these reviews. My buying decesions are soley based off of Gamespot reviews. If it's below a 6.5, that game does not get considered.

StuGuinness
StuGuinness

Plain and simple, a product should be complete when hits the shelves. Not just thrown out there to be fixed on the go. It seems Bethesda is becoming more worried about a release date than putting out a solid product. Fallout New Vegas had horrible issues on its release date, so I took it back. Screw waiting on a patch. Get it right the first time.

TheLast1Left
TheLast1Left

I do appreciate the updates and patches that are latter available to fix issues that can only be found after hundreds of thousands of gamers have explored every aspect of a game. But, I do not think they should release a broken game with the intent of fixing it. If you know its broken release it after it is fixed. It also seems crazy to me that they do not even consider the gamers out there that still don't have an internet connection on their system. They are paying $60 just like the rest of us but have a broken game that may never get fixed.

popki
popki

Unfortunately jecht, a lot of people do rely on reviews, or at least the scores. There are tons of more casual gamers that still buy games that won't spend time reading reviews, but will look at scores at sites like gamespot or more commonly, metacritic. Regardless, what a whiney blog post. I keep trying to convince myself that Gamespot today is as good as it was four years ago...but it's just not. It has devolved into the embarrassment that IGN used to be back then, partially because of wishy washy approaches like this. What's being said in this post about how they receive game is true, but that doesn't mean it's "hard" or requires a different approach. It just requires picking an approach and sticking to it. You want to review retail copies on retail hardware with real world multiplayer? That's fine, then your reviews will be "true" but will come out later than competitors, but it gives you a chance to make the review blog more active and something to highlight on the front page. There will always be arguments for one way over another, because there's not really a right answer. But other sites don't seem to be struggling with the philosophy behind what they do; maybe Gamespot should consider that the best practice is simply having one you stick with.

terrain600
terrain600

Imagine if God of War 3 got a low rating. Now that's when Gamespot's taking things too far

terrain600
terrain600

@gamerxx89 it's a good game. Gamespot sucks

gamerxxx89
gamerxxx89

i swear to god i got lost planet 2 game bt still i hvnt played it judt becoz of review of gamespot.

Sepewrath
Sepewrath

@Diablo-B Yeah you can do that, but what's the point? If you update your review a month later after giving the game a 6.5 for being a broken mess, no one is going to care. Everyone would have already moved on to the next big game and deemed that game as trash. Hell you could redo the review a week later and it is too late. I don't know if this is the point their trying to make with this article exactly, but what I take from it, is people cant look at reviews like the be all end all; not anymore at least. Online play, patches, DLC, the game changes and even if the review does, the question is will anyone really notice. So even going as far as GS does(while admirable) is a system that is too old fashioned and inadequate for the current market.