360 achievements unlock developer tips, DLC dates

GDC 2010: Researcher Bruce Phillips' detailed study of 18,000 players' habits, multiplayer usage, and achievement acquisition is helping Microsoft Game Studios make better games.

Who Was There: Bruce Phillips (User Researcher, Microsoft).

What They Talked About: Many people take their Xbox 360 achievements seriously. Nobody takes them more seriously than Bruce Phillips--because studying achievements is part of his job. As head of Microsoft's player-behavior research project, the Ph.D. in psychology spends his time studying how players play games on Xbox Live. By looking at how--and when--thousands of gamers rack up achievements, choose multiplayer modes, and slog through single-player campaigns, Microsoft is collecting information it hopes will help developers make better games.

Phillips gets his data from "The Panel," a group of roughly 18,000 volunteers drawn from Microsoft Game Studios' beta-testing program. Though about half are in the US, Panel members are spread all over the world, including such far-off regions as India, South America, Indonesia, and Sub-Saharan Africa. All are over 18 years old, and all willfully let Microsoft have access to their Xbox Live information in exchange for early access to games and free software.

In return, Microsoft gets data--mountains of it. Phillips' team knows every single Panel member's achievement record, which lets them track all manner of player behavior. They know when Panel members stop playing a game, they know how many multiplayer hours they've logged, they know when they're online, and they know when they're offline. The one thing they don't know is how Panel members interact with their friends, since that would violate the privacy of said friends who are not Panel members.

One might think all this data is being monitored via some sinister, supersecret, WOPR-like supcomputer that acquires it through some secret back door in Xbox Live. That would be wrong, according to Phillips. He said the information is collected via HTML parsing of public gamertag profiles on Xbox.com and via an XML feed provided by the Microsoft Community Developer Program. Though it's not open to the public, any developer can apply to get access to the tool to access the XML feed, and Phillips said third-party publishers also track data from there. The applications collect data every 15 minutes, and data of online players every five minutes.

Phillips said his first job after setting up the Panel was to make sure its data was accurate. So he compared its general gameplay statistics to those of Xbox Live, which boasts more than 23 million users. Both the Panel and overall XBL user base played the same games in the same amounts, with popularity bar graphs of the two being nearly identical. Phillips showed off a chart of achievement completion rates, which also showed the two in sync.

SINGLE-PLAYER STATS & GAME COMPLETION
Confident that the Panel data was statistically sound, Phillips and his team began to use it to answer questions about gamers' habits. The first metric was game completion. Using achievements as markers, Microsoft can determine how far members of its Panel get in a single-player campaign before quitting--and what difficulty they played the game at. By finding out how much of a game consumers play until they quit, the company hopes it can isolate the issue that caused players to stop--and keep it out of any future Microsoft Game Studios titles.

93.22 percent of Halo 3 players finish the first chapter. 71.82 percent finish the whole campaign.

As an example, Phillips showed the completion rates of two games--Halo 3 and Battlefield: Bad Company. His team measured the number of players who earned the first-chapter campaign-completion achievement in each game and how many completed the game. Some 93.22 percent of Halo 3 players finished the first chapter on normal difficulty, compared to 71.82 percent who finished the campaign.

By contrast, Battlefield: Bad Company saw only 39 percent of players finish the campaign in its entirety. He showed that each campaign-completion achievement saw a marked drop in continuing players--the first chapter was 70 percent, the second was 58 percent, and the third was 51 percent. "That's a very steep drop-off early on," he said. Phillips said Rainbow Six Vegas 2 had an even higher rate of attrition.

Microsoft can also track players' preferences via the achievement meta-data included with achievements. They can tell if the achievement is awarded due to completion, collection, questing, or gameplay. This also helps their in-house development team figure out what kinds of missions to put into games, since they can see which are the most popular.

Another graph presented by Phillips showed how a game--in this case Call of Duty: World at War--is played for the first 30 days. He presented a graph that showed that virtually all single-player achievements are earned in the first four to five days after a game is first played. After that, Zombie mode picked up speed, leaving co-op campaign a distant fourth after multiplayer.

MULTIPLAYER UNDER A MICROSCOPE
Unfortunately for single-player purists, by far the most popular game modes are multiplayer. Phillips and his team track the Panel's online play with a metaphorical microscope, tracking the popularity of each game's various modes and how they are used over time. Team Deathmatch dwarfed all other multiplayer modes for Call of Duty: World at War, with the others changing places at the bottom as people would try out different modes.

Call of Duty: World at War has a "game tenure" of 132 days.

Microsoft also measures which multiplayer maps the Panel plays. Phillips displayed a dizzyingly detailed chart that showed what percentage of the Panel played particular World at War maps, when they played them, and in which modes they played them.

The rate at which achievements are earned is also closely studied. For Modern Warfare 2 multiplayer, people earn about five achievements on average in the first 30 minutes, but then it levels off, with five achievements earned every 1.5 hours. Phillips said Modern Warfare 2 was an example of a game that did a good job rewarding players for continuing to play by continuing to dole out achievements, albeit at a slower rate.

"GAME TENURE" & DATING DLC
One of the most important concepts Phillips and his team have come up with by studying achievement data is "game tenure." This is the duration of time between the day players first fire up a game in their Xbox and the last time they play it.

The vast majority of game tenure loss happens in the first few days, due to people not liking the game or renting the game. Then it flattens out, since people keep coming back to play it over and over again. One of the games with the best tenure was Rock Band 2, and another was Call of Duty: World at War, with a whopping 132 days. The game with the highest game tenure? Geometry Wars: Retro Evolved, which players are still frequently playing nearly four and a half years after its release.

Nearly four and a half years on, Geometry Wars is still being played often.

Why is game tenure important? Because according to Phillips' research, DLC is more successful when it is released while players are still actively playing a game. That's why so many DLC packs are being released within weeks, days, or even hours of a game's launch. A good recent example of this (not provided by Phillips) is Dragon Age, which had DLC on day one and a full-fledged expansion, Awakening, just four months after the game hit the market. (A sequel also appears to be scheduled less than a year and a half after the original debuted.)

WHY?
Though Phillips' team definitely knows how to detect patterns of behavior from Xbox Live data, its ultimate job is summed up in a simple question: "Why?" Why do players stop playing when they do? Why do they prefer one multiplayer mode over another? Why will they spend hours pursuing one achievement but leave others untouched?

To answer those questions, the Panel is bombarded with surveys about their behavior. These surveys are sent out automatically and correspond to behavior triggers in a game. Most interestingly, they can be sent out days, weeks, or months after the event occurred, allowing developers at Microsoft Game Studios to pick the brains of players while fashioning a sequel or DLC for a particular title.

Quote: "Now, for the cool stuff."--Phillips, showing a slide of Arthur Fonzarelli from Happy Days.

Takeaway: Many people consider achievements "nerd cred." Microsoft and other publishers consider them clues to be closely studied on how to make a better game.

Written By

Want the latest news about Halo 3?

Halo 3

Halo 3

Discussion

70 comments
stugiff
stugiff

If they actually studies the online habits of a widespread audience they would have realised gamers like to create their own rules in online multiplayer matches. Forza 3 would be a enjoyable online experience instead of those stupid bloody "hoppers", I like to race better players than myself as it helps me get better, instead i get lumped in a lobby of ass-wipes who see my team pre-fix and proceed to take me out on track or boot me from the lobby. The only thing microsoft are interested in is taking your money and giving you crap in return.

Pablo620
Pablo620

To say that Microsoft invented achievements is ridiculous. Games had achievements well before MS made the system we know have. Every innovation comes from another idea. MS does deserve credit for what they moved it to

calm_syrface
calm_syrface

@armydominican69 yea i know the PS3 has trophies but they got that idea from the 360. My eyes lit up the first time I got an achievement, I'm pretty sure everyone else did too. The trophy system is just not the same because xbox live users love to show off their gamerscore, can anybody agree?

goez3000
goez3000

@ -Saigo- i couldn't agree with you more, there is a really good article about the success of warcraft and how the vp of blizzard uses strategy to master the tenure business... also love your avatar pic

-Saigo-
-Saigo-

Definitely an interesting thing to study...but if M$ was interested in creating a game with the longest "Gaming Tenure" they should be studying World of Warcraft.

Andrew_Wiggin
Andrew_Wiggin

I'd let someone study my gaming habits if I got free software... Where can you sign up?

calm_syrface
calm_syrface

I actually chose a 360 over a PS3 simply cause of Xbox Live and the achievements...Microsoft brought back points in a whole new way. Yall know yall like to hear that PLOP! and then "Achievement"

HollowNinja
HollowNinja

This is actually pretty cool. While I own a PS3 and not an Xbox, it's great to see that someone is studying what we like.

okassar
okassar

I didn't know companies went this far to analyze gamer behavior :/ I guess if it means a better product in the end that's great lol, but still creepy.

ferntfactor
ferntfactor

cuz it's their business and they want to improve all the time and be more profitable. thought this article was interesting. go WAW!

buft
buft

i just want to play video games, why does there have to be all these sciences involved ?

zyxomma100
zyxomma100

So... they actually pay some Ph.D. a ton of money for all this? Hell, if MS matched my current wages, I could give them tons of advice based on what all the achievement site members complain about and it would probably be more useful than what net benefit they get out of this guy. All of that "great info" that's in the article should have a big DUH tag attached to it.

big_and_dumb
big_and_dumb

You guys realize that it isn't big brother right? Maybe I'm missing sarcasm here but its a volunteer program. A study. So they can (hopefully) constantly improve things.

PixelAddict
PixelAddict

As an aside, I totally dig that they referenced how the World at War Nazi Zombies mode was more popular than the multiplayer. Nobody anywhere (other than Treyarch) talks about Nazi Zombies AT ALL. I personally thought it was better than the Left 4 Dead rubbish and started to worry that my friends and I were the only ones who played or cared about the Nazi Zombies add-on.

Riprock
Riprock

Just remember that they are only tracking those in the Beta program, those who agreed to be tracked. Big Brother isn't watching everyone. But yeah, it sounds like a pretty sweet deal. And I like getting as many achievements as I can that are "within range" (reasonably attainable if I pay attention, not the crazy ones where you have to jump through hoops or spend enormous time to get).

Angelspit_CH
Angelspit_CH

"Microsoft is collecting information it hopes will force players to buy more games." There, I fixed it for you guys.

Hey_Jay
Hey_Jay

Points junkies we are indeed.

fanboy999
fanboy999

Wow I never thought of achievements as trackers for gaming behavior. Be careful what you play now that big brother watches.

Pete5506
Pete5506

Something so easy, and yet people still try to get the points they can

drokmore
drokmore

Big brother is watching...

Gamesmasterx
Gamesmasterx

Wow, early games and free software, just for allowing them to know everything about what games you play, what modes and for how long? That's not even fair, lol. Gaining lots for losing nigh nothing.

raahsnavj
raahsnavj

"nerd cred" - How is it nerd cred when you can just boost them with some other loser to make it easy to pull off? I like the idea to study them to find out where people quit playing and why. They should include profile information where players can review games they have played, similar to the my nintendo network that would hopefully give clues to the puzzle. Keep it tied to the gamercard. Let us rate the achievements and give feedback. As it stands right now though, most achievements seem to be little more than a fake reward for fake bragging rights or a goofy way to pretend you have stats about behavior where the researcher gets to make all the assumptions regardless of the circumstances.

ninjaroach81
ninjaroach81

Oh, man...I hope that this doesn't lead to even shorter campaigns...this could be bad...

Aesahaettr
Aesahaettr

That was very interesting I would love to have known about that job type before I gave up on psychology to focus on music.

BenGenio
BenGenio

I used to play a lot to unlock achievements (unlocking more than 250G a day), but after a while it loses the purpose of playing the game for just the fact that the game is fun. So I stopped focusing on the achievements, I only collect them if it makes sense, and doesn't ruin the experience.

Trogeton
Trogeton

they need to put more than that range of 50 achievements standard on a game and 1000 gs points.. they should stop being lazy and move that up to 2000 and 100 achievements

green_dominator
green_dominator

Does this mean that the kill 1,000 enemies/people achievement reign of terror has come to an end??

jimbo913
jimbo913

As a psychologist working at a place I hate, and given my love of video games, I have to say that I want his job.

wolf0
wolf0

did not know this very cool i love me some achievements

ThAdEa82
ThAdEa82

Also, though I have GeoWars2, its a cool guy

ThAdEa82
ThAdEa82

yeah yeah. Im calling it. When multiplayer achievements pick up, they will be on a logarithmic scale

Aaron__1127
Aaron__1127

i hope not, kameo sucked lol. but i tell you what they can look at my achievement data, if it means i get to play some game betas lol

mayceV
mayceV

wait so if I get a lot of people to play kameo 2 it will be seen? and get a chance of a sequel?

Worm209
Worm209

Waldkrieger Posted Mar 12, 2010 12:44 pm GMT @Worm209 While your comment is obviously sarcastic and unrealistic I will say: Maybe you are unaware. But, some people actually enjoy to just set down, relax, escape, and play a game without a bunch of over the top challenge. Life is quite challenge enough. I am a hardcore gamer, but I personally do not enjoy playing games on harder settings. Why? Because that is not why I play games. I play them to relax, enjoy, escape. Not get pissed off and redo sections 50 times. That's just a waste of time and id rather go read a book. So, having games split into two sections, one for the easier, one for the harder, is actually a good thing. This way both sides are happy as there are different types of players. The same should go for achievements. One section for single players, one section for multi-players. Your gamercard could reflect that you are a more single player person and unlocked all single player achievements. -------------------------------------------------- Yes games should cater for both types of player in some way or other, i never said they shouldnt. I was suggesting that if they continue going the way they ARE going, it is only the casual player that is catered for. Not the other way round (which you thought i was saying) I just tell it like it is

whynotdairy
whynotdairy

I played fallout for over 200 hours. Not necessarily to get all of the achievements (which I did) but becuase the world was believable and intense. Not for a million dollars would i suffer through the halo 3 campaign again just to collect al the skulls or whatever...same goes for odst.

bschroth
bschroth

interesting. i'd think this would be a cool topic to go further into. and do the same for trophies of course. and steam.

pete_merlin
pete_merlin

I still play geo wars to this day lol Great game

sonicare
sonicare

Interesting read. Makes sense about why some games have DLC only weeks to months after release. Used to wonder about that, but this research explains it.

PixelAddict
PixelAddict

I was hoping they were using the aggregate achievement data to improve games (or make games better suited to gaming tendencies). This is good for the industry and good for us gamers. Finally somebody's paying attention to our gaming habits and game makers have tangible proof of their suspicions. I would totally be on the XBL panel.

Witchsight
Witchsight

Stats and videogames... some of my favorite things, interesting read. Ive turned off notifications for Achvs though, and i dont even think of them until after ive beaten a game. Just my personal preferance, but it feels much more natural to me.

immortality20
immortality20

I'm one of the rare people who only care about single player. Most multiplayer is tacked on and unneeded.

ParisSun
ParisSun

I hope the conclusion they gather from all of this is more multiplayer and less single player hours. If that happens, I'll stop playing video games.

BladeTrinity114
BladeTrinity114

The best achievements I have come across in a game have got to be the original Mass Effect. They were tough, but they were accessible at the same time. I still only have 485G on it, and I've completed it twice.

Waldkrieger
Waldkrieger

@Worm209 While your comment is obviously sarcastic and unrealistic I will say: Maybe you are unaware. But, some people actually enjoy to just set down, relax, escape, and play a game without a bunch of over the top challenge. Life is quite challenge enough. I am a hardcore gamer, but I personally do not enjoy playing games on harder settings. Why? Because that is not why I play games. I play them to relax, enjoy, escape. Not get pissed off and redo sections 50 times. That's just a waste of time and id rather go read a book. So, having games split into two sections, one for the easier, one for the harder, is actually a good thing. This way both sides are happy as there are different types of players. The same should go for achievements. One section for single players, one section for multi-players. Your gamercard could reflect that you are a more single player person and unlocked all single player achievements.