GDC 2009: Quality testing Halo 3

SAN FRANCISCO--Two years of consumer research. One thousand game testers. Thirty-five laboratory tests--close to double the amount many big-name titles have. Halo 3 was, according to Microsoft Game Studios Games User Research Group engineer John Hopson, the largest research project done by the division on a single title. Hopson used his speech at the Game Developers Conference recently to outline the mammoth testing process behind one of 2007's biggest hits, outlining what worked, what didn't, and how the final game changed as a result.

Consumer testing for Halo 3 started well before the game's launch in September 2007--almost two years, in fact, going all the way back to November 2005. Hopson, whose group had been responsible for testing the previous two Halo games, said Halo 3 was by far the most complex task of the trio. Where the original Halo had only eight laboratory studies, which focused only on the first hour of gameplay, Halo 2 ramped it up considerably by having 25 lab studies that focused on the entire single-player campaign. But even this was to be dwarfed by the third game in the franchise, which covered both single-player and multiplayer for a total of 35 separate studies.

Early testing focused on usability and was conducted well before Halo 3 was in any serious playable form. Researchers would have one-on-one sessions with testers who would try out specific discrete aspects of the game such as menu navigation before moving up to testing key moments within the game. One of the first to be tested was the first Scarab tank battle.

In initial tests, researchers basically let players loose against a Scarab, dropped some weapons on the ground, and then waited to see what strategies they would devise to bring the gigantic walking tank down. From this, Hopson's group realized that most testers didn't realize the Scarab's weak points, so the developers at Bungie changed how the target reticle moved over the weak points to give players a better indication. Some audio clues--in the form of dialogue prompts--were also added to the final game to help push players in the right direction.

But by far the most effective method of consumer testing, according to Hopson, was getting the game's developers in the same room as a group of testers while they were playing the game. "All of my research is nothing compared to having the developer being there to see how the players suffer as they play through their world," Hopson said, tongue only slightly planted in cheek. "It was very powerful stuff, but very hard to watch."

In fact, bringing the player experience closer to what the developer intended was the whole aim of Hopson's Games User Research Group when testing Halo 3. He said his team didn't try to tell Bungie how to make the game, but rather asked them what their objective was, testing it on consumers, and then seeing how closely the two aligned.

One of the key tools the team used was a visualisation map that plotted out every place testers died on any given level, as well as what they died from. This, Hopson said, gave Bungie a better idea of how the game flowed, whether there were any difficulty walls that players were coming across, and generally how well the game was playing. And speaking of difficulty, Hopson revealed that all of the testing for Halo 3 was done on normal difficulty, as the research was aimed at what the average to below-average player experience would be.