• 13Dec 08

  Concepts For Making Music Through Videogames (a new way of looking at gaming)

  Making Music Through Video Games

  Written by Michael Manning

  mickdadude@gmail.com

  The Future's in Our Hands
  

  Just as author Isaac Asimov defined a series of rules and guidelines towards the future creation of artificial intelligence, so too does it seem that Apple are defining their own set of guideline as to the future of how we perceive what a portable device can be. With a strong enthusiasm for user created content and interactivity with its portable hardware and an aesthetic that is universally accepted, Apple are quickly making its mark as the leader of mobile devices, so much so that rival mobile phone and handheld gadget companies seem to be copying the template set by apple providing its users with similar features. Now that the machine is getting into the hands of more and more people, how does the software reflect the capabilities of the device? With such a large user base and effective use of new technologies, for the innovative programmer there is a potential for not only financial success through the digital shop the 'app store' but also for the invention of new ideas.

  Since its release on June of 2007 there have been thirteen million iPhone's sold. In comparison, the total number of Microsoft Xbox 360's sold after one year was one point five million units. I'm using the Xbox to compare because this is a machine that has an online service which matches that of the iPhone in terms of how consumers interact with it and each other. The Xbox has a functionality about it which could potentially enable its live connected user base to invent new software. It is the same for PC and Mac also but those platforms do not have a structured service or a universal interface which links each user to each other. The PC could use a service like Steam which is a game software distribution and player communication service. Which brings users together in a way that allows them to communicate, play games or share information with each other, but not all PC users use steam, and not all PC users use the same machines or operating systems to run it on. The Xbox live service or apple's handheld technology brings people together simply because all users have that service/device in common; this is quite a powerful thing.

  Firstly, as is the case with the iPhone, every user is familiar with the device, how it functions and how it links its various technologies together. Let's take an upcoming iPhone video game, Drug Lords. This is a fully online connected video game. The objective is to buy and sell drugs on the virtual street; the game uses the iPhone's in built messaging system to talk with both the computer AI and real life players.Here you can buy, sell or get a loan of money from the other players/AI.There is an exciting feature which uses the inbuilt GPS system to display where all the other real-life players are in your area. So you can see physically who is playing the game and where, some would say this is an infringement of privacy or personal space but you have the option to turn the feature on or off, it is in fact by default off. The game uses the familiar iPhone interface as a means of playing the game, so you don't need to learn much to start playing, it is as seamless as it would be to access your emails or send a text message. Fantastically, none of this interferes with your other iPhone apps, so you won't be getting messages from other drug lords while you are not playing the game. So here is a game that couldn't exist in the same way as it does here on the iPhone, by taking advantage of the inbuilt messaging, internet and GPS features the developers, (A-Steroids,) have crafted something unique.

  According to thinkabdul.com, the iPhone's hardware comes equipped with an ARM1176JZF chip with processing capabilities of up to 700MHZ, 8GB of Flash Memory, a 3.5inch colour screen, 2.0 Megapixel Camera, Wi-Fi and Bluetooth, rechargeable battery, Accelerometer and Touch Screen architecture. This is a relatively powerful machine as compared with other Phones or PDA's of its liking. There are other devices which surpass it for raw hardware statistics like the Nokia F800 but none can compare to Apples software distribution networks iTunes and the App Store.

  The iPhone itself runs off a modified version of OSX called OSX Mobile, this OS comes pre installed with a YouTube channel, GPS Mapping system, Weather, Clock, Calculator, Stocks channel, Calendar as well as the iPod, E-Mail, Phone and SMS functions. So out of the box already there is a suite of applications which make use of the phones capabilities. On the App Store you will find 3,456 third-party applications. This range from content made by independent developers or LTD companies, fantastically either party can decide what to charge for their content or to release it for free.

  Even though there is a large quantity of content readily available for the iPhone, a great deal of it are muddled, pick up and drop ****toys like a 50 pence Zippo lighter application which is simply an interactive animation of a lighter opening. Edge, a very respected gaming magazine did a feature on the iPhone and its applications saying, 'Despite the instantaneous success of the App Store as a money-making machine, there's something indistinct and gently underwhelming about a lot of its more playful output to date. IPhone games aren't bad, many of them just aren't terribly memorable; you may touch them, shaking, prodding, japing and tilting your way through marble runs, trivia quizzes and side scrolling shooter, but they often fail to have any lasting impact in return.' The music applications fall under the same bracket in that they do not provide a service like Fring which is a Hotmail and Skype messaging service. Also the line between music and game software's for handhelds have defiantly blurred in recent times as exemplified with Bloom and the work of Toshio Iwai.

  Toshio Iwai

  Toshio Iwai was born in 1962 in Kira, Aichi Prefecture, Japan. He has been described generally as an interactive media and installation artist. He studied Plastic Art and Mixed Media at the University of Tsukuba where he began producing installation art which combined early animation techniques with modern ones. In 1985 he won the Grand Prize at the 17 Contemporary Japanese Art Exhibition for his piece Time Stratum 2, Iwai himself explains 'In this installation, I placed 120 paper human figures on a motorised spinning disk. I set up a video monitor above them, while strobeing the light down, the paper figures all burst into motion. By using a video monitor as a strobe light, I could change the colour and speed of strobe lights with the music. The images are transformed with changes in the light.' This was taken from an autobiographical essay called A Short History of the Works. Toshio created his first videogame in 1987 called Otocky.

  Otocky has been credited by the Game Innovation Database, as the first Media art videogame. The game was developed for the Famicom Entertainment System and features colourful, abstract graphics which are impressive for the Famicom. You essentially play a side-scrolling shooter where you guide, what looks like an odd, alien superhero...Or something like that! The simple graphics of the time leave a lot for interpretation; it is only recently that gamers have not had to leave much to the imagination as they play. So as you control this superhero you can fire in multiple directions around you, up, down, left, right and the diagonals in between. It is here that the game fuses music with your actions. When you fire the multicoloured balls from around you, you trigger the sounds of the melody which accompanies the bass and drum backing track. So as you fly around and shoot things you are also dictating the performance of the melody. There is a blog post from a developer who works for Hand Circus which is a micro-studio which creates games and exhibits. In this post you can find his very informed reaction to playing Otocky including a video which I recommend you see.

  The post entitled Ode to "Otocky" (by Toshio Iwai) compares the game to that of a similar, more popular music/game hybrid Rez which was released first on the Sega Dreamcast and Sony Playstation 2 and is now featured on the Xbox Live Arcade store, Rez was created by Tetsuya Mizuguchi in 2002 (I also recommend you experience this game).'When I first played Rez, the connection between the audio and your ability to affect that audio as a player really made an impact on me. The way that each level was constructed so that elements were created to guide you to add percussive events to the soundtrack and so that sections of the level corresponded to sections of the main track felt so unique and such a suitable marriage.'

  The poster who, strangely is unnamed goes on to say, 'It's only after playing Otocky, created in 1987(!) that you realise that Rez certainly wasn't the first to explore this area. One of the things I love so much about Otocky is that you can't really go wrong with the composition you create. As you pass through the level, you pass through successive chord progressions, and the sounds generated by firing map onto a single note from the current chord, so it's always in tune. It also snaps to the nearest beat so that the note always hits on time - so while you feel total ownership of the notes created, the notes always sound good.'

  Toshio was a resident artist at the Exploratorium in San Francisco, California from 1991-92 where two of his works are now a permanent fixture of the museum, Well of Lights and Music Insects. Toshio describes Music Insects, 'You can see four small Insects walking on the screen. These Insects have their own sound. They are kind of walking musical instruments. Using the mouse, you can choose a colour from this palette; you can draw something like paint software. If an insect crosses a colour dot, the insect makes a sound and light pattern.'

  The concept behind Music Insects led to his work for the Nintendo DS which uses a similar interface. Electroplankton was released in 2005 and offered ten different interactive pieces where the player can compose their own music within the governed rules of the area. Edge describes the game as 'off the wall...In one part of the game, neon-pink tadpoles are shot toward a water fern. When a tadpole hits a leaf it makes a dulcet tone. Using the stylus, players can rotate the leaves to create a tune when the little guys hit them.' Other areas include; places where you spin in turn six disk-like creatures to create padded overtones, each generating different colours. A four channel sampler where you can make noises into the DS inbuilt microphone and the fish on-screen record the data and play it back looped. Electroplankton is an incredible example of how music software can make full use of the specific hardware it was designed for. This is not my idea of a fully fledged sequencer disguised as a game, but it's defiantly a creative, improvisational music focused game powerful enough to use as a performance tool. One of my main gripes with the game is the lack of an ability to record your performances; this requires an external device to record the sound and a separate one to record the visuals. The lack of this may have been due to the limited memory capacity of the Nintendo DS hardware.

  Unfortunately on release Electroplankton did not do so well only selling 5,000 copies in its first week in Japan, although it is speculated that this is because one of the DS flagship titles Nintendogs was sold in the same week, which reached figures of 150,000 copies sold. Aside from that though, Electroplankton has gone on to achieve critical success and a solid cult following, it is a game which is firmly placed in history as a fine example of interactive sound media.

  Toshio's latest work is in takes form of the Tenori-On. This develops the things he's learnt over the years with software and applies them to hardware. Released in 2006 and Co-produced by Yamaha this £684 hardware digital instrument could well be Toshio's proudest moment. It features a 16 x 16 LED button matrix which acts simultaneously as an input controller and display. Through this you can play and record over sixteen layers through multiple modes. You can give your melodies and beats characteristics which you could never have done with previous hardware for example; you can set up a sequenced loop, then press the line of a melody in time with this loop. The device will remember and repeat this line but, as the track bar reaches those notes again you could set it to make them bounce. So they bounce with a sort of gravity onto the bottom of the device. As well as this the device features the same LED's on the back which you can hold up to show to an audience. In an interview with Future Music, Toshio explains. 'I wanted to create a very unique hardware and software together. This is not just a musical interface, this is also a musical instrument...I have created software for the Nintendo DS, Electroplankton for example. But writing software is so easy, anyone can do it, but making hardware at the ''big factory'' [pause] 'it's a kind of dream!'

  So it would seem that Toshio Iwai has his heart set on the hardware/software synergy, I personally couldn't find that more encouraging for such a visionary developer of software. The ability to create purpose-built hardware for his software puts him in an extremely rare position which very few artists can find themselves in. If an artist is to have such power however, I'm glad that it is Toshio Iwai. He has constantly proven himself to be a figure of whom to take great inspiration from, and if the day comes when we are creating vastly different interfaces to make music on, we must take note of this man's body of work.

  Concept

  From my research over the years I have found there to be many ways to interpret sound. The information I've learned has completely re-shaped the way I approach my audio projects and, generally how I hear the landscape around me. It is my intention to create music software for the iPhone and other handhelds in the guise of a videogame, the ideas/concepts I have written take direct inspiration from what I'm about to talk about. I feel that these findings should be one of the cornerstones of when it comes to creating any audiovisual composition. So I'm going to briefly talk about listening modes and other concepts.

  French composer, film maker and critic Michael Chion in his book Audio-Vision talks about the three listening modes. 'When we ask someone to speak about what they have heard, their answers are striking for the heterogeneity of level of hearing to which they refer. This is because there are at least three modes of listening, each of which addresses different objects. We shall call them casual listening, semantic listening and reduced listening.' Casual listening, he describes, consists of listening to a sound in order to gather information about its cause. Semantic listening is what refers to a code or language to interpret a message; he uses spoken language as an example. This listening mode is similar to something that sound artist R Murray Schafer calls signals. The book, Tuning of the World and now sourced from a previous paper of my own says how signals are sounds which are used to communicate; cities are full of them, from the beeps you hear telling you it is safe to cross the road to the 'ding' the elevator speaks when you reach your desired floor. Finally, Chion talks about reduced listening, 'Pierre Schaeffer gave the name reduced listening to the listening mode that focuses on the traits of the sound itself, independent of its cause and of its meaning. Reduced listening takes the sound as itself the object to be observed instead of as a vehicle for something else.

  I don't want to elaborate too much on the subject but with this is mind you can break apart these new and emerging software's of Brian Eno and Toshio Iwai and place the audio into these separate categories.I feel that it is important to be aware of the information provided from Michael Chion and R Murray Schafer for when software is created with audio in mind. This allows you to think more deeply about what your sounds are doing in the software, what role they play and the meaning of their existence. I do not think that sounds should be placed just because they sound nice but they should have a meaning of their own; why do they sound nice? What is the relevance of its niceness within the context of the programme? This is only really pertinent for the music software which behaves more like a videogame, obviously for a programme such a Cubase, the ideal for these programmes is for the user to have total creative freedom, to start from a blank canvass.

  My realisations through the research I have found have shown that in order to create some software for which to make music on, aimed at non-musicians but in a fun, easy and vastly different way the creator has to inhibit certain limitations on the user. In Bloom, the notes available for you to play are pre-picked from a scale, so where you can choose to play a note, you do not have the freedom that you would have with a piano. It is impossible in Bloom to play the 'wrong note' if such a thing exists. So Musicality, is something you have to be aware of, most non musical people wouldn't know what keys fit with what modes or scales, in that regard there has to be a musical structure already in place within the programme. Making something fun is what I've learned from Electroplankton, if you can wrap up a neat, detailed and powerful music creation package within a videogame or similar format you then give the opportunity to people who would never before have much interest in composition or music in general to try it for themselves, even if they do not realise this themselves as they play!

  Finally, in preparation for the new technological evolution, dare I say, revolution, that is happening, designers, artists, musicians and programmers should take a distinctly different approach to software development. These new devices such as the iPhone or even handheld game consoles have their hardware designed in such a way that completely new ideas which couldn't have existed elsewhere can emerge from it. If we take the examples set by Brian Eno and Toshio Iwai including the fundamentals of how to work with sound, as talked about from Michael Chion and R Murray Schafer, plus a keen eye on what is to come, impressive, unimagined things can come. This could potentially change the way that people see, work and play with music and the world around them.

  I'm going to finish on a quote from Kevin Kelly, of Wired magazine 'Imagine Van Gough being born before the technology of cheap oil paints, imagine Hitchcock without the technology of film. Somewhere today, there are millions of young children being born who's technology of self-expression has not yet been invented. We have a moral obligation to invent technology so that every person in the world has the potential to realise their true difference.'

  • Posted Dec 13, 2008 6:02 pm PT
  • Category: Editorial
  • 0 Comments
  • Tag
  • Print
  • Email
  • Facebook
  • Digg

See Previous Blog Posts »