MUSIC FOR NEW MEDIA

Dec 1, 2000 12:00 PM, Peter Drescher

Tips and techniques for composing interactive music. Music has always been a linear art form. No matter what the venue, style, instrumentation, or medium, a piece of music always has a beginning, middle, and end, performed by musicians and listened to by an audience on an immutably sequential timeline.

Interactive media have changed the paradigm by letting the audience control the musical experience. User actions modify the music in computer games and Web sites. How can a composer create compelling music for a realm in which unpredictability and constant change reign supreme?

I spoke with several prolific high-profile composers in the game-music business to compare their Internet audio ideas and writing styles with my own. While most thought that good music makes for good soundtracks, they also agreed that composing interactive music presents challenges different from those you find when writing in more traditional genres. From games to the World Wide Web, thinking of an interactive musical score as a two-dimensional map or network seems to be the most useful approach.

The explosion of Internet-audio technologies has made the Web a hotbed of new interactive music programs. Macromedia's Flash and Shockwave are ubiquitous, and they allow cartoon-style soundtracks, sound effects triggered by mouse clicks and rollovers, and mixing of multiple audio streams. Although these browser plug-ins contain a high-quality MPEG audio-compression utility, their interactive music controls can be somewhat rudimentary. This isn't really surprising, because they focus mainly on low-bandwidth graphics.

On the other hand, the Beatnik technology (www.beatnik.com) was specifically designed for interactive applications. The proprietary Rich Music Format (RMF) contains a wide variety of controls that you can access from an HTML page through JavaScript. This enables the user to mute and unmute tracks; change volumes, instrument patches, and tempos; and switch between multiple files. (For an introduction to Beatnik and its interactive capabilities, see this month's "Desktop Musician: The Beatnik Player.")

One of the most popular Beatnik programs is Mixman's eMix (formerly GrooveGram), an online interactive remixer. Basically, eMix runs a large number of tracks simultaneously and lets the user mute and unmute them. Code records when the various tracks are turned on or off, so you can save your version of the mix and even e-mail it to a friend.

THE QUEEN AND WEIRD AL

Last year, I was contracted by Beatnik to produce RMF content for two remixes: Queen Latifah's rap song "Latifah's Had It Up 2 Here" and a Weird Al Yankovic parody called "Pretty Fly for a Rabbi." I was provided with samples from the original multitrack recordings (vocals, drums, guitar, and so on), which I had to edit and arrange so they could function together in appropriate and interesting ways while still retaining the flavor of the original linear recording. That meant creating up to 16 tracks of looped MIDI data that triggered the guitar and vocal samples and played bass, drum, and keyboard lines using the built-in Beatnik bank instruments, all at the appropriate times.

The challenges of adding interactivity was different for each song. The Latifah tune had a fairly simple structure: a bass line; a straightforward hip-hop beat; a few vocal, chorus, and string-ensemble lines; and the rap itself. The eMix format pairs two tracks in an A/B manner; you can turn on one of the tracks or turn off both. The limited source material went on the A track, and I wrote a new tune for the B track, which was intended to run in contrasting parallel to the original song. The tune consisted of a new bass line, a double-time drumbeat, a bluesy piano groove, and some additional percussion effects. The trickiest part was interleaving the rap and other vocals so they wouldn't interfere with each other if both tracks were turned on. You can hear the results at www.beatnik.com/remix/bn/up2here/standalone-launch.html. (Mac users running Microsoft's Internet Explorer cannot currently access this Web page.)

The Weird Al Yankovic song had a driving rock 'n' roll feel, offset by absurd lyrics peppered with Yiddish cliches. While it didn't have a lot of harmonic movement (basically just I-IV-V-I), one section did go into a half-time backbeat, and I used it to populate track B. I also added a klezmerlike clarinet to track A and accordion (Weird Al's signature instrument) riffs to track B. For the vocals, I had the basic verse lyrics on one track and created a novelty conglomeration from sampled bits and pieces to place on the other track. The result was an awful lot of bizarrely mismatched material, but because comedic juxtaposition was the name of the game, this remix ended up being quite amusing. Check it out for yourself at www .beatnik.com/emix/rabbi/standalone-launch.html. (As with the Queen Latifah remix, Mac users running Internet Explorer will not be able to access the site.)

METAMORPHOSIS

The most difficult part of putting together the eMix files was wrestling the linear parts into 16- and 32-bar loops that would work in harmony. I have always believed that it would be much easier to write music specifically designed to remix in interesting ways than it would be to "interactivize" linear songs. Drawing inspiration from a section of M. C. Escher's Metamorphosis II - which depicts bees that morph into fish that morph into birds - I wrote my own remixer, which you can find at www.twittering.com/metamorphosis.

The piece contains three different songs in three different styles. "Bees" is modal with a Latin beat, "Fish" boasts an up-tempo pop feel, and "Birds" is a bluesy/funky number in 6/8. Each song has similar track layouts (ostinato, melody, bass, chords, and percussion), so you can mix and match the various parts. For example, you could play the bass line from "Bees" with the "Fish" melody to the rhythm of "Birds." An author's mix provides a mechanism for cycling through 24 different track combinations that gradually morph the music from one song to the next. Users can also toggle tracks on and off to create their own mix versions.

JavaScript commands contained in the music-object.js file control interactivity. When the page finishes loading into the browser, the RMF file begins playing in a loop with all tracks muted. The various parts can then be turned on or off in response to mouse clicks - I set up that option using the setTracksMute() function. This simple but effective method of creating interactive music experiences is useful for a wide range of web applications, including soundtracks that change when visitors go from page to page within a site, audio interfaces that respond to user input, and online gaming.

HARMONY AND RHYTHM

I designed my remixer to provide as many musical possibilities as it could without creating complete cacophony. The most important part of this process was choosing key areas and chord progressions that would contrast without clashing. Keeping everything in a single key with the same chords is the easiest way to do this, and it works well with certain kinds of grooves. However, I wanted the piece to surprise the listener with unexpected harmonic and rhythmic movement.

I started working on "Bees" first, using a simple F maj/G maj/Am chord progression with a standard Latin-style bass line. Next, to create contrast with the first section's somewhat moody feel, I wrote the peppy, happy "Fish" tune in the key of C major with a II-V turnaround and a flowing flute solo. Then came the funky "Birds" section, which I put in the key of D minor, adding a bluesy Hammond B-3 riff and a fast, edgy synth-guitar solo. Thus, by putting all three songs in related "white note" keys, I minimized harmonic conflict, because a melody in C major or A minor usually functions over a Dm chord.

I also wanted three different rhythmic styles. The first section's Latin percussion ensemble plays a standard "2-3 clave" mambo beat. The rock backbeat of the second section can alternate between single and double time, easing the rhythmic transition from the mambo to the third section's 6/8 shuffle. Because a 150 bpm tempo continues throughout the piece, the bars of the 6/8 section are 112 times longer than those of the 4/4 sections. That means the bar lines don't line up all the time (in fact, they line up only every other time), which creates some interesting temporal shifts in the melodic and chordal parts when users switch back and forth between the sections.

SETTING LIMITS

Being limited to related keys or simple loops does not have to be creatively stifling; in fact, it's quite the opposite when viewed as an aesthetic challenge. In his book Poetics of Music, Igor Stravinsky says the "sea of possibility" is so huge and daunting that it's unwritable. But if you narrow your focus to, say, a single pebble on the shore, that you can write. "My freedom will be so much the greater and more meaningful the more narrowly I limit my field of action and the more I surround myself with obstacles," writes Stravinsky.

Letting the user click tracks on and off raises the issue of how to limit the choices to prevent too dense a mix, because turning everything on at once results in a tumultuous racket. Though I defined a few limits in "Metamorphosis" (for example, you can have only one bass line at a time), I usually leave the options open so the listeners can produce a "serendipitous cacophony" if they wish. Choosing what and when to play is one of the most important aspects of creating music.

Jeff Lipton of the Internet-audio company Sonicopia takes a different approach when producing "sonified" web sites, which are intended for a mostly nonmusician audience. The Sonicopia site (www.sonicopia.com) is a good example: a one-chord groove plays as background music on most of the pages, while various menu items and rollovers produce different musical notes. Users play along with the song by moving the mouse, and the notes are constrained to a pentatonic scale to prevent dissonances.

Lipton also extracts phrases from the background composition and applies them to different links. That way, mouse clicks can trigger flourishes in the song's key, and they can also trigger melodic motifs associated with the graphic elements. "It all works together and gives the user a sense of control over the composition, but within a narrowly defined set of parameters that you've set out," he explains.

MAPPING THE GAME

Musical scores for computer games present different kinds of challenges and limitations for composers, though many of the issues of how and when a musical transition takes place are similar to those surrounding Web-based applications. These days, many popular computer games are less like Space Invaders or Tetris and more like interactive movies. This style of "digital entertainment" frequently uses puzzles, characters, and shoot-'em-up scenes to move a story line forward.

In these situations, game music composers usually adopt a cinematic approach, producing scores intended to set up and sustain a mood. However, they must throw away any notion of locking to picture with SMPTE time code or employing film-style timing. During game play, the player can stay on a level for an indeterminate period of time and then branch to one of several environments. How then can a composer arrange for the music to flow appropriately from one section to the next?

Many game composers think of the score as a map or network, as opposed to the more traditional linear film-score form. Each level or environment is blocked out, and connections between the blocks delineate all the possible ways the player can go. The result - which can end up looking like a New York City transit system map - describes all the music that needs to be composed and helps the composer decide how to deal with the transitions between the pieces.

THE UNPREDICTABLE Composer Clint Bajakian calls this process "preparing for the unpredictable." To pull off this seemingly impossible task, composers create an ambient background score followed by a big flourish when the transition occurs. When properly executed, this tactic sends emotional information to the player and also smooths over any harmonic or rhythmic inconsistencies, creating the illusion of a seamless but responsive piece of music.

Tommy Tallarico, one of the industry's most prolific composers, describes the process. "Let's say you've got a Tomb Raider-type game where you're trapped in a cave, looking for an exit," Tallarico says. "I might have a 2-minute looping ambience in the background until the character pulls a lever to open a stone door. Then I'll quickly fade out the ambience and play a 4-second music sting, like a harp glissando or string crescendo, then quickly fade in a 30-second loop of suspense music while the character walks down the corridor behind the door. When the hallway opens into a big room - where 30 guys are waiting to attack - I'll hit with a big orchestral cue."

Tallarico isn't all that concerned with making the transitional flourishes happen on the beat or even in the same key. He avoids spending a lot of time and energy trying to get the pieces to fit together perfectly. Because the transitions are in response to the game play and are frequently intended to alert or surprise the user, a degree of abruptness can be a good thing. Also, the human ear wants to hear a musical flow and therefore tries to make the connection. "I'm a huge believer in just writing a great tune," Tallarico says. "It may not even match exactly what's going on at any given time in the game, but that's okay. If the player is really excited about what he's listening to, he's going to want to reach the next level just to hear the next tune."

MULTIPLE VIEWS

Bajakian takes an almost three-dimensional approach to interactive game soundtracks. He believes that aesthetically they are more akin to sculpture and architecture than to symphonies or film. Unlike an audience member who passively watches a movie from beginning to end, a player views a game from multiple angles while wandering around a virtual environment. The music must enhance that immersive experience.

Using multiple versions of the same tune is one technique for producing that feeling of immersion. In Peter McConnell's film noir score for the LucasArts game Grim Fandango, a song featuring a Benny Goodman-style clarinet plays in a secret casino hidden behind a cafe. As the Manny Calavera character moves through the game environment, the player hears the song in a variety of ways, not only in the casino, but also filtering through the cafe's walls, coming up from the floor of Manny's office, and broadcasting over an intercom system. This effective use of sound design makes the music seem to fit into a real-world setting.

Composer Michael Land also used multiple song versions in his score for Monkey Island II: LeChuck's Revenge. One of the environments is a shop-lined street, and a 32-bar song loop is heard as the player goes from place to place. However, each shop has its own version of the song, with different instrumentation and moods appropriate for the characters and atmosphere. The transitions from version to version take place on the bar lines. The subtle and effective technique makes each change practically unnoticeable, weaving a coherent musical structure while introducing sonic variety.

VARIETY OF STYLES The biggest challenge a game composer faces is the loop. "Repetition is the curse of interactive audio, and music is the first thing you will notice repeating," says game-music composer George Sanger. Just do the math: if a puzzle or level contains a 2-minute music loop and finding the solution or killing all the bad guys takes 20 minutes, the player will have to listen to the same tune ten times before moving on. (no wonder so many players reach for the Turn Music Off switch.) The average game can take as long as 40 hours to finish, which translates into an enormous amount of music, much more than a film needs.

Fortunately, a long-standing model for dealing with large-scale dramatic musical scores exists: Wagnerian opera. The Wagnerian concept of the leitmotiv (a musical phrase or chord progression associated with a character, place, or object) can help organize the huge amount of music required for a game. Orchestral music often works well for this application, thanks to its broad and flexible palette of tone colors. The percussive qualities of many of the instruments can also produce a wide range of textures and effects. Bajakian believes that the symphony orchestra lends a greater credibility to the game's subject matter. McConnell says that you can "practically cut from anything to anything else" in an orchestral score, an important advantage when he's editing a John Williams score for a LucasArts Star Wars game.

The game industry's other popular musical genre is techno/industrial electronic music, such as the score to the hit movie The Matrix, with its hard-driving drumbeat, dark textures, and heavy use of guitars. Although this overused style is on the verge of becoming passe, its powerful beat neatly solves some transitional problems. A strong rhythmic pulse can easily carry a player from one scene to the next, even if there's a momentary timing glitch or change in tempo. However, variety becomes even more important when the beat is so constant.

LOOP THE LOOP It's a real challenge to create a looping piece that doesn't sound like it's looping. One technique in Land's score for The Dig employs key changes and chord progressions in measures that are not multiples of 4. "When you combine odd phrase lengths with modulation, you can really hide a loop," Land says. "There's an art to writing ambient music where you create the effect of sparseness and use random bubblings of ideas that don't really have a coherent thrust to them to create a mood." McConnell expresses a similar sentiment, which echoes the old movie cliche that good scores aren't heard. "With puzzle games, it's good to learn how to graze," McConnell says, "so that after the stinger, the music fades into the background and becomes a sort of noncommittal audio wallpaper."

Tallarico has developed a technique for more energetic soundtracks. "I don't use a song structure," he says. "If you have only two minutes, there's no need to go to a chorus. I'll never play the same part twice if I'm writing a song that's going to loop for a while. The entire two minutes is your verse, and I try to fill it with unique parts." Tallarico's approach brings to mind composer Aaron Copland's experiences as a student of Nadia Boulanger. Copland writes, "She had but one all-embracing principle: the desirability of creating what she called la grande ligne - the long line in music. Much was included in that phrase, the sense of forward motion, of flow and continuity in the musical discourse, a feeling for inevitability."

Sanger also has a "more is better" attitude. "I get much more mileage out of just composing more music," he says. "The changes between two different musical phrases are much more satisfactory to me than changing the mix or muting tracks. In games, where so little thematic material is budgeted, it's best not to repeat at all."

THE INTERACTIVE FUTURE Writing music for interactive media is a fledgling art form. The production tools and delivery platforms are so new compared with those of traditional musical styles that it's difficult to predict what forms interactive music might take down the road. Nonetheless, a few of the current trends point the way toward future developments.

On the Web, it's all about bandwidth. Ubiquitous broadband access will eventually eliminate the need for highly compressed, audio-compromised files and allow for a much wider instrumental palette. Ultimately, a silent web site will resemble a silent movie. There will be many more opportunities to produce a broader range of music, sound effects, and voice-overs in the interactive medium. However, until standards evolve for delivery format and SMPTE-like synchronization, the Web will continue to be a Wild West-type frontier, with competing audio technologies shooting it out for market dominance.

In the game industry, scores produced by wavetable synthesis and MIDI are giving way to digital audio soundtracks. (this may happen on the Web as well, but not anytime soon.) As the speed and data-storage capabilities of game platforms increase, multiple-synched streams of CD-quality stereo audio will give composers and sound designers much greater fidelity and flexibility. But the most important development could be a system of standards that smooths the integration of music and sound effects into an interactive context - without involving the programmers.

New forms of music will certainly evolve to take advantage of new media capabilities. It will be interesting to hear what happens next.

---

Clint Bajakian composed scores for Outlaws, Indiana Jones and the -Infernal Machine, and Star Wars: Dark Forces, among many other games. He recently started C. B. Studios, an audio-production facility.

Michael Land and Peter McConnell were composers for LucasArts for ten years, where they scored The Dig, Grim Fandango, the Monkey Island series, and numerous Star Wars titles. They recently founded CoolerTool.com and are working on a better way to make digital media function interactively.

Jeff Lipton is a composer and engineer who has produced web-based interactive audio for MGM, McDonald's, KFC (PepsiCo), and PBS. He's vice president of sonification at Sonicopia, an Internet-audio startup located at San Francisco's DubeyTunes Studios.

George Sanger, aka the Fat Man, is a legend in the game-audio industry. He has produced soundtracks for Wing Commander, The 7th Guest, and Putt Putt Saves the Zoo and hosts Project Bar-B-Q, a Texas-style audio technology think tank.

Tommy Tallarico, of Tommy Tallarico Studios, is one of the most prolific composers in the game industry, with more than 120 titles to his name. He has released five CDs of his game music and won numerous industry awards. He hosts the weekly video-game television show The Electric Playground.