The Outer Limits

Aug 1, 2000 12:00 PM, MARTY CUTLER, GINO ROBAIR, AND BEAN

Over the years, innovative approaches to using physical gestures in electronic music have been highlighted in the pages of EM. Although some of the technologies we’ve covered have had mainstream commercial success, many technologies have remained—either by design or by accident—on the fringes.

However, like artistic and musical works, the commercial success of a new controller shouldn’t be the sole criterion for judging its worth. Like hammers and chisels, controllers are merely tools—albeit, in some cases, rather sophisticated tools that may require a major paradigm shift in order to understand their full potential. Ultimately, these new tools are only a means to a musical end; a controller’s effectiveness at getting across musical ideas will be the greatest factor in its success.

David Wessel, esteemed researcher of gestural controllers and director of the Center for New Music and Audio Technologies (CNMAT) at the University of California at Berkeley, put it best when he recently noted, “We’re on the verge of a controller renaissance.” This is primarily due to the growing number of musicians and engineers fighting to keep electronic music a unique medium of expression rather than a means of mimicking established forms. Many composers and musicians are using input devices that retain elements of traditional instruments, while others are tracking gestures in new ways by measuring motion, light, gravity, temperature, air pressure, proximity, and anything else you can imagine. Rarely in the history of music has there been so much work—and such varied results—in instrument development.

With that in mind, we decided it was high time to survey the current approaches to alternative input devices for electronic music. Some of the approaches are of the most personal and intimate kind, whereas others are geared for the mass market. This article will cover both extremes, as well as the universe of approaches in between. Commercially available MIDI controllers based strictly on conventional instruments, such as percussion pads, guitar, bass, and wind controllers, and variations on the standard piano keyboard (such as accordion controllers) will not be covered.

IT’S ALL ABOUT INPUT

Over the centuries, every culture has developed ways to express itself musically. By contrast, the field of electronic music, just over a century old, is perhaps moving into its adolescent stage, metaphorically speaking. Few of the earliest performance-based electronic instruments have survived this short test of time, the notable exceptions being the theremin and the Ondes Martenot. But until recently, technology imposed strict limitations on what a performer could do in real time.

These days, the tools needed to analyze and use multiple streams of input data in musical contexts are readily available and shrinking in size and price. In fact, everyday computer input devices, such as joysticks and graphics tablets, are sophisticated and inexpensive enough to work in musical contexts. And as musical-instrument transducer systems get smaller and more powerful, acoustic and gestural sensors are becoming easier to use with traditional instruments.

Already, external cable connections are beginning to disappear on instruments; external cables will disappear completely as microtechnologies become more available. Imagine the possibilities presented by miniature computers (including power supply and wireless transmitter) that are less than a cubic millimeter in size. This technology is what Kris Pister at U.C. Berkeley’s Department of Electrical Engineering and Computer Sciences terms “smart dust” (http://robotics.eecs.berkeley.edu/~pister/SmartDust): microtechnology that, perhaps within this decade, will have a major impact on real-time performance applications.

MIDI OR BUST

Even though technologies change rapidly, well-designed instruments never become obsolete.

Recent technological progress has allowed designers to use powerful processors in smaller spaces and to overcome difficulties in precise pitch extraction; however, they still come up against the bandwidth limitations of MIDI. No matter how small and unobtrusive the sensors are, MIDI still imposes a speed limit of 31.25 Kbps. And although faster data-transmission schemes abound, none have gotten enough popular support to dethrone the MIDI 1.0 specification. Yet engineers and artists have worked around the problems presented by MIDI to create input devices that allow new and exciting ways of making music.

CONTROL SURFACES

Fig. 1 : The MTC Express, from Tactex Controls, can track the Velocity and position of several stimuli simultaneously.

MTC Express. One of the newest and most promising touch surfaces being developed is a 3-D controller that uses smart fabric—a soft material developed by the Canadian Space Agency that contains an array of sensors interconnected by a network of fiber-optic cables. The MTC Express ($495), from Tactex Controls (www.tactex.com), is a square of smart fabric, measuring 5.75 by 3.75 inches, that is covered with a padded surface and housed in an anodized aluminum slab (see Fig. 1). The entire unit weighs just 17 ounces.

The MTC Express can track multiple contact points within a 2-D (x-y) field, with a sensitivity of around 100 dots per inch. It also measures 256 levels of pressure. The surface of the smart fabric is covered with numerous taxels, each connected to a pair of fiber-optic cables. Because of the sensor density, the MTC Express can track very subtle physical gestures.

Light from an LCD is sent down one of the cables and is returned over the other. Sensors are used to track the deformation of the fibers by measuring the amount of light sent back down the cables when the fabric is touched. As pressure is exerted on the surface of the MTC Express, variations in light are returned to the sensors.

FIG. 2: More than just a ribbon controller, the Kurzweil ExpressionMate can send Note On and a variety of controller messages.

ExpressionMate. At first, the Kurzweil ExpressionMate ($549) looks like the ribbon controllers that were used with analog synths in the ’60s and ’70s (see Fig. 2). But unlike older ribbons, this one can send an impressive array of MIDI messages. The ribbon surface is divided into three sections, each individually assignable. The unit also includes three 16-step arpeggiators that can send and receive data on separate MIDI channels. The control box can be conveniently mounted on the keyboard or on a mic stand.

Jam Bass. The Jam Bass ($253.50), by Kellar Bass Systems, is a MIDI controller (with internal synth) that attaches to the neck of an electric guitar or bass. The control surface is made up of two rows of 14 pads that mimic the fret layout below the E and A guitar strings. The pads are played with the thumb while the other fingers are on the fretboard. Using a ribbon cable, the control surface is attached to the Circuit Pack, which contains the processor, synth, and audio and MIDI outputs. You can change Synth Voice and Performance modes using the pads.

FIG. 3: David Wessel’s custom Thunder interface. Notice that some of the controllers have two sliders: one for vertical finger position, the other for pressure.

Thunder. Although it debuted more than a decade ago, the Thunder ($1,990), by Buchla and Associates, gives the performer a sophisticated touch-control surface. The thunderbird-style surface design is the result of ergonomic considerations—the layout comes from tracings done around the designer’s hands, and the long, featherlike control strips sit nicely under the fingers. Each of these control strips can track two control dimensions: pressure and position (see Fig. 3). All ten fingers can send pressure and position data simultaneously. Although the Thunder is no longer manufactured, a limited number of the controllers are still available directly from Buchla and Associates.

A more recent development by the company is the Marimba Lumina ($2,995), an instrument that combines the familiar design of a 31/2-octave mallet controller with advanced electronic technology and Buchla-style ingenuity. Designed by Donald Buchla with input from percussionists Mark Goldstein and Joel Davel, the Marimba Lumina is manufactured by Nearfield Multimedia, specialists in precision antenna technology.

The Marimba Lumina comes with four color-coded, foam-covered mallets that contain tuned circuitry. Embedded in each bar, strip, and pad on the surface of the instrument is a radio antenna that can track and identify the mallets, allowing each mallet to have independent control functions. Although cosmetically it bears some resemblance to a more conventional mallet controller, the Marimba Lumina is a highly sophisticated instrument capable of mapping a variety of responses to performance gestures over its various control surfaces.

A special “Gold Edition” Marimba Lumina ($8,000) is a 41/2-octave version of the instrument that features gold-plated bars and a curved frame that allows players to reach the furthest notes easily. (See “What’s New” in the October 1999 issue of EM.) Both instruments include a Yamaha DB51 XG synthesizer, so they can be used without an external sound source.

Finally, there is the Marimba Lumina 2.5 ($1,995), a new 21/2-octave version of the instrument (see “What’s New” in this issue).

Starr Labs controllers. Harvey Starr, who heads Starr Labs (http://catalog .com/starrlab), has designed a number of variations on the MIDI guitar controller theme. Avoiding the common pitch-to-MIDI schemes used in guitar controllers, Starr’s devices are more user interfaces than MIDI guitar controllers. Starr’s controllers often combine guitarlike fingerboards with keys on the neck instead of strings. Sometimes a breath controller and joystick are thrown in for good measure.

The following examples by Starr Labs represent only a fragment of the company’s output. Any of the configurations can be customized to fit the needs of the musician.

FIG. 4: Instruments from Starr Labs can be customized with various controllers to fit the performer’s needs. Shown here are the Mini-ZXS and Mini-ZS, which include Velocity-sensitive string triggers.

The Mini-Z ($1,395) is a Velocity- and Pressure-sensitive 24-fret fingerboard designed for tapping. Options include a 4-way joystick, a breath controller, and a programmable strip along the side of the neck that can add Modulation, Pitch Bend, or crossfades between sounds. The Mini-ZS ($1,795) includes the optional joystick and a set of six Velocity-sensitive string triggers. The Mini-ZX ($1,695) adds a set of drum machine– style trigger pads. The Mini-ZXS ($1,995) contains all of the above: joystick, trigger pads, and string triggers (see Fig. 4). Note that the prices quoted above are for basic models only; any added options or customization costs extra.

Starr Labs’ MT-48DD ($2,195) was originally designed for bassist Billy Sheehan, who wanted pedals that were better suited for bass playing than the traditional pedal configuration. The unit consists of a 4 5 12 array of 2-inch rubber mounds that are playable with mallets as well as with feet. Each mound is individually programmable and can send a MIDI event or group of events—up to eight per mound—including notes, chords, or even SysEx messages. The floor unit connects to a stand-mountable programmer that holds up to 32 user programs.

A collaboration between Harvey Starr, Stephen Taylor, and microtonalist Ervin Wilson has produced the Wilson 990 Generalized Keyboard Controller ($7,500). Inspired by the Generalized Keyboard that R.H.M. Bosanquet developed in 1875, it is designed to work both as a microtonal performance controller (with the ability to map multiple tonal systems) and as a more traditional controller offering multiple mini-keyboards on a single surface.

The Wilson 990 includes nine ranks of 90 keys. Each rank is assigned its own MIDI channel—in fact, each key can transmit multiple user-defined MIDI messages. The instrument comes with tuning software that is compatible with E-mu and Ensoniq sound modules, as well as Symbolic Sound’s Kyma system.

Groups of keys can be defined for different musical purposes, and different setups can be edited, stored, and recalled using a provided editor/librarian program. In addition to the banks of keys, the Wilson 990 has four assignable sliders and a 4-way programmable joystick. A smaller 288-key version ($3,200) is also available.

Besides these commercially available products, there are a couple of proprietary touch-sensitive controllers that offer interesting solutions to specific performance requirements.

JamODrum. Developed by Tina “Bean” Blaine and Tim Perkis at Interval Research in Palo Alto, California, the JamODrum is meant to inspire people to engage in spontaneous, collaborative music making. In fact, the community drum circle became a metaphor that guided the form and content of the team’s work. The designers also intended the JamODrum as a way for participants to explore the relationships between rhythm and graphics. The resultant object—a 7-foot circular table that includes six MIDI drum pads and doubles as a video-projection surface—was something that people could gather around for jamming.

Blaine and audio engineer Kris Force spent several months slicing, dicing, and processing thousands of samples to create a custom library for the project. During the JamODrum’s development phase, several interaction methods were available. For example, in the call-and-response method, the sequencer plays short rhythmic patterns that trigger synchronized flashing of the “call” area in the center of the screen. The call patterns are followed by space for players to copy the pattern, directed by response cues. “Your Turn” indicators allow everyone at the table to play together, to be split into subgroups, or to support solo sections. Once the players catch on to the system of when to play and when to listen, opportunities emerge for more-experienced players to improvise within the form. Although some players have found this rhythmic learning approach too structured to be entertaining, others have enjoyed its “Simon says” aspect.

Of the many interaction methods explored, the JamODrum designers found that call-and-response was the most successful in bringing novice and expert players together for musical collaboration.

A JamODrum installation that scales from 6 to 12 participants is currently on exhibit at the Experience Music Project in Seattle. A three-person JamODrum was recently donated by Paul Allen/Vulcan Ventures to the Entertainment Technology Center at Carnegie Mellon University in Pittsburgh (www.etc.cmu.edu).

Talking Stick. Under the direction of Bob Adams, a small group of musicians and technologists at Interval Research—including Geoff Smith, John Eichenseer, Jesse Dorogusker, Mark Goldstein, and guitarist/producer Michael Brook—joined forces to create a touch-sensitive, tubular instrument called the Stick (no relation to the Chapman Stick). The project combines customized circuitry with an array of force-sensitive resistors (FSRs) that take multiple data measurements with one touch. Because Brook intended the Stick to be played in a way similar to an acoustic bass, a vertical strip of FSR linear potentiometers was used in lieu of a fretboard. Location and velocity information is determined by the amount of surface pressure applied by the player’s fingers. To independently measure force and position, the team developed a custom library of Max patches for the Stick.

With plans under way for her work Songs and Stories from Moby Dick, Laurie Anderson, with the help of Bob Bielecki and in collaboration with Interval Research, extended the capabilities of the Stick controller. Repurposed specifically for Anderson’s hybridized approach to music, movement, and spoken word, the new Talking Stick features a linear potentiometer and a pressure-sensitive actuator for the manipulation of sampled audio, as well as a wireless transmitter for sending control information.

Anderson uses the Talking Stick to evoke the clicking patterns in the language of sperm whales and the creaks and groans of a ship. John Eichenseer, Lukas Girling, and Dominic Robson used Cycling ’74’s Max/MSP to create a variety of granular synthesis patches for the short sound fragments. During performance, these fragments are modified and resequenced in real time.

Besides the four Talking Sticks featured in Moby Dick, three other Sticks are in existence—one at Stanford University, one at the Berklee College of Music in Boston, and the one in the possession of Michael Brook.

MIDIBall. Fans rushing a stage at a concert in Tokyo inspired Candice Pacheco, cofounder of D’Cückoo, to design a gigantic beach ball that creates music as the audience bats it around. The MIDIBall, a wireless 5-foot sphere, converts radio signals into MIDI commands that trigger audio samples and real-time 3-D graphics with every blow. The biggest challenge was finding a plastic material that would be durable enough to protect embedded sensors and withstand heavy hitting, yet would appear to float.

The MIDIBall also required wireless technology with a threshold sensitive enough to reliably interpret a range of raps, slaps, and punches without double-triggering. After experimenting with several different sensors, Pacheco ended up inserting an RF transmitter in a sleeve sewn into the middle of the MIDIBall. The MIDIBall debuted at the Grateful Dead’s Mardi Gras show at the Oakland Coliseum in 1992.

STRINGS ’N’ THINGS

Research and development in the field of bowed-string controllers have been going on for some time; examples include IRCAM’s (Institute of Research and Coordination in Acoustics and Music) SuperPolm for violinist Suguru Goto, Tod Machover’s work with the hypercello, and Peter Beyls’s use of multiple infrared sensors on violins at Brussels University in Belgium.

However, the international ambassador of the extended violin is Jon Rose. Over the years, the English-born inventor/performer has implemented a number of technological innovations for the instrument and has also developed some unique “deconstructed” bowed-string instruments. Rose’s interactive setups, developed with help from STEIM (STudio for Electro-Instrumental Music), combine an accelerometer on the bowing arm, an ultrasound sensor mounted on the bow, a bow-mounted sensor that measures bow pressure on the strings, and three MIDI footpedals (www.euronet .nl/users/jrviolin).

BOLD AS GLOVE

Sophisticated glovelike controllers continue to be popular with performers of electronic music. At the end of the ’80s, Tom Zimmerman and Jaron Lanier’s DataGlove had scored some notoriety. When Mattel used the technology in its own junior version of the controller—called the PowerGlove—musicians such as Mark Trayle were able to hack into the greater potential of the $79 toy.

The Hands. In 1984 in the Netherlands, Michel Waisvisz began performing with an instrument he helped develop called The Hands. The original controller was made up of a group of keys and sensors mounted under his fingers and thumbs. The data collected by the sensors was translated into MIDI using a microcomputer and custom software that eventually became the SensorLab, marketed by STEIM. Over the years—with engineering help from Frank Balde, Johan den Biggelaar, Bert Bongers, Peter Cost, Tom Demeijer, Wim Rijnsburger, and Hans Venmans—Waisvisz has made incremental improvements to his gestural hand controllers. But Waisvisz keeps technological upgrades to a minimum. This allows him to master the performance techniques necessary for exploring the limits presented by the controller.

The latest version, Hands II, is a more refined version of the original controller—with upgraded wiring and components—that measures finger, hand, and arm movements. The distance between the hands is also measured, using ultrasound sensors. Hands III is currently being developed at STEIM by Jorgen Brinkman.

Lady’s Glove. In 1991, Laetitia Sonami (www.sonami.net) began her work on the Lady’s Glove by attaching magnetic sensors to the fingers of latex gloves. Since then, the Lady’s Glove has gone through a series of radical design changes, most recently with the help of engineer/designer Bert Bongers through a sponsorship from STEIM.

FIG. 5: Laetitia Sonami’s Lady’s Glove includes sensors that detect the distance between the glove and the floor.

In its present implementation, the Lady’s Glove (see Fig. 5) is a full-length Lycra glove with an accelerometer that measures hand speed; numerous motion and pressure sensors; and ultrasound transmitters and receivers that detect the distance between the glove and the floor.

Sonami uses the Lady’s Glove in live performances to control sound, mechanical devices, and lights via MIDI—mostly in solo situations but also in improvisations with other instrumentalists. Her current setup includes a STEIM SensorLab processor and a Mac laptop running Max/MSP.

WE SING THE BODY ELECTRIC

For decades, composers have been using electrical signals from the body as a source for electronic music. Most of these systems were built using parts originally designed for medical or scientific purposes. Although artists working with biofeedback continue in this manner today, a number of companies are marketing systems that are directly applicable to music.

BioMuse. Researchers Hugh S. Lusted and R. Benjamin Knapp have created BioMuse (www.biocontrol.com), an interface that analyzes and interprets the signals from up to eight simultaneous bioelectric sensors and translates them into MIDI data.

BioMuse has been used primarily with electromyographic (EMG) sensors that measure the flexion and extension of muscles. However, other sensors can be used with the system, including those that read eye movement (electrooculographic) and brain waves (electroencephalographic, or EEG).

FIG. 6: The BodySynth can track up to 12 EMG sensors simultaneously for translating muscle movement into MIDI data.

BodySynth. The BodySynth ($1,499), developed by Ed Severinghaus and Chris Van Raalte, uses EMG sensors attached to the body of the performer (see Fig. 6). The basic setup comes with four EMG sensors, a two-position switch, the Body Unit, a wireless transmitter, and a remote processor that handles the necessary A/D conversion and signal processing and includes a collection of algorithms for musical handling of the MIDI data.

Input from each of the four EMG sensors can be mapped to any of the remote processor’s eight MIDI output channels (or system channels, as the designers call them). The BodySynth can handle up to 12 EMG electrodes simultaneously.

Once they are attached to the body, the EMG sensors are plugged into the Body Unit, a device roughly the size of a cigarette pack that is worn by the performer. The Body Unit comes with four EMG amplifiers and a gain control for each of the four channels. The performer also wears a wireless transmitter that sends the signals from the Body Unit to the remote processor. Each BodySynth is configurable to fit the needs of the user.

FIG. 7: All of the I-CubeX sensors come prewired and ready to use—no soldering is necessary. As many as 32 sensors can be used with the system at one time.

I-CubeX. The I-CubeX ($625), manufactured by Infusion Systems (www .infusionsystems.com), is a comprehensive system that can translate up to 32 analog voltages into MIDI data from a varied host of gestural and environmental sensors (see Fig. 7). The I-CubeX Digitizer handles the I/O; besides MIDI information, the unit can also output 1-bit voltages (that is, 0 or 5 volts). In performance, the I-CubeX can be used with or without a computer. It has editors for both the Mac OS and Windows 95 that allow you to store sensor setups, enabling it to function as a stand-alone unit.

Each I-CubeX system comes with a Turn sensor and a See actuator. Additional sensors are available that measure temperature, light, pressure, acceleration, distance, and proximity; all are specifically created for use with the I-CubeX. For example, the TapTile is a 12-inch-square pad that measures pressure when it is stepped on or danced on. Another device, the TouchGlove, contains six sensors—one used in the palm and five on the fingertips.

GoFly/IRFly. Infernal Devices (www.infernaldevices.com) offers two environmental sensors in its Sensopede series of products. The GoFly ($95) is a tiny heat sensor (1.25 by 1.5 inches) that can be used to detect the motion of people in a room. The IRFly Ranging Detector ($59) is a 1.25-by-1.5-inch infrared detector that ignores other light and heat sources. Both devices work with processors made by Infusion Systems, with Beehive Technologies’ ADB I/O, and with STEIM’s SensorLab (see the sidebar “Preaching to the Converters”).

IBVA. IBVA Technologies (www.ibva .com) sells a system that couples EEG brain-wave sensors with Mac software specifically tailored to musical applications. The single-channel IBVA (which stands for Interactive Brainwave Visual Analyzer) Core system ($1,300) includes a headband with electrode sensors, a wireless transmitter and receiver, and a Pin Input extension pack for connecting other types of biofeedback sensors.

Software accompanying the Core system includes the Step 1 Expansion Pak (containing various control applications), Alps+ for brain wave–to–MIDI control, and a software synthesizer. In fact, the IBVA system complies with the General MIDI specification.

MIDIVox. An interesting technique for tracking the voice nonacoustically involves electroglottography (EGG). An EGG sensor measures laryngeal behavior through changes in electrical impedance across the throat. The MIDIVox ($1,295) uses EGG sensors for converting pitch and intensity data from the larynx as MIDI, binary, analog, and gate output. Invented by SynchroVoice, the MIDIVox is now available from HealingMusic.net.

The MIDIVox is made up of two components: a neck band (available in blue or black) with four biosensors, and a 1U rack-mount interface module. The hypoallergenic neck band is wrapped around the singer’s neck and affixed with Velcro so that two of the sensors are on either side of the Adam’s apple. The neck band attaches to the processor with a ribbon cable. Since its review in the July 1992 issue of EM, the MIDIVox has been upgraded with new motherboards and EPROMs, larger biosensors, and a new Velcro neck band.

TOUCHE PAS

Until the 20th century, the sense of touch has been one of the most important elements in playing musical instruments. With few exceptions (primarily the voice and the aeolian harp), earlier instruments required physical contact to make them work. Recently, however, this fundamental principle has changed.

Some gestural controllers operate by measuring the capacitance of an object. The capacitance (or ability to hold an electrical charge) varies based on the object’s distance from an adjacent object. Once the measurements of these changes are recorded, they can be converted to MIDI information with the assistance of A/D converters.

Other controllers use beams of light or ultrasound. For example, when an object is moved within a field of infrared light, optical sensors measure the increasing and decreasing amount of reflected light and generate MIDI data based on the values produced by these measurements. (For a more technical description, see Scott Wilkinson’s article “Interactive Light” in the December 1995 issue of EM.) This particular technology is not a recent development—several MIDI guitar controllers have been developed that measure reflected light as an alternative to pitch-to-MIDI conversion, where fingers placed on the fretboard interrupt an infrared beam, and the altered length of the beam is then converted to a MIDI Note message.

Radio Drum/Radio Baton. Developed by Bob Boie and Max Mathews, the Radio Baton and the Radio Drum track the 3-D movement of two or more batons over a base unit. The ends of the batons are covered with copper tape and topped with large foam balls that resemble mallets. Each baton transmits a discrete frequency that is localized by measuring the electrical capacitance between the tip of the baton and the array of receiving antennas embedded in the base unit.

The system allows performers to predefine behaviors of the batons/mallets. For example, Mathews wrote a conductor program to provide new ways of interpreting and performing traditional music scores. This software, coupled with the Radio Baton, enables singers and soloists to create their own orchestral accompaniment, process their voice, or work with algorithmic compositions. A program written by Andrew Schloss lets a performer conduct Standard MIDI Files. In this mode, the baton sends motion information to a computer, which then interprets the baton’s signals and sends MIDI commands to a synthesizer for playback.

An interesting example of a work created using the Radio Drum is composer David Jaffe’s pairing with Schloss in the duo Wildlife. With Jaffe playing a Zeta violin and Schloss using the Radio Drum, two computers interpret and respond to each player’s actions, superimposing the output of one instrument onto the other. For example, a glissando on the Zeta violin can change the pitch of notes played on the Radio Drum, or the drum can modify the output of the Zeta. Performing in such situations requires the musicians to develop new interactive skills, especially when their musical intentions are wrested away from them by other musicians.

FIG. 8: The Big Briar Ethervox theremin is a world-class performance instrument with extensive MIDI capabilities.

Ethervox MIDI theremin. The theremin is arguably one of the finest and best-known gestural controllers. Big Briar’s Ethervox (www.bigbriar .com) offers several enhancements to the traditional theremin, including the addition of more complex waveforms and a filter that can be modulated by pitch (see Fig. 8). The Ethervox theremin can control external sound sources and can be played from a MIDI sequencer.

In many ways, the Ethervox is fairly straightforward in its MIDI implementation. Pitch Bend is transmitted, as is Control Change 7 (volume). The device can also send and receive Note On, Note Off, Velocity, Program Change, and System Exclusive messages. As you would probably guess, the Ethervox sends a constant stream of Pitch Bend messages. Fortunately, the user can scale the updating of Pitch Bend and Volume information to thin out the MIDI data stream.

Big Briar’s most popular theremin, the Etherwave ($369 assembled; $299 kit), doesn’t have MIDI capabilities but does boast a 5-octave range as well as controls for waveshape and brightness, and it comes with an instructional videotape.

Dimension Beam. Developed by Interactive Light, the Dimension Beam uses an infrared beam to track the position of an object moving within its field. This invisible beam is shaped like an elongated egg balanced on one end, and it has 128 layers that can be assigned specific MIDI values.

One of the interesting aspects of the Dimension Beam is that, although the device may have seemed a bit esoteric when it was first released in the mid-’90s, it has since found commercial acceptance: Roland licenses the technology (now referred to as DBeam, and no longer sold to the public by Interactive Light) for use in many of its products, including the SP-808 Groove Sampler, the HPD-15 HandSonic drum controller, and the MC series of Groove Boxes.

Lightning II. Buchla and Associates’ Lightning II ($1,990) is a spatial controller that features a pair of wireless wands, a half-rackspace processor, and a stand-mountable remote receiver. The Lightning II uses infrared trigonometry to track the vertical and horizontal position as well as the velocity of each wand. It divides the performance space into eight Zones, configured in a 4 5 2 array. A Stimulus from either wand can be assigned to each of the Zones. Stimuli include movement within a Zone; entering or exiting a Zone; and clicking, double-clicking, or releasing the buttons. You can also use footswitches with the Lightning II, for yet another level of control.

The processor contains a 32-voice, 18-bit Kurzweil MASS sample-playback chip that provides a General MIDI sound bank. Each postage stamp–size memory card can store 30 presets. A preset can hold up to 40 patches that contain user-definable mapping of gestures to responses.

FIG. 9: The ultrasonic sensing capabilities of the Soundbeam 2 make the system ideal for situations that require noninvasive biofeedback.

Soundbeam. Developed more than a decade ago by EMS, the English company famous for its popular Synthi and VCS3 analog synthesizers, the Soundbeam (www.soundbeam.co.uk) is an ultrasonic MIDI-control system favored in special-education situations where noninvasive biofeedback is needed (see Fig. 9). With the recent release of the Soundbeam 2 ($2,777), EMS has enhanced the feature set of the original model to include the ability to track the speed of a moving object within the ultrasonic field, as well as proximity and on/off status. The Soundbeam 2 also allows you to divide each ultrasonic beam into 64 sections, each of which can be assigned its own notes, chords, or MIDI control parameter.

Up to four ultrasound beams can be used simultaneously, each with its own MIDI channel. The Switchbox ($250) sends data from eight additional controllers (such as switches and joysticks) to the Soundbeam 2 controller, and it works simultaneously with the beams. The beams have a variable range from 2 to 20 feet, whereas sensors can be placed more than 150 feet away from the controller. The Soundbeam 2 comes with presets containing pitch sequences, as well as a large bank for user-defined sequences.

OptiMusic. The OptiMusic system (www.optimusic.com) uses reflected, visible light beams as controllers and is also useful in physical-therapy situations. The system comes in two versions: the single-beam OM-L1 and the 7-beam OM-L7. Both versions are controlled by the OM-PCI light–to–MIDI control software.

The lights sense the reflection from performers moving within the beams. The color, shape, angle, sensitivity, and distance between the lights is customizable. The OptiMusic system allows up to 99 notes per beam and can work with up to 32 light beams. It will be available in September.

HOT OFF THE SHELF

One thing changing the alternative-controller landscape is that standard control devices from the graphics and gaming worlds are becoming increasingly powerful, plentiful, and lower in price. And, more and more of them feature Universal Serial Bus inputs. “In fact,” notes CNMAT’s David Wessel, “USB is just now beginning to mature as we speak.”

Among the devices popular with electronic musicians are the Wacom graphics tablets. For example, Wacom’s UD-1212-R offers, among other things, 32,000 points along an x-y axis and pressure and angle sensitivity. In addition, each UltraPen—the little stylus that is used as an input device—can have its own ID.

Matt Wright, musical-systems designer at CNMAT, uses a Wacom tablet when performing. In recent performances with Wessel and Pakistani singer Shafqat Ali Khan, Wright used the tablet to play additive-synthesis sound descriptions (using MSP) of Khan’s voice. Wright uses templates to indicate how the sound descriptions are mapped horizontally across the tablet. This allows him both to scrub through the sound in either direction (with the pitch kept independent of the scrubbing speed) and to visually locate particular musical parts so he can set the UltraPen directly onto them.

“Joysticks have also matured over the years,” comments Wessel, “and some include enough control options—buttons, triggers, movement direction—for sophisticated music making.” After some research into the subject, composer Bob Ostertag suggested to Wessel and CNMAT that the Cyborg 3D USB by Saitek (www.saitek.com) was one such controller worthy of investigation in this area.

BRAIN OPERA

Perhaps the most prolific source of research into alternative controllers is MIT’s MediaLab, under the direction of composer Tod Machover and Joseph Paradiso. The MediaLab’s many developments have moved beyond the ivory tower and are now used in prestigious venues around the globe.

Paradiso is the technology director of the Things That Think consortium, a group captivated with the infusion of intelligence into everyday objects. He also leads MIT’s Responsive Environments group, which has developed controllers used in musical/graphical board games, as well as “smart sneakers” that, when worn during performances, endow dancers with the ability to produce continuous musical output.

Since the mid-1980s, Machover has been passionate about augmenting the expressivity of traditional instruments—keyboards, strings, and percussion—with computer systems that measure and interpret human expression and feeling. These “hyperinstruments,” which are tailored specifically for professional musicians, expand the capabilities of existing instruments and redefine the ways in which people interact with objects that may or may not typically be associated with making music. As sensors, software, and signal processing have become more sophisticated, the development of hyperinstruments has evolved to include interactive musical instruments for amateur musicians as well.

In 1996, Machover created a work that blends live hyperinstrumentalists with audience participation—both live and via the Internet—in the polymorphous production known as the “Brain Opera”(http://brainop.media.mit .edu). After two years of touring, the Brain Opera and its associated gadgetry are still undergoing constant revisions and upgrades. Many of the controllers described here will become permanent installations in Vienna’s House of Music when it opens this summer.

For some attendees of the Brain Opera, the most engaging aspect is not the actual performance but the opportunities for hands-on musical interaction. This takes place within an eclectic assortment of sculptures known collectively as the Mind Forest.

FIG. 10: The translucent, pressure-sensitive rubber pads of the Rhythm Tree are an inviting interface for sonic exploration in the Mind Forest.

Rhythm Trees. From the gigantic pods of the Rhythm Trees hang 300 translucent rubber drum pads that trigger vocal samples when struck (see Fig. 10). Implanted in each pad is a pressure-sensitive piezo-electric strip that can accommodate a wide range of strike velocities. Networks of 30 pads are routed to a PC running custom software under Windows NT. As performers strike the pads, the software looks for patterns between players and generates new rhythms in response, creating a kaleidoscopic array of sounds, lights, and images in the process.

Singing and Speaking Trees. The Singing Trees respond with audiovisual feedback to the tonal quality of notes sung next to them. A well-sung note results in a calm, atmospheric response from the system, while badly sung notes receive a more complex response. Speaking Trees, on the other hand, allow people to record stories, memoirs, lyrical phrases—in other words, to speak on any topic at all. Speaking Trees automatically edit and process the recordings and incorporate the sound bytes into each Brain Opera performance.

Harmonic Driving. One sculpture that closely approximates an arcade-style video game is the Harmonic Driving controller. Responding to video generated by Rolf Rando’s 3-D rendering system on an IBM RS/6000 computer, participants use a steering wheel to navigate through a multicolored course complete with bends, turns, and potholes. Melodies and harmonies are created that correspond to the path taken; icons placed in the course signify “hot” or “cool” musical tracks. The driver determines whether the ride is a rhythmic journey or a wandering detour of ambient sound. In the true spirit of a video game, the player is given a skill rating at the end.

FIG. 11: Tod Machover of MIT’s MediaLab performs in the Sensor Chair, which senses the position and movement of the performer’s arms and upper body.

Sensor Chair. Normally, one would think of a chair as something you sit on while making music with an instrument. But the Sensor Chair captures movement of the arms and upper body and converts those motions into music while the performer is sitting in a chair (see Fig. 11).

Using the Sensor Chair, the performer becomes an extension of a transmitting antenna embedded in the chair’s cushion. Four receiving antennas mounted on poles around the Sensor Chair allow body gestures to control sound. Joe Paradiso, Neil Gershenfeld, and Ed Hammond developed the chair’s hardware, and Pete Rice and Eran Egozy devised the interpretive software. The mystical nature of this new controller even attracted the magicians Penn & Teller, who used it in a routine to highlight the innovations of music, magic, and machines.

Gesture Wall. Movement from a participant’s entire body modifies the on-going sounds and projected images as they stand near the Gesture Walls. Beneath the floor are plates that transmit a low-voltage electrical signal to the participant’s body through his or her shoes. Sensors mounted around the projection screen above the walls receive these electrical signals as they leave the body. This allows the instrument to follow precisely even the subtlest of gestures.

Future Music Blender. After wandering through the Mind Forest, visitors to the House of Music will have an opportunity to explore a new addition to the Brain Opera—the Future Music Blender. Chris Dodge, Alex Westner, Peter Colao, and Ed Hammond worked with Tod Machover to create a sonic sculpture designed to collect music samples that could easily be incorporated into a performance medley. While some of the sounds are activated from a prepared database, other samples will be generated by visitors to the Mind Forest. Using a Sensor Chair retrofitted with a “multimodal mixer” for control, people will be able to access, select, and play samples. Simply waving a hand in the air will enable a visitor to “blend” sounds that are then complemented by musical accompaniment (generated through custom algorithmic software) to create larger compositions.

Music toys. The latest round of controllers is disguised as “music toys.” With names like Simple Things, the Big Thing, and Music Shapers, they are made from as many squeezable, stretchable, and reconfigurable parts and materials as possible. (Even Play-Doh is used as a conductive material to manipulate sounds.) The ultimate goal behind the creation of these controllers is to distribute them to music-education programs in five host cities (New York, Boston, London, Berlin, and Tokyo). The culmination of the project will be a series of Toy Symphony compositions and performances with local symphony orchestras.

As the name implies, Simple Things make simple sounds, such as individual notes or sample playback, and are intended to be handheld stand-alone devices. Josh Strickon, Abie Flaxman, Tristan Jehan, and Diana Young use infrared links to exchange sounds between Simple Things and to synchronize groups of Simple Things.

Music Shapers are fabrics, furniture, balls, and spatial instruments that offer fun, tactile ways to explore different aspects of music. Designed by Maggie Orth and Gili Weinberg, these malleable instruments provide new ways to physically shape and manipulate musical sounds and textures by measuring the exertion of force and pressure on stretchable objects. This tangible approach to playing music is inspired by the desire to create flexible interfaces to complement the multidimensional aspects of synthesizers and computer-generated music.

The Big Thing—a multitiered structure designed by Orth, Weinberg, Dan Overholt, and Mary Farbood—is intended to be the brain that controls the network of interconnected music toys. The Big Thing enables young people to compose, arrange, and perform music in a 3-D construction kit, letting them experience musical expression by changing the physical relationships between a variety of sensing objects, connectors, and “chunks.” Each chunk contains sounds or commands that can be reorganized by moving, twisting, and interconnecting with other chunks to create “islands” of sound. The sensor objects can be used to add touch sensitivity or gesture-controller functions. Obviously, children will need a bit of coaching before they can fully understand the capabilities of these giant Lego-like controllers and music toys, but the prospects for collaboration are encouraging.

There are many more controllers—too many to list in this article—that have also been an outgrowth of development throughout MIT’s MediaLab. Among these are “musical bottles” that control different sounds and patterns of colored light when the stoppers are removed; musical threads that are sewn into a denim jacket and emit sounds; and gesture-tracking digital batons that provide position, orientation, and surface-pressure information.

Conductor’s Jacket. Wearable computers are not an essential part of most people’s wardrobe—unless you happen to be hanging around Teresa Marrin Nakra. Interested in finding new avenues for musical and emotional expression while studying at the MediaLab, Marrin Nakra mounted a collection of EMG sensors to her upper body, with the wires strapped inside a “conductor’s jacket” to collect a variety of physiological measurements.

FIG. 12: Boston Pops conductor Keith Lockhart wears the Conductor’s Jacket during 1998’s Tech Night at the Pops.

After taking readings on heart rate, respiration, skin conductivity, temperature, and muscle tension, Marrin Nakra hoped to find a correlation by mapping these measurements to the gestures and rhythmic timing of conducting. She was able to convince a number of well-known conductors, including Keith Lockhart of the Boston Pops, to try out the Conductor’s Jacket in performance so that data pertaining to these unknown relationships could be collected (see Fig. 12). To her surprise, Lockhart was more than willing to get wired for his audiences; he even requested a flashier jacket than the one that was originally provided.

STEIM

As a research center dedicated to the performing arts, STEIM (STudio for Electro-Instrumental Music) has been a hotbed of alternative-controller research (www.steim.nl). Notable developments include The Hands, developed with Michel Waisvisz, as well as the Sweatstick and the MIDI Conductor. Important computer applications for use in live performance have also come from STEIM, such as LiSa, for live sampling; BigEye, which converts video into MIDI data; and Image/ine, for real-time video manipulation.

A number of prominent performers using live electronics have developed highly personalized instruments at STEIM, including Jon Rose, Bob Ostertag, Laetitia Sonami, Miya Masaoka, and Kaffe Matthews.

STEIM also features an exhibition called the Electro Squeek Club, where visitors can experience firsthand a collection of audio and video pieces. Some are completed installations, while others are in development at the center. Visitors are encouraged to explore the individual properties of each of the exhibits, which include Crackle Boxes, Babblephones, the Electronic Baby Mirror, and the BeBop Table. The artists represented include Michel Waisvisz, Bert Bongers, Jorgen Brinkman, and Tom Demeyer, among many others.

IRCAM

The research center in Paris known as IRCAM (Institute of Research and Coordination in Acoustics and Music) has been at the forefront of music and technology for decades. Research into musical input structures has led to the development of several controller prototypes, including the SuperPolm and an extension of the computer mouse, dubbed Jerry.

This and other information has been assembled for a CD-ROM titled Trends in Gestural Control of Music, by Marc Battier and Marcelo Wanderly, and is available from the Electronic Music Foundation (www.cdemusic .org). Sections of the disc are dedicated to performance issues, definitions of gesture, and gestural analysis. It also includes a roundtable discussing the “Present and Future of Gestural Control in Music,” with contributions by such luminaries in the field as Donald Buchla, Mark Goldstein, Joel Chadabe, Tod Machover, Teresa Marrin Nakra, Robert Moog, Jean-Claude Risset, Laetitia Sonami, and Michel Waisvisz.

FINAL PERSPECTIVE

It is interesting to note that many of the controllers surveyed here are continually being refined—some could even be considered works in progress. Some readers may feel that there is a gratuitous use of new technologies behind some of these new musical-input devices, but deeper investigation reveals that musical results are the chief motivating force behind most of the controllers.

Although some of the input devices in this survey are theatrical, the ultimate goal is for the results to transcend the novelty of the visual aspects that a device presents. In most cases, the artistic success of a particular controller will depend on its transparency in the creation of the music: the main purpose behind all of these devices is the natural and immediate translation of physical gestures into music.

Some of the controllers we’ve examined here are based on the highly personal goals of their developers, whereas others are created with mainstream applications in mind. Performers who want to develop their own controllers need to define the gestures they wish to use and then find the best way to measure these gestures and translate the results into a useful data format, such as MIDI. Sometimes a series of gestures may require more than one type of sensor.

A number of companies have already developed the various components needed in an interactive control system (such as sensors, signal converters, and software). All that inquiring musicians, dancers, or artists need to do is spend some time configuring their own systems and learning how to use them.

Ever since Marty Cutler started as assistant editor for EM, he has referred to his banjo as a “real-time 5-string arpeggiator.” Armchair thereminist Gino Robair is an associate editor for EM. Bean’s music-making methods include sneaking into schools around the Bay Area with her group, RhythMix. Special thanks to David Wessel, David Jaffe, Donald Buchla, Alex Artaud, Peter Elsie, Joel Chadabe, Mary Gallardo, Miya Masaoka, Pamela Z, Bela Fleck, Bob Applebaum, and Jimi Tunnell.

ALTERNATIVE CONTROLLERS IN EM

Reviews	Issue
Big Briar Ethervox	05/98
Buchla and Associates Lightning II	08/96
Buchla and Associates Thunder	08/90
Cycling ’74 MSP	10/98
Interactive Light Dimension Beam	07/96
Kurzweil ExpressionMate	05/00
Nearfield Multimedia Marimba Lumina	06/00
SynchroVoice MIDIVox	07/92
Articles	Issue
BodySynth/BioMuse	09/97
CNMAT/Wacom tablet/Buchla Thunder	10/98
D’Cückoo	02/98
Hydra/Dimension Beam	11/97
Lady’s Glove	06/98
MIDIBall/Interactive stage show	05/97
Soundmapping/GPS/Interactive luggage	06/99
Tech Page: “Interactive Light”	12/95

GET WITH THE PROGRAM

One of the most popular commercially available software applications for sound artists who need flexibility in mapping physical gestures to MIDI is an object-oriented programming language called Max (Mac; $395) available from Cycling ‘74 (www.cycling74.com). Created in 1987 by Miller Puckette at IRCAM and later developed into a commercial product by David Zicarelli, Max provides a graphical user interface for combining the basic building blocks used in an object-oriented environment. A Windows version is on the horizon.

Cycling ’74’s MSP lets you create, analyze, and process audio and is designed for use with Max. The full version of MSP ($295) requires Max 3.5.8 or higher. A free runtime version of MSP, which comes with the free MaxPlay application, allows you to play, but not create, MSP patches.

PREACHING TO THE CONVERTERS

A number of converters are available for translating analog signals to digital ones.

The SensorLab (www.steim .nl/sensor.html) is a voltage-to-MIDI converter developed at STEIM for use with any type of interactive controller. Although popular with many artists, the SensorLab is currently out of production. Keep an eye on the developer’s Web site for further details.

The ADB I/O ($199) by Beehive Technologies (www.bzzzzzz.com) allows you to use up to eight input sensors with a Mac. You can connect up to four units, for a total of 32 I/O channels. The device accepts sensors from Infernal Devices and Infusion Systems and can work with Hypercard, Supercard, AppleScript, Macromedia Director, Symantec C/C++, and Cycling ’74’s Max. A beta version of a Max object created by David Zicarelli for the ADB I/O can be downloaded for free from the Beehive downloads page.

To accommodate different types of sensors (both gestural and environmental), IRCAM has begun marketing its AtoMIC Pro ($665). The AtoMIC Pro translates sensor information (electrical voltages) into MIDI data. The device was designed to be open-ended, so it can work with any type of controller or signal. It has 32 analog inputs (using two multipin connectors), eight digital inputs and outputs on multipin connectors, one MIDI input, and four MIDI outputs.

CNMAT has developed a low-latency, high-quality multichannel interface for use with laptop computers in live performance situations. Code-named the Rimas Box (after its primary developer, researcher Rimas Avizienis), the interface uses the 100BaseT Ethernet port to communicate with the computer, allowing the device to simultaneously handle 64 channels of sample-synchronous control-rate gesture data, 10 time-stamped MIDI I/O streams, and up to 10 channels of 24-bit audio. “Latency measurements show that we can get signals into and back out of Max/ MSP in less than 7 milliseconds,” says CNMAT director David Wessel. “One of the most important features of our interface is that data from sensors is treated as signals that are synchronized at the sample level with the audio. This provides the user with a very high degree of control intimacy.”

The Rimas Box, which is designed to sit neatly under a laptop computer, is currently being manufactured in a limited quantity for final beta testing.