Say It With Pictures

Sep 1, 2008 12:00 PM, By Dennis Miller

           
 

EIGHT PROGRAMS THAT CONVERT IMAGES TO MUSIC

BONUS MATERIAL
The UPIC System

Everybody knows that a picture is worth a thousand words. But did you know that a picture can also buy you the frequencies of a thousand oscillators? Using any of several modern music applications, you can convert the data that forms a picture on the computer screen into information that will generate an audio file. In most cases, the program uses the vertical position of each pixel of the image to control frequency and one or more color (RGB) values for parameters such as amplitude and stereo position.

In this article, I'll look at a number of programs for both Mac and Windows that allow you to perform this alchemy. Included are Adobe Audition 3, Thomas Baudel's HighC 2.2, Camel Audio's Cameleon 5000 1.5, Rasmus Ekman's CoagulaLight 1.66, Nicolas Fournel's AudioPaint 2.1, Image Line FL Studio 8, U&I Software MetaSynth 4, and VirSyn Poseidon 1.4. I'll also cover Mark Coniglio's Isadora 1.2.9, which can convert images to sound but is better suited to work in the opposite direction (see the sidebar “Izzy Gets Down”). I'll start with a general overview of the field before looking at each program individually.

Note that AudioPaint, Coagula, and HighC are standalone applications whose only role in life is the conversion of images to sound and, perhaps, vice versa. For Audition, Cameleon, FL Studio, MetaSynth, and Poseidon, image-to-sound conversion is just one of many features. Also, many of the ideas that drive image-to-sound software stem from research by Iannis Xenakis, a Greek composer whose UPIC system was among the first to allow musicians to draw the data used to generate sounds (see the online bonus material “The UPIC System” at emusician.com). Xenakis used imagery in creative ways when composing both his acoustic and electronic works.

Roundabout

FIG. 1: This image shows the original bitmap (left) that Audition used to produce the spectrum on the right.

All of the programs in this roundup share the ability to convert an image into an audio file, but the range of editing and processing features you'll find varies widely. For example, all except HighC allow you to import a preexisting bitmap image (typically a BMP or PICT file, but in some cases other formats as well), and all but Cameleon, FL Studio, and Poseidon let you modify or process an image before using it to generate a sound file. Audition, FL Studio, MetaSynth, and Poseidon convert the graphics file into a 2-D sonogram display, which shows frequency on the y-axis and time on the x-axis and uses intensity (brightness) to represent amplitude (see Fig. 1). HighC uses a hybrid musical-score/piano-roll metaphor and, like the original UPIC system, provides tools for drawing the gestures and shapes that will control musical parameters. Coagula and MetaSynth also provide tools for drawing an image from scratch, while AudioPaint lets you generate a new image automatically with its configurable Lines & Curves and Clouds of Points tools.

Deciding how to extract data from the bitmap and how to use the extracted data is a big part of these programs' toolkits. Poseidon takes each pixel on the y-axis of the image and assigns it a frequency value within the range of 20 Hz to 22.05 kHz, then uses additive synthesis to generate a new sound from the sum of those values. The amplitude of each partial changes over time depending on the brightness of the pixel, and each pixel accounts for about 3 ms of the new sound's duration. Most of the other programs work in a similar manner, though several — for example, AudioPaint and Audition — let you choose how the program will interpolate from one amplitude value to the next. FL Studio lets you specify an arbitrary number of partials (up to 999) for the resynthesis regardless of the size of the original image, and AudioPaint, Audition, Coagula, FL Studio, and MetaSynth let you set an arbitrary frequency range over which new partials will be generated.

Some of the programs — for instance, Audio-Paint, Audition, and FL Studio — let you determine whether the new partials are distributed in a linear or log fashion (linear distributes the frequencies in increments of hertz, while log uses increments of cents). MetaSynth goes quite a bit further by including a large number of tuning options. You could, for example, space the partials of the new sound in steps of different types of traditional scales (whole tone, major/minor, and so on), using various microtonal increments (from 4 to 1,024 divisions to the octave, with more than 1,000 scales included), or, like AudioPaint and Poseidon, using scales in Scala format. (Currently over 4,000 different scales are available at the Scala Web site, xs4all.nl/~huygensf/scala/.) HighC lets you create your own pitch/frequency scales, but you have to create a list of tuning increments by typing it. (I'll discuss the other programs, including the additional extraction parameters they offer, in their respective sections.)

FIG. 2: Cameleon displays the converted image using adjustable sliders that represent the amplitude for each partial (top). A separate set of sliders is available for adjusting the noise components (bottom).

In addition to using additive to generate the new sound, FL Studio offers a nonadjustable form of granular synthesis. HighC and MetaSynth can use additive along with several other synthesis methods, including granular and FM. AudioPaint and MetaSynth will create a new sound by using the extracted data to control the playback parameters of samples (pitch-shifting and time-stretching, for instance).

Cameleon can display the converted image as an editable two-dimensional (frequency and amplitude) spectral plot and offers handy tools to manipulate the newly generated harmonic spectrum prior to synthesizing the sound. It also has the ability to use information extracted from an image to generate discrete bands of noise and allows you to alter the amplitudes of both the harmonic partials and noise components individually (see Fig. 2). Audition also lets you add a bit of random offset to the individual frequency components as part of the conversion process.