Say It With Pictures

Sep 1, 2008 12:00 PM, By Dennis Miller

           
 

EIGHT PROGRAMS THAT CONVERT IMAGES TO MUSIC

BONUS MATERIAL
The UPIC System

Like Reaktor and other modular programming environments, Isadora offers a large number of video and audio modules (called Actors) that you connect on the main work area. A basic configuration might consist of a QuickTime movie running via the MoviePlayer, outputting to a video-processing module (a rectangular tiler, for example), which is then connected to the Projector Actor for desktop viewing. You could also enable output to an external device and capture it to a DV video camera or project it onto a large screen.

FIG. A: This image shows the main work area in Mark Coniglio’s Isadora. The program’s Actors (modules) appear on the left and are connected on the right. This configuration uses Actors to track the frequency and amplitude of an incoming sound source and map them to parameters controlling the amount of displacement that is applied to the source movie file. The final output is shown at the bottom right of the screen.

But things can get a lot more interesting really quickly. For instance, using the Sound Frequency and Sound Level Watchers, you could map the amplitude level of a user-defined frequency band of an incoming audio signal to one or more parameters of a video effect or image generator (see Fig. A). That way, the loudness of the incoming audio could control the amount of displacement in a displacement module or determine the number of particles in a particle animation created in real time by the program.

There are dozens of modules for mixing, generating, and processing both images and audio, as well as support for the FreeFrame plug-in standard (freeframe.org), so you can add additional commercial or free video-processing plug-ins, too. You'll also find math and I/O modules, and you can build your own interfaces using elements such as sliders, knobs, and dials. The best part is that you can freely mix modules of nearly any type, using data extracted from an audio signal as the input to a video-effect parameter or vice versa.

Even if you only want to work with audio modules, you can design a wide range of networks in which elements interact in innovative and unusual ways. For example, using the Core version of the program (which adds support for AU plug-ins), you could track the frequency output of one audio file, then use that information to determine the amount of delay or perhaps the pan position (or both) of a second sound. The same information, perhaps scaled by some factor, could also control another audio-effect parameter and of course a video effect simultaneously.

Isadora is supported by a terrific manual (PDF only), an active users forum, and a number of getting-started tutorials. If you want to explore combining video and audio or are simply looking for unique ways to interconnect audio parameters, then give it a try. A free trial version, as well as educational pricing ($275) on the commercial version, makes it well worth a look.

ONLINE LINKS

Christopher Penrose’s HyperUpic system
music.princeton.edu/winham/PPSK/hyper.html

The vOICE home page
seeingwithsound.com

The story of the ANS synthesizer
theremin.ru/archive/ans.htm

Sonification of images
scientific-computing.com/features/feature.php?feature_id=58

IanniX download page
sourceforge.net/project/showfiles.php?group_id=174402