Web Bonus! Watch a video of this entire process HERE.
THE PROCESS of scoring music to picture is an exciting challenge. Whether you’re
using a broad approach based on mood or underscoring every available movement,
you are defining your new work within the formal restrictions of the movie itself.
It becomes your timeline—your beginning, middle, and end.
One can view this process as either constraining or liberating. However, I find
it to be liberating in the sense that, once a formal structure is defined, there is one
less thing to worry about. The timeline ahead of you can help inspire musical ideas
or themes. You can use the onscreen action to dictate tempo and activity, or not. It
all depends on the mood you’re trying to create or underscore.
|Fig. 1. First steps.
One of the main advantages of scoring to video within a DAW is the ability
to accurately match a musical cue’s tempo to the action. For example, it allows
you to place the musical downbeat on the exact frame of a new scene. Even if the
music you compose ends up being played by live musicians, they can record to the
click track (or metronome) within the session, and then you can line up the tracks
with your video reference as closely as desired. This process is now an industry
standard for film, commercials, animation, and video games, and it is the best way
to work quickly and efficiently.
Of course, nothing can replace a sharpened No. 2 pencil and manuscript
paper for some things—notes, ideas, condensed, or even full scores. And using
those tools as a starting point can often be the best approach. But the usefulness
of a DAW is undeniable.
In this article, I will show you how to begin
a project in Pro Tools, sketch out the musical
parts to fit the action using MIDI, then create a
score in Sibelius from the MIDI data, which I
use when I replace the virtual instruments with
live musicians. As a demonstration, I will use
a video of Voodoo Vince, the creation of game
designer Clayton Kauzlaric for Microsoft’s
original Xbox. This scene shows Vince the
Voodoo Doll as a young man, drawn in the style
of early 20th century animation. (Watch a video
demonstrating these steps at emusician.com.)
Building Your Project Begin by creating
a new Pro Tools session. I typically run my
sessions at 96kHz/24-bit, but it is possible that
for video compatibility concerns a 48kHz/24-
bit session might be used. If you are creating
content for clients, it’s always best to check
with them and export your final bounced movie
at the sample rate and bit depth they request
(see first steps, Figure 1). Next, import the video
you intend to score. Use the pull-down menu
File>Import >Video (see Figure 2).
|Fig. 2. Importing video.
Then, locate and select the video. A pop-up
window will ask you where you want to locate
the imported video. Choose Session Start to
have it begin at the beginning of the session
timeline (see Figure 3).
At this point, the video you’ve selected will
appear both as a track on your edit window
and as a moveable, floating video window. The
frame rate for this video is at 29.97 frames
per second. (The frame rate shows up in the
movie track window after it is imported; see
I will change the frame rate from 30
fps to 27.97 to match the video by going to
Setup>Session, and then selecting the proper
frame rate (see Figure 5).
|Fig. 3. Session Start.
Once the movie has been imported and
the session is playing the movie at the correct
frame rate, the image in the movie window will
reflect where the cursor is set in the timeline
of the video when you drag the cursor over the
movie track (see Figure 6).
Tempos and Hits Now I’m ready to develop
an overview for the tempos that I’ll use for
various sections of the video, by creating a
timeline with markers for quick reference.
I will also insert MIDI instruments into the
session in order to sketch out some musical
themes and ideas that underscore the action
and enhance the mood and intent of the video.
|Fig. 4. Frame rate after import.
| Fig. 5. Setting frame rate.
|Fig. 6. Image at cursor.
|Fig. 7. Changing the tempo.
|Fig. 8. Checking the Tempo window.
For this part, I made sure to have the
conductor icon lit in the Transport window
so that the session’s tempo will follow the
tempo track in the ruler above the edit
Starting with the default tempo of 120
BPM, I view the opening credits with an
audible click track. Once the main character
enters, I change the tempo in order to match
his footsteps, by opening the Tempo Change
dialog box on the top of the track window of
the Edit Window (see Figure 7).
Depending on your setup preferences from
previous sessions, this dialog window may or
may not show up. If not, you’ll need to select
the drop-down window next to the Bars/Beats
header and check the tempo window (see
Clicking on the arrow sign next to Tempo
will open up the Tempo Window timeline
window, which allows a variety of ways to
manipulate tempo; using the pencil tool for
accelerandos and decelerandos, using the
trim tool to select and change the tempo of an
entire region, etc. (see Figure 9).
Clicking the plus sign to the right of the
tempo bar will open a dialog box that will
allow you to change tempo from any selected
point in your timeline (see Figure 10).
I type in various BPM values until I find the
one that works best.
|Fig. 9. Tempo window timeline.
|Fig. 10. Changing tempo from timeline.
In this case, I want to have the downbeat
hit when Vince takes his first step on the
ground. To do this, I click on the Meter Bar in
the ruler above the Edit window’s timeline to
open the Meter Change dialog box. Instead
of changing the meter, I deselect the box that
says Snap to Bar. This creates a new measure
exactly where I want the music to begin,
which in this case is where Vince takes his first
step. The time signature—in this case, 4/4—
changes to italics because you are not in Snap
to Grid mode. From here on out, the other
measures follow from that starting point until I
institute another change.
Now I can try different musical ideas
against the tempo. In this case, to match his
gait, I build a piano part that has the feel of a
Raymond Scott piece.
As Vince slips on a banana peel, I determine
that a bar of 7/16 in a slightly slower tempo
covers his somersault. From there, I return
to the original time signature and tempo
once his foot hits the ground again. To make
this transition work, I insert a 7/16 measure
using the Meter Change dialog box within
the Tempo Ruler, immediately followed by
a change back to 4/4. To change the tempo,
I select across the 7/16 section and use the
Trimmer tool to slightly lower the level of the
tempo line (see Figure 11).
Throughout the 4/4 part, I have added a
jazzy swing to the piano part.
As I develop the rough piano part, I decide
to add 3-note chordal stabs and melodic fills
to underscore the humorous onscreen action
(in this case, calamities taking place with the
folks Vince passes on the street). To keep track
of the hit points, I place Memory Location
markers by placing the cursor when I want to
set a marker, then hitting the Enter key on my
numeric keypad. A dialog box opens, letting
me name the marker. The name will appear
next to the yellow triangluar Memory Location
marker above the timeline.
Once I have placed and named the Memory
Locations, I open the Memory Location
window (under the Window tab or by typing
command/control-5 on the numeric keypad).
Now I can click on the name of a Memory
Location in that window and the curser will
jump to that marker so that I can play the
session from that point.
|Fig. 11. Trimming tempo.
After placing markers to determine the
exact hit points, I can complete the sketch
using MIDI sounds for the piano and
Ochestration Time Now it’s time to expand
the orchestration using virtual instruments,
which I will later replace with live musicians.
For this project, I used Propellerhead Reason
as the source of the instruments for my virtual
orchestra—violin, trumpet, alto saxophone,
and trombone—using ReWire to slave the
sounds to my Pro Tools session. In this case,
a benefit to using Reason is that it isn’t very
CPU intensive, so my session plays back easily
as I add musical parts to my sketch within
Now I can take the 3-note chord stabs from
my original MIDI piano part and re-distribute
them among my virtual instruments. I begin by
giving the top notes to the violin, and copying
and pasting them to the violin’s instrument
track. Then I do the same with the middle
notes of the chords, which go to the trumpet
track. The bottom notes are copied to the alto
sax part. Next, I re-quantize the orchestral
part with 77% swing to match the piano part,
and then delete those pitches from the piano
part so that there are no doubled notes.
I decided to double the piano’s left-hand
part with an upright bass patch; to do this, I
simply select the entire MIDI track from that
part of the piano roll and paste it into the bass
track. Then, I use the Trim tool to lengthen the
MIDI notes of the bass so that they are more
audible when combined with the percussive
piano part. This adds to the Raymond Scott-like
sound I’m looking for.
|Fig. 12. The Tracks window.
|Fig. 13. Selecting Notation Display Track Settings.
Export the MIDI Data Once the musical
sketch is complete, I export the MIDI parts to
Sibelius, the full-featured notation software
where I can convert the raw notation into
readable charts for the musicians.
Select the tracks you want to convert to
notation, then open the Score Editor window
(accessed by choosing Window>Score
Editor). In this case, I have selected the
violin and winds. By default, each MIDI
track is displayed as a grand staff, as
if it were a keyboard part, so I need to
reconfigure the staves to fit each instrument:
On the left hand side of your Score Window,
go under the Tracks window (see Figure 12)
and select Notation Display Track Settings
(see Figure 13).
Select the proper clef and staff for each
instrument, which will be reflected in the
score. The score will reflect concert pitch
for all instruments, and once the score
is exported to Sibelius, instruments that
need to be transposed will be transposed
Now I can export the data to Sibelius and
clean it up further to make it presentable.
Under File, I choose Export>Sibelius, then
select a place to save the document where you
will be able to easily find it.
Cleaning up the Score Because I
deselected Snap to Bar in the Meter Change
dialog earlier in the project, the exported
score will have some truncated measures
that don’t accurately reflect what I want the
musicians to see. To correct this, I create a
duplicate score with accurate measures and
time signatures, and then copy the data from
the exported score.
I copy the first measure up to the point
where a truncated measure appears. Once I
paste it into the new document, I will change
the rhythmic values of the last bar to fit into
the time frame that I want for the musicians. I
do this for each section of the score as needed.
Once that is done, I can place the tempo
markings. To do this, go to Create>Text>
Tempo, then control-click and select the
tempo for each section. For the sections
where I used a swing feel in the Pro Tools
session, I need to alter the notation in order
for the rhythmic values to be accurately
portrayed. In Sibelius, this is referred to as a
Start by going to the Create menu>Text>
Other System Text>Metric modulation. Now
control-click to get the window of modulation
options. In this case, I select the one showing
a pair of barred eighth notes next to a quarter-note-
plus-eighth-note triplet. Once that is
positioned in the appropriate place, I manually
clean up the notation. (An easier way to do
this would be to duplicate the MIDI tracks
with swing eighths in Pro Tools and straighten
out the timings so that they appear as straight
eighths when you export them.)
From there, I can check out the
individual parts and see how they look
before printing them.
Recording Parts, Mixing, and Exporting
a Quicktime File I prepared and printed
the parts that I’ve designated for recording
with live musicians, and hired a violinist (Alisa
Rose), a trumpet player (Chris Grady), an alto
saxophonist (Sheldon Brown), a trombonist
(Andy Strain), and a drummer (John Hanes) to
replace their equivalent MIDI counterparts.
I had the horn players play most of their
parts together; it really helps the final result
come off as a well-rehearsed ensemble when
the musicians can practice articulations and
dynamics together. Alisa and John came in
separately to record their parts.
Microphones are up, levels are set, and
once I’ve played the arrangement for the
musicians, we begin tracking. Sometimes
players will want to hear their MIDI part in
the headphones, but most of the time, they
don’t. Once they know what to play, they are
usually better off listening to themselves and
other live players. It’s more musical, and that’s
the reason that I decided to use live musicians
in the first place. For me, some MIDI
instruments—the upright bass, the piano, etc.—
will sound fine, and when combined with the
magic touch of skilled human interpretation,
these MIDI tracks take on a new life.
When everyone was done tracking, I began
mixing. At this point, I made the decision to
add sound effects to underscore some of the
action of the video. Because I wanted to have a
clean session with just the music, I mixed the
audio first and opened up a new session with
the final stereo mix and some extra tracks for
File>Import Audio (see Figure 14) will
open up a browser window where you can
navigate to your sound effects libraries and
copy them to your session. (Do not just “add”
them to your session unless you don’t want
them to show up the next time you open up
Then you can move your sound effects
along the timeline until they sync with the
action you wish to underscore (see Figures
15 and 16).
Once you’re done mixing the sound
design with the track, the last step is
exporting the movie to a QuickTime
(.mov) file by selecting the region in
your edit window that you wish to
create your movie from, and going to
File>BounceTo>QuickTime Movie. You
may want to use QuickTime Pro to adjust
the resolution of the movie to make it
web ready, but in most cases, YouTube
prefers the high-res version, and will
compress it on that end. When delivering
to clients, find out their requirements and
accommodate them appropriately. You can
check out the final result, in its entirety, at
This was a really fun scoring project, and
the musicians had a great time working on it.
Adding real performers to music created in a
MIDI environment can really breath life into
your score. Now you’re armed with the tools to
try this yourself, so take the leap!
Bonus! Watch a video of this entire process HERE.
Steve Kirk is a composer whose work has
been featured in TV, film, and video games.
He owns and operates Steve Kirk Pop
Studios in Oakland, CA, and teaches guitar
and composition both privately and at the
Blue Bear School of Music and Community
Music Center, both in San Francisco.