Implementation of a 3D Virtual Drummer

Martijn Kragtwijk, Anton Nijholt, Job Zwiers

Department of Computer ScienceUniversity of Twente

PO Box 217, 7500 AE Enschede, the NetherlandsPhone: 00-31-53-4893686Fax: 00-31-53-4893503

email: {kragtwij,anijholt,zwiers}@cs.utwente.nl

ABSTRACT

We describe a system for the automatic gen-eration of a 3D animation of a drummer play-ing along with a given piece of music. The in-put, consisting of a sound wave, is analysedto determine which drums are struck at whatmoments. The Standard MIDI File format isused to store the recognised notes. From thishigher-level description of the music, the an-imation is generated. The system is imple-mented in Java and uses the Java3D API forvisualisation.

1. INTRODUCTION

In this paper we describe preliminary resultsof our research on virtual musicians. Theobjective of this project is to generate ani-mated virtual musicians, that play along witha given piece of music. The input of thissystem consists of a sound wave, originatingfrom e.g. a CD or a real-time recording.

There are many possible uses for an appli-cation like this, ranging from the automaticgeneration of music videos to interactive mu-sic performance systems wheremusicians playtogether in a virtual environment. In the lastcase, the real musicians could be located ondifferent sites, and their virtual counterpartscould be viewed in a virtual theatre by a world-wide audience. Additionally, our departmentis currently working on instructional agentsthat can teach music, for which the work wedescribe in this paper will be a good founda-tion.

For our first virtual musicians application,we have restricted ourselves to an animateddrummer. However, the system is flexibleenough to allow an easy extension to otherinstruments.

AnimationGenerator

Percussion Recognizer

Audio Signal

MIDIEvents

3D Animation

Figure 1: An overview of the system

As figure 2 shows, the total task can beseparated into two independent subtasks:

• An analysis of the sound signal and tran-scription of the percussion part. Thesystem has to determine which drumsare hit, at what moments in time. Con-centrating on percussion sounds has cer-tain advantages and disadvantages; thisis further discussed in section 2.

• The creation of the the movements ofa 3D avatar playing on a drum kit. Amore detailed explanation on this partis given in section 3 and 4.

2. THE PERCUSSION RECOGNISER

This part of the system is responsible for thetranslation from a ‘low level’ description ofthe music (the sound wave) to a abstract, ‘highlevel’ description of all percussion sounds thatare present in the signal. These recognisednotes are stored as MIDI events.

Many attempts in the field of musical in-strument recognition concentrate on pitched

sounds [1]. As explained in [9], this is a ratherdifferent task than recognising percussive sounds,which have a sharp attack, short duration,and no clearly defined pitch. As shown in [9],individual, monophonic samples of drums andcymbals can be classified very well. In thisapproach, a few frames of the spectrum, mea-sured from the onset of the sounds, werematchedagainst a database of spectral templates.

In our highly polyphonic, ‘real-life’ situa-tion, however, the input signal may containmany percussive sounds played simultaneously,and non-percussive instruments (such as gui-tar and vocals) may be mixed through thesignal as well. Therefore, special techniquesare needed to separate the percussive soundsfrom the other sounds. Other researchershave already tried to solve the same problem[13, 14]: Sillanpaa et. al. subtract harmoniccomponents from the input signal to filterout non-percussive sounds. Furthermore, theystress the importance of top-down process-ing: using temporal predictions to recognisesoft sounds that are partially masked by loudersounds [14]. Puckette’s Pure Data programhas an object called bonk that uses the dif-ference between subsequent short-time spec-tra, to determine whether a new ‘attack’ hasoccurred.

We are still developing this part of the sys-tem, therefore we cannot yet present a finalsolution of this problem. We plan to solvethe problem of polyphony by adding exam-ples that consist of multiple sounds playedtogether to the collection of spectral templates.For example, a bass drum, snare drum and hi-hat played together. For an off-line situation,where the complete input signal is alreadyknown, we plan to apply clustering methodson all fragments of the signal that contain astrong attack. This is based on our hypothe-sis that specific drum sounds will sound verysimilar throughout a piece of music. Thisis especially plausible for commercial record-ings, and/or in the case that the music con-tains sampled drum sounds.

3. BASIC ALGORITHMS

In this section, we describe how our systemgenerates animations automatically. The var-ious algorithms discussed here are kept rathersimple on purpose, to maintain a clear viewon the system as a whole. In section 4, moreadvanced techniques (that give better results)will be explained.

Figure 2: An overview of the system

3.1. Overview of the system

A general overview of the animation gener-ation is shown in figure 2. An abstract de-scription of the animation (in this case, a listof time-stamped MIDI events) is transformedinto a ‘concrete’ animation. This lower-leveldescription of the animation is defined in termsof ‘key frames’ [4] that can directly be used bythe graphical subsystem to animate objectsin the scene.

Our implementation uses the Java3D en-gine for visualisation purposes [7]; the geom-etry of the 3D objects we have used is hasbeen created using Virtual Reality ModelingLanguage (VRML, [15]).

3.2. ‘Pre-calculated’ versus ‘real-time’ anima-tion

In our current off-line implementation, thepiece of music to be played is completely knownin advance as a list of MIDI events. Therefore,the entire animation can be computed beforeit is started. In a real-time situation, wherethe system has to respond to incoming MIDIevents, this would not possible. In that case,a short animation should be constructed andstarted immediately for each note that occursin the input.

A great advantage of pre-calculating theentire animation is that the transitions be-tween strokes will be much smoother: foreach note we already know which drum willbe struck next, and the arm can already startmoving towards that drum.

3.3. Polyphony Issues

Monophonic instruments (such as the trum-pet or the flute) are relatively easy to animate,because each possible sound corresponds toexactly one ‘pose’ of all fingers, valves, etcetera,and only one pose can be active at each mo-ment in time. Highly polyphonic instruments(such as the piano) are much more difficult,because there are many different ways (‘fin-gerings’) to play the same piece of music, anda search method is needed to find a good so-lution [8]. The drum kit could be viewed inbetween these two extreme examples: up tofour sounds can be started simultaneously.

3.4. Drum Kit Model

In this section we will describe the parame-ters that are used to model different drumkits.

3.4.1. Event Types

The General MIDI standard [11] defines 47different percussive sounds. The standard in-cludes different versions of the same sound,for example “Crash cymbal 1” and “Crash cym-bal 2”. Our application should treat both eventsin the same way.

Additionally, there are six different tom-tom sounds (“Low floor tom”, “High floor tom”,“Low tom”, “Low-mid tom”, “Hi-mid tom”, “Hightom”), while a ‘real’ drum kit usually only has2 or 3 tom-toms. It may be clear that we haveto define a smaller set of ‘drum event types’in the drum kit model . The MIDI events fromthe input file can then be mapped onto thesedrum event types.

Our current implementation distinguishesbetween the following drum event types: BASS,SNARE, RIM, HIGHTOM, MIDTOM, FLOORTOM,CRASH, RIDE, RIDEBELL, SPLASH, CHINA, HI-HATOPEN, HIHATCLOSED, HIHATPEDAL, COW-BELL.

The drum event types do not neccesarilyhave to have a one-on-one correspondencewith the objects in the 3D scene, because 2or more event types can belong to the samedrum/cymbal, with a different ‘hit point’. Agood example of this are the ‘RIDE’ and ‘RIDE-BELL’ events: both are played on the ride cym-bal, but we speak of a ride bell (or cup bell)when the stick hits the small ‘cup’ at the cen-ter of the cymbal (this gives a bell-like sound,hence the name).

3.4.2. Other Parameters

Other parameters that are defined in the drumkit model:

• For each event type, a preferred hand:-1 (“left”) or 1 (“right”).

• For each event type, a parameterminTimeGapthat determines how fast that particu-lar event type can be played with onehand. This parameter will be explainedin more detail in section 3.6.

3.5. MIDI Parsing

First, the list of MIDI events is transformedinto a list of DrumEvent objects according tothe mapping defined in the drum kit model(see section 3.4). The class DrumEvent is anextension of the AbstractEvent class (seeappendix ??). Besides having a type code, aDrumEvent has an associated velocity veleventin the range [0..1].

Secondly, the list of DrumEvent events isparsed to remove double events 1 and dis-tribute the events over the different animatedobjects. Objects in the scene respond to asubset of drum event types.

Three new event lists are created (one forthe hands, one for the left leg and one for theright leg) and the DrumEvents from the orig-inal list are distributed between them. Theevent list that is used for the hands will laterbe subdivided for the left and right hand; thisis discussed in section 3.6.

3.6. Event Distribution

Drum events that can be played by both hands(i.e. all events except BASS and HIHATPEDAL)need to be distributed between the left andright hand in a natural looking way.

3.6.1. Hand assignment

The first algorithm that we have tested, wasdesigned to be as simple as possible. It isbased on the following principles:

1. No more than two events, that are playedwith the hands, can have the same timestamp.

1some MIDI files that we used contained ‘doubleevents’, that is: multiple events on the same chan-nel, with the same time stamp, the same note numberand the same velocity. These extra events do not con-tain new information, nor do they increase the velocity,therefore we can discard them.

2. For each event type, there is a preferred(default) hand that should be used if pos-sible.

3. When playing fast rolls, both hands shouldbe used.

In our system, these principles were im-plemented in the following way:

• Whenmore than two events (that shouldbe played with the hands) are found tohave the same time stamp, all but twoare deleted.

• A parameter defaultHandeventType isspecified for all event types. In our im-plementation, the SNARE and RIM eventshave the default hand set to ‘left’, while‘right’ is the default hand for all otherevents.

• A parameterminTimeGap is defined,that determines how fast an event canbe played with one hand. This parame-ter can have a different value for differ-ent event types, Because the tendencyto alternate hands varies from one drumtype to another. For example, the hi-hatis usually played with the right hand;only in very demanding situations (fastrolls) both hands will be used. On theother hand, hand alternation on the hightom is much more common.

These principles are implemented in algo-rithm 3.1. It consists of two phases:

1. default hand assignment

2. hand alternation

Algorithm 3.1 A simple algorithm for eventdistribution

iterate over all events e:hand(e) := preferredHand(type(e))

iterate over all triplets of subse-quent events (e1,e2,e3):if hand(e1)=hand(e2)=hand(e3)

ANDTime(e2) -

Time(e1) <= minTimeGap(type(e1))OR

Time(e3) -Time(e2) <= minTimeGap(type(e3))then

hand(e2) := otherHand(hand(e2))

3.7. Pose Creation

Figure 3: The graphical poser interface, ap-plied to the left arm

A graphical user-interface (GUI) is providedto create ‘poses’ manually. Figure 3 shows ascreenshot of the GUI applied to the left arm.A pose consists of a set of angles or trans-lation values: one for each degree of free-dom. With the horizontal sliders, the usercan change these values.

Figure 4: ‘MID TOMUP’

Figure 5: ‘MID TOMDOWN’

For each limb, two posews should be spec-ified for each drum event type that it sup-ports: the ‘DOWN’ pose (the exact situationon contact) and the ‘UP’ pose (the situationjust before and just after the hitting moment).Examples of ‘UP’ and ‘DOWN’ poses are shownin figures 4 and 5.

Once a good position is achieved, it can bestored in the pre-defined list of poses. Theentire list can be saved to disk, to preservethe information for a next session.

3.7.1. Motivation

We have chosen for manually setting the posesthrough a GUI interface, instead of using mo-tion capture [16] or inverse kinematics forthe following reasons:

Costs: Motion capture equipment is expen-sive, and requires a complete setup with

a real drum kit that matches the 3D kit.If one would want to change somethingin the 3D drum kit (for example, movinga tom-tom) the whole capturing wouldhave to be done all over again.

Simplicity: there are only a small number ofposes, and they have to be set only oncefor a new drum kit configuration.

Flexibility: besides the setting poses for thearms and legs, the interface can also beused for the hi-hat stand and pedal, thecymbal stands, the parts of the bass pedal,and giving the snare, bass drum and tom-toms their position and orientation inthe 3D scene.

3.7.2. Implementation

In the object source files, we have to definethe parameters of the object, such as the de-grees of freedom and the corresponding ro-tation / translation axis. For example:

• arm:– the shoulder can rotate around itslocal X,Y and Z axis;

– the elbow can rotate around its lo-cal X and Y axis, to make the lowerarm twist and the elbow bend, re-spectively;

– the wrist can rotate around its lo-cal X and Z axis

• hi-hat:– the pedal can rotate around its lo-cal Z axis

– the upper part (the stick to whichthe upper cymbal is attached) canbe translated along the Y axis.

3.8. Key Frame Generation

In this section, the transformation from ‘ab-stract events’ (DrumEvents) to ‘concrete events’(key frames) is discussed. Because a differentapproach is used for the limbs and the cym-bals, they are discussed separately:

3.8.1. Avatar Animation

The poses that were created with the GUI in-terface (see section 3.7) are used to createkey frames for the animation of the limbs.For each arm and leg, its abstract time line

(that contains only drum events that shouldbe played by that arm/leg) is parsed in thecorrect temporal order. For each abstract an-imation event e, a Stroke is added to the an-imation time line. A Stroke consists of three‘concrete animation events’ (i.e. key frames):(ebefore, econtact , eafter ).

The parameter delta is a constant thatdetermines the time between the key frameswithin a stroke (100ms is a useful value). Seefigure 6 for a graphical representation of aStroke that will be used throughout this chap-ter.

Figure 6: A basic Stroke, consisting of keyframes ‘before’, ‘contact’ and ‘after’

If the time gap between subsequent ani-mation events e1 and e2 is less then delta,their key frames will overlap, and special carehas to be taken. We distinguish between twocases:

• If e1 and e2 are of the same event type(e.g. both are ‘SNARE’ events), the lastkey frame of e1 and the first key frameof e2 are replaced by an interpolatedkey frame eNew: the less time betweene1 and e2, the closer the new key framewill be to the ‘DOWN’ key frame, as canbe seen from figure 7.

Added key frames

key

fram

e sp

ace

UP

type(e1)DOWN

type(e1)

e2e1e0

time

Figure 7: New key frames in the case of over-lapping events of the same event type

• If e1 and e2 are of different event types(e.g. a ‘SNARE’ and a ‘HIGHTOM’ event),more time is needed to bring the arm

from the ‘after’ key frame of e1 to the‘before’ key frame of e2. To accomplishthis, the time difference between e1contactand e1after , and between e2before ande2after is shortened. A parametera (0 <a < 1) determines the fraction of thetime between the events that is used formoving the arm from e1after to e2before.

1−a2

1−a2

a

e2

e1UP

DOWN

type(e1)

type(e1)

UP

DOWNtype(e2)

type(e2)

key

fram

e sp

ace

time

Figure 8: New key frames in the case of over-lapping events of different event types

3.8.2. Drum Kit Animation

The event list, containing all DrumEvents fromevery limb is used to animate the 3D drumkit.

3.8.2.1. Pedals

The bass pedal and the hi-hat are animatedthrough the same kind of Stroke objects aswe use for the arms and legs. Because thepedals and the feet have their ‘UP’ and ‘DOWN’key frames at exactly at the same momentsin time, the illusion is created that the feetreally ‘move’ the pedals.

3.8.2.2. Cymbals

For the animation of the cymbals we use Vi-bration objects, that contain a number of keyframes starting at the ‘contact’ time stamp ofa cymbal event. These key frames are com-puted by rotating the cymbal object aroundits local X and / or Z axis. The angles aresampled from an exponentionaly decaying si-nusoid:

maxalpha

time

0angl

e

Figure 9: angle(t)

angle(t) = ηtαmax sin(βt)

In the above equation,

• αmax represents the maximum angle

• η is the damping factor of the vibration(0 < η < 1): low values for η result in afast decay.

• β determines the speed of the vibration:a higher value for β corresponds to ashorter swing period.

Overlapping Vibrations are much easierto deal with than overlapping Strokes. Whenthe first time stamp of a new Vibration fallswithin the time range of an previous Vibra-tion, the remaining events (key frames) aredeleted2.

time

e3e1 e2

maxalpha

angl

e

0

Figure 10: overlapping vibrations for events[e1,e2,e3]

4. IMPROVEMENTS

In this section, some advanced techniques willbe discussed that extend the system as de-scribed in section 3. These techniques aredesigned to make the motion of the virtualdrummer appear to be more ‘natural’ and ‘re-alistic’. One should keep in mind, however,

2Note that this will sometimes cause a sudden dis-continuity in the angle, when a new vibration overridesan existing one at a moment that the angle was not 0.In practice, however, this effect is hardly noticed; prob-ably because the viewer’s eye already expects a sharpchange in the motion of the cymbal, once it gets hit bythe stick.

that although some general rules can be fol-lowed, there is no ‘perfect’ solution: differentdrummers will have their own playing style.Differences may lie in

• the parts of the drum kit: how manyand what type of cymbals, toms etc. areused?Is there one bass drumwith a single pedal,one bass drum with a double pedal, ortwo bass drums with two seperate ped-als?

• the setup of the drum kit: ‘normal’ (withthe hi-hat on the left side and the lowesttom on the right side, this setup is usedby right-handed players) or ‘mirrored’(for left-handed drummers)? Where arethe cymbals placed?

• The hand patterns used on a certain ‘roll’:LLRR, LRLR, LRRL, etc.

• ‘grip’, the way of holding the drum sticks:either ‘matched’ 3 or ‘traditional’ 4?

• the way of striking the drums: are thepalms of the hands kept vertical or morehorizontal?

4.1. Event Distribution

In our basic algorithm (see section 3.6), therewas a maximum of two simultaneous eventsthat were played by the hands. In this sec-tion, we will show how this constraint can bereleaved by using the hi-hat pedal in specificsituations. First, however, another constrainton the contents of the event list will be dis-cussed.

4.2. Simultaneous hi-hat events

Our input list of MIDI events is not bound byany ‘real-world’ constraints, and may there-fore contain any number of simultaneous events,even when this would be impossible to playon a real drum kit. Consider the set of possi-ble hi-hat events {HIHATCLOSED, HIHATOPEN,

3in matched grip, both hands hold their stick be-tween thumb and index finger

4the traditional grip is often used by jazz drummers.The right hand grip (for right-handed players) is thesame as with matched grip, while the left hand holdsthe stick between thumb and index finger and also be-tween ring and middle finger

HIHATPEDAL} 5 : only one of them can beplayed at a time.

We must therefore ensure that the eventlist that is used to create the animation con-tains no more than one hi-hat event at eachmoment in time. This is taken care of in theMIDI parsing stage: whenever two or three hi-hat events have the same time stamp, one willbe kept and the others are discarded. Whichevent is kept and which ones are removed isa rather arbitrary choice.

4.2.1. The hi-hat pedal as a substitute

Human drummers often use the hi-hat pedalto play the hi-hat sounds when they have toplay two other sounds on the same time aswell. This is implemented in our system inthe following way: If three or more DrumEventshave the same time stamp (not counting BASSevents), and one of them is a HIHATOPEN orHIHATCLOSED event 6 , this event is replacedby a HIHATPEDAL event with the same timestamp and velocity.

4.2.2. Hand assignment

The hand assignment algorithm described insection 3.6 is easy to model and gives satis-factory results in most situations. However,a number of problems arise:

• when two simultaneous events have thesame default hand (for example, MID-TOM and LOWTOM), the original algo-rithm would remove one of the eventsfrom the list, even when the other handcould have played that event.

• in some cases, the arms are crossed whenthis is not necessary: consider for ex-ample a fast sequence HIHATOPEN-RIDE-HIHATOPEN. Both RIDE and HIHATOPENhave ‘right’ as default hand, and the handalternation algorithm will assign the RIDEevent to the left hand. Most drummers,however, would in this case prefer toplay the HIHATOPEN with the left handand the RIDE with the right hand.

5This is a strongly simplified view of reality, as hu-man drummers are able to play much more different hi-hat sounds than these three. For example, playing withthe hi-hat cymbals almost closed sounds entirely dif-ferent than both HIHATOPEN and HIHATCLOSED. How-ever, the three event types that we consider in ourmodel are the only three that are included in the Gen-eral MIDI specification, and are used in most situations.

6Note that there can only be one such event, becauseof the filtering as explained in section 4.2.

Our second algorithm, that solves theseshortcomings, uses default hand assignmentsfor all possible pairs of events. For exam-ple, we can define that whenever RIDE andHIHATOPEN are played together, the RIDE isplayed with the right hand and the HIHATwith the left. We should keep some flexibility,as these constraints do not have to be equallystrong for all pairs: for example, SNARE+CRASHcan be played as left-right just as easy as right-left.

The drum kit model is extended with afunction pair(eventType, eventType), thatreturns a floating-point value in the range [-1..1]. The semantics of this value are as fol-lows:

−1 ≡ strictly left-right0 ≡ don’t care1 ≡ strictly right-left

The improved hand assignment algorithmuses just the pair(a, b) function for simulta-neous events. For events [e1, e2] with a timegap ∆t greater that zero, the default handvalues are taken into account as well.

For each event with index I in the eventlist, a hand assignment value is calculatedtwice: in the pair [event(I-1),event(I)] and inthe pair [event(I),event(I+1)]. Afterwards, thesetwo values are averaged to yield the final handassignment value for event(I).

For a pair [e1,e2] the hand assignment val-ues (hand(e1), hand(e2)) are calculated inthe following way:

∆t =Time(e2)− Time(e1)

hand(e1) =ρ∆tpair(e1, e2)+ (1− ρ∆t)× defaultHand(e1)

hand(e2) =ρ∆t(−pair(e1, e2))+ (1− ρ∆t)× defaultHand(e2)

The decreasing exponential function ρ∆t(0 < ρ < 1) ensures that the default hand val-ues are taken more into account when thereis more time between e1 and e2, at the sametime lowering the influence of the pair-wisehand preference.

4.2.3. Shortest path methods

A third possible solution to the hand assign-ment problem might be found in shortest-path methods, as used in [8, 10]. These meth-ods consist of the following steps:

1. generate all possible solutions

2. assign a distance value to each solution(e.g. based on distances between drums,penalties for using a certain hand for acertain event type, etcetera)

3. take the solution with the lowest dis-tance value.

Problems with this approach lie in the designof a good distance function, and in the largenumber of possible solutions7. We have not(yet) implemented a shortest-path algorithmin our system.

4.3. Key Frame Generation

4.3.1. Drum Elasticity

In a real drum kit, one can observe that somedrums or cymbals are more ‘elastic’ than oth-ers, i.e. the drum stick ‘bounces’ more on oneobject than on another. Besides the object it-self, the elasticity is also dependent on theway of playing: the stick will bounce backmore on the hi-hat when it is played ‘closed’then when it is played ‘open’.

To simulate this phenomenon, we extendthe drum kit model with an elasticity param-eter eleventType in the range [0..1] for eachdrum event type. The value of eleventTypedetermines how far the drum stick shouldbounce back to its initial position after con-tact. In this definition, 0 means “no elastic-ity” while 1 corresponds to “maximum elas-ticity”. The elasticity values are now used inthe following way: for each stroke, the TRafterkey frame is interpolated between the ‘UP’and the ‘DOWN’ pose:

TRbefore =TRUPTRcontact =TRDOWNTRafter =TRDOWN + eleventType

× (TRUP − TRDOWN)

From this, one can easily deduce that

eleventType = 0→ TRafter = TRcontacteleventType = 1→ TRafter = TRUP

7This is of exponential complexity, as n events canbe distributed over the 2 hands in 2n ways

4.3.2. Note Velocities

In the basic algorithm (see section 3.5), wedid not take the velocities velevent of the DrumEventsinto account. It would of course be more con-vincing to use different animations for differ-ent velocities. use different animations fordifferent velocities will result in a more ‘nat-ural’ behavior: the ‘UP’ position should becloser to the drum surface for softer notes,and further away in the case of loud notes.The key frames [TRbefore, TRcontact , TRafter ]that make up a Stroke can therefore be de-fined as follows (see also figure 11):

TRbefore =TRDOWN + velevent × diff

TRcontact =TRDOWNTRafter =TRDOWN + velevent × eleventType

× diff

diff =TRUP − TRDOWN

vel=1.0el=0.25

vel=0.5el=1.0

vel=0.5el=0.5

DOWNTR

TRUP

vel=1.0el=0.5

Figure 11: The effect of different velocity andelasticity values.

4.3.3. Extra avatar animation

In this section, a number of extensions arediscussed that animate parts of the avatarthat were not animated at al in the basic sys-tem. This helps a great deal to make theavatar look ‘alive’.

4.3.3.1. The headThe head of the avatar is animated, to cre-

ate the effect that the avatar ‘follows’ his handswith his eyes. First, we create poses for thehead: one for each event type that is sup-ported by the hands. These poses rotate thehead so that the eyes are pointed at the as-sociated drum / cymbal. If we then use allevents that are played by e.g. the right handto create a key frame time line, the head ap-pears to ‘follow’ this hand.

4.3.3.2. The neckThe neck joint is used to make the avatar

nod with his head on the beat: ‘UP’ and ‘DOWN’

poses are defined for the neck joint, and foreach ‘beat’ note a Stroke is created. We haveused the SNARE event on the left hand as anapproximisation of beat notes.

Finding the ‘real’ beat in a MIDI file is farfrom trivial, and many other researchers haveaddressed this problem [3, 2, 5, 6]. Our sys-tem could very well be integrated with an in-telligent beat detector to create even betterlooking behaviour.

4.3.4. Key Frame Interpolation

After the basic key frames are set, the motionis fine-tuned by inserting extra key frames ac-cordiapplying a different interpolation scriptbetween certain key frame types (‘before’ /‘con-tact’ /‘after’). These scripts can also be differ-ent for each joints.

The example scripts shown in figure 12create rather convincing results, because thestick moves slightly ‘behind’ the hand, givingin a whip-like motion. These interpolationscripts are derived by observing the motionof a human drummer.

time

stam

p

time

stam

p

next

curr

ent

time

stam

p

time

stam

p

next

curr

ent

time

stam

p

time

stam

p

next

curr

ent

elbowwriststick

key framecurrent

key framenext to ‘contact’

from ‘before’to ‘before’

from ‘after’to ‘after’

from ‘contact’

Figure 12: example interpolation scripts forthe elbow and the wrist and stick joints

4.4. Implementation Notes

The Java3D API is used for the implementa-tion, because it is platform-independent andsupports a wide range of geometry file for-mats. Moreover, the our virtual theatre [12] iscurrently being ported from VRML to Java3D.

The SMF format (StandardMIDI File) is usedas intermediate file format between the per-cussion recogniser and the animation gener-ator. A great advantage of using the SMF is,that it allows us to use MIDI files (which arewidely available on the WWW) to test the an-imation generator independent from the per-cussion recognizer.

For the synchronisation of the animationand the sound, a seperate thread is used, whichlooks up the current audio position and ad-justs the start time of the animation accord-ingly.

5. CONCLUSION

We have chosen for a GUI-based pose editorand script-based key frame interpolation. Ascreenshot is shown in figure 3. This provesto be a very flexible solution, since there areonly a small number of poses, and they haveto be set only once for a new drum kit config-uration. The system could be extended withmotion capturing, dynamics and inverse kine-matics to create even more realistic behaviour,but at the cost of losing simplicity and flexi-bility. The interpolation scripts create natu-ral motion, while the hand assignment algo-rithm ensures the arms will not cross. Mo-tion capture would require the setup of thevirtual drum kit to exactly match the setupof the ‘real’ kit, so changes cannot easily bemade.

The animation results can be viewed atour web site: http://wwwhome.cs.utwente.nl/ kragtwij/science/

6. REFERENCES

[1] A. T. Cemgil and F. Gurgen. Classi-fication of musical instrument soundsusing neural networks. Technical re-port, Department of Computer Engi-neering, Bogazii University, IstanbulTurkey, 1997.

[2] P. Desain and H. Honing. Quantiza-tion of musical time: A connection-ist approach. Computer Music Journal,13(3):56–66, 1989.

[3] P. Desain and H. Honing. Can music cog-nition benefit from computer music re-search? from foot tapper systems tobeat induction models. In Proceedingsof the ICMPC, pages 397–398, Liege: ES-COM, 1994.

[4] J. D. Foley, A. van Dam, S. K. Feiner, andJ. F. Hughes. Computer Graphics: Princi-ples and Practice. Addison-Wesley Pub-lishing Company, second. edition, 1990.

[5] M. Goto and Y. Muraoka. Music un-derstanding at the beat level: Real-timebeat tracking for audio signals. In Work-ing Notes of the IJCAI-95 Workshop onComputational Auditory Scene Analysis,pages 68–75, Montreal, Aug. 1995.

[6] M. Goto and Y. Muraoka. A real-timebeat tracking system for audio signals.

In Proceedings of the International Com-puter Music Conference, pages 171–174,Sept. 1995.

[7] The Java3D API. http://java.sun.com/-products/java-media/3D/.

[8] J. Kim. Computer animation of pianist’shand. In Eurographics ’99 Short Papersand Demos, pages 117–120, Milan, 1999.

[9] M. Kragtwijk. Recognition of percus-sive sounds using evolving fuzzy neuralnetworks. Technical report, Universityof Otago, Dunedin, New Zealand, July2000. Report of a practical assignment.

[10] T. Lokki, J. Hiipakka, R. Hanninen, T. Il-monen, L. Savioja, and T. Takala. Real-time audiovisual rendering and con-temporary audiovisual art. OrganisedSound, 3(3):219–233, 1998.

[11] The general midi specification.http://www.midi.org/about-midi-/gm/gm1sound.htm.

[12] A. Nijholt and J. Hulstijn. Multimodal in-teractions with agents in virtual worlds.In N. Kasabov, editor, Future Directionsfor Intelligent Systems and InformationScience, Studies in Fuzziness and SoftComputing, chapter 8, pages 148–173.Physica-Verlag, 2000.

[13] M. Puckette. Pure data: Recent progress.In Proceedings of the Third Intercol-lege Computer Music Festival, pages 1–4,Tokyo, 1997.

[14] J. Sillanpaa et al. Recognition of acousticnoise mixtures by combined bottom-upand top-down processing. In Proceed-ings of the European Signal ProcessingConference EUSIPCO, 2000.

[15] Web3d consortium.http://www.vrml.org/.

[16] V. B. Zordan and J. K. Hodgins. Track-ing and modifying upper-body humanmotion data with dynamic simulation.In Computer Animation and Simulation’99. Springer-Verlag Wien, 1999.