Style synthesis of human body motion based on learnedspatio-temporal synergies.

Albert Mukovskiy1, Martin A. Giese1

1Section Computational Sensomotorics, Hertie Institute for Clinical Brain Research, &Centre for Integrative Neuroscience, University of Tubingen, Germany

July 27th, 2007

Presentation details

Slides and video presentation for the poster session at10th Tubingen Perception Conference (TWK 2007),10th Tubinger Wahrnehmungskonferenz, Tubingen, Germany,Friday 27th - Sunday 29th July 2007.

The abstract is published as

Mukovskiy A., Giese M.A. (2007) ”Style synthesis of human body motion basedon learned spatio-temporal synergies.” Bulthoff H.H., Chatziastros A., MallotH.A., Ulrich R. (eds): Proceedings of the 10th. Tubinger Perception Conference(TWK 2007) , Knirsch, Kirchentellinsfurt, 152.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 2 / 24

Abstract

Abstract

Research in motor control suggests that complex full-body movements might be controlled by the combination of motorsynergies, i.e. smaller control units that encompass only limited sets of degrees of freedom [1]. For the generation of perceptuallybelievable complex full-body movements it seems intriguing to devise algorithms that exploit the concept of synergies for thesynthesis of body movements in computer animation. Compared to simpler methods for movement synthesis, which typicallyspecify the same motion style for all degrees of freedom, such approaches would permit a more flexible design of animatedcharacters, where e.g. different movement styles can be specified for the upper and the lower body. Such animated stimuli arehighly interesting for psychophysical experiments that investigate how the movement styles of different body parts are combinedin perception.

We propose a new algorithm for motion morphing that is based on synergies that are derived from motion capture data by

unsupervised learning. The learning of synergies is based on the idea that jointly controlled degrees of freedom should show

similar timing variations over different action styles. Exploiting a probabilistic framework for dynamic time warping derived from

methods in proteomics [3,4], we characterize the time warps between the trajectories of different action styles separately for each

degree of freedom. Our algorithm determines synergies by clustering degrees of freedom with similar time warping functions

exploiting a probabilistic similarity measure. We also present an extension of a method for the modeling of complex motion styles

by linear combination of prototypical trajectories [2]. The new algorithm is suitable for the separate variation of motion styles for

individual synergies. This provides higher flexibility for the approximation of novel motion styles based on small amounts of

motion capture data, however resulting in very natural animations. In a psychophysical experiment, animation results obtained

with the new algorithm are compared with standard methods for motion synthesis. In addition, the proposed method is suitable

for investigating how different spatial motion components contribute to the perception of style properties, like emotions.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 3 / 24

Motivation

Motivation

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 4 / 24

Motivation

Research in motor control suggests that complex full-body movements mightbe controlled by the combination of motor synergies, i.e. smaller controlunits that encompass only limited sets of degrees of freedom[Bernstein,1967].

For the generation of perceptually believable complex full-body movementsit seems intriguing to devise algorithms that exploit the concept of synergiesfor the synthesis of body movements in computer animation.

Compared to simpler methods for movement synthesis, which typicallyspecify the same motion style for all degrees of freedom, such approacheswould permit a more flexible design of animated characters, where e.g.different movement styles can be specified for the upper and the lower body.

Such animated stimuli are highly interesting for psychophysical experimentsthat investigate how the movement styles of different body parts arecombined in perception.

Existing methods for the interpolation synthesis of trajectories morph all DoFs(joints) together. =⇒ Limited flexibility, large dictionaries required.

−→ Goal : Find clusters of DoFs (synergies) with ’similar’ style changes.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 5 / 24

Motivation

Example : Karate technique

Styles of arms and legs movements might bedifferent.

Modeling of all style combinations requires enoughprototypes to span the space of all combinations.

Combinatorial explosion for > 2 components.

Compute space-time alignments separately for eachjoint for morphings of the reference movement.

DoF in the same synergy result in the similar timewarping functions.

Motivation : Synergies are jointly controlled DoF=⇒ Their ’timing’ should change together.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 6 / 24

Methods

Methods

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 7 / 24

Methods

Dynamic Time Warping (DTW)

DTW establishes spatio-temporalcorrespondence, defining time-warpthat optimally alligns the sequencewith a reference sequence withnormalized timing.

[Rabiner & Huang(1993), Giese & Poggio(2000)]

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 8 / 24

Methods

Probabilistic DTWCost of pointwise alignement oftrajectories xi and yj for the discrete timepoints i and j is s(xi , yj).Cost of Alignment Path A containinglattice points (i , j) satisfies Bellmancondition:S(A) =S(AStart=⇒(i,j)) + s(xi , yj) + S(A(i,j)=⇒End)

Thermodynamic orprobabilistic formalism:

Partition function:(β is inverse temperature)Z (β) =

∑A e−βS(A)

Path probability:Prob(A;β) = e−βS(A)/Z (β)

[Miyazawa(1994), Koike etal(2004)]

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 9 / 24

Methods

Probabilistic DTW

Probabilities for pointwisealignments are given by the densityof all alignment paths through (i , j)

Algorithm:

1. Recursive cyclic propagation of auxillarysubsums of partition function using DP

(β →∞ ⇒ Viterbi algorithm)2. Compute probabilities for each pointwise

alignment - the probability of the cyclicpath through the point (i , j)

Pi,j = ZForwardi,j ZBackward

i,j

Lattice’s partial subsums, diagonal, vertical and horizontal(e.g. for forward propagation):

ZMi,j = (ZM

i−1,j−1 + ZVi−1,j−1 + ZH

i−1,j−1)e−βs(xi ,yj )

ZVi,j = (ZM

i,j−1 + ZHi,j−1)e−βg0 + ZV

i,j−1e−βgext

ZHi,j = (ZM

i−1,j + ZVi−1,j )e

−βg0 + ZHi−1,j e

−βgext ,

where g0 and gext depend on λ, and

Zi,j = ZMi,j + ZV

i,j + ZHi,j

Miyazawa(1994)

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 10 / 24

Methods

Maps comparison measures

In our approach we compare morphingmaps for joints angles, rather thenangle trajectories. This is equivalent tomethods comparing Markov Modelsthat generate such trajectories. Oneprobabilistic map represents a canonicalensemble of Markov Models for a given”temperature”. To measure thesimilarity of such maps we chose theprobability distance, integrated overtime, after cyclic alignment of the maps.

Some PDFs are characterized by multiple local peaks (ridges),and not a delta-function like path. In this case also the secondbest matches might specify valid correspondence betweenPDFs. This makes the method more robust against pathsambiguities - contrasting with standard matching methodswith DTW.

Introduction of new coordinates:diagonal time t and its orthogonalcompliment h, rewrite Z (T istime-length): Z (t, h) = Zmod(t−h,T ),h

We have for the distance betweenalignment paths distributions for jointangles m and n:Dm,n = mint′,h′ [

RKL(Zm(t + t′, h + h′), Zn(t, h))dt],

where (t′, h′) is shift and KL(f (h), g(h)) is Kullback-Leiblerdivergence between 1-dim probability distributions f (h) andg(h).

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 11 / 24

Methods

Pulling of alignment maps for composite morphings

A joint angle which belongs to one synergy in mapping along one principaldirection in feature space may belong to another synergy for another direction.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 12 / 24

Methods

Pulling of deterministic alignment maps

The exact pulling for deterministic maps can be derived from the logarithmicrepresentation of the maps, reconstructing the underlying Lie group structurefrom few given maps examples, using regularization. For the 1st orderapproximations (for maps as functions of t, c.f.[Sternberg(1964)])τv∗(t) =

∫τw ([dτv (t)/dt − 1]/[dτw (t)/dt])dt

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 13 / 24

Methods

Pulling of a probabilistic cyclic map along deterministic

For the pulling operations the probabilistic maps (in form of alignment pathspartition functions) are regarded as probabilities flows, which preserve the totalmarginal probabilities for each time slice. The main challenging problem here is torenormalize the results to the same given effective temperature.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 14 / 24

Methods

Representation of the emotion feature-space

1st approachIf all style combinations are given asexamples, use all pairwise maps. Pullthem back to form classes forCorrespondence Analysis.

2nd approachEach map from the individualexamples to the reference motion(here it is neutral walk sample) definesa single class for a CA using MDS.

For both two datasets from the Results section below we take only two morphs from the reference motion towards 2 different

styles. Each of these styles is characterized by changes of only one synergy: either synergy of the legs or of the arms. So, 1st and

2nd approaches are equivalent– here the only 2 classes are produced with single example in each class.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 15 / 24

Results

Results

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 16 / 24

Results

Toy problem:

Walking vs. Marching (with or without cell-phone)

Walking with phone Mixed prototype

Normal walking Marching

(Click on figures to play movies)

Weblinks for AVI movies: Normal Walking, Marching, Walking with cell-phone, synthesized: Marching with cell-phone.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 17 / 24

Results

Toy problem:

The videos of morphing of separate synergies :

1. Arms morphing (synergy 1): fromwalk with phone to normal walk

AVI movie file

2. Legs morphing (synergy 2): fromwalk with phone to march with phone

AVI movie file

(Click on figures to play movies)

If you cannot play movies online, download them into the same folder, where PDF file is stored locally.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 18 / 24

Results

Prototypes for the different styles of marching

5 natural captured trajectories:4 different marching styles and areference motion (normal walking);the marching styles combine 2 armmovement styles (straight and flexing)and 2 leg movement styles (withstraight and flexing knees) in all 4possible pairings.

a. arms and legs flexing (AVI)

b. arms straight, legs flexing (AVI)

c. arms flexing, legs straight (AVI)

d. arms and legs straight (AVI)

e. reference - normal walking (AVI)

a. b.

c . d .

e.(Click on figures to play movies)

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 19 / 24

Results

Synergy clusters recovered from different styles of marching

Two ways to compute discriminated distances for 1-dimensional MDS.

A. For two morphs from the reference motion towards two different motion samples we have two sets of TWF: τ1n and τ2

n ,

where index n specifies the joint angle. We introduce: all pairwise distances D1,1m,n between time-warping functions τ1

m and τ1n of

two joints angles m and n in the same morph 1; D2,2m,n - the same for morph 2;

D2,1m,n - is where morph of angle trajectory m belongs to morph 2, but morph of angle n belongs to morph 1; and D1,2

m,n - angle

m belongs to morph 1, n belongs to morph 2. To obtain the distance measure that optimally separates sets of joint angles whichco-vary in the same morph but vary differently in two different morphs, we introduce the normalized mixed distance matrixD∗m,n = (D1,1

m,n + D2,2m,n)/(D1,2

m,n + D2,1m,n). Next, we use the produced distance matrix D∗m,n for 1d MDS.

B. Another way to obtain the normalized discriminated distances for MDS is the following. We use single class distances

matrices D1,1 and D2,2 to compute their scattering matrices C1,1 and C2,2. First principal components (correspondent to the

first maximal singular values for C1,1 and C2,2) provide new coordinates xi and yi . Next, we remap data to the transformed

hyperbolic coordinates ξi = xi yi and ηi = x2i − y2

i and compute the matrix D∗∗ of Euclidian distances in the new coordinates.

The result of 1d-MDS for D∗∗ is on the right plot. Both methods provide the same clustering result if the synergies are

independent - the set of joint angles of one synergy is stable in both morphs.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 20 / 24

Results

Synergy clusters recovered from different styles of marching

Matrices of pairwise distances between joint angles.

Matrix of distances D; rows and columns represent joint angles re-odered alongthe projection of the 1st principal component (one-dimensional MDS).The two alternative methods are described in the previous slide.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 21 / 24

Results

Morphing results for the 4 examples of marching

The synergies underlying these four prototypes are estimated by clustering of timewarping paths. The estimated synergies can then be recombined to generate novelmovement patterns that look quite natural:

a. b. c.

a. Separate morphing of the legs synergy followed by the morphing of armssynergy (AVI)

b. Morphing of arms synergy followed by morphing of legs synergy (AVI)

c. Morphing by classical motion blending without spatially local motioncomponents (prototypes a =⇒ c =⇒ d), (AVI)

(Click on figures to play movies)

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 22 / 24

Acknowledgements

Acknowledgements

Supported by DFG, HFSP and the Volkswagenstiftung.

Authors thank L. Omlor for help with the motion capture of walking withcell-phone.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 23 / 24

References

References

[1]. Bernstein, N. (1967) ”The Coordination and Regulation of Movements.” Oxford : Pergamon.

[2]. Giese, M.A., Poggio, T. (2000) ”Morphable models for the analysis and synthesis of complex motion patterns.” Int. J. ofComputer Vision, 38, 59-73.

[3]. Koike, R., Kinoshita, K., Kidera, A. (2004) ”Probabilistic description of protein alignments for sequences and structures.”Proteins, 56, 157-166.

[4]. Miyazawa, S. (1994) ”A reliable sequence alignment method based on probabilities of residue correspondences.” ProteinEng., 8, 999-1009.

[5]. Rabiner, L., Huang, B.H. ”Fundamentals of speech recognition.” Prentice Hall Inc., Englewood Cliffs, N.J., 1993.

[6]. Sternberg, S. (1964) ”Lectures on Differencial Geometry.” Prentice Hall Inc., Englewood Cliffs, N.J., 1964.

[7]. Sugiyama, M. (2006) ”Local Fisher Discriminant Analysis for Supervised Dimensionality Reduction.” Proc. Of the 23rd Int.Conf. on Machine Learning, Pittsburg, PA, 2006.

[8]. Yu, Y.-K., Hwa, T. (2001) ”Statistical significance of probabilistic sequence alignment and related local HMMs.” J. Comp.

Biol. 8, 249-282.

Mukovskiy, Giese (HIH) TWK 2007 July 27th, 2007 24 / 24