A Hybrid Grammar-Based Approach forLearning and Recognizing Natural Hand Gestures

Amir Sadeghipour and Stefan KoppFaculty of Technology, Center of Excellence ‘Cognitive Interaction Technology’ (CITEC),

Bielefeld University, P.O. Box 100131, D-33501 Bielefeld, Germany

Abstract

In this paper, we present a hybrid grammar formalismdesigned to learn structured models of natural iconicgesture performances that allow for compressed repre-sentation and robust recognition. We analyze a datasetof iconic gestures and show how the proposed Feature-based Stochastic Context-Free Grammar (FSCFG) cangeneralize over both structural and feature-based varia-tions among different gesture performances.

IntroductionNatural gestures are becoming increasingly popular as ameans to interact with technical systems, from tablet de-vices, to smart TVs, to social robots. However, the ges-tures that can be used are severely restricted to what canbe recognized robustly. Although there have been advancesin vision-based motion tracking, these have been usuallyconfined to predefined, reliably discernable movement pat-terns. Successes in this area include Hidden Markov Models(HMM) (Yamato, Ohya, and Ishii 1992; Yoon et al. 2001;Turaga et al. 2008), HMM in combination with the Kalmanfilter (Ramamoorthy et al. 2003), or the Ordered MeansModel (OMM) (Großekathofer et al. 2012). However, al-though these gestures are often reminiscent of familiar phys-ical motions, they remain artificially defined gesture com-mands.

Natural gesturing, on the other hand, has for the most partresisted attempts at reliable recognition and interpretation.One reason is the wide variability or apparent lack of struc-tural patterns that can be tapped by classifiers. For instance,iconic gestures are naturally performed during communica-tion to refer to objects or events by depicting aspects of theirvisual-spatial properties. However, one can refer to a ball bydrawing a circle with an index finger, with either hand orboth hands simultaneously, clockwise or counter-clockwise,slow or fast, small or big, once or repeated several times.Hence, recognizing and interpreting an iconic gesture re-quires learning discriminative models that can recognize thedifferent variants without overgeneralization. Furthermore,it needs to be robust against minor motion deviations or mo-tion tracking noises.

Copyright c© 2014, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

A number of researchers have tried to develop linguis-tically motivated grammar-based approaches that can cap-ture highly complex hierarchical structures, and recognizehuman actions and activities (Bobick and Ivanov 1998;Pastra and Aloimonos 2012). In this view, there is a gram-matical structure behind nonverbal behavior that can be usedto recognize or “parse” valid performances. This idea buildson the strong assumption that human actions and move-ments consists of primitives (like morphemes or words inlanguage) that are arranged in compositional structures ina limited number of ways. However, when looking at ges-tures or, more generally, human motor actions, it is not clearwhat such primitives may be, whether they are stable, andwhat they are composed of. Hence, most grammar-basedapproaches for activity recognition have used a syntax thathas been predefined by experts (Ivanov and Bobick 2000;Moore and Essa 2002; Chen, Georganas, and Petriu 2008).

In this paper, we present an approach to learning struc-tured models of gesture performance that allows for a com-pressed representation and robust recognition of naturaliconic gestures. We propose a hybrid approach that, at thesame time, strives to extract the structural-syntactic patternsof gestures while identifying low-level statistical regularitiesthat constitute primitive building blocks. After reviewing re-lated work on grammar-based approaches used in computervision and behavior recognition, we propose Feature-basedStochastic Context-Free Grammars (FSCFG) as a hybridapproach that extends Stochastic Context-Free Grammars(SCFG; proposed by Stolcke 1994) by using sets of prob-abilistically defined features as terminal symbols. The pro-posed hybrid grammar has been applied to learning and rec-ognizing diverse natural iconic gestures from noisy move-ment data delivered by standard tracking sensors. Before re-porting the classification results, we describe our recordedcorpus comprising 1739 iconic gestures performed in refer-ence to twenty different objects. We also analyze the strate-gies the participants used to depict an object iconically, aswell as the variations among their gestures.

Related workGrammar-based formalisms have become increasingly pop-ular in research on vision-based activity recognition dur-ing the last 15 years (see Chanda and Dellaert 2004 for asurvey). Decomposing complex behaviors into primitive ac-

tions (symbols), and given ways of performing a behavioror an activity are both described as strings of symbols. Therules of a grammar model determine which combination ofprimitive actions comprises a valid performance of a behav-ior. There are several reasons why a grammar model is ap-pealing to the representation of complex activity patterns.Grammar can be elegantly represented, its structure is inter-pretable, and it can be used to formulate concise descrip-tions of action patterns. Many studies have applied syntac-tic approaches to recognizing different types of nonverbalbehavior, i.e. the Finite State Machine (FSM) for hand ges-ture recognition (Hong, Turk, and Huang 2000) or Context-Free Grammar when used to represent and recognize hu-man actions and interactions (Ryoo and Aggarwal 2006;Kitani, Sato, and Sugimoto 2006).

To address the uncertainty created by noisy sensors orcomputer vision, syntactic approaches have been extendedto include probabilities early on. Stochastic Context-FreeGrammar (SCFG) (Stolcke 1994) have been applied to dif-ferent vision-based applications such as simple hand ges-ture recognition Ivanov and Bobick (2000) or surveillancein parking. Most of these approaches have used the Earley-Stolcke parsing algorithm (Stolcke 1994) for efficient prob-abilistic parsing. Minnen, Essa, and Starner (2003) use sim-ilar techniques for activity recognition during the Tower ofHanoi task, and Moore and Essa (2002) to recognize multi-agent activities in blackjack card games. All of these sys-tems define the applied grammar syntax manually and haveapplied it only for task recognition at levels of rather com-plex actions with clear-cut compositionality with respect toboth units and structures. None faced the challenge of learn-ing a grammar from samples of action performances. Thereare two exceptions. They are work by Kitani, Sato, and Sug-imoto (2006) on very simple activity recognition, and byZhang, Tan, and Huang (2011) who learned SCFGs for ap-plications such as the recognition of gymnastic exercises,traffic events and multi-agent interactions.

The idea of attributed grammars was originally proposeby Knuth in 1968 for linguistic frameworks. The idea ofcombining statistical and syntactic approaches using linguis-tic frameworks for nonlinguistic applications can be tracedback to the 80’s, when Tsai and Fu (Tsai and Fu 1980;Fu 1986) proposed attributed grammars as a means to in-tegrate statistical regularities into syntactic pattern analysis.However, many of the applied grammar formalisms in be-havior analysis have not integrated statistical and syntac-tic aspects within the same approach. That said, they haveprocessed data at two separate levels. The first is low-levelsegmentation and symbolization, where statistical methodssuch as HMMs are used (Zhang, Huang, and Tan 2006;Chen, Georganas, and Petriu 2008; Ivanov and Bobick2000). The Second involves the high-level recognition oflonger range pattern with the aid of syntactic methods suchas SCFG. Exceptions to this method are studies that haveproposed attributed grammar formalisms in context such asactivity recognition (Damen and Hogg 2009) or the detec-tion of abnormal events when parking a car (Joo and Chel-lappa 2006).

Definition of Feature-Based StochasticContext-Free Grammar

Our proposed framework, FSCFG, is an extension of theprobabilistic SCFG framework proposed by Stolcke (1994).An SCFG is defined by the following symbol sets:

• Σ, a finite set of terminal symbols.• N , a finite set of non-terminal symbols.• S ∈ N , a start symbol.• R, a finite set of rules, each of the form X → λ with the

left-hand side X ∈ N and the right-hand side λ ∈ (N ∪Σ)∗. Each rule is associated with a probability P ∈ [0, 1](shown in brackets in front of each rule).

Based on this definition, an FSCFG is defined by the fol-lowing additions (see Figure 1):

• F , a finite set of n features {f1, . . . , fn}. A feature set isdefined as F={f1=v1, . . . , fn=vn}.

• Each terminal t ∈ Σ is represented by a weighted set of lfeature sets: t={(Fi, wFi

) | i=1, . . . , l; wFi∈]0, 1[ }.

• Each feature of a terminal is weighted for all its featuresets equally, given by {(fi, wfi) | i = 1, . . . , n;wfi ∈ R}.

Through the definition of n features used to form l featuresets, FSCFG allows for feature-based representation of datasamples in an n-dimensional feature space. On this basis, asillustrated in Figure 1, each terminal is not represented asan atomic symbol, but as the prototype of a cluster with nfeatures. The importance of the i-th sample (i.e. feature set)to the cluster is determined by (Fi, wFi

), and the importanceof the i-th feature within a cluster is determined by (fi, wfi).

Accordingly, a given string of symbols to an FSCFG isan ordered set of n-dimensional samples. For instance, inthe case of working with movement streams, a symbol cor-responds to the feature-based representation of a movementsegment. In this way, statistical (feature-based) and syntac-tic (structure-based) processing are unified within a hybridframework that learns not only rule-based syntactic modelsof symbols, but also takes into account the statistical rela-tions in the underlying features spaces.

Parsing with FSCFGIn contrast to SCFG, FSCFG can compute the similaritybetween two symbols (or terminals) in an n-dimensionalfeature space. Hence, while parsing with an FSCFG, thematch between a terminal against an input symbol is com-puted probabilistically, depending on the measured sim-ilarity between the parsing terminal and the parsed in-put symbol in their n-dimensional feature space. Specif-ically, parsing the i-th symbol xi = {Fxi} of aninput string x = {x1, . . . ,xm}, through a terminalt={(F1, wF1), . . . , (Fl, wFl

)} can be measured probabilis-tically (as opposed to binary true/false matching in SCFG).The parsing probability is given by

p(xi|t)=l∑

j=1

wFjg(xi|Fj), (1)

Statistical

Syntactic NT1 NT2 NT3

NT11 NT12

S

t11 t12

...

fnf2f1 ...

wF

wf

SCFG represenation

S ⇒ NT1 NT2 NT3 [1]NT1 ⇒ NT11 NT12 [0.5]NT1 ⇒ t1 [0.5]NT11 ⇒ t21 [1]NT12 ⇒ t22 [1]NT2 ⇒ t3 [1]NT3 ⇒ t4 [1]

F1Feature sets:

Features:

t1 t2 t3

F2 Fl

Figure 1: Hybrid model of FSCFG, where terminal symbolsin the syntactic structure serve as the interface to the statis-tical feature-based representations.

where g(xi|Fj) is a multidimensional Gaussian distribu-tion. In order to keep this computational step simple andefficient, we make the naıve assumption that the featuresare statistically independent of each other. Hence, the co-variance matrix between n features is diagonal, and in thisway the multidimensional Gaussian is defined as product offeature-specific Gaussians:

g(xi|Fj)=

n∏k=1

wfk gauss(fk,xi|µfk , σfk), (2)

where the mean of Gaussian function µfk is set to the fea-ture value of fk,Fj

; and the standard deviation σfk can beset for each feature separately (i.e. equal to the standard de-viation of each feature in the training data). As a result ofapplying this equation, the parsing probability computed ina “scanning step” indicates how well a symbol is parsed by aterminal. Thus in FSCFG, the scanning step of xi through tmultiples both forward and inner probabilities with p(xi|t).

Besides parsing, an FSCFG grammar can be learned fromsamples of strings, in a supervised manner by exploring bothspaces of possible structures and parameter values. Learn-ing the structure of an FSCFG corresponds to the process offinding the optimal set of rules. The parameters that need tobe optimized during learning are P (the probability of eachrule),wF (the weight of each feature set), andwf (the weightof each feature) for each terminal.

Learning the structure of an FSCFGIn order to find an optimal set of rules, first an initial setof rules are generated which fit a given sample of stringsmaximally. To this end, for each symbol in the string, wegenerate a terminal with a single feature set t={(F1, wF1)}.In addition, a non-terminal X and a lexical rule X → t [1]is generated for each terminal. In the end, a start rule S →X Y . . . [1] comprising the entire given string – by produc-

ing the sequence of all non-terminals – is added to the set ofrules.

Upon initialization, the structure is generalized by apply-ing the merge and chunk operators proposed by Stolcke. The

merge operator merge(X1, X2) = Y replaces all occur-rences of the non-terminals X1 and X2 with a new non-terminal Y . The chunk operator chunk(X1 . . . Xk) = Yreplaces all occurrences of the ordered sequence of the non-terminals X1 . . . Xk with a single new non-terminal Y andadds a new chunking rule Y → X1 . . . Xk to the grammar.These operators simplify the grammar by decreasing its De-scription Length (DL) (Rissanen 1983). The loss measureduring this process is the negative logarithm of the Bayesianposterior parsing probability. The likelihood term, which in-dicates how well the given samples fit the learned model, isset to the parsing probability of the samples; the prior proba-bility of a grammar is set to its DL, which is proportional tothe code length in bits needed to store the grammar’s rules.In this way, by using a Bayesian loss measure, the grammarstructure is modified towards a trade-off between general-ization (or simplicity) and fitting of the model to the givendata.

Given the Bayeisan loss measure for an FSCFG, similarlyto SCFG, different search strategies can be applied to find arelatively optimal grammar structure. (1) In best-first searchall candidates of merge and chunk operators are pooled andthe best one locally is selected. To overcome local minima,if the loss begins to increase, further modification steps arechecked for a sudden drop of loss (look-ahead steps). (2)Multilevel best-first search is very similar to the best-firstsearch, except that it searches for the best merge and chunkcandidates at two different levels. Only after the best localmerge candidate is selected, is the best local chunk candi-date chosen. (3) In beam search, instead of always choosingthe best local modification candidate, different possible se-quences of merge and chunk operators are checked as a treeof possible moving paths in the structure space.

When providing samples to learn the structure of anFSCFG, the process of adding new rules to the grammar andoptimizing them by searching the structure space, is repeatedfor each sample that cannot be parsed by the grammar witha high enough probability.

Learning the parameters of an FSCFGThe parameters of an FSCFG are learned and optimized dur-ing both learning the structure and parsing new input strings,after a default initialization at the beginning.

Rule probabilities (P ): Similarly to SCFG, the proba-bility of each rule is determined from how often the rulehas been invoked for parsing, normalized by the sum of allinvocations of the rules with the same left-hand side non-terminal.

Weights of features (wf ): The weights of features areset for each terminal individually, and they are set for allfeature sets of a terminal equally. This means, wf refers tothe weight of feature f in the terminal t for all its l featuresets {Fi|i = 1, . . . , l}. wf is defined inversely proportionalto the standard deviation of the values of f among all featuresets, given by

wf (t)=1

std({f ∈ Fi|Fi ∈ t; i = 1, . . . , l}) + 1(3)

Hence, the higher the variance of a feature within a termi-nal, the less discriminative is the feature for the terminal andthe less it contributes to the parsing through that terminal. Inother words, during parsing, each terminal is more sensitiveto its less variable features and in this way an FSCFG distin-guishes between variant and invariant features for each ter-minal. The sensitivity of each terminal to the given featuresdepends on the parsed input symbols and can lead to differ-ent weightings at different positions of the grammar rules.Thus, some variant features for a terminal may be countedas invariant for other terminals and vise versa.

Weights of feature sets (wF): Computing the weightof each feature set of a terminal employs a counterwhich is normalized by the sum of its values in eachterminal. Initially, this counter is set to one yieldingt={(F1,

1l ), . . . , (Fl,

1l )}. This set of feature sets can be ex-

tended in two ways: (1) During parsing, when terminal tparses a symbol xi, the symbol – which is represented asa single feature set – is added to the terminal, with an initialcounter of one. In this way, parsing reshapes the terminalsof a grammar towards the features of the parsed symbols.(2) During learning the structure of a grammar, when merg-ing two lexical non-terminals merge(X1, X2) = Y withX1 → t1 and X2 → t2, the right-hand side terminals – e.g.t1={(F1, 1)} and t2={(F2, 1)} – are also merged. This re-sults in a new rule Y → t, where the new terminal t isa cluster of the old terminals: t={(F1,

12 ), (F2,

12 )}. These

two incorporation steps for feature sets during both parsingand learning may lead to terminals that are too large withtoo many feature sets and therefore additional computationalcosts. To increase the efficiency, we perform a pruning stepthat combines similar feature sets. In this case, the counterof a new feature set is set to the sum of the counters of theones replaced. As a result, the new feature set gains moreweight and therefore more influence in the representation ofits terminal.

Through the computation and optimization of the pa-rameters wF and wf during both learning and parsing, anFSCFG learns the set of its terminals continuously, dynam-ically and incrementally. This is a useful feature when deal-ing with continuous input strings without a clear defini-tion of compositional elements as symbols, such as contin-uous human motion data. In many studies that have appliedgrammar-based approaches to learning models of human ac-tions, a symbolization step has been performed as prepro-cessing for the input data. For instance, Wang, Lorette, andBouthemy (1991) and Ivanov and Bobick (2000) appliedHidden Markov Models (HMMs) to learn symbols as pro-totypes of movement segments before applying a SCFG. InFSCFG, by representing each terminal as weighted featuresets, there is no clear cut off between these two processinglevels and the statistical symbolization of terminals is ho-mogeneously integrated in the learning process of syntac-tic structure. Furthermore, such an integrated symbolizationprocess also takes the learned syntax of the grammar intoaccount, and may lead to different symbolization results fordifferent parts of a the grammar.

Handling uncertain inputAn important challenge in grammar learning and parsing isuncertainty in the input data. This may lead to deletion er-rors when an expected symbol is missing in the input stream,insertion errors when symbols occur spuriously, or substitu-tion errors when a symbol is parsed with the wrong terminal.Since FSCFG can parse any symbol by any terminal throughfeature-based parsing, the substitution error is handled im-plicitly. To deal with the insertion and deletion errors duringparsing, we introduce special symbols (cf. Ivanov and Bo-bick 2000):• A new terminal skip ∈ Σ with no feature set• A new non-terminal SKIP ∈ N , mapped to the skip

terminal through the lexical rule SKIP → skip.• A new terminal ε ∈ Σ which is used to create null pro-

duction rules of the form X → ε for any X ∈ N .skip handles insertion errors as it treats all symbols

equally and is assigned a small probability. Consequently,a parsing path through skip is costly and is only used if oth-erwise parsing would fail. To enable this path, we add to anylexical ruleX → t the new alternative rule X → SKIP X .In addition, the start rule of the form S → λ receives thealternative rule S → λ SKIP . Finally, a new lexical ruleSKIP → skip is added to the grammar. ε handles dele-tion errors because it forces its associated non-terminal tobe ignored during parsing. Additionally, we add to each lex-ical rule X → t, the alternative rule X → ε with a smallprobability. In this way, the parsing algorithm ignores theoccurrence of the terminal t only if need be.

Note that, from parsing to structure learning, deletion andinsertion errors exchange their roles as cause and effect.Learning a grammar rule based on an input string with aninsertion error results in adding an erroneous lexical rulewhich then causes a deletion error when parsing a correctstring. On the other hand, when learning a rule from a stringwith a deletion error, the missing lexical rule in the learnedgrammar causes an insertion error when parsing a correctstring. Using these error handling symbols when confrontedwith noisy data, the FSCFG framework should entertainboth hypotheses that either the learned grammar structureis incorrect or the given input string is noisy. Hence, whena given input string uses a deletion handling rule such asX → ε, the structure of the grammar is optimized by addingan alternative for each rule containingX in the form of Y →λXν, by omitting the missed non-terminal as Y → λν. Fur-thermore, in the case of using a skip-rule, the correspondingterminal skip and non-terminal SKIP are renamed to newsymbol names and are inserted into the set of grammar rules.

These error handling rules result in a growing FSCFGwhich also contains rules and symbols for noisy data. How-ever, after providing enough training data, the noisy ruleswill be used less for parsing and will result in low probablerules. In a successive pruning step after learning, these rulescan be removed from the grammar as needed.

3D Iconic Gestures DatasetTo test our grammar framework, we have recorded a datasetof 1739 iconic gestures performed by 29 participants (20

males and 9 females; from 9 different declared ethnicities) todepict different objects. All gestures were recorded using theMS KinectTM. Participants were presented with twenty vir-tual 3D models of simple and complex objects on a screen(see Figure 2). One of the objects moved to the center ofthe screen, enlarged and rotating, and participants signaledwhen they felt ready to perform a gesture for that object.Then, the object disappeared and a photo of an addresseeperson was shown to whom the object had to be depictedgesturally. After the participants retracted their hands orgave a verbal signal, the face disappeared and the next ob-ject was shown (order randomized). The participants weretold that their gestures would be videotaped and shown toother participants, who would have to recognize the objectfrom the same set. Each object was shown three times. Thisprocedure resulted in approximately 87 gestures per object,each recorded in color video, depth video and 3D motion ofthe skeleton (in 30 fps). This three-dimensional iconic ges-ture dataset (3DIG) has been made available online1.

Sphere Egg Ellipsoid Cylinder Rectangular box Cube Cone Pyramid Trapezoid Parallel trapezoid

House Apple Vase Bulb Bottle Chair Table Umbrella Dumbbell Teapot

Figure 2: 3D object models used in the study; in this situa-tion the cube is to be described next.

Analysis of the gestures performedAnalyses of the video data revealed that the participantsused four different representational techniques to depict vi-sual or functional features for each object: (1) Drawing the3D or 2D contours of an object in the air. (2) Enacting anaction performed on an imaginary object (e.g. “throwing”to refer to the ball). (3) Static posturing with the hand(s)held still in order to form an object (e.g. forming a con-cave circular shape with both hands). (4) Dynamic postur-ing where drawing and static posturing are combined, as ifthe gesturer is touching the surface of an imaginary object.Figure 3 shows the use of each technique among gesturesperformed for each object or object set. Note that withina single gesture performance, different techniques could beused sequentially or simultaneously. As shown, the drawingtechnique was the most dominant choice followed by dy-namic posturing. Static posturing was used more frequentlyfor simple objects with abstract geometrical shapes, whereas

1http://projects.ict.usc.edu/3dig/

Structuralvariability

Examples of variation

Degree ofsimplification

Drawing a 3D shape or a 2D projection.

Ordering First, referring to the triangle shape of a coneand then to its circular bottom, or vise versa.

Repetition Drawing the round shape of a circle once,twice or three times.

Handedness Drawing a circle with one hand or both hands.

Feature-basedvariability

Examples of variation

Direction Drawing while moving a hand upward ordownward, clockwise or counter-clockwise.

Velocity Drawing fast or slow.Size Drawing a small or a big circle.Position Drawing in front of head or chest.Projection Drawing a horizontal or vertical projection.Form Making curved movements or straight ones.

Table 1: Structural and feature-based variabilities amongiconic gesture performances.

the enacting technique was used preferentially for complexeveryday objects.

Except for static posturing, the wrists movements madethe greatest contribution to the gestural depiction. We thusconcentrate in the following section on wrist trajectories ingesture space, which can be better captured with low-costtracking systems such as MS Kinect TM than, e.g., hand-shape. Our goal is to learn a hybrid grammar for iconic ges-ture trajectories.

0

0.2

0.4

0.6

0.8

1

AppleBottle Bulb

ChairCone

Cube

Cylinder

Dumbbell Egg

Ellipso

idHouse

Paralle

l trap

ezoid

Pyramid

Rectan

gular box

SphereTable

Teapot

Trapez

oid

UmbrellaVas

e simple objectscomplex objects0

0.2

0.4

0.6

0.8

1

Applied representational techniques per object set

Enacting

0

0.2

0.4

0.6

0.8

1

Simple

objects

Complex objec

ts

BulbChairCone

Cube

Cylinder

DumbbellEgg

Ellipso

idHouse

Paralle

l trap

ezoid

Pyramid

Rectan

gular box

SphereTable

Teapot

Trapez

oid

UmbrellaVas

e

simple objects complex objects0

0.2

0.4

0.6

0.8

1

Applied representational techniques per object set

DrawingDynamic posturingStatic posturingEnacting

simple objects complex objects0

0.2

0.4

0.6

0.8

1

Applied representational techniques per object set

DrawingDynamic posturingStatic posturingEnacting

100%

80%

60%

40%

20%

Figure 3: The rates of different representational techniquesused to refer to each 3D virtual object.

The structural property most commonly acknowledgedfor gestures is the division of a gesture into different phases(Kendon 1972): (1) pre-stroke preparation to move thehands from a rest position to a start position; (2) stroke, themeaningful and most effortful phase; (3) post-stroke retrac-tion to move the hand back to a rest position. Furthermore,even during the stroke phase, some parts of the movementmight be transitional (e.g. when continuing to draw in adifferent location). This means that in order to interpret acommunicative iconic gesture, the irrelevant parts (i.e. pre-stroke, post-stroke, transitional sub-movements) need to be

ignored. This structural property should be identified whenlearning a grammar for gestures.

To get a better idea of the variabilities involved in ges-ture, we began by analyzing the variation among the gesturesobserved in the corpus. Table 1 reports the most prominentstructural (or syntactic) variabilities, which can lead to verydifferent gestures performed for the same object. Below thelevel of syntactic variation, there are spatiotemporal varia-tions of features that represent statistical variabilities. At thislevel, the spatiotemporal features can be invariant and thuscharacteristic for a specific gesture class (e.g., a curved tra-jectory for round objects), or they can be variant and thuslikely irrelevant with respect to a specific technique (e.g.movement direction while drawing). In the next section, wereport on how FSCFG is able to cope with both kinds ofvariability while generalizing over different gestures, sep-arating different performing ways and determining variantand invariant features from seemingly meaningless aspectsof a gesture.

ResultsWe applied the FSCFG framework to learn a generalizedmodel from the gesture performances performed for each ofthe twenty objects in the 3DIG dataset. The learned grammarmodels were then tested by classifying the unseen gestureperformances, based on their parsing probabilities.

Before learning a grammar model of hand gestures, thecontinuous wrist movement trajectories needed to be seg-mented into discrete symbols. Inspired by the concept ofguiding strokes (Kopp and Wachsmuth 2004) as segmentsof movement trajectories of hand gestures, we segmentedthe continuous trajectories on the minima of their velocityprofiles. Since wrist movements are slower while changingdirection or drawing a sharp curve, the resulting segmentswere in the form of simple curves or straight lines. Then,each segment was normalized as follows (see the figure inTable 2). First, the start position of the segments was trans-lated to the origin. Second, the segments were rotated aboutall three spatial axes and mapped on the x-y plain. Finally,segments were resized while maintaining their width-heightproportions. The normalized trajectory was then resampledat equal distances on the x axis, whose y coordinates repre-sented the form of the trajectory in 5 dimensions. As shownin Table 2, each segment is represented by 18 features (as asingle feature set). These extracted features reflect the statis-tical variabilities that were shown in Table 1.

Learning FSCFG models of gesturesOne FSCFG model was learned for all gesture performancesfor each object. At the beginning of the learning process foreach model, a small subset of gestures (e.g. three perfor-mances) was used to initialize a maximally fitting FSCFG.For this purpose, the following set of symbols and rules wereadded to the grammar: a terminal symbol for each segment,a lexical rule to produce each terminal, a rule producing themovement sequence for each hand, and a start rule produc-ing the whole gesture as a sequence of left and right handnon-terminals successively.

l

x

z hn

1.0

y1

y

y2

y3y4y5

Features Dim. nr. Calculation/NotionSamples’ heights 5 (y1, y2, y3, y4, y5)

Start to end vector 3 l = (xl, yl, zl)

Bounding box 2 (‖ l ‖, ‖ h ‖)Normal vector 3 n = (xn, yn, zn)

Direction of concavity 3 ∈ {−1, 1} for each dim.Average speed 1 ‖ l ‖ / durationStart time 1 start time of movement

Table 2: Extracted features from each movement segment.The blue arrow at the top represents a segment of a wristmovement trajectory, and the red arrow shows the same tra-jectory after normalization.

After this initial batch mode, the remaining samples weregiven to the algorithm one by one in online mode. In thismode, each training sample is first parsed, and if the parsingprobability is above a given threshold, the grammar rules areadopted in three steps: (1) updating the rule probabilities, (2)adding the parsed segments as new feature sets to the pars-ing terminals, and (3) updating the weights of features andfeature sets (wF and wf ). In case the parsing probability isbelow the threshold, first the grammar is extended by thenew rules to fit the given sample maximally. Consequently,the resulting suboptimal structure is optimized by applyingmerge and chunk operators, and in this way, searching thestructure space according to the Bayesian loss measure. Inthis application of FSCFG, applying the naıve but fast best-first search algorithm with two look-ahead steps, we couldachieve classification results as accurate as the multilevelbest-first and beam search algorithms.

Figure 4 shows an example of an FSCFG model learnedfrom drawing gestures performed for the sphere object. Thegrammar possesses three start rules that represent three mainways of performing a gesture for sphere. Each of the startrules produces one or two non-terminals, whereas each ofthe non-terminals produces the movement of either the leftor right wrists as strings of lexical non-terminals. Thesestrings represent more specific ways of performing the samegesture.

Each lexical non-terminal produces a terminal that con-siders the regularities at the level of features. As illustrated,each terminal is the prototype of a group of feature sets,where the strength of each connection (i.e. wF) indicatesthe weight of the corresponding feature set for each termi-nal. It can be seen that some of the terminals link to one

MG16 => t13 (20)[1.0] MG20 => t17 (20)[1.0] MG19 => t16 (20)[1.0] MG18 => t15 (20)[1.0] MG17 => t14 (13)[0.54] | MG33 MG17 (11)[0.46] MG22 => t19 (16)[1.0] MG21 => t18 (12)[0.86] | SK24 MG21 (2)[0.14] MG15 => t12 (18)[0.41] | MG34 MG15 (26)[0.59] NT15 => t11 (4)[1.0] NT16 => t13 (23)[0.51] | SK21 NT16 (22)[0.49] NT17 => t14 (4)[1.0] NT18 => t15 (4)[1.0] NT19 => t16 (4)[1.0] NT20 => t17 (10)[1.0] NT22 => t19 (10)[1.0] SK21 => sk18 (22)[1.0] SK22 => sk19 (42)[1.0] MG33 => t29 (19)[1.0] SK24 => sk21 (2)[1.0] MG34 => t30 (33)[1.0] SK26 => sk23 (1)[1.0] NT28 => t25 (3)[1.0] NT30 => t27 (3)[1.0] NT31 => t28 (3)[1.0] NT32 => t29 (3)[1.0] NT33 => t30 (3)[1.0] NT34 => t31 (2)[1.0] NT35 => t32 (2)[0.67] | SK39 NT35 (1)[0.33] MG40 => t35 (4)[0.80] | SK40 MG40 (1)[0.20] NT37 => t34 (2)[1.0] NT39 => t36 (2)[1.0] SK39 => sk34 (1)[1.0] SK40 => sk35 (1)[1.0] SK41 => sk36 (1)[1.0]

S => L24 R24 (24)[0.69] R33 (9)[0.26] L39 (2)[0.05] L24 => MG16 MG20 MG19 MG18 MG20 MG16 MG19 MG18(20)[0.45] L24 SK22 (20)[0.45] NT15 NT16 NT17 NT18 NT19 (4)[0.10] L39 => NT33 NT34 NT35 MG40 NT37 MG40 NT39 (2)[1.0] R24 => MG17 MG22 MG21 MG17 MG15 MG21 MG15 (6)[0.24] R24 SK26 (1)[0.04] NT20 MG22 NT22 (10)[0.40] MG17 MG22 MG21 MG15 MG21 MG15 (7)[0.28] MG15 (1)[0.04] R33 => MG33 NT28 MG34 NT30 NT31 NT32 (3)[0.30] R33 SK41 (1)[0.10] MG33 NT28 MG34 NT32 (5)[0.50] MG33 NT28 NT32 (1)[0.10]

wf wF

(counter)[probability]

Features ( f1, …, f18)

alpha_signaxay

sample_1_ysample_3_ysample_5_ysample_7_ysample_9_y

sample_maxynormal_vec_xnormal_vec_y

velocityresize_factor

start_time_idxstart-end_xstart-end_y

f F

Figure 4: An example of a learned FSCFG for sphere.

or two strong feature sets (such as MG15, MG21 and MG22)that they represent their typical movement segments. In con-trast, non-terminals such as MG17 or MG40 are representedthrough a set of equally influential feature sets. These ter-minals represent parts of a gesture that vary widely and arethus most likely less informative (e.g. the pre-stroke or post-stroke phases). At the next level of statistical regularities,feature sets are connected to the same features but with dif-ferent weights (i.e. wf ). As illustrated, some of the featureshave a higher impact on their feature sets and consequentlyon the representation of their terminals than others. Smallsets of highly-weighted features thereby represent “invari-ant” features, i.e. features that are most characteristic of aparticular terminal (and thus a specific part of a gesture).

In sum, the feature-based representation of terminals gen-eralizes over many spatiotemporal deviations in differentparts of gesture performance. Grounded in this statisticalrepresentation, the syntactic grammar rules generalize overthe structural variants among different gesture performancesfor the same object.

Classification resultsTo evaluate the FSCFG models of gestures quantitatively,we used them to probabilistically parse unseen gestures andto classify them according to the highest parsing probabil-ity. To this end, we divided the given corpus into differentsubsets of gesture performances based on a manual anno-tation of the gesture videos. Since the representation of ges-tures as spatial wrists movement trajectories is an underspec-ified representation of iconic gestures, it was expected thatthe classification algorithm would achieve relatively moreaccurate results for gestures performed with drawing tech-

nique than for example with static postures, where the ab-sent hand postures play a decisive role. We tested the classi-fication results for the following subsets of gestures: (1) Thesubset of simple synchronized drawing gestures (367 ges-tures) in which the drawing technique was used, both handsmoved synchronously, no part of the gesture was repeatedand a two-dimensional projection of the referred object wasdepicted; (2) 2D drawing subset (543 gestures) with onlydrawing gestures with two-dimensionally depicted shapes;(3) Drawing subset (702 gestures) that contained gesturesperformed only with the drawing technique; (4) The sub-set at least one drawing (909 gestures) refers to all gestureswith at least one part performed with drawing technique; (5)Drawing or dynamic posturing subset (1303 gestures) con-sisted of all gestures performed only with one of these tech-niques; (6) No static posturing (1507 gestures) was the sub-set of gestures in which no static posturing technique wasapplied; and finally (6) all gestures contained all 1739 ges-tures in the 3DIG corpus.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False positive rate

True

pos

itive

rate

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Simple sync. drawing2D drawingDrawingAt least one drawingDrawing or dynamic pos.No static pos.All gesturesBaseline

(AUC=0.996)(AUC=0.979)

(AUC=0.953)(AUC=0.851)

(AUC=0.811)(AUC=0.824)

(AUC=0.816)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

data1

data2

data3

data4

Chance level (AUC=0.5)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Simple sync. drawing2D drawingDrawingAt least one drawingDrawing or dynamic pos.No static pos.All gesturesBaseline

Figure 5: The ROC curves of classification using FSCFGs. AROC curve represents the mean of twenty ROCs, each rep-resenting the classification result of gestures for an object.

Figure 5 shows the receiver operating characteristics(ROC) graphs of two-fold cross-validation results, for eachsubset of the gesture performances. The results show thatFSCFG classified drawing with relatively high performance.A considerable drop in classification accuracy occured whenthe gesture performances with underspecified techniqueswere added to the dataset.

Figure 6 shows the confusion matrix of the classificationresults of the drawing subset. Many of the confusions oc-cur between gestures for objects that share many visuospa-tial properties. For instance, many of the gestures for sphereand ellipsoid, cube and rectangular box, or cone and pyra-mid are in fact performed ambiguously, because of a toorough depiction or their similar 2D projection. These con-

sphere:1cone:2ellipsoid:3rectangular_box:4parallel_trapezoid:5cube:6trapezoid:7cylinder:8pyramid:9egg:10apple:11table:12chair:13umbrella:14dumbbell:15vase:16bulb:17bottle:18teapot:19house:20

81310000000010010200030011014000000200015160000002100000230110020000000000000000000001300000000000000011151741110020000002000000240000000000000110002022100000000020000000016000000000000000000001600000000001000000000100000000001001110100022000010200000000000018000000000000000000008000000000000000000007000000100101102000001722062101120003001001800200000000000300000140000000000000000000080000000000000000000030

Predicted label

0102030Teapot 20

Dumbbell 19Umbrella 18

Table 17Chair 16Bottle 15Bulb 14Vase 13

Apple 12House 11

Par.trapezoid 10Trapezoid 9

Pyramid 8Cone 7Cube 6

Rectang.box 5Cylinder 4Ellipsoid 3

Egg 2Sphere 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

30

0

10

20

Predicted labels

Target labels:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

sphere:1

egg:2

ellipsoid:3

cylinder:4

rectangular_box:5

cube:6

cone:7

pyramid:8

trapezoid:9

parallel_trapezoid:10

house:11

apple:12

vase:13

bulb:14

bottle:15

chair:16

table:17

umbrella:18

dumbbell:19

teapot:20

8 0 3 0 1 0 1 0 0 0 0 0 1 0 2 1 0 0 0 00 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 05 2 6 0 0 0 1 0 0 0 1 1 0 2 3 0 0 0 0 01 1 0 2 0 2 1 2 0 0 2 0 0 0 0 0 0 0 0 01 0 0 0 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 01 1 1 1 5 7 1 1 4 1 2 0 0 0 0 2 0 0 0 00 0 0 1 0 1 3 4 0 1 1 0 2 0 0 0 0 0 0 0

0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 24 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 13 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 30 0 0 0 0 0 0 0 0 01 0 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 00 2 0 1 0 0 1 0 1 1 6 0 17 2 2 0 0 0 0 02 3 0 0 1 2 1 0 0 1 2 0 1 8 0 1 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 0 3 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18 0 0 0 01 0 0 1 1 1 0 0 0 1 2 0 0 0 1 2 2 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8

Confusion matrix

Predicted label

0

5

10

15

20

25

30

Figure 6: Confusion matrix of the FSCFG classification re-sults, performed on drawing gestures.

fused classes are in fact a result we had hoped for, since theeventual goal is to learn a grammar that carves out gesturalstrategies to iconically depict object properties and not spe-cific individual objects. For example, a single FSCFG modelfor all performances for sphere and ellipsoid can be learnedto represent a generative gesture model for round objects.Other confusions can be found with complex objects suchas the table object, which are referred to by relatively veryfew but diverse drawing gestures, resulting in very differenttraining and test samples.

In order to evaluate the classification performance of theFSCFG models, we compared them to the performance ofother methods on the same datasets (Hidden Markov Mod-els2 and Support Vector Machines3), and human judgmentperformance (see Figure 7). The features we used for HMMwere instantaneous features of movement trajectories, suchas spatial position, velocity and acceleration at each timestep for each wrist. As features for SVM, we took the his-togram of the features of all segments of a given gesturesample. The idea of these so-called HoGS features wasproposed in a previous work on this dataset (Sadeghipour,Morency, and Kopp 2012), in which – in contrast to this ap-plication – the best features were selected for each pair ofSVMs separately. As shown in Figure 7, the FSCFG modeloutperforms the other algorithms in all subsets of the ges-ture dataset. Moreover, we found that normalizing the pars-ing probability of each class for each gesture performance tothe sum of all parsing probabilities from all twenty FSCFGmodels improved the classification accuracy significantly, asshown in Figure 7 under the notion of “norm. FSCFG”.

Human judgment for recognizing the performed gestures

2Using the Bayes Net Toolbox for Matlab, available athttp://code.google.com/p/bnt/, with 5 hidden statesand 8 Gaussian mixtures.

3Using the LIBSVM library (Chang and Lin 2011), availableat http://www.csie.ntu.edu.tw/˜cjlin/libsvm/,with ν-SVM type (ν=0.01) and radial basis kernel type (γ=0.01).

1 2 3 4 5 6 70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

2D drawing Drawing At leastone drawing

Drawing ordynamic pos.

No staticpos.

All gestures Simplesync. drawing

F1 /

Hua

mns

' cor

rect

gue

ss ra

te

1 2 3 4 5 6 70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

FSCFGnorm. FSCFGHMMSVMHumans − color videosHumans − skeleton videosHumans − wrist videosChance level

1 2 3 4 5 6 70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

FSCFGnorm. FSCFGHMMSVMHumans − color videosHumans − skeleton videosHumans − wrist videosChance level

Figure 7: Comparing the classification performance (F1) ofFSCFG to different methods and human judgment base-lines, given different subsets of gestures.

represents a base-line for this classification task. We carriedout an online study in which the participants were asked toguess which of the twenty objects each gesture refered to.Each of the 240 participants watched 10 color videos of ges-ture performances, then 10 videos showing rendered skele-tons of the gesturers, and finally 10 videos showing only thewrists movements as two moving dots. We excluded partic-ipant who did not complete the task to the end, or who an-swered at least one test question incorrectly4. The remaining179 participants together guessed each gesture performancein each of the three demonstration conditions at least once.Notably, as shown in Figure 7, normalized FSCFG modelsachieved better recognition rates than humans in the firstthree subsets of drawing gestures, for which the extractedfeatures were designed. Further, the performance of normal-ized FSCFGs is in all subsets was better than human judg-ments in the skeleton or wrist conditions.

ConclusionWe have presented a hybrid grammar-based approach to cap-ture the structural and feature-based characteristics of natu-ral iconic gestures. To address the large variability and weakcompositionality in gesturing, our FSCFG models simulta-neously leverage low-level statistical regularities and high-level syntactic patterns during learning and recognition. Fur-thermore, the extension of the parsing algorithm to deal withuncertain input allowed for learning of noisy human motiondata. We extracted features from wrist movement trajecto-ries and achieved reliable classification results of drawinggestures, in which wrist movements depicted the contoursof objects. The FSCFG models of these gestures are gener-alized interpretable representations of their samples. In clas-sification, these models not only outperformed other classi-fication methods in gesture recognition, but they were alsomore accurate than human judgment on color videos of thedrawing gestures.

4Three videos were shown randomly, in which the participantswere asked to click on a specific object in textual form.

Acknowledgements This research is supported by theDeutsche Forschungsgemeinschaft (DFG) in the Center ofExcellence EXC 277 in ‘Cognitive Interaction Technology’(CITEC).

ReferencesBobick, A., and Ivanov, Y. 1998. Action recognition usingprobabilistic parsing. In Computer Vision and Pattern Recogni-tion, 1998 IEEE Computer Society Conference on, 196–202.Chanda, G., and Dellaert, F. 2004. Grammatical methods incomputer vision: An overview. Technical report, Georgia Insti-tute of Technology.Chang, C.-C., and Lin, C.-J. 2011. LIBSVM: A library for sup-port vector machines. ACM Transactions on Intelligent Systemsand Technology 2:27:1–27:27.Chen, Q.; Georganas, N. D.; and Petriu, E. 2008. Hand gesturerecognition using haar-like features and a stochastic context-free grammar. Instrumentation and Measurement, IEEE Trans-actions on 57(8):1562–1571.Damen, D., and Hogg, D. 2009. Attribute multiset grammarsfor global explanations of activities. In BMVC, 1–11.Fu, K. S. 1986. A step towards unification of syntactic andstatistical pattern recognition. IEEE Trans Pattern Anal MachIntell 8(3):398–404.Großekathofer, U.; Sadeghipour, A.; Lingner, T.; Meinicke, P.;Hermann, T.; and Kopp, S. 2012. Low latency recognition andreproduction of natural gesture trajectories. In ICPRAM (2)’12,154–161.Hong, P.; Turk, M.; and Huang, T. S. 2000. Gesture modelingand recognition using finite state machines. In Automatic Faceand Gesture Recognition, 2000. Proceedings. Fourth IEEE In-ternational Conference on, 410–415. IEEE.Ivanov, Y., and Bobick, A. 2000. Recognition of visual activi-ties and interactions by stochastic parsing. Pattern Analysis andMachine Intelligence, IEEE Transactions on 22(8):852–872.Joo, S.-W., and Chellappa, R. 2006. Attribute grammar-basedevent recognition and anomaly detection. In Computer Visionand Pattern Recognition Workshop, 2006. CVPRW ’06. Confer-ence on, 107–107.Kendon, A. 1972. Some relationships between body motionand speech. New York: Pergamon Press.Kitani, K. M.; Sato, Y.; and Sugimoto, A. 2006. An mdl ap-proach to learning activity grammars. Technical Report 376,IEICE - The Institute of Electronics, Information and Commu-nication Engineers, Tokyo, Japan.Knuth, D. E. 1968. Semantics of context-free languages. Math-ematical systems theory 2(2):127–145.Kopp, S., and Wachsmuth, I. 2004. Synthesizing multimodalutterances for conversational agents: Research articles. Com-put. Animat. Virtual Worlds 15(1):39–52.Minnen, D.; Essa, I.; and Starner, T. 2003. Expectation gram-mars: leveraging high-level expectations for activity recogni-tion. In Computer Vision and Pattern Recognition, 2003. Pro-ceedings. 2003 IEEE Computer Society Conference on, vol-ume 2, 626–632.Moore, D., and Essa, I. 2002. Recognizing multitasked activi-ties from video using stochastic context-free grammar. In Eigh-teenth national conference on Artificial intelligence, 770–776.

Menlo Park, CA, USA: American Association for Artificial In-telligence.Pastra, K., and Aloimonos, Y. 2012. The minimalist grammarof action. Philosophical transactions of the Royal Society ofLondon. Series B, Biological sciences 367(1585):103–117.Ramamoorthy, A.; Vaswani, N.; Chaudhury, S.; and Banerjee,S. 2003. Recognition of dynamic hand gestures. Pattern Recog-nition 36(9):2069–2081.Rissanen, J. 1983. A universal prior for integers and estimationby minimum description length. Annals of Statistics 11(2):416–431.Ryoo, M. S., and Aggarwal, J. 2006. Recognition of compositehuman activities through context-free grammar based represen-tation. In Computer Vision and Pattern Recognition, 2006 IEEEComputer Society Conference on, volume 2, 1709–1718.Sadeghipour, A.; Morency, L.-P.; and Kopp, S. 2012. Gesture-based object recognition using histograms of guiding strokes.Proceedings of the British Machine Vision Conference, 44.1–44.11. BMVA Press.Stolcke, A. 1994. Bayesian Learning of Probabilistic Lan-guage Models. Ph.D. Dissertation, University of California atBerkeley, Berkeley, CA.Tsai, W. H., and Fu, K. S. 1980. Attributed grammar - a toolfor combining syntactic and statistical approaches to pattern-recognition. Ieee Transactions on Systems Man and Cybernet-ics 10(12):873–885.Turaga, P.; Chellappa, R.; Subrahmanian, V. S.; and Udrea,O. 2008. Machine recognition of human activities: A survey.Ieee Transactions on Circuits and Systems for Video Technology18(11):1473–1488.Wang, J.; Lorette, G.; and Bouthemy, P. 1991. Analysis ofhuman motion: A model-based approach. In 7th ScandinavianConference on Image Analysis, Aalborg.Yamato, J.; Ohya, J.; and Ishii, K. 1992. Recognizing humanaction in time-sequential images using hidden markov model.In Computer Vision and Pattern Recognition, 1992. Proceed-ings CVPR’92., 1992 IEEE Computer Society Conference on,379–385. IEEE.Yoon, H. S.; Soh, J.; Bae, Y. J.; and Yang, H. S. 2001. Handgesture recognition using combined features of location, angleand velocity. Pattern Recognition 34(7):1491–1501.Zhang, Z.; Huang, K.; and Tan, T. 2006. Complex activity rep-resentation and recognition by extended stochastic grammar.In Proceedings of the 7th Asian conference on Computer Vi-sion - Volume Part I, ACCV’06, 150–159. Berlin, Heidelberg:Springer-Verlag.Zhang, Z.; Tan, T.; and Huang, K. 2011. An extended gram-mar system for learning and recognizing complex visual events.Pattern Analysis and Machine Intelligence, IEEE Transactionson 33(2):240–255.