A Constrained Probabilistic Petri Net Framework for Human Activity Detection in Video*

IEEE TRANSACTIONS ON MULTIMEDIA 1

A Constrained Probabilistic Petri Net Frameworkfor Human Activity Detection in Video

Massimiliano Albanese, Rama Chellappa, Fellow, IEEE, Vincenzo Moscato, Antonio Picariello,V. S. Subrahmanian, Pavan Turaga, Student Member, IEEE, and Octavian Udrea, Student Member, IEEE

Abstract—Recognition of human activities in restricted settingssuch as airports, parking lots and banks is of significant in-terest in security and automated surveillance systems. In suchsettings, data is usually in the form of surveillance videos withwide variation in quality and granularity. Interpretation andidentification of human activities requires an activity model thata) is rich enough to handle complex multi-agent interactions,b) is robust to uncertainty in low-level processing and c) canhandle ambiguities in the unfolding of activities. We presenta computational framework for human activity representationbased on Petri nets. We propose an extension – ProbabilisticPetri Nets (PPN) – and show how this model is well suited toaddress each of the above requirements in a wide variety ofsettings. We then focus on answering two types of questions:(i) what are the minimal sub-videos in which a given activity isidentified with a probability above a certain threshold and (ii) fora given video, which activity from a given set occurred with thehighest probability? We provide the PPN-MPS algorithm for thefirst problem, as well as two different algorithms (naivePPN-MPA and PPN-MPA) to solve the second. Our experimentalresults on a dataset consisting of bank surveillance videos andan unconstrained TSA tarmac surveillance dataset show that ouralgorithms are both fast and provide high quality results.

Index Terms—Algorithms, machine vision, Petri nets, surveil-lance.

I. INTRODUCTION

THE task of designing algorithms that can analyze humanactivities in video sequences has been an active field of

research during the past ten years. Nevertheless, we are stillfar from a systematic solution to the problem. The analysisof activities performed by humans in restricted settings isof great importance in applications such as automated se-curity and surveillance systems. There has been significantinterest in this area where the challenge is to automaticallyrecognize the activities occurring in a camera’s field of viewand detect abnormalities. The practical applications of such asystem could include airport tarmac monitoring, monitoringof activities in secure installations, surveillance in parkinglots etc. The difficulty of the problem is compounded byseveral factors: a) at the lower-level, unambiguous detectionof primitive actions can be challenging due to changes in

Authors listed in alphabetical order. Vincenzo Moscato andAntonio Picariello are with the Universita di Napoli “Federico II”,Napoli, Italy, Email: {vmoscato, picus}@unina.it. MassimilianoAlbanese, Rama Chellappa, V. S. Subrahmanian, Pavan Turagaand Octavian Udrea are with the Institute for Advanced ComputerStudies, University of Maryland, College Park, MD 20742. Email:{albanese, rama, vs, pturaga, udrea}@umiacs.umd.edu.

Research funded in part by AFOSR grants FA95500610405 andFA95500510298, ARO grant DAAD 190310202 and NSF grant 0540216.

illumination, occlusions and noise; b) at the higher-level,activities in such settings are usually characterized by complexmulti-agent interaction; c) the same semantic activity can beperformed in several variations which may not conform tostrict statistical or syntactic constraints.

In real life, one usually has some prior knowledge of thetype and semantic structures of activities that occur in a givendomain. In addition, one may also have access to a limitedtraining set, which in most cases is not exhaustive. Thus, thereis a need to leverage available domain knowledge to designsemantic activity models and yet retain the robustness ofstatistical approaches. The fusion of these disparate approaches– statistical, structural and knowledge based – has not yet beenfully realized and has only been gaining attention in recentyears. Our approach falls in this category – we exploit domainknowledge to create rich and expressive models for activitiesbased on Petri nets, and augment Petri nets with a probabilisticextension to handle (a) ‘noise’ in the labeling of video data, (b)inaccuracies in vision algorithms and (c) allow for departuresfrom a hard-coded activity model.

Based on this activity model, given a segment s of a video,we can then define a probability that the segment s containsan instance of the activity in question. We then provide fastalgorithms to answer two kinds of queries. The first type ofquery tries to find minimal (i.e., the shortest possible) clips ofvideo that contain a given event with a probability exceeding aspecified threshold – we call these Threshold Activity Queries.The second type of query takes a video segment and a set ofactivities, and tries to find the activity that most likely occurredin the segment – we call these Activity Recognition Queries.

A. Prior Related Work

The study and analysis of human activities has a wide andrich literature. On the one hand there has been significant effortto build algorithms that can learn patterns from training data –which can be broadly classified as statistical techniques – andon the other hand, there are the model-based or knowledge-based approaches representing activities at a higher level ofabstraction. A recent survey of computational approaches foractivity modeling and recognition may be found in [1].

Statistical approaches: A large body of work in the ac-tivity recognition area builds on statistical pattern recognitionmethods. Hidden Markov Models (HMMs) [2], [3] and theirextensions such as Coupled Hidden Markov Models (CHMMs)[4] have been used for activity recognition. Bobick and Davis[5] developed a method for characterizing human actions based

2 IEEE TRANSACTIONS ON MULTIMEDIA

on the concept of “temporal templates”. Kendall’s shape theoryhas been used to model the interactions of a group of peopleand objects by Vaswani et al. [6]. Statistical approaches suchas these require a training phase in which models are learntfrom training examples. This inherently assumes the existenceof a rich training set which encompasses all possible variationsin which an activity may be performed. But, in real lifesituations, one is not provided with an exhaustive training set.Moreover, it is not possible to anticipate the various ways anactivity may be performed. Statistical techniques do not modelthe semantics of activities and may fail to recognize the sameactivity performed in a different manner. They are also notwell suited to model complex behaviors such as concurrency,synchronization and multiple instances of activities.

Structural approaches: Unlike statistical methods whichassume the availability of a training set, these methods usuallyrely on hand-crafted models from experts or analysts. Sincethe models are hand-crafted, they are often intuitive and canbe related to the semantic structure of the activity. A methodfor generating high-level descriptions of traffic scenes wasimplemented by Huang et al. using dynamic Bayes networks(DBNs) [7], [8]. Hongeng and Nevatia [9] proposed a methodfor recognizing events involving multiple objects using tem-poral logic networks. Context free grammars have been usedby [10] to recognize sequences of discrete events. Ivanov andBobick [11] used Stochastic Context Free Grammars (SCFG)to represent human activities in surveillance videos. Albaneseet al. [12] use stochastic automata to model activities andprovide algorithms to find subvideos matching the activities.Shet et al. [13] have proposed using a Prolog based reasoningengine to model and recognize activities in video. Castelet al. [14] developed a system for high-level interpretationof image sequences based on deterministic Petri nets whichwere also used by Ghanem et al. [15]. Though structuralmethods have distinct advantages over statistical techniquesin terms of representative and expressive power – CFG’s,DBN’s and logic-based methods do not easily model con-currency, synchronization and multiple instances of activities.To address these issues, we use the Petri net formalism torepresent activities. Further, using a probabilistic extension tothe standard Petri net model, we show how we can integratehigh-level reasoning approaches with the inaccuracies inherentin the low-level processing.

Hybrid approaches: Techniques that can leverage therichness of structural approaches, incorporate domain knowl-edge from experts or knowledge bases and retain the ro-bustness of statistical approaches have been gaining moreattention in recent years. Ivanov and Bobick [11] use stochasticcontext free grammars (SCFGs) which are better able to handleinaccuracies in lower level modules and variations in activities.SCFGs have also been used by Moore and Essa [16] torecognize activities involving multiple people, objects, andactions. Attribute grammars have also been used to combinesyntactic and statistical models for visual surveillance appli-cations in [17]. Though grammar based approaches presenta sound theoretical tool for behavior modeling, in practice,the specification of the grammar itself becomes difficult forcomplex activities. Further, it is difficult to model behaviors

such as concurrency, synchronization, and resource sharingusing grammars. Petri nets are well suited for these tasks andhave been traditionally used in the field of compiler designand hybrid systems. A probabilistic version of Petri nets basedon particle filters was presented by Lesire and Tessier [18]for flight monitoring. Possibilistic Petri nets which combinepossibility logic with Petri nets have been used for modelingof manufacturing systems in [19].

The aforementioned papers do not explicitly address thefollowing problems: (i) developing efficient algorithms tofind all minimal segments of a video that contain a givenactivity with a probability exceeding a given threshold and(ii) developing efficient algorithms to find the activity (froma given set of activities) that is most likely in a given videosegment.

Contributions: First, we introduce a probabilistic PetriNet representation of complex activities based on atomicactions which are recognizable by image understanding al-gorithms. The next major contribution is the PPN-MPS1 algo-rithm to answer threshold queries. Our third major contributionis the naivePPN-MPA and PPN-MPA2 algorithms to solveactivity recognition queries. We evaluate these algorithmstheoretically as well as experimentally (in Section VI) on twothird party datasets consisting of staged bank robbery videos[20] and unconstrained airport tarmac surveillance videos andprovide evidence that the algorithms are both efficient andyield high-quality results.

Organization of paper: First, in Section II we defineprobabilistic Petri nets (PPN) and discuss representation ofcomplex activities using the PPN framework. We then presentefficient algorithms to solve the Most Probable Subsequence(MPS) problem in Section III, and the Most Probable Activity(MPA) problem in Section IV. Section V presents our imple-mentation. Section VI describes experimental results on twothird party datasets consisting of staged bank robbery videosand unconstrained airport tarmac surveillance videos. Finallyin Section VII, we present concluding remarks.

II. MODELING ACTIVITIES AS PROBABILISTICPETRI NETS

Petri Nets were defined as a mathematical tool for describ-ing relations between conditions and events. Petri Nets areparticularly useful to model and visualize behaviors such assequencing, concurrency, synchronization and resource shar-ing. David et al. [21] and Murata [22] provide comprehensivesurveys on Petri nets. Several forms of Petri nets (PN’s) suchas Colored PN’s, Continuous PN’s, Stochastic timed PN’s,Fuzzy PN’s etc. have been proposed. Colored PN’s associatea color to each token, hence are useful to model complexsystems where each token can potentially have a distinctidentity. In Continuous PN’s, the markings of places are realnumbers and not just integers. Hence, they can be used tomodel situations where the underlying physical processes arecontinuous in nature. Stochastic timed Petri nets [23] associate,to each transition, a probability distribution which represents

1Probabilistic Petri Net – Most Probable Subsequence.2Probabilistic Petri Net – Most Probable Activity.

ALBANESE et al.: A CONSTRAINED PROBABILISTIC PETRI NET FRAMEWORK FOR HUMAN ACTIVITY DETECTION IN VIDEO 3

Frame 1 Frame 2 Frame 3 Frame 4

Frame 5 Frame 6 Frame 7 Frame 8

Fig. 1. Video sequence of a staged bank attack.

the delay between the enabling and firing of the transition.Fuzzy Petri nets [24] are used to represent fuzzy rules betweenpropositions.

In real-life situations, vision-based systems have to dealwith ambiguities and inaccuracies in the lower-level detectionand tracking systems. Moreover, activities may not unfoldexactly in the sequence as laid out by the model that representsthem. The aforementioned versions of Petri nets (continuous,stochastic, fuzzy) are not well suited to deal with thesesituations. The proposed probabilistic Petri net model is well-suited to express uncertainty in the state of a token or associatea probability to a particular unfolding of the PN.

A. Labeling of Video Sequences

There has been significant effort in computer vision toaddress the problem of detecting various types of elementaryevents or actions from video data, such as a person enteringor leaving a room or unloading a package. We assume theexistence of a finite set A of action symbols correspondingto a set of atomic or primitive actions. Image understandingalgorithms3 are used to detect these atomic actions. In gen-eral, the action symbols can be arbitrary logical predicatesa : Ok → {true, false}, where O is the set of objects in thevideo and k is the arity of the action symbol a. To provide afew examples used for the TSA dataset, pickup(P : Person,O : Object) returns the value true if person P has picked upobject O in the current frame and false otherwise; similarly,occludes(O1 : Object, O2 : Object) returns true if and onlyif O1 occludes O2 in the current frame4.

For the remainder of the discussion, without loss of general-ity 5, we will assume that action symbols are propositional innature. The image processing library associates a subset of A

3Although some of the image understanding methods cited in this paperreturn a numeric value in the interval [0, 1], they can be readily transformedinto methods that return either true or false based on a user-specified threshold.Alternatively, one can learn such a threshold via a classification procedure: (i)mark certain images as being a “true” and certain images as being a “false”;(ii) invoke a classification algorithm to automatically learn a threshold thatseparates the annotated images into a true or false class.

4According to the occlusion detection algorithm.5For a finite object domain, we can “flatten” predicates of any arity to the

propositional case.

with every frame (or block of frames) in the video sequence.Formally, a labeling is a function ` : v → 2A, where v is avideo sequence and 2A is the power set of A. For example,consider the labeling of the video sequence in Fig. 1.

Example 1: Let A = {outsider–present, employee-present, outsider-absent, employee-absent, outsider-insafezone, employee-insafezone}. The labeling of thevideo sequence in Fig. 1 w.r.t. A is given by:`(frame1) = {outsider−absent, employee−absent}`(frame2) = {outsider−present, employee−absent}`(frame3) = {outsider−present, employee−present}`(frame4) = {outsider−insafezone, employee−insafezone}`(frame5) = {outsider−absent, employee−absent}`(frame6) = {outsider−insafezone, employee−insafezone}`(frame7) = {outsider−present, employee−present}`(frame8) = {outsider−absent, employee−present}In this example, the presence of the outsider-presentlabel for frame 2 means that an outsider (a person whois not an employee) is present in that frame. Similarly,employee-present in frame 3 means a person whose faceis recognized as that of an employee is present in frame 3,whereas employee-insafezone means an employee is nearthe bank safe (on the left-hand side of the image).

B. Constrained Probabilistic Petri Nets

We define a constrained probabilistic Petri net (PPN) as thetuple PPN = {P, T,→, F, δ} where

1) P and T are finite disjoint sets of places and transitionsrespectively, i.e., P ∩ T = ∅.

2) → is the flow relation between places and transitions,i.e., →⊆ (P × T ) ∪ (T × P ).

3) The preset of a node x ∈ P ∪ T is the set {y|y → x}.This set is denoted by the symbol .x.

4) The postset of a node x ∈ P ∪ T is the set {y|x → y}.This set is denoted by the symbol x..

5) F is a function defined over the set of places P , thatassociates to each place x a local probability distributionover {t|x → t}. F : x 7−→ fx(t), where x ∈ P andfx(t) is a probability distribution defined over the postsetof x. For each place x ∈ P in a given PPN, we denoteby p∗(x) the maximum product of probabilities over all


paths from x to a terminal node. We will often abusenotation and use F (x, t) to denote fx(t).

6) δ : T → 2A associates a set of action symbolswith every transition in the Petri Net. Intuitively, thisimposes constraints on the transitions in the Petri Net –transitions will only fire when all the action symbols inthe constraint are also encountered in the video sequencelabeling.

7) ∃x ∈ P s.t. x. = ∅. This means that there exists at leastone terminal node in the Petri net.

8) ∃x ∈ P s.t. .x = ∅. This means that there exists at leastone start node in the Petri net.

C. Tokens and Firing of transitions

The dynamics of a Petri net are represented via ‘markings’.For a PPN (P, T,→, F, δ), a marking is a function µ : P →N that assigns a number of tokens to each place in P . Forexample, µ(p1) = 2 means that, according to marking µ, thereare two tokens in place p1. For a PPN representing an activity,a marking µ represents the current state of completion of thatactivity.

The execution of a Petri net is controlled by its currentmarking. A transition is enabled if and only if all its inputplaces (the preset) have a token. When a transition is enabled,it may fire. When a transition fires, all enabling tokens areremoved and a token is placed in each of the output placesof the transition (the postset). We use µ0 to denote the initialmarking of a PPN. As transitions fire, the marking changes.A terminal marking is reached when one of the terminalnodes contains at least one token. In the simplest case, allthe enabled transitions may fire. However, to model morecomplex scenarios we can impose other conditions to besatisfied before an enabled transition can fire. This set ofconditions is represented by δ.

Example 2: Consider the example of a car pickup activityrepresented by the probabilistic Petri Net shown in Fig. 2. Inthis figure, the places are labeled p1, . . . , p7 and transitionst0, . . . , t5. We use µ0 to denote the initial marking depicted inthe figure in which all places except p0 have 0 tokens. In thisPN, p0 is the start node and p7 is the terminal node. Transitiont0 is unconstrained and it is always fired, adding a token inboth p1 and p2. When a car enters the scene, t1 fires and atoken is placed in p3 and we have a new marking µ1 such thatµ1(p3) = 1 and µ1(p) = 0 for any p 6= p3. The transition t3is enabled in this state, but it cannot fire until the conditionassociated with it is satisfied – i.e., when the car stops. Whenthis occurs, the token is removed from p3 and placed in p5.This leads to a new marking µ2 such that µ2(p5) = 1 andµ2(p) = 0 for all p 6= p5. Similarly, when a person enters theparking lot, a token is placed in p4 and transition t4 fires afterthe person disappears near the car. The token is then removedfrom p4 and placed in p6. Now, with a token in each of theenabling places of transition t5, it is ready to fire when theassociated condition – i.e., car leaving the parking lot – issatisfied. Once the car leaves, t5 fires and both the tokens areremoved and a token placed in the final place p7. This exampleillustrates sequencing, concurrency and synchronization.

Fig. 2. Probabilistic Petri net for Car-pickup.

In the above discussion, we have not yet discussed theprobabilities labeling some of the place-transition edges.Note that the postsets of p1, p2, p3, p4, p5, p6 contain multipletransitions. The ‘skip’ transitions are used to explain awaydeviations from the base activity pattern – each such deviationis penalized by a low probability. For example, after the carenters the scene, suppose that instead of stopping in a lot,the car goes around in circles. In this case, the skip transitionis fired, meaning a token is taken from p3 and immediatelyplaced back. However, the token probability is decreased bythe probability labeling the edge to the skip transition. Theprobabilities assigned to skip transitions control how tolerantthe model is to deviations from the base activity pattern.

All tokens are initially assigned a probability score of 1.Probabilities are accumulated by multiplying the token andtransition probabilities on the edges. When two or more tokensare removed from the net and replaced by a single token– for instance when t5 fires – then the probability for thenew token is set to be the product of the probabilities of theremoved tokens. We will use the final probability of a token ina terminal node as the probability that the activity is satisfied.

Example 3: Fig. 3 depicts another constrained PPN mod-eling two possible ways in which a successful bank robberymay happen. Uncertainty appears after both the employee andthe outsider are inside the zone of the bank safe (at placep5). According to the model, in 60% of the cases both theattacker and the employee will enter the safe6 (p6), whereasin 35% of the cases, the attacker enters the safe alone (p7).In either case, a successful robbery is completed once theoutsider leaves the scene (p9). Note the use of skip transitionsto model possible “noise” in the labeling (i.e., labels that donot correspond to any of the currently enabled transitions) asexplained previously.

In this paper, we do not address the problem of how tolearn the PPN. If a training set is available, an expert user canspecify the structure of the PPN, i.e., he/she can specify theP , T , → and δ components of a PPN, and the system can usethe training set to learn the probability distribution F using

6Visually, this means that they will temporarily disappear from the scene.


Fig. 3. Constrained PPN modeling a bank attack.

(a) (b)

Fig. 4. (a) Before firing; (b) after firing of transition t1.

standard statistical methods. Learning the entire PPNs fromthe training set is a challenge.

D. Rules for token probability computation

We denote by αk the value or probability associated with atoken at place pk. When a token is placed in one of the startnodes, it is associated with a value of 1 (or some other priorprobability, for instance, as reported by the low-level visionmodule). Fig. 4 shows how the value associated with a tokengets updated as a transition fires. In this figure, transitions t1and t2 are both enabled. fp1 is the local probability distributionassociated with the place p1. If transition t1 fires, then thevalue of the token in place p2 becomes α2 = α1 ∗ fp1(t1)as shown in Fig. 4. The probability computation rules forsynchronization and concurrency are illustrated in Fig. 5 andFig. 6 respectively.

(a) (b)

Fig. 5. Concurrency: (a) before firing; (b) after firing.

(a) (b)

Fig. 6. Synchronization: (a) before firing; (b) after firing.

p1 p2

permitted-action

forbidden -action

1.00

0.95

t1

dead -

end

(a)

p1 p2

less_than_eq(current _frame-last_trans _frame, 20)

0.95

t1

1.00

greater _than(current_frame-last_trans_frame, 20)

dead -

end

(b)

Fig. 7. (a) Forbidden actions; (b) temporal constraints.

E. Enhancing PPN expressiveness: Forbidden actions andTemporal durations

In this section, we will describe how PPNs can be usedto model “forbidden actions” – actions which trigger therecognition of a partially completed activity to be nullified– and constraints on the temporal duration of the activity.

Forbidden Actions: In several applications, it is importantto be alert to suspicious behavior that should immediatelytrigger an alarm. We show how forbidden actions can bemodeled in Fig. 7(a). This is possible when the set of forbiddenactions is mutually exclusive to the set of permitted actions.Essentially, for each forbidden action fa that may occur in agiven state, we add a transition with probability 1 conditionedon fa to a place that is a dead-end (i.e., no terminal markingcan be reached from it).

Temporal Duration: Activities are often constrained notjust by a simple sequence of action primitives, but also by theduration of each action primitive in the sequence. In general,we may associate a probability distribution over the ‘dwell’time to each place in the Petri net. We constrain the dwelltime to be within a given interval. We have built macros thathandle frame computation. For instance, current frame isthe number of the current frame, whereas last trans frameis the number of the last frame at which a transition wasfired for the current activity. Intuitively, Fig. 7(b) imposes thecondition that the transition t1 fire within twenty frames ofthe last transition that was fired. If this does not occur, thenwe again transition to a dead-end, as in the case of forbiddenactions.


F. Activity recognition

We now define a ‘PPN trace’ and what it means for a videosequence to satisfy a given PPN and provide a method forcomputing the probability with which an activity is satisfied.

Definition 1 (PPN trace): A trace for a PPN (P ,T ,→,F ,δ)with initial marking µ0 is a sequence of transitions〈t1, . . . , tk〉 ⊆ T such that:(i) t1 is enabled in marking µ0.

(ii) Firing t1, . . . , tk in that order reaches a terminal marking.For each i ∈ [1, k], let pi = Π{x∈P |x→ti}F (x, ti). We say thetrace 〈t1, . . . , tk〉 has probability p = Πi∈[1,k]pi

7. Let pmax

be the maximum probability of any trace for (P, T,→, F, δ).Then 〈t1, . . . , tk〉 has relative probability p

pmax. Intuitively, the

relative probability measures how close that trace is to theideal execution of the PPN, i.e., the execution that leads tothe maximum possible absolute probability.

Example 4: Consider the PPN depicted in Fig. 3.〈t1, t2, t3, t4, t6, t7, t9〉 is a trace with probability .464 and arelative probability of 1. 〈t1, t2, t3, t4, t5, t8, t9〉 is a trace withprobability .271 and relative probability .584.

Definition 2 (Activity satisfaction): Let P=(P, T,→, F, δ)be a PPN, let v be a video sequence and let ` be the labelingof v. We say ` satisfies P with relative probability p ∈ [0, 1]iff there exists a trace 〈t1, . . . , tk〉 for P such that:(i) There exist frames f1 ≤ . . . ≤ fk such that ∀i ∈ [1, k],

δ(ti) ⊆ `(fi) AND(ii) The relative probability of 〈t1, . . . , tk〉 is equal to p.

We will refer to a video sequence v satisfying an activitydefinition, as an equivalent way of saying the labeling of vsatisfies the activity. For reasons of simplicity, if ` is thelabeling of v and v′ ⊆ v is a subsequence of v, we will alsouse ` to refer to the restriction of ` to v′.

Example 5: Consider the video sequence in Fig. 1 with thelabeling in Example 1, and the PPN in Fig. 3. The labeling inExample 1 satisfies the PPN with relative probability 1 sincethe conditions for the trace 〈t1, t2, t3, t4, t6, t7, t9〉 are subsetsof the labeling for frames 1 – 8.

We now define the two types of queries we are interestedin answering.

The Threshold Activity Query Problem. (PPN-MPS)Given a constrained PPN, a video sequence v and a probabilitythreshold pt, find the minimal subsequences of v that satisfythe PPN with probability at least pt. Intuitively, such queriesare of interest for identifying specific portions of the videosequence in which a given activity occurs.

The Activity Recognition Query Problem. (PPN-MPA)Given a set of activities {P1, . . . ,Pn} and a video sequencev, determine which activity (or set of activities) is satisfied bythe labeling of v with maximum probability. Intuitively, suchqueries are useful in scenarios in which we are interested infinding which activity most likely occurs in a given sequence.

III. ALGORITHMS FOR THRESHOLD ACTIVITY QUERIES

We start this section by presenting an efficient algorithm forthe PPN-MPS problem. In the next section we show how an

7Without loss of generality we are assuming that the initial probabilityassigned to tokens in the start nodes is 1.

iterative variant of PPN-MPS can be used to naively solve thePPN-MPA problem – we call this naivePPN-MPA.

The PPN-MPS algorithm (Algorithm 1) simulates the con-strained PPN forward. The algorithm uses a separate structureto store tuples of the form 〈µ, p, fs〉, where µ is a validmarking, p is the probability of the partial trace that leads tothe marking µ and fs is the frame when the first transition inthat trace was fired. The PPN-MPS algorithm takes advantageof the fact that we only need to return minimal subsequencesand not the entire PPN traces that lead to them, hence using thelast marking as a state variable is sufficient for our purposes.

The algorithm starts by adding 〈µ0, 1, 0〉 to the store S(line 1). This means we start with the initial marking (µ0),with the probability equal to 1. The value 0 for the startframe means the first transition has not yet fired. Wheneverthe first transition fires and the start frame is 0, the currentframe is chosen as a start for this candidate subsequence(lines 8–9). The algorithm iterates through the video frames8,and at each iteration analyzes the current candidates stored inS. Any transitions t enabled in the current marking from Sthat have the conditions δ(t) satisfied are fired. The algorithmgenerates a new marking, and with it a new candidate partialsubsequence (line 7). If the new probability value p′ (lines 13 –14) is still above the threshold (line 15) and can remain abovethe threshold on the best path to a terminal marking (line 16),the new state is added to the store S. This first pruning doesaway with any states that will result in low probabilities. Notethat we have not yet removed the old state from the store –nor should we at this time, since we also need to fire any skiptransitions that are available, in order to explore the space ofsolutions. This is done for any enabled skip transition on line26. At this point we also prune away any states in S9 that haveno possibility of remaining above the threshold as we reach aterminal marking (line 27).

Example 6: We illustrate the working of PPN-MPS on thePPN in Fig. 3, the labeling in Example 1 and pt = 0.5.Initially, S = 〈µ0, 1, 0〉. We use µi to denote the markingobtained after firing ti in Fig. 3. At frame 1, since transitiont1 has no precondition, it will fire and we will have one newelement in S (along with the existing initial state) – 〈µ1, 1, 0〉.Note that the starting frame is still set to 0 – this is done so thatthe start of the subsequence is marked only when a labelingcondition is satisfied. At frame 2, transition t2 fires and weadd 〈µ2, .95, 2〉 to S. At this point, transition t3 is enabledbut cannot fire because its condition employee-present is notsatisfied. Hence a skip transition fires, but that generates a statewith probability .05, which is under the threshold and hencethrown away immediately. We will ignore the firing of skiptransitions for the rest of the example, as they are immediatelypruned away due to the low probabilities generated. Transitiont3 does fire at frame 3 and adds 〈µ3, .9025, 1〉 to S. Transitiont4 is now enabled and its conditions satisfied at frame 4, henceit is fired and a new state 〈µ4, .8145, 1〉 is generated. At frame5, transition t6 is satisfied (and t5 is not), hence it is firedand the state becomes 〈µ5, .4880, 1〉. Finally, t7, t9 are fired

8In our implementation, we group frames with identical labeling into blocksand iterate over blocks instead which leads to enhanced efficiency.

9Pruning always keeps 〈µ0, 1, 0〉 in S.


Algorithm 1 Threshold Activity Queries (PPN-MPS)Input: P = (P, T,→, F, δ), initial configuration µ0 , video sequence labeling `, probability threshold ptOutput: Set of minimal subsequences that satisfy P with relative probability above pt1: S ← {〈µ0, 1, 0〉}2: R ← ∅3: maxp ← the maximum probability for any trace of P4: for all f frame in video sequence do5: for all 〈µ, p, fs〉 ∈ S do6: for all t ∈ T enabled in µ s.t. δ(t) ⊆ `(f) do7: µ′ ← fire transition t for marking µ

8: if fs = 0 and δ(t) 6= ∅ then9: f′ ← f

10: else11: f′ ← fs12: end if13: p∗ ← Π{x∈P |x→t}F (x, t)

14: p′ ← p · p∗15: if p′ ≥ pt ·maxp then16: if ∃x ∈ P s.t. µ(x) < µ′(x) ∧ p′ · p∗(x) ≥ pt ·maxp then17: S ← S ∪ {〈µ′, p′, f′〉}18: end if19: end if20: if µ′ is a terminal configuration then21: S ← S − {〈µ′, p′, f′〉}22: R ← R ∪ {[fs, f]}23: end if24: end for25: for all skip transition t ∈ T enabled do26: Fire skip transition if no other transitions fired and update 〈µ, p, fs〉27: Prune 〈µ, p, fs〉 ∈ S s.t. ∀x ∈ P s.t. µ(x) > 0, p · p∗(x) < pt ·maxp

28: end for29: end for30: end for31: Eliminate non-minimal subsequences in R

32: return R

at frames 6, 8 and bring the final state to µ8, .4642, 1〉, leadingto a relative probability of 1 (we remind the reader that theelements of S hold the absolute probability).

The following theorem states that the PPN-MPS algorithmreturns the correct result.

Theorem 1 (PPN-MPS correctness): Let P=(P, T,→, F, δ)be a PPN with initial configuration µ0, let v be a videosequence and ` its labeling, and let pt ∈ [0, 1] be a probabilitythreshold. Then for any subsequence v′ ⊆ v that satisfiesP with probability greater than or equal to pt, one of thefollowing holds:(i) v′ is returned by PPN-MPS OR

(ii) ∃v′′ ⊆ v that is returned by PPN-MPS such that v′′ ⊆ v′.Proof Let v0 ⊆ v be a subsequence of v that satisfies the

activity definition, but is not returned by PPN-MPS. If v0 has asubsequence returned by PPN-MPS, we are done. Otherwise,since v0 satisfies the activity definition, according to Definition2, there exists a trace 〈t1, . . . , tk〉 and corresponding frames(in v0) f1 ≤ . . . ≤ fk such that ∀i ∈ [1, k], δ(ti) ⊆ `(fi).We will show by induction that the algorithm will fire alltransitions (ti)i∈[1,k]. Since 〈µ0, 1, 0〉 always remains in S,and t1 is executable from µ0 and δ(t1) ⊆ `(f1), then t1 mustbe fired on line 7 and f1 becomes the current frame (initialcondition). Assume that all transitions up to (ti)i∈[1,k−1] havebeen fired; we want to show that tk must fire next. Assumethat the new state in S induced by the firing of tk−1 issk−1 = 〈µk−1, pk−1, f1〉 such that tk is enabled in µk−1.Since (ti)i∈[1,k] is a trace, then when frame fk is analyzed,there must exist a state sk = 〈µk−1, p, f1〉 ∈ S (if multiplesuch states exist, then we choose the one with the highestprobability)10. Let pk be the probability of firing tk (which isenabled and has δ(tk) satisfied on line 6) and let p0 be theprobability of the trace. Then p · pk ≥ p0

maxp≥ pt, (condition

on line 15 holds). Also, for all x ∈ P that receive a token as tk

10p may be different than pk by firing skip transitions.

fires, p · pk · p∗(x) ≥ p0maxp

≥ pt (condition on line 16 holds);line 17 therefore executes and the new marking µk (includingthe effects of firing tk) is added to S (induction step). Weshowed by induction that a trace of probability greater thanor equal to pt corresponding to a subsequence of v0 has beenexecuted.

Theorem 2 (PPN-MPS complexity): Let P=(P, T,→, F, δ)be a PPN with initial configuration µ0, such that the numberof tokens in the network at any marking is bounded by aconstant k11. Let v be a video sequence and ` its labeling.Then PPN-MPS runs in time O(|v| · |T | · |P |k).

Proof. We will first look at how many possible markingsthere are if the PPN is bounded by k. If we denote the numberof possible markings for a PPN with n nodes which is boundedby k by xn,k, then by fixing one token (n possibilities), wehave that xn,k = n·xn,k−1. This leads to xn,k = nk−1∗xn,1 =nk. The algorithm process each frame in the video (line 4), ateach iteration looking at every element of S (line 5); however,there are at most a constant number of elements in S foreach marking, therefore the size of S is O(|P |k). For everyelement in S, we look at at most |T | transactions enabledfor that marking, hence the complexity result. We normallyexpect that in regular activity definitions, the degree k of thepolynomial is small; also, for a fixed activity definition, thealgorithm is linear in the size of the video.

We should point out that the probabilities in the PPN alreadyenforce a bound on the number of tokens in the network.Assume that there are no transitions with a probability of1 (for instance, by adding skip transitions at every place).Let c be the upper bound of the probabilities in P . Newtokens are generated as a result of transitions with a postsetof cardinality strictly greater than 1: one can generate at most|P| − 1 tokens on each such transition, with a net increase of|P|−2 (at least one token is removed when a transition fires).Furthermore, for a threshold pt, each token can be used to fireat most log pt

log c12 transitions before its probability is lower than

pt. Assuming the initial number of tokens is |µ0| =∑

x∈P

µ0(x),

this means we can have at most |µ0| · (|P| − 1)log ptlog c tokens

in the network. Intuitively, markings with a higher numberof tokens are more unlikely and they are pruned away, thuspreventing an unbounded increase in the number of tokens.

Handling Multiples Instances: Note that PPN-MPSalready handles a certain degree of parallelism in the activitiesoccurring in the video. This is a direct consequence of thefact that 〈µ0, 1, 0〉 is always a member of S, hence a newactivity could be started at every frame. However, PPN-MPS does not directly handle the case in which multipleactions that start an activity (i.e., enable a transition from〈µ0, 1, 0〉) occur in the same frame. In our implementation, wehandled this case by duplicating the PPN for each action of aframe that can potentially start a new activity. The maximum

11This is called a bounded PPN – i.e., one that does not generate an infinitenumber of tokens. We will see that the probabilities in the PPN may enforcea variant of this rule.

12The cumulative probability when firing k transitions is at most ck , whichmust be greater than or equal to pt in order for the marking not to be pruned.Therefore k ≤ log pt/ log c.


number of such actions per frame is maxf∈v(|`(f)|). Hence,taking the complexity of the scene into account, the worst-case complexity becomes O(maxf∈v(|`(f)|) · |v| · |T | · |P |k).However, in our two datasets we only needed duplication inunder 1% of the frames, and maxf∈v(|`(f)|) ≤ 3 in all cases.

IV. ALGORITHMS FOR ACTIVITY RECOGNITION

Let us now consider the PPN-MPA problem. Let{P1, . . . ,Pn} be a given set of activity definitions. If weassume that there is only one activity Pl which satisfies thevideo sequence v with maximal probability, a simple binarysearch iterative method – which we shall call naivePPN-MPA– can employ PPN-MPS to find the answer:

1) Begin with a very high threshold pt.2) Run PPN-MPS for all activities in {P1, . . . ,Pn} and

threshold pt.3) If more than one activity definition has a non-empty

result from PPN-MPS, then increase the threshold by1−pt

2 and go to step 2.4) If no activity has a non-empty result from PPN-MPS,

decrease the threshold by pt

2 and go to step 2.5) If only one activity Pl has a non-empty result from PPN-

MPS, return Pl.Clearly, this method can be improved to account for the

case in which multiple activity definitions are satisfied by thevideo sequence with the same probability. Even under theseconditions, the algorithm requires multiple passes through thevideo (for each run of PPN-MPS) until a delimiter thresholdis found. We will now present a much more efficient method– the PPN-MPA algorithm.

Algorithm 2 Activity Recognition Queries (PPN-MPA)Input: Set of PPNs {(Pi, Ti,→i, Fi, δi)}i∈[1,m] , initial configurations µ0

i , video sequence labeling `

Output: Set of PPNs satisfied by ` with maximal probability1: for all P ∈ {(Pi, Ti,→i, Fi)}i∈[1,m] do

2: S ← S ∪ {〈µ0i , 1,P〉}

3: end for4: maxp ← 05: R ← ∅6: for all f frame in video sequence do7: for all 〈µ, p,P〉 ∈ S do8: for all t ∈ T enabled in µ s.t. δ(t) ⊆ `(f) do9: µ′ ← fire transition t for marking µ

10: p∗ ← Π{x∈P |x→t}F (x, t)

11: p′ ← p · p∗12: if p′/maxPp ≥ maxp then

13: if ∃x ∈ P s.t. µ(x) < µ′(x) ∧ p′ · p∗(x)/maxPp ≥ maxp then

14: S ← S ∪ {〈µ′, p′,P〉}15: end if16: end if17: if µ′ is a terminal configuration then18: S ← S − {〈µ′, p′,P〉}19: if p′/maxPp > maxp then

20: R ← {P}21: maxp ← p′/maxPp22: else if p′/maxPp = maxp then

23: R ← R ∪ {P}24: end if25: end if26: end for27: for all skip transition t ∈ T enabled do28: Fire skip transition if no other transitions fired and update 〈µ,P, p〉29: Prune 〈µ, p,P〉 ∈ S s.t. ∀x ∈ P s.t. µ(x) > 0, p · p∗(x)/maxPp < maxp

30: end for31: end for32: end for33: return R

PPN-MPA (Algorithm 2) uses a similar storage structureas PPN-MPS (tuples of the form 〈µ, p,P〉, where P is theactivity definition to which marking µ applies). This time

we are no longer interested in the start and end frames ofthe subsequences matching one of the activity definitions.Instead, we maintain a global maximum relative probability(maxp) with which the video satisfies any of the activitiesin {(Pi, Ti,→i, Fi, δi)}i∈[1,m]. maxp is updated any time abetter relative probability is found (lines 19–21); R maintainsa list of the activity definitions that have been satisfied withthe relative probability maxp. At the return statement on line33, we are guaranteed that the activity definitions in R aresatisfied by the video with maximal probability.

Theorem 3 (PPN-MPA correctness): Let {(Pi, Ti,→i, Fi,δi)}i∈[1,m] be a set of PPNs, let v be a video sequence and `its labeling. Let P1, . . . ,Pk be the activity definitions returnedby PPN-MPA. Then the following hold:(i) ∀i ∈ [1, k], let pi be the relative probability with which

` satisfies Pi. Then p1 = . . . = pk.(ii) ∀P ′ ∈ {(Pi, Ti,→i, Fi, δi)}i∈[1,m] − {P1, . . . ,Pk}, `

does not satisfy P ′ with relative probability greater thanor equal to p1 = . . . = pk.

Proof. The proof of correctness for PPN-MPS shows thatthe algorithm does not “miss” transitions, but ignores onlythose that cannot end up above the threshold. In this case,the threshold is the variable maxp, which means the onlytransitions being ignored are those for which there is alreadya “better” activity (satisfied with the probability maxp) in R;this is sufficient for condition (ii). Condition (i) is satisfiedby the fact that maxp always increases (lines 19–24) and allelements in R must be satisfied with the same probability (line22).

Theorem 4 (PPN-MPA complexity): Let {(Pi, Ti,→i, Fi,δi)}i∈[1,m] be a set of PPNs, let v be a video sequence and `its labeling. PPN-MPA runs in time O(m · |v| · |T | · |P |k).

Proof. The same reasoning as for PPN-MPS applies hereas well, with the addition of the factor m which accounts forthe size of S when we analyze m different activities.

Similarly to PPN-MPS, PPN-MPA will clone the PPNs eachtime multiple actions in a single frame can potentially start anew activity. Taking into account the maximum number ofsuch actions would yield a complexity of O(maxf∈v(|`(f)|) ·m · |v| · |T | · |P |k), however in practice we only needed toclone the PPNs very rarely.

V. SYSTEM IMPLEMENTATION

Our PPN-MPS, naivePPN-MPA and PPN-MPA algorithmswere implemented in Java. In addition, our image processinglibrary is based on the People Tracker framework [25], [26],with several modifications outlined in this section.

People Tracker13 uses object detection and motion algo-rithms to track persons, groups of persons, vehicles and otherobjects in a given sequence of images. From a functional pointof view, the framework consists of four components:

1) The Motion Detector subtracts the background from thecurrent image and thresholds the resulting differenceimage to obtain moving objects (“blobs”).

2) The Region Tracker tracks all blobs identified in the pre-vious step by a frame-by-frame matching algorithm. The

13Freely available under a GPL license at http://www.siebel-research.de.


next position of a blob is predicted using a first-ordermotion model and then matched against the positions ofthe blobs in the new frame.

3) The Head Detector is used to identify blobs correspond-ing to people by creating a vertical projection histogramof the upper part of every tracked region. This is basedon the algorithms of Haritaoglu et al. [27].

4) The Active Shape Tracker uses B-spline 2D shape mod-els to resolve the type of a tracked region to a person,group of persons, vehicle, package, etc.

In addition to these, the following extensions were made:1) Shadows reduced the accuracy of detecting object splits

and merges. To take care of this, we implemented thealgorithm of [28] for detection of moving shadows.

2) We implemented the face recognition algorithm of [29].The algorithm detects faces in a frame using a wavelettransform based approach and then compares featuresof detected faces with faces stored in the database. Wechose this particular face recognition algorithm becauseof its speed and good accuracy.

3) To accurately answer queries of type: “Is object Oin frame f?”, we implemented the occlusion detectionalgorithm of [30].

Based on the above framework and the aforementionedextensions, we implemented a number of low-level predicatesto answer simple queries such as:

1) Is object O present in frame f (in a particular zoneZ)? This is accomplished through the object matchingalgorithm in the Region Tracker module.

2) Is object Oi occluding Oj in frame f?3) Are objects Oi and Oj splitting/merging in frame f?4) What is the type of object Oi (person, vehicle, group of

persons, etc.) in frame fi?5) Is object O identical to the person P from the database?All the above queries are typically answered with a degree

of confidence reported as a probability by the underlyinglow-level module. Finally, we have combined these low-levelpredicates into a set of action symbols in order to generatethe higher-level labeling our algorithms take as input. As anexample, a frame f is associated with the the action symbolemployee-present used in Fig. 3 if and only if an object oftype person is detected in that frame, and the person’s facematches one of the faces in a database of employees.

VI. EXPERIMENTAL RESULTS

Our experiments were performed on two datasets. The firstdataset (Bank) consists of 7 videos of 15–20 seconds in length,some depicting staged bank attacks and some depicting day-to-day bank operations. The dataset has been documented in [20].Fig. 1 contains a few frames from a video sequence depictinga staged bank attack. Fig. 8(top) contains a few framesfrom a video sequence depicting regular bank operations.The second dataset consists of approximately 118 minutesof TSA tarmac footage and contains activities such as planetake-off and landing, passenger transfer to/from the terminal,baggage loading and unloading, refueling and other airportmaintenance.

We compared the precision and recall of our algorithmsagainst the ground truth provided by human reviewersas follows. Four human reviewers analyzed the videos inthe datasets. The reviewers were provided with VirtualDub(http://www.virtualdub.org), which allowed them to watch thevideo in real time, as well as browse through it frame by frame.They received detailed explanations on the activity model, aswell as sets of activity definitions and were asked to mark thestarting and ending frame of each activity they encountered.An average over the four reviewers was then used as theground truth for the purpose of computing precision and recall.We compared the subvideos identified by the reviewers withthose returned by our algorithms above a given probabilitythreshold.

All experiments were performed on a Pentium 4 2.8 GHzmachine with 1 GB of RAM running SuSE Linux 9.3. Thelabeling data generated by the low-level vision methods wasstored in a MySQL database. The labeling was compressedfor blocks of frames, as neighboring frames often tend tohave identical labeling. Running times include disk I/O foraccessing the labeling database, but not the time taken togenerate the labeling.

A. Experimental evaluation for the Bank dataset

For the Bank dataset, we defined a set of 5 distinct activitiesand designed the associated PPNs. They are:(a1) Regular customer-bank employee interaction.(a2) Outsider enters the safe.(a3) A bank robbery attempt – the suspected assailant does

not make a getaway.(a4) A successful bank robbery similar to the example in

Fig. 3.(a5) An employee accessing the safe on behalf of a customer.For the PPN-MPA experiments, we grouped these into dif-ferent sets of activities as: S1 = {a5}, S2 = {a3, a5},S3 = {a3, a4, a5} and S4 = {a2, a3, a4, a5}.

For the PPN-MPS algorithm, we observed empirically thatin the majority of cases, at most one instance of the activityoccurs at any given time. For instance, it is highly unlikelythat we have two bank attacks (a3, a4) interleaved in the samevideo sequence; although we might have a number of regularcustomer operations (a1), we assumed that the special eventswe are interested in monitoring are generally rare. We tookthis into account by developing two different versions of thealgorithm. The first is the one presented in Section III; thesecond version optimizes the algorithm by assuming there canbe no interleaving of activities in a video; we will denotethe latter with the suffix NI (no interleaving). This approachgenerally leads to an improvement in running time, at theexpense of precision.

Running Time Analysis: In our first set of experiments,we measured the running times of PPN-MPS, naivePPN-MPAand PPN-MPA for the five activity definitions described abovewhile varying the size of the labeling (the number of actionsymbols in the labeling14). The running times are shown in

14We remind the reader that the videos are 15-20 second long and thatlabeling has been compressed from frame to block level.


Frame 1 (Bank) Frame 2 (Bank) Frame 3 (Bank) Frame 4 (Bank)

Frame 1 (TSA) Frame 2 (TSA) Frame 3 (TSA) Frame 4 (TSA)

Fig. 8. (top) Video sequence of regular customer interaction at a bank; (bottom) Sample frames from the TSA dataset.

Fig. 9. As expected, the non-interleaving (NI) version ofPPN-MPS is more efficient than its counterpart. naivePPN-MPA exhibits a seemingly strange behavior – running timeincreases almost exponentially with labeling size, only to dropsuddenly at a labeling size of 30. On further analysis of theresults, we discovered that at this labeling size the activitydefinitions first begin to be satisfied by the labeling. Whenno activity definition is satisfied, naivePPN-MPA performsmany iterations in order to decrease the threshold close to 0.Somewhat surprisingly, it is comparable in terms of runningtime with PPN-MPA when the activity is satisfied by at leastone subsequence.

Accuracy: In the second set of experiments, we measuredthe precision and recall of PPN-MPS, naivePPN-MPA andPPN-MPA w.r.t. the human reviewers. In the paper describingthe dataset, Vu et al. [20] only provide figures for recallbetween 80% and 100% (depending on the camera angle),without further details. In this case, the minimality conditionused by the PPN-MPS algorithm poses an interesting problem– we found that in almost all cases, humans will choose asequence which is a superset of the minimal subsequencefor an activity. In order to have a better understanding oftrue precision and recall, we compute two sets of measures:recall and precision at the frame level (Rf and Pf ) andrecall and precision at the event (or activity) level (Re andPe). Assume that both the human reviewer and the algorithmreturn a subsequence corresponding to an activity. We willdenote the subsequence returned by the algorithm A andthe one returned by a human reviewer H . Precision/recallmeasures at the frame level15 are defined as: Pf = |H∩A|

|A|and Rf = |H∩A|

|H| . However, we can relax this conditionby stating that any subsequence returned by the algorithmthat intersects a subsequence returned by a human reviewershould be counted as correct. Let the set of sequences returnedby the algorithms be {Ai}i∈[1,m] and the set of sequencesreturned by the human annotators be {Hj}j∈[1,n]. Then wecan write precision as Pe = |{Ai|∃Hj s.t. Ai∩Hj 6=∅}|

m and recall

15Only with respect to one subsequence, but easily extensible to manysubsequences

as Re = |{Ai|∃Hj s.t. Ai∩Hj 6=∅}|n .

Fig. 10(a) shows the frame precision and recall for thetwo versions of the PPN-MPS algorithm. We notice thatprecision is always very high and generally constant, whilerecall has a rather surprising variation pattern – namely,recall is monotonic with the threshold. The explanation forthe apparently anomalous behavior comes from the fact thatfor low thresholds, the number of candidate subsequences isrelatively high, hence the minimality condition causes fewerframes to appear in the answer. The monotonic behaviorpattern stops when the threshold becomes high enough (.45in this case). Furthermore, we see that the non-interleavingversion of the algorithm has a slightly higher recall than thestandard version. We observed that human annotators usuallyfocus on detecting one activity at a time and very seldomconsider two or more activities that occur in parallel. Hence,the NI version of the algorithm is informally “closer” tohow the reviewers operate. With respect to the PPN-MPAalgorithm, we see a very good match in terms of the activitiesreturned by the human annotators and the ones returned byPPN-MPA for the S4 = {a2, a3, a4, a5} – the most completeset of activities. Poorer performance on the other activity setsis explained by the limitation in the possible choices of thealgorithm. For instance, if a successful bank robbery occursin the video sequence, PPN-MPA cannot return it from thechoices in S2 = {a3, a5}, and thus returns the closest possiblematch. Lastly, event precision was always 100% for PPN-MPS, meaning the algorithm did not yield any false positives.Recall was greater than 90% for the entire dataset for bothPPN-MPS and PPN-MPS-NI.

Temporal Overlap with Groundtruth: Since precisionand recall at the event level was very high, we investigatedto what extent the sequences returned by the reviewers andby PPN-MPS actually overlap. To see why this is relevant,consider an 8 frame-video such as the one in Fig. 1. Assumethat for activity (a5), PPN-MPS returns the subsequence fromframe 1 to frame 4, whereas the human reviewer returns thesubsequence between frames 4 and 8. The event precision andrecall measures match the result returned by PPN-MPS withthat of the reviewer, but the 1-frame overlap between the two


(a) PPN-MPS (b) PPN-MPA and naivePPN-MPA

Fig. 9. Comparison of running times for PPN-MPS, naivePPN-MPA and PPN-MPA on the Bank dataset.

Video S1 S2 S3 S4 Human1 (a5,.61) (a3,.99) (a3,.99);

(a4,.99)(a4,1) (a4,1)

2 (a5,.61) (a3,.99) (a3,.99);(a4,.99)

(a4,1) (a4,1)

3 - (a3,.2) (a3,.2) (a4,.61) (a4,.75);(a2,.75)

4 (a5,1) (a5,.99) (a5,1) (a5,1);(a4,1);(a2,1)

(a4,1)

5 (a5,.61) (a3,.99) (a3,1);(a4,.99)

(a2,1);(a4,.99)

(a4,1)

6 - - - - -7 - - - - -

(a) (b)

Fig. 10. (a) Comparison of frame precision and recall for the two versions of the PPN-MPS algorithm – PPN-MPS and PPN-MPS-NI; (b) comparison ofresults returned by PPN-MPA algorithm and human subjects. Results indicate a good match of the obtained results with human annotators.

TABLE ITEMPORAL OVERLAP OF SEQUENCES RETURNED BY HUMAN REVIEWERS

AND BY THE PPN-MPS ALGORITHM FOR pt = .7.

Video Reviewer 1 Reviewer 2 Reviewer 3 Reviewer 4|H∩A||H|

|H∩A||A|

|H∩A||H|

|H∩A||A|

|H∩A||H|

|H∩A||A|

|H∩A||H|

|H∩A||A|

1 .82 1 .83 1 .84 1 .93 1

2 .36 1 .37 1 .38 1 .41 1

3 .74 1 .83 .93 .67 .97 .72 1

4 .75 1 .78 1 .76 1 .94 .96

5 .56 .99 .58 .98 .58 1 1 .74

is proof to the contrary. We therefore measured the amount ofoverlap between sequences from reviewers and PPN-MPS forall probability thresholds in .1 increments between .1 and 1.Results were similar for all thresholds; we show the overlapfor pt = .7 in Table I for the five video sequences containingbank attacks. The table gives the overlap as a percentage ofthe sequence returned by the algorithm and by the humanreviewer. The data supports our initial intuition that reviewerstend to return larger sequences than the algorithm, since inmost cases the result of the algorithm is a subset of that ofthe reviewers – indicated by the number of 1 values on theright-hand column for each reviewer.

B. Experimental evaluation for the TSA dataset

For the TSA dataset, we used a set of 23 activities includingflight take-off and landing, embarkation or debarkation of pas-

sengers, baggage handling and other maintenance operations.We evaluated PPN-MPS, naivePPN-MPA and PPN-MPA onthe dataset in an attempt to establish their performance ona large dataset, as well as to find evidence for our earlierconclusions drawn from the evaluation on the Bank dataset.The labeling of the TSA dataset consists of 1716 actionsymbols and was obtained in two stages. In the first stage,we ran the low-level image processing primitives to obtainthe initial labeling in approximately 3 hours. In the secondstage, we semi-automatically processed the outcome of theprevious process for about 3 more hours – for instance, todetermine blocks of frames with identical labeling insteadof using individual frames. The ground truth contains 77instances of activities. naivePPN-MPA and PPN-MPA weregiven as input the entire set of activity definitions.

Running Time Analysis: First, we evaluated the runningtime of the algorithms for different labeling sizes. We obtainedthe data points by analyzing the subvideos starting from thebeginning of the TSA footage up to a point where |`(v)| = N .We varied N in increments of 200 action symbols from 0to 1716. The results in Fig. 11(a) show the same type ofvariations as in the Bank dataset. Note that the entire video isprocessed by PPN-MPS in 346 seconds and by naivePPN-MPA and PPN-MPA in a little under 10 minutes16. Weanalyzed the videos that correspond to relatively stable values

16We remind the reader that the labeling had been precomputed in a MySQLdatabase.


(a) (b)

Fig. 11. (a) Variation of running time of PPN-MPS, naivePPN-MPA and PPN-MPA with |`(v)|; (b) variation of running time of PPN-MPS with pt

Fig. 12. Answer quality for PPN-MPS in terms of precision/recall.

in both PPN-MPS and PPN-MPA and to the spike regions innaivePPN-MPA (between 400 and 800, and 1000 and 1400action symbols respectively). We found that these are “quiet”portions of the video, when there are very few activities onthe tarmac. We observed that the binary search algorithm innaivePPN-MPA causes it to deteriorate in terms of perfor-mance whenever there are very few activities being completed.The results for PPN-MPS in Fig. 11(a) are provided for athreshold value of 0.7. We repeated the experiments for allthreshold values between 0.1 and 0.9 and we obtained thesame curve characteristics (Fig. 11(b)).

Accuracy: Second, we evaluated the precision and recallof PPN-MPS and PPN-MPA on the TSA dataset. To evaluatePPN-MPS, we ran PPN-MPS for every of the 23 activitiesand compared the subvideos returned by PPN-MPS to thoseof the ground truth, both at the frame and at the block level.We repeated the process for different thresholds between 0.1and 0.9. The results are displayed in Fig. 12. We notice that theideal probability thresholds (from the point of view of answerquality), both in the case of frame-based and block-basedprocessing, are between .6 and .7. To evaluate PPN-MPA,we divided the video into equal segments of approximately2 minutes and 35 seconds and then ran PPN-MPA with theentire set of 23 activity definitions as input. We then comparedthe most probable activities obtained with those from theground truth. The results of PPN-MPA were correct (from

TABLE IIACCURACY FOR LOW-LEVEL PROCESSING TASKS

Task T1 T2 T3 T4 T5

Accuracy 87.9% 68.6% 76.5% 65.4% 61.5%

a precision point of view) for 44 of the 46 segments. Thetwo false positive segments were due to false positives inocclusion processing. In terms of recall, PPN-MPA missedfour activities; we observed that the false negatives onlyoccurred for activity definitions that were almost subparts oflarger activities. As we mentioned before, due to its greedynature, PPN-MPA has a bias for shorter activities. We alsomeasured the amount of overlap between the results of PPN-MPS and those of the ground truth. For |H∩A|

|H| , the values were

of .85, .89, .91, .86 respectively, and for |H∩A||A| the values were

1, .97, 1, .93.Impact of low-level processing: We also studied the

impact of errors in the low-level image processing algorithmson higher-level reasoning using the Bank data. Specifically,we analyzed the accuracy of the following low-level tasks: a)object tracking (T1), b) object detection (T2), c) merge/splitdetection (T3), d) occlusion detection (T4), e) face recognition(T5).

For all these tasks, we computed the average accuracyin the same set of experiments used to evaluate PPN-MPSprecision and recall for a probability threshold equal to .65.The results are shown in Table II. It is important to note thatthese recognition percentages are only indicative of the relativedifficulty of each of the tasks. Performance on each of the taskscan be easily improved by careful domain-specific retrainingof classifiers (for face recognition for instance) and estimationof detection parameters for individual deployments.

We also considered how the errors in the five tasks propagateto the higher level algorithms. We found that T1 and T2

had by far the greatest impact – approximately 30% of themisses in PPN-MPS and PPN-MPA were caused by errorsin object tracking, whereas 45% were caused by the mis-classification of object types. Two factors contribute to therelatively smaller impact of T3–T5. First, usually errors in tasksT3–T5 are localized to a relatively small number of frameswhere the camera angle or lighting may be detrimental to the


detection algorithms. However, misses in earlier frames mightbe corrected later on, when conditions in the video change.As an example, even if face recognition fails to identify aperson from the database due to shadows, occlusions, etc.,when the algorithm succeeds in a successive block of frames,activity detection in PPN-MPS and PPN-MPA will still betriggered. Second, as we initially suspected, the presence ofskip transitions in the model allows us to filter out part of thenoise produced by the low-level primitives. However, misses inT1 (not detecting an object altogether) or T2 (misclassifying anobject) are generally not recoverable through skip transitions.

VII. CONCLUSIONS

There has been extensive work in computer vision onthe problem of modeling and recognizing human activities.Numerous techniques have been developed to learn activitymodels – both statistical (e.g. HMM’s) and knowledge based(e.g. logic based). In this paper, we build upon such methods.We propose the concept of a Constrained Probabilistic PetriNet using which it is possible to specify activities that canbe executed in a multiplicity of ways. Moreover, constrainedPPNs use “skip” transitions to account for the fact thatactivities in the real world may vary from a hard-coded modelof those activities.

We then propose very fast algorithms for answering thresh-old activity queries. These queries ask us to find minimal seg-ments of a video that contain a given activity with a probabilityexceeding a given threshold. To our knowledge, we are the firstto provably develop algorithms to find such minimal segments.We also show that the algorithm’s complexity is reasonable.We further show experimentally that our algorithm is bothefficient and highly accurate. We also propose a very fastalgorithm for activity recognition – finding the activity thatis maximally probable in a given video (or video segment).Again, we develop a provably correct algorithm. We analyzethe complexity of the algorithm and experimentally show thatour algorithms are both fast and accurate.

Future work may focus on two important problems. First,can we develop methods to learn these PPNs automaticallyfrom a sample of annotated video? Here, human subjectswould annotate a set of videos as containing or not containinga given activity, and the problem would be to learn a PPNfrom such annotations. Second, suppose we are given PPNsdescribing “similar” activities. Is it possible to take multiplesuch PPNs and compose them together into a similar activitythat generalizes them into one?

REFERENCES

[1] R. Chellappa, N. P. Cuntoor, S. W. Joo, V. S. Subrahmanian, andP. Turaga, Understanding Events: How Humans See, Represent, and Acton Events. Oxford University Press, January 2008, ch. ComputationalVision Approaches to Event Modeling.

[2] H. Buxton and S. Gong, “Visual Surveillance in a Dynamic andUncertain World.” Artif. Intell., vol. 78, no. 1-2, pp. 431–459, 1995.

[3] F. Bremond and M. Thonnat, “Analysis of Human Activities Describedby Image Sequences,” in Proc. of Intl. Florida AI Research Symp., 1997.

[4] Q. Cai and J. Aggarwal, “Tracking Human Motion Using MultipleCameras,” in Proc. of 13th Intl. Conf. on Pattern Recognition, 1996.

[5] A. Bobick and J. Davis, “Real-time Recognition of Activity UsingTemporal Templates,” in Proc. of IEEE Workshop on Appln. in ComputerVision, 1996, pp. 39–42.

[6] N. Vaswani, A. Roy Chowdhury, and R. Chellappa, “Activity Recogni-tion Using the Dynamics of the Configuration of Interacting Objects.” inProc. of 2003 IEEE Conf. on Computer Vision and Pattern Recognition(CVPR ’03), vol. 2, 2003, pp. 633–642.

[7] T. Huang, D. Koller, J. Malik, G. Ogasawara, B. Rao, S. Russell, andJ. Weber, “Automatic Symbolic Traffic Scene Analysis Using BeliefNetworks.” in Proc. of the 12th Natl. Conf. on Artificial Intelligence(AAAI ’94), 1994, pp. 966–972.

[8] J. Pearl, Probabilistic Reasoning in Intelligent Systems. MorganKaufmann, 1988.

[9] S. Hongeng and R. Nevatia, “Multi-Agent Event Recognition,” in Proc.of the 8th Intl. Conf. on Computer Vision (ICCV ’01), 2001, pp. 84–93.

[10] M. Brand, “Understanding Manipulation in Video,” Proc. of the 2nd Intl.Conf. on Automatic Face and Gesture Recognition, pp. 94–99, 1996.

[11] Y. Ivanov and A. Bobick, “Recognition of Visual Activities and Inter-actions by Stochastic Parsing.” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 22, no. 8, pp. 852–872, 2000.

[12] M. Albanese, V. Moscato, A. Picariello, V. S. Subrahmanian, andO. Udrea, “Detecting Stochastically Scheduled Activities in Video,” inProc. of the 20th Intl Joint Conf. on Artificial Intelligence (IJCAI ’07),January 2007, pp. 1802–1807.

[13] V. D. Shet, D. Harwood, and L. S. Davis, “VidMAP: Video Monitoringof Activity with Prolog,” in Proc. of the IEEE Intl. Conf. on AdvancedVideo and Signal based Surveillance (AVSS), 2005, pp. 224– 229.

[14] C. Castel, L. Chaudron, and C. Tessier, “What is going on? A High-LevelInterpretation of a Sequence of Images,” in Proc. of ECCV Workshopon Conceptual Descriptions from Images, Cambridge, U.K., 1996.

[15] N. Ghanem, D. DeMenthon, D. Doermann, and L. Davis, “Representa-tion and Recognition of Events in Surveillance Video Using Petri Nets,”in Proc. of the 2nd IEEE Workshop on Event Mining, CVPR2004, 2004,p. 112.

[16] D. Moore and I. Essa, “Recognizing Multitasked Activities from Videousing Stochastic Context-Free Grammar,” in Proc. of Proc. of the 18thNatl. Conf. on Artificial Intelligence (AAAI ’02), 2002, pp. 770–776.

[17] S. W. Joo and R. Chellappa, “Recognition of Multi-Object Events UsingAttribute Grammars,” in Proc. of IEEE Intl. Conf. on Image Processing,October 2006, pp. 2897–2900.

[18] C. Lesire and C. Tessier, “Particle Petri Nets for Aircraft ProcedureMonitoring Under Uncertainty,” in Proc. of the 26th Int. Conf. onApplications and Theory of Petri Nets (ICATPN ’05). Springer Verlag,June 2005, pp. 329–348.

[19] J. Cardoso, R. Valette, and D. Dubois, “Possibilistic Petri Nets,” IEEETrans. Syst. Man Cybern., vol. 29, no. 5, pp. 573–582, 1999.

[20] V. T. Vu, F. Bremond, and M. Thonnat, “Automatic Video Interpretation:A Novel Algorithm for Temporal Scenario Recognition,” in Proc. of the18th Intl. Joint Conf. on Artificial Intelligence (IJCAI ’03), 2003.

[21] R. David and H. Alla, “Petri nets for Modeling of Dynamic Systems ASurvey,” Automatica, vol. 30, no. 2, pp. 175–202, 1994.

[22] T. Murata, “Petri Nets: Properties, Analysis and Applications,” Proceed-ings of the IEEE, vol. 77, no. 4, pp. 541–580, 1989.

[23] M. A. Marsan, G. Balbo, G. Chiola, G. Conte, S. Donatelli, andG. Franceschinis, “An Introduction to Generalized Stochastic PetriNets.” Microelectronics and Reliability, vol. 31, no. 4, pp. 699–725,1991.

[24] S.-M. Chen, J.-S. Ke, and J.-F. Chang, “Knowledge RepresentationUsing Fuzzy Petri Nets.” IEEE Trans. Inf. Theory, vol. 2, no. 3, pp.311–19, sep 1990.

[25] N. T. Siebel and S. Maybank, “Fusion of Multiple Tracking Algorithmsfor Rbust People Tracking,” in Proc. of the 7th European Conf. onComputer Vision (ECCV 2002), vol. IV, May 2002, pp. 373–387.

[26] ——, “The ADVISOR Visual Surveillance System,” in Proc. of theECCV 2004 workshop Applications of Computer Vision (ACV ’04), May2004, pp. 103–111.

[27] I. Haritaoglu, D. Harwood, and L. S. Davis, “W4: Real-Time Surveil-lance of People and Their Activities, journal = IEEE Trans. Pattern Anal.Mach. Intell., volume = 22, number = 8, year = 2000, pages = 809–830.”

[28] A. Prati, I. Mikic, M. M. Trivedi, and R. Cucchiara, “Detecting MovingShadows: Algorithms and Evaluation,” IEEE Trans. Pattern Anal. Mach.Intell., vol. 25, no. 7, pp. 918–923, 2003.

[29] Y. Zhu, S. Schwartz, and M. Orchard, “Fast Face Detection UsingSubspace Discriminant Wavelet Features,” in Proc. of IEEE Conf. onComputer Vision and Pattern Recognition, vol. 1, June 2000, pp. 636–642.

[30] S. Khan and M. Shah, “Tracking People in Presence of Occlusion,” inProc. of Asian Conf. on Computer Vision, January 2000.

A Constrained Probabilistic Petri Net Framework for Human Activity Detection in Video*

Documents

Transcript of A Constrained Probabilistic Petri Net Framework for Human Activity Detection in Video*