Abstract—Previously, most human tracking system mainly focuses on conventional imaging system that depends on the illumination and has a limited field of view. In this paper, we propose to introduce a novel surveillance system that is Thermal Catadioptric Omnidirectional (TCO) Vision System. The proposed system is able to realize the surveillance in all-weather and big field of view conditions. For human tracking in TCO vision, the most of existing contour based feature cannot be used directly unless unwarp the distorted omnidirectional image into the traditional rectangle panoramic image for rectification. In this paper, a contour coding based rotating adaptive model is proposed, which is developed based on the characteristic of TCO vision. The proposed model not requires unwarping but only use relative angle based on location of target in the omnidirectional image to change the sequence of sampling from the proposed model. Based on this model, contour feature is extracted, which contains geometric location (r, ) and gradient information g(r, ) of the target. The extracted feature is fed into Support Vector Machine (SVM) for classification. For tracking purpose, we integrate the obtained classification posterior probability of SVM with the likelihood observation of particle filter to realize tracking in TCO vision. Finally, a series comparative experiments and quantitative analysis are presented to verify the feasibility and performance of proposed tracking method.

I. INTRODUCTION N these decades, the automatic surveillance system becomes more and more popular in a large variety of

applications due to the improvement of automation in the modern society. In computer vision community, human tracking is always a hot research topic, which form the important part of surveillance system. The previous surveillance system [1] mainly focuses on the conventional vision system that has a limited field of view and depends on the illumination. In this paper, we try to explore a novel surveillance system for human tracking in thermal catadioptric omnidirectional (TCO) vision to realize surveillance under all-weather and wide field of view conditions. The proposed novel surveillance system can offer a much wider field of view and work independent of illumination, which permit to acquire more information from the environment. In order to overcome the illumination restriction, the thermal camera is employed for it can detect the warm-blooded animal (include human) in any illumination condition, no matter day and night, rain and fog.

Yazhe Tang , Youfu Li, Tianxiang Bai and Xiaolong Zhou are with the Department of Mechanical and Biomedical Engineering City University of Hong Kong, Kowloon, Hong Kong, China, (corresponding author to provide phone: 852-3442-8227, e-mail: yztang2@student.cityu.edu.hk & meyfli@cityu.edu.hk).

Furthermore, the price of thermal camera has reduced dramatically, so it becomes popular in civilian application over the recent years. To achieve a big field of view, the catadioptric omnidirectional sensor is able to meet our requirement, which can reflect the surrounding light (360 degree in horizontal direction) into a single camera. Therefore, the hardware of proposed surveillance system mainly consists of a thermal camera, a catadioptric omnidirectional mirror and a stand (Fig. 1). A representative of TCO image is given in Fig. 2. For the merits of the proposed surveillance system, it could have a wide range of applications, such as surveillance in the airport, security in the military, monitor in public place, wild animal conservation, automatic drive assistance and so on.

Fig. 1. Configuration of TCO system.

Fig. 2. A Representative of TCO image.

Tracking in TCO vision have some big challenges due to only limited features can be used in thermal imagery and severe inherent distortion in catadioptric omnidirectional vision (COV). To the best of the author's knowledge, there are very limited works on tracking in TCO vision can be referenced by our work. Based on the characteristic of thermal camera, contour feature can be considered as a stable feature in thermal vision, and gray level information is useful too, but texture is very limited and unstable in thermal vision. Even tracking in thermal imaging is difficult compare to

A Rotating Adaptive Model for Human Tracking in Thermal Catadioptric Omnidirectional Vision Yazhe Tang, Youfu Li, Tianxiang Bai and Xiaolong Zhou

I

978-1-4577-2138-0/11/$26.00 © 2011 IEEE 2435

Proceedings of the 2011 IEEEInternational Conference on Robotics and Biomimetics

December 7-11, 2011, Phuket, Thailand

traditional visible imaging but driven by its merit of illumination independent, and decreasing price of it, more and more researchers began to focus on thermal imagery vision. Recently, there have a flurry of excellent works on human detection and tracking in thermal vision. [2] employed Support Vector Machine (SVM) for classification and used Kaman filter to integrate with mean shift for tracking pedestrian in thermal imagery. In [3], the author presented a two-stage template-based method combined with an Adaboosted classifier for pedestrian detection. [4] integrated SVM [5,6] with Histogram of Oriented Gradient (HOG) [7] to detect the pedestrian in thermal imagery. In [8], the author used a generalized expectation-maximization (EM) algorithm to separate infrared images into background and foreground layers and incorporated with SVM for classification, and then presents a graph matching-based method to meet the tracking purpose.

In addition, more researchers began to concentrate on omnidirectional vision detection and tracking for its wide field of view. Especially in the situations which require a global field of view, omnidirectional vision has its unique advantage compare to conventional vision. Previously, researchers did some excellent works in this area. In [9], a fisheye omnidirectional tracking system is introduced, the author used optical flow to detect the target and employed color feature based kernel particle filter (KPF) to realize the single target tracking in omnidirectional vision. [10] presented a catadioptric omnidirectional surveillance system which use of multi-background modeling and dynamic thresholding to make a outdoor tracking in clutter to spot the sniper in the battlefield. [11][12] used of particle filter incorporates with color feature to realize the tracking in COV. [13] introduced a catadioptric omnidirectional pedestrian recognition system for vehicle automation, which proposed a method of boosted cascade of wavelet-based classifiers combined with a subsequent texture-based neural network. In the human detection and tracking community, the most of state-of-art methods are based on contour feature which includes the oriented gradient information. On account of inherent distortion of COV in its radial direction, gradient direction information is varying as the human target change the location in omni-image. So, conventional human detection and tracking method can not apply into omnidirectional vision directly. Therefore, some researchers prefer to unwarp the original catadioptric omnidirectional image into panoramic image to rectify the distorted omnidirectional image into traditional rectangle image first [13] [14], and then apply the conventional algorithm for human detection and tracking purposes (Fig. 3). Those methods are time consuming due to interpolation and coordinate transformation. In addition, unwarpping has the risk to split the target that locates at the border of the panoramic image. So, we prefer to track the human target in the original TCO image directly. In this paper, we present a contour coding based rotating adaptive model in TCO vision, which can apply into original omnidirectional image directly

but not require to unwarp the original omnidirectional image into panoramic image.

Fig. 3. Flowchart of conventional human tracking method in omnidirectional vision

In this paper, we mainly address the tracking in TCO vision. So, the proposed system is initialized manually in detection phase. In the main content of this paper we present the proposed rotating adaptive model which can directly apply into original TCO vision to extract the contour feature. And then, the extracted feature is classified by the SVM as the traditional method [4][7][15]. Moreover, the proposed tracking method intends to take advantage of the SVM classification posterior probability to relate the observation likelihood of particle filter for efficient tracking in TCO vision. The main flowchart of proposed tracking method in TCO vision is shown in Fig. 4. For the reason of no existing TCO vision dataset in public, a dataset including TCO videos and extracted human samples is formed. The extracted samples are used to train the SVM classifier, and TCO videos are employed for testing. In addition, polar coordinate is adopted to fit the COV. Finally, a series comparative experiments and quantitative analysis are presented to verify the performance of proposed tracking method in TCO vision.

Fig. 4. Flowchart of proposed human tracking method in TCO vision

The paper is structured as follows. In section II, present the novel model in TCO vision, introduce a general theoretical foundation of Support Vector Machine and particle filter. Section III presents the quantitative analysis and discusses the experimental results. Finally, conclusion will be followed.

II. PROPOSED TRACKING ALGORITHM In this paper, we propose to address the human tracking in

original TCO vision. However, tracking in TCO vision is difficult due to there are several big challenges have to be overcome, such as limited features can be used in thermal image and inherent distortion in COV. In this section, we propose to introduce a new tracking method based on a novel rotating adaptive model that is developed on the characteristic of TCO vision. In addition, the proposed method integrates SVM with particle filter for efficient tracking in TCO vision. The detail of algorithm is shown as following.

2436

A. A Novel Rotating Adaptive Model

Fig. 5. The proposed model extracts the feature fro

Fig. 6 The rotating adaptability of the proposed moBased on the characteristic of TCO

features can be used for tracking, information and contour feature. Howevcontour based feature can not apply directly due to inherent distortion of COSo, we propose to introduce a novel corotating adaptive model for human tracThe proposed model is very suitable for not requires to unwarp the disomnidirectional image into conventionalimage for rectification. It just requires aon the characteristic of omnidirectional sampling sequence of contour. The propis extracted from the gradient map basmodel which consists of a series of coseparated by a group of radial lines intoEach cell includes geometric informatioinformation g(r, ). Based on the charactposition in the omnidirectional image cor relative to the reference line in omnidir

6). Therefore, the model can change theautomatically based on the relative angleprinciple, the proposed feature can accomof target in COV very well. On accourotating adaptive model involves neithcoordinate transformation in unwarpin

om a gradient map

odel vision, only limited such as gray-level

ver, the conventional into original COV

V in radial direction. ontour coding based

cking in TCO vision. TCO vision due to it

storted catadioptric l rectangle panoramic a relative angle based

vision to change the posed contour feature sed on the proposed oncentric circles and o multi-cells (Fig. 5). on (r, ) and gradient teristic of COV, each rresponds to an angle rectional image (Fig. e initial sample point e . According to this mmodate the rotation unt of the proposed

her interpolation nor ng process, it has a

potential capability to extracmeet the requirement of realprinciple of this proposed proposed novel model is veomnidirectional vision due toCOV.

B. SVM Classification PosSVM [16] [17] is a very

classification. It can discrimcorrectly based on a trainedsample set is provided

, , l is the The SVM works by maximclasses in feature space as fol

1min ( , , )2

NT

ii

J b Cξ = +w w w

Subject to the constraints[ ( ) ] 1 ,T

i i iy bϕ ξ+ ≥ −w x Where i is the slack variab

is the mapping from input spthe Lagrangian of (1), the orig

1 1, 1

1max 2

l l

i i j i ji i j

y y kα α α= = =

−

Subject to the constra

10

l

i ii

yα=

= 0≥iα

Where ( , ) ( ) (i ik ϕ ϕ= ⋅x x x xi=1,2,…l, is the Lagrange coe(2). The hyperplane function

Ti i

1( ) y x x + b

m

ii

f x α=

=

Where m is the number oobtained hyperplane functionclassification posterior probab

( )1( 1 )

1 expP y f

Af B= =

+ +It expresses the posterior

classified into class 1 (positivthe probability of sample be cset).

C. Particle Filter Particle filter [19-21] is al

Carlo methods (SMC), and iadvantage of nonlinear/nframework, the particle filter at time k, given t

kk zzzz ,,, 21:1 = up to

( )1:11 −− kk zxp at time k-1 is avcan be obtained recursively prediction stage makes use

ct the contour feature fast and l-time for tracking. Base on the feature, we can conclude the ery suitable to apply into the

o it well use the characteristic of

sterior Probability y useful algorithm for pattern minate the unlabeled samples d model. Suppose the training

asnumber of the training samples.

mizing the margin between two llows [17]:

1

N

iξ=

(1)

s: 0

ble, C is the penalty factor, ( )ϕ ⋅pace to feature space. By taking ginal problem can be derived as:

( , )ik x x (2)

aints:

)x is a kernel function, and i efficients which is obtained from can be expressed as:

(3)

of support vectors. Further, the n (3) can be used to derive the bility [18] defined as follow:

) (4)

probability of this sample be ve set). Oppositely, 1-P(y=1|f) is classified into class -1 (negative

lso known as Sequential Monte it has been widely used for its non-Gaussian. In Bayesian is recursively obtain the state xk

the available observations time k. Suppose posterior

vailable, the posterior ( )kk zxp :1 by prediction and update. The

e of the probabilistic system

2437

transition model ( )1−kk xxp to predict the posterior probability of time instant k.

Prediction step:

( ) ( ) ( ) −−−−− = 11:1111:1 kkkkkkk dxzxpxxpzxp (5)

At time instant k, an observation zk is available, the posterior ( )kk zxp :1 at time instant k can be obtained through update prior by Bayes’s rule.

Update step:

( ) ( ) ( )( )1:1

1:1:1

−

−=kk

kkkkkk zzp

zxpxzpzxp (6)

Where ( )kk xzp is the observation likelihood that influences the weight distribution of the particle. In this paper, the observation likelihood can be obtained from the SVM classification posterior probability.

At time instant k, each particle represents a hypothetical state i

kx , with corresponding weight ikw . So, the principle of

particle filter is to utilize finite discrete N weighted particles{ } Ni

ik

ik wx ,,2,1, = to approximate the continuous posterior

distribution. Of course, a large number of particles can approach to the true distribution with a greatest degree. However, the computational cast is increased as the N increasing. So, that is a trade-off problem in the practical situation.

D. Likelihood As aforementioned, if an extracted local feature be

classified by SVM, the result of (4) expresses the extent of extracted feature be positive (class 1). That is pp=P(yi=1|f) represents the probability of sample xi to be classified into positive set (class 1). In contrast, pn=1-P(yi=1|f) is the probability of negative set (class -1). So, the achieved SVM classification posterior probability can be employed to obtain observation model ( )kk zxp :1 as shown in the following equation:

( ) ( )2exp dxzp ikk ⋅−∝ λ (7)

( )ikk

ik

ik xzpww 1−∝ (8)

Where is covariance and d is defined by (9). The obtained observation likelihood ( )kk xzp is used to relate weight wi of the particles in (8). If the extracted sample is classified into positive set by SVM, the proposed algorithm calculates the distance d. Oppositely, the algorithm ignores the result if the extracted sample is classified into negative set. The obtained d expresses the distance between candidate sample and standard positive sample. In other word, if distance d getting small, the sample has a higher probability to be positive and the weight getting to big.

d=1-pp if positive sample (9)

Finally, a contour coding based rotating adaptive model is proposed. The extracted contour feature is classified by the

SVM. The obtained SVM classification posterior probability is integrated with the particle filter to relate the observation likelihood of particle for tracking purpose in TCO vision. According to above, the proposed tracking method should able to track the human in TCO vision efficiently. The detailed experiments are shown below.

III. EXPERIMENT In the beginning of this paper, the configuration of TCO

system has been introduced. In this section, a series of comparative experiments will be presented. In tracking community, color feature based tracker has been verified that has a good performance in the most of situations. In addition, the color of human target is salient against to the background in thermal vision. Moreover, color feature is non-directional due to it just depend on the color-level distribution. That is to say, the normalized color feature is invariable when the target is rotating in the TCO vision. So, the color based tracker has been applied into tracking in thermal vision and omnidirectional vision, respectively [2][9]. Therefore, we prefer to compare experiment performance of proposed tracking method with the tracker of color feature based particle filter. In this paper, we adopt the gray level information as the feature due to there is only single channel gray-level information can be employed in thermal image. Through test in a number of different experiments, the performance, effectiveness and stability of proposed tracking method are verified.

1) “P-PF” is the proposed tracking method based on rotating adaptive contour model, which utilizes the SVM classification posterior probability to impact the observation likelihood of particle filter.

2) “C-PF” is the gray-level information based particle filter.

For a fair comparison, we set the same parameters for above two tracking methods, such as particles number N, covariance and so on. The dynamic models of the trackers are random walking models which represent as kkk vxx += −1

, where vk is a zero-mean Gaussian random variable, and the state of particle is (xk, yk, sk), k=1, 2…n. xk and yk are the coordinates of particle and sk is the scale of particle in time k. For the proposed method, the SVM classifier has been well trained with 750 positive and 960 negative samples. Because there is without the available TCO vision dataset in public, we have to extract the samples manually. For limited space, we just show three experimental results with one shot in the daytime and two in the nighttime. In the nighttime, the human target has a higher gray value compare to background than it in the daytime. That is to say, the C-PF should more easily to capture the target in nighttime. Therefore, the experimental configuration is favorable for C-PF. In this paper, it is should be noted that we just test single and simple multiple targets to discuss the feasibility of the Proposed method in TCO vision. After that, more challenge situation involved occlusion in TCO vision will be extended in our future work.

2438

Fig. 7 The performance of P-PF with the different model configuration, Radius Interval (RI) and Angle Spacing (AS), (RI unit: pixel, AS unit: degree)

Fig. 8 RMSE of P-PF and C-PF with different number of particles

In order to verify the performance of P-PF effectively, the quantitative analysis is given in the following. The average root mean square errors (RMSE) with different particle numbers are presented to show the stability of proposed method during tracking. As mentioned in previous, the proposed rotating adaptive contour model is adopted to handle the rotation of target. However, the performance of proposed method depends on the configuration of the model, such as the radius interval and the angle spacing. We use 5 groups of configuration to quantize the performance of P-PF as shown in Fig. 7. It can be seen from the Fig. 7 that all the RMSE values of P-PF are less than 3.2 and the configuration with the RI=3 (3 pixels length in radial direction), AS=10 ( 10 degree angle spacing) gives a better performance among of those configurations in generally. Furthermore, it achieves the best performance when the particle number is 300. Therefore, we employ the configuration with RI=3, AS=10 of P-PF to compare with the C-PF in the following comparative experiments. The RMSE of P-PF and C-PF with the different number of particles is given in Fig. 8. It is shown from Fig. 8 that the P-PF has a better performance than C-PF in TCO vision. C-PF with 400 particles achieved the best performance while P-PF with only 300 particles gives much better performance. Combined with Fig. 7 and Fig. 8, all the RMSE values of P-PF are less than 3.2 but the C-PF is between 5.6 and 7.2. Therefore, it is clear that the P-PF has a very good ability to well focus on the center of human target

but the C-PF very often happen to bias. To further discuss the stability and performance of P-PF, we propose to analyze the following three experiments in detail.

In experiment I, a person walks around the TCO-sensor in the daytime, and we used 100 particles in the tracker. At the beginning, two trackers are initialized respectively, and they tracked the target successfully. After several frames later, C-PF tends to drift due to the impact of background, and lost the target at frame 90 finally (Fig. 9 (a)). The reason of C-PF failed is because the gray-level information of foreground is similar with the background in the daytime that is not distinct enough for C-PF to capture the target. Although it is difficult to track the target for C-PF, P-PF still can work well to track the human target from beginning to the end on account of contour feature is adopted. Therefore, the contour feature is stable in the TCO vision. During the tracking process, C-PF cannot well focus on the center of target (Fig. 9 (b)) due to the influence of the background that has the similar gray value distribution with the target. So, C-PF very often happens to bias. However, the P-PF can capture the target very well as shown in Fig. 9. (c)(d).

In experiment II, two people walk around the TCO sensor in the nighttime. In the evening, the foreground of image is more distinct compare to the image in daytime because the ambient temperature is decreased. So, the interference of background is possibly reduced greatly. Therefore, the performance of the two trackers is likely to improve than before. However, the experiment shows the P-PF works very well (Fig. 10(c), (d)) but the C-PF often happens to bias and lose one of two targets finally(Fig. 10 (a), (b)). The reason of C-PF to fail is because low number of particles (100 particles) and only limited gray level information can be used in both experiment I&II. Through this experiment we can conclude that P-PF is stable and robust compare to C-PF, especially in the case of a small number of particles.

In experiment III, a person moves around the TCO sensor in the nighttime as the experiment II. In order to improve the performance of C-PF, we increase the number of particles to 400. In this experiment, both of P-PF and C-PF survive to the end of experiment. Moreover, P-PF can well focus on the center of target (Fig. 11(c), (d)). However, C-PF happens to deviation frequently (Fig. 11(a), (b)). One of reasons C-PF has poor performance is only limited single channel gray-level information can be used instead of three channels RGB information. In addition, serious background interference misguides the tracker to the wrong place. Finally, only P-PF can work well to track the target in TCO vision.

Finally, the system is implemented in Matlab on a PC with an Inter Pentium 2.7GHz and 2G memory, we achieve 2.13s per frame (300 particles) without optimization. So, the proposed system should have a great potential to meet the requirement of real-time for tracking. In summary, the above three experiments and quantitative analysis can effective verify the rotating adaptive contour model based P-PF has a good performance on tracking the human target in TCO vision even in the situation of small number of particles.

2439

Fig. 9 Experiment I. From left to right: (a), (b), (c), (d)

Fig. 10 Experiment II. From left to right (a), (b), (c), (d)

Fig. 11 Experiment III. From left to right (a), (b), (c), (d)

IV. CONCLUSION In this paper, we present a novel tracking system. The

proposed tracking system integrates the rotating adaptive contour model with the SVM and particle filter to perform human tracking in TCO vision. Finally, the proposed tracking algorithm has been verified that achieves a good performance on tracking the human in TCO vision through a variety of experiments.

ACKNOWLEDGEMENT This work was supported by the Research Grants Council

of Hong Kong (Project No. CityU 118311) and the University Grants Committee (Grant No.SEG_CityU 01)."

REFERENCES [1] P. Viola, M. J. Jones, and D. Snow. Detecting pedestrians using patterns

of motion and appearance. In Proc. ICCV, Volume 2, pages 734-741, 2003

[2] F. Xu, X. Liu, K. Fujimura, Pedestrian detection and tracking with night vision, IEEE Trans. Intell. Transport. Syst. 6 (2005) 63-71

[3] J. Davis, M. Keck, A two-stage approach to person detection in thermal imagery, Proc. Workshop on Applications of Computer Vision, 2005, IEEE OTCBVS WS Series Bench

[4] F. Suard, A. Rakotomamonjy, A. Bensrhair, Pedestrian detection using infrared images and histograms of oriented gradients, Intelligent Vehicles Symposium 2006, June 13-15, Tokyo, Japan

[5] C. J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 2, 121-167, 1998

[6] V. Vapnik. Statistical Learning Theory. Wiley, 1998 [7] N. Dalal, B. Triggs, Histogram of Oriented Gradients for Human

Detection, CVPR, 2005. volume 2, pp 886–893 [8] C. X. Dai, Y. F. Zheng, X. Li, Layered representation for pedestrian

detection and tracking in infrared imagery, Computer Vision and Image Understanding. 106 (2007) 288-299

[9] S. Y. Yang, G. W. Min and C. Zhang, Tracking unknown moving targets on omnidirectional vision, Vision Research, 49 (2009), pp362-367

[10] T. E. Boult, X. Gao, R. Micheals, M. Eckmann, Omni-directional visual surveillance, Image Vis. Comput. 22, (2004), pp 515-534

[11] J. C. Bazin, K. J. Yoon, I. Kweon, C Demonceaux, P Vasseur, Particle filter approach adapted to catadioptric images for target tracking application, IEEE, BMVC, 2009.

[12] O. A Jaime and B.C Eduardo, Omnidirectional vision tracking with particle filter, ICPR 2006.

[13] W. Schulz, M. Enzwiler and T. Ehlgen, Pedestrian recognition from a moving catadioptric camera, Proceedings of the 29th DAGM conference on Pattern recognition 2007, pp.456-465

[14] A. L. C Barczak, J. O. Jr and V. G. Jr, Face tracking using a hyperbolic catadioptric omnidirectional system, Res. Lett. Inf. Math. Sci, 2009 (13), pp55-67

[15] W. L. Ye, H. P. Liu, F. C. Sun and M. Gao, Vehicle tracking based on co-learning particle filter, IROS, 2009, pp.2979-2984

[16] V. Vapnik, The Nature of Statistical Learning Theory, Spring-Verlag, New York, 1995

[17] C. J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 2, 121-167, 1998

[18] J. C. Platt. “Probabilistic outputs for support vector machines and comparisions to regularized likelihood methods”. in Advances in Large Margin Classifiers, Alexander J. Smola, Peter Bartlett, Bernhard Scholkopf, Dale Schuurmans, eds., MIT Press, (1999)

[19] M. S. Arulampalam, S Maskell, N Gordon and T Clapp, A Tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking, IEEE Transactions on Signal Processing, (50), 2002, pp 174-188

[20] A. Doucet, J.F.G. de Freitas and N.J.Gordon, “An introduction to sequential Monte Carlo methods,” in Sequential Monte Carlo Methods in Practice, A. Doucet, J.F.G.de Freitas and N.J.Gordon, Eds. New York: Springer-Verlag, 2001

[21] M. Isard, A. Blake, Condensation-conditional desity propagation for visual tracking, IJCV, vol.29,no.1,1998, pp.5-28

2440