International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

Experiments in Haptic-Based Authentication of Humans Mauricio Orozco, Matthew Graydon, Shervin Shirmohammadi,

and Abdulmotaleb El Saddik Multimedia Communciations Research laboratory

University of Ottawa {morozco,shervin,abed}@mcrlab.uottawa.ca

Abstract With the rapid advancement of the technological revolution, computer technology such as faster processors, advanced graphic cards, and multimedia systems are becoming more affordable. Haptics technology is a force/tactile feedback technology growing in disciplines linked to human-computer interaction. Similar to the increasing complexity of silicon-based components, haptics technology is becoming more advanced. On the other hand, currently available commercial haptics interfaces are expensive, and their application is mostly dedicated to enormous research projects or systems. However, the trend of the market is forcing haptic developers to release products for use in conjunction with current keyboards and mice technologies. Haptics allows a user to touch, fell, manipulate, create, and/or alter simulated three-dimensional objects in a virtual environment. Most of the existing applications of haptics are dedicated to hone human physical skills such as sensitive hardware repair, medical procedures, handling hazardous substances, etc. These skills can be trained in a realistic virtual world, and describe human behavioural patterns in human-computer interaction environments. The measurement of such psychomotor patterns can be used to verify a person’s identity by assessing unique-to-the-individual behavioural attributes. This paper explores the unique behaviour exhibited by different users interacting with haptic systems. Through several haptic-based applications, users’ physical attributes output data from the haptic interface for use in the construction of a biometric system. 1. INTRODUCTION Haptic research has been building the theory that relates the intersection of human perception and computer technology through software and hardware components. Recently, the word “Haptics”, which is derived from the Greek verb “to touch” or to handle, has appeared in vast scientific literature. Haptics technology has developed tactile human-computer interfaces and devices that mimic the human sense of touch. This technology of touch can be seen in great a variety of applications from tele-medicine, interactive virtual reality, tele-presence, and tele-manipulation. This technology has already been explored in contexts as diverse as modeling and animation, geophysical analysis, dentistry training, virtual museums, assembly planning, surgical simulation, and remote control of scientific instrumentation. Recently, haptics technology has been explored in the context of biometric interfaces [1]. Such exploration describes characteristic parameter values that are consistent to particular user when he/she gets used to handling a haptic device. Similarly, users leave a personalized mark with the daily use of devices such as mobile phones, keyboards, PDAs, etc. The analysis of these human patterns has been investigated by biometrics. Biometric systems produce an interface that allows identification of individuals based on behavioral or physiological characteristics [2]. The most common implementations of such technology recognize people based on their fingerprint, voice, iris, or face image. Applications of such systems are vast, and range from national security applications to access control. Haptics devices allow for the characterization of personalized user-specific physical and biological parameters, similar to those that are deployed by human interaction with equipment such as signature pads, keyboard buttons, and telephone buttons. From the user-haptic-based interactions, parameters such as measurements of force exerted, position, and velocity are obtained. With measurements of these parameters, we investigate the possibility of identifying a user with a rigorous level of precision. The quantification and measurement methodology described in this study provides an opportunity for the evaluation of haptics suitability as a mechanism to be implemented in a biometric system. We implement several haptic-based applications that, from a biometric point of view, yield recognizable patterns that represent conventional interactions between humans and machines: dialing phone codes, signing virtual cheques, and solving a virtual maze. Through such haptic-based applications,

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

we investigate the consistency of personalized data describing human psychomotor patterns. These patterns are quantified and classified for authentication purposes. This identification procedure can be extended to a continuous process for participants interacting with haptic systems. We believe in the possibility of automatically discriminating and characterizing participants based on this data. We also identify relationships between the methodology used for authentication purposes and the nature of the haptic based applications. We conclude that the potential use of haptics as a biometric system interface depends on different issues, such as the nature of the application specifications, the pattern recognition theory used, the algorithm used in the process of feature extraction and selection, and also in the precision and perception of the haptic devices. 2. THE ROLE OF HAPTIC TECHNOLOGY 2.1. Introduction Recently, innovations in technology have led to an increase in research and demand for haptic user interfaces. The term haptics refers to the expanding discipline that is associated with the study of touch through human computer interaction. The development of haptic interfaces is motivated by the fact that these devices provide an intriguing opportunity to develop multisensory feedback and communication systems that present data in an intuitive manner [8].By incorporating haptic feedback and sensing units to multimodal virtual reality systems, there is a potential to increase the veridicality of such environments [9]. Fields such as tele-surgery and tele-operation are posed to gain substantially through the development of haptic user interfaces. The demand for such systems has fueled research in computational haptics. Lately, some prototypes have become commercially available to the market. Applications of this technology have rapidly spread to devices applied to graphical user interfaces (GUI’s), Games, Multimedia Publishing, Scientific Discovery and Visualization, Arts and Creation, Editing Sound and Images, Vehicle Industry, Engineering, Manufacturing, Tele-robotics, Tele-operation, Education and Training, Medical Simulation and Rehabilitation. Despite of this wide application spectrum, only a few disciplines have developed robust platforms for a concrete usage of haptics. Requirements of stability, perception, and precision are important issues to be considered for a system’s success. By adding force feedback to system design in conventional 3D designing software platforms such 3ds max®, maya® and rhino™, free-form virtual modeling has been endowed with some of the greatest benefits haptics technology can offer [3]. Haptics technology has also been successfully deployed in the medical field, a discipline contributing significantly to the demand for development and implementation of haptic interfaces. A VR based surgical simulator system with force feedback enriches and enhances traditional education in endoscope surgery [4]. Moreover, these commercially available products are also released with specialized instruments for force-feedback manipulators in laparoscopic surgery simulators [5]. Surgical planning and training tasks have also been developed to support the endovascular surgery to radically change the traditional methods for official recognition [6]. The list of research literature and haptic surgical simulators prototypes implemented in this domain is increasing and there is out of scope to enumerate all. 2.2. Motivation This innovative concept of introducing haptics systems to security systems to identify users has not been investigated yet. In other words, to the best of our knowledge, no other work has examined haptics from a biometric point of view. However, the aforementioned concept is somewhat based on traditional behavioral biometric systems, such as keystroke dynamics, speaker recognition, and dynamic signature verification [10, 11, 12, 13]. In the past two decades, keystroke dynamics research has been studied in terms of latency timings [10]. These studies describe a particular pattern defined by a simple approach, given by the keystroke durations. In the same manner, dynamic signature verification technologies have been extensively researched; however, force feedback has not yet been incorporated [12] [13]. Our set of haptic-

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

based applications presents a study that is similar to these previous works, but it extends the concept by using programmable haptic interfaces that provide measurements of physical parameters in real time. This system includes parameters such as user’s force exerted, speed, and pen position during the haptic training. These parameters are used to obtain a biometric profile of each user. 3. SYSTEM OVERVIEW 3.1. General Architecture Based on the general authentication process for biometric systems, we identify four core subsystems: haptic devices and applications, data acquisition, feature generation/selection, and classifier design/ evaluation. From Figure 1, we see that the first step involves a biometric enrollment process described by two subsystems: (1) haptic based applications, and (2) data acquisition. In the former, tasks are defined where haptics delivers an appropriate human-sensor system interface for capturing a person’s behavioral characteristics. In the data acquisition subsystem a recording process is carried out, resulting in biometric raw data in unprocessed format. After acquisition, the data is handed to the feature generation subsystem (FGS). Here, a quantitative characterization of the state of the haptic system is formed. The system’s state properties are described by an array of vectors, each corresponding to different features to be analyzed. In addition, within the FGS a preprocessing task is required before data is ready to be processed in the feature selection processes. Outliner removal, data normalization, and missing data detection is accomplished via this preprocessing interface. The next procedure, called feature selection, receives a given number of features and selects those of the greatest user-classificatory worth. Using relative entropy the reduction of state vector dimensionality is reduced, and using chi-squared tests features are extracted. For authentication, a set of biometric algorithms are used in the classifier design procedure. From a great variety of pattern recognition methodologies, we have chosen the well-know PCA, DTW, Neural Networks, and K-Means methods to classify haptic user feature vectors extracted from raw data characterizing their psychomotor patterns. This final component of the system process results in an authentication decision based upon match score metrics.

Figure 1: Overall architecture of verification system using haptics to identify human movements’ patterns

The architecture presented in Figure 1 is based on the implementation of biometric signatures within the data acquisition, feature generation/selection, and authentication processes. This approach permits the importation of other methodologies and also enables a classification of biometric signatures using a variety of complex single-modal and multimodal biometric systems in the Information and Technology(IT) field according with [7].

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

3.2. Data Acquisition Data acquisition starts with the physical attribute to be measured. This physical attribute could be the 3D world coordinates that describe a trajectory, the pressure applied to an object, or many other things. A set of virtual environments with haptic-based applications was developed by accessing an application-programming interface providing direct access to the tracking device during various events [14]. The software application set was developed in a combination of Python scrip code and VRML-based scene graph module. The behavioural of the hapto-visual applications were determined by the python procedural process, to handle certain events and output the data to a flat file; meanwhile, the 3D scenario was defined by VRML-node-fields approach. Over a period of 4 weeks, a database of profiles was created. There is variation within different types of persistent storage. Our database is composed of a set of flat files, which were collected on a workstation MS-OS 2000 with a XENON processor at the DISCOVER Lab of the University of Ottawa. The data we collected describes various distinct physical parameter haptic system features as functions of time. The performance results reported here are based on the measurements of these parameters: xyz position, orientation (Θ), time (t), force (F) and torque (T) acquired from the output of the haptic device. We use measurements of such physical attributes to characterize the instantaneous state of the haptic system.

An example of a series of profiles generated using data gathered from haptic sessions where different users dialed a predefined set of telephone codes using a virtual phone model application (see Figure 2). Dialing expertise was not a requirement in this study, although almost all participants were familiar with mobile phone user interfaces. In this application, users were asked to dial the 7 telephone numbers that appeared on the right side of the screen. This virtual environment provided a mechanism of testing individual skills in dialing and handling concentration in such process. The software application implemented for capturing particular characteristics recorded data at discrete time points. Specifically, data was recorded when users made contact with the key buttons of the virtual telephone using the pen device. Volunteers who took part in this experiment performed the same dialing code ten times. The data files recorded 3D world coordinates, the force and torque applied by the stylus on the virtual keys of the phone, as well as the pen rotation angle. In a preliminary analysis of the data, we analyze the differences in the force exerted by four users when pressing different telephone buttons.

Figure 2. Profile of the force exerted during the dialing code for 4 users from the group.

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

At first glance four user profiles of the force applied to each of the virtual keys is variant from each key to another as it is illustrated by the Figure 2. We assume that such characteristic is due to the position of a particular key in the virtual phone. Note also that there are some points where the force values are very similar in particular telephone virtual keys. This similarity makes the feature extraction process more difficult; however, we note that the overall force applied by for each user is different in each case.

3.3. Haptic-Based Applications The purpose of our set of haptic applications was to provide an integrated and comprehensive framework for the design of modular multimodal biometric systems. We were inspired by the current methods used to recognize and authenticate behavioural patterns, such as authentication via keystroke dynamics, dynamic signature recognition and handwritten signature dynamics [10] [11][12].Using the VRML-node fields’ approach, we implemented a virtual phone, a maze solving process and virtual cheque application. Firstly, we used a virtual phone to capture the keystroke rhythms of different users, a natural choice biometrics based on daily human-device interaction. Similar to the typing pattern described when a user interacts with a keyboard; we argue similar patterns are described when dialing a phone code.

Figure 3. Virtual Phone Haptic Based application where users dial code phones. The user is required to dial the 7 numbers that appear

on the right side of the screen.

Another behavioural pattern can be described via the handwritten signature. The way a person signs his or her name is known to be a characteristic of that individual [15]; although, even handwritten signatures are a behavioural metric known to change over the time. These changes are influenced by conditions such as an individual’s physical and emotional states. Nevertheless, handwritten signatures have been widely accepted as method of verification. Therefore, we implemented a virtual cheque environment with a haptic feedback. The design of the cheque was built on an elastic membrane surface with particular features [14]. In a similar manner as conventional dynamic signature verification technologies, the virtual cheque application records a wide array of the physical parameters describing the way in which a user performs his or her handwritten signature (see Figure 4).

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

Figure 4: The virtual check haptic-based application. The user is required to perform his/her hand written signature by using the haptic

stylus in the area selected

In addition to the virtual phone and cheque application, we developed a virtual maze application. The software’s main purpose was to record the hand movements of different users during different maze solving events. The experiment involved the participant completing a flat 2D maze, which was placed on one surface of a 3D cube. To prevent a user from passing through the walls of the maze, the walls were made ‘sticky’. Generating using the haptic-tool, the maze trajectory was visually represented using a line path (colored in blue) to join all the recorded positions of the pen throughout the maze (see Figure 5). Each person performed the exact same maze 10 times, one trial immediately after the other. Participants were given the opportunity to practice the maze before the trials were actually recorded, as the ability to solve the maze was not being judged.

Figure 5 . Screenshot of a user navigating the maze. The stylus path is indicated with the blue line. The picture besides shows the

colour codes of the Maze Recording Process

3.4. Experimental Set up The subjects wore stereo glasses, enabling them to be able to see through a semi-transparent mirror. The glasses and the mirror became the interface where graphics and haptics were co-located, allowing users to

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

both see and feel objects. Within the virtual environments, graphics depicted a realistic mobile phone, a virtual cheque, and a maze with haptic characteristics. Virtual phone’s key buttons were dynamically deformable and provided a real pressing sensation. The walls of the maze were made sticky to prevent a user from passing through them. The virtual cheque was placed on a 2D rigid surface of a cube that bent as the pen that bent as the pen applied pressure to it.

Before each trial, subjects were instructed to explore the scenario and get used to the haptic device and its feedback sensations. 3.4.1 Virtual Phone The trial started when the subject began dialing the seven telephone numbers. When carrying out the skill mode test, the numbers appeared on the right side of the screen. To signify a call, the subject had to press the “green-phone” key. Then, after a short interval of time, the subject pressed the “red-phone” to terminate the call. Consequently, the user became able to carry on with the next telephone number, until the seventh number was dialed. In the second testing mode, the user dialed a familiar number five times. 3.4.2 Maze This experiment involved the participant completing a flat 2D maze placed on one surface of a 3D cube. The software began to record data when the pen device made contact within a reasonable radius of the starting point of the maze. The end condition of the recording process was met when the pen made contact within a reasonable radius of the exit point of the maze. In order to inform the user when recording was taking place, colour coding was also added to the application. Prior to starting the maze, it was coloured red. While recording a trial through the maze, it was coloured white. When recording completed, it was coloured green (see Fig 5). 3.4.3 Virtual Cheque This experiment starts by asking users to perform their handwritten signatures in a virtual check application. A virtual cheque was built on an elastic membrane surface, and the user was asked to navigate the stylus through it. Pen position, force exerted, and velocity were computed from the haptic-based application, providing the input to obtain a user’s patterns. The software recorded data just when the pen made contact with the cheque, and stopped when no more contact was detected. 4. QUANTIFYING HAPTIC-PSYCHOMOTOR PATTERNS Feature extraction is one of the most important factors in pattern recognition problems, and it was indeed our main concern in the preliminary stage of haptic biometrics. We studied the construction of a pattern vectors to describe the unique way that different haptic users interacted with virtual objects. The choice of the features defined by these pattern vectors significantly affects the performance of user classification. In the general pattern recognition problem, selected features are often refined in an attempt to achieve the desired level of performance. We implemented such refining processes in the feature extraction process, described in section 4.2. The use of haptics for identifying individuals’ patterns borrows approaches that belong to conventional biometric field research. This is due to the nature of the experiments we performed, all of which in one way or another involve an assessment of hand dexterity. Physical parameters quantifying a user’s interactions with haptic devices are considered as biometric features linked to unique biological and physical attributes [1] [12]. It is well-known that signature verification literature elucidated the existence of special parameter values, which are inherent to a particular person. These values can be determined when the signature is repeated several times [12]. This consistency is mainly created by natural motion and practice over time, which creates a recognizable pattern, where practically no mental interference is introduced. Based on this concept, we sought to discover the special physical parameters that distinguish different haptic users. 4.1. Feature Selection

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

The quantitative characterization of a system’s instantaneous state involves measurements of its defining properties. Besides the actual 3D position of the haptic tool, the state of the haptic-user interface is defined by the following properties: velocity (v), force (f), torque (n), and angular orientation of the stylus (θ). Therefore, we construct a state vector r = (vx, vy, vz, fx, fy, fz, nx, ny, nz, θ) to describe the state of the interface at some moment in time. During the data acquisition process of our experiments, we sample the state vectors at a rate of 70 samples/s. Hence, we acquire a sequence of state vectors which describe physical attributes defining users’ interactions with various virtual objects (phone, cheque, and maze). We fit these sequences to multivariate Gaussian distribution functions, defining each users’ state vector probability distribution on the state space. With these distributions, we define the probability of measuring some vector r given that the interface is being controlled by a certain user. Choosing disjoint subsets of the state space, we repeat this process and calculate the relative entropy between inter-person p(r) and intra-person q(r) probability distributions on the state space. Such a technique provides a mechanism to evaluate the user-classificatory worth of different physical parameters. Also referred to as the Kullback-Leibler divergence, the relative entropy between inter-person p(r) and intra-person q(r) probability distributions on a space S is given by:

D (p||q) = ∫ p(r) log (p(r)/q(r)) dr. [27]. S

From an information theory viewpoint, we interpret D (p||q) as a measure of how much information is contained in the assumption of a distribution p on S, when the distribution is actually q on S. Put in the context of our study, D (p||q) assesses the information gained through the false assumption that the state of the haptic system is independent of its user. The choice of logarithmic base used in the integration of p(r) log (p(r)/q(r)) dr corresponds to the units used to measure information [18]. In our study, we choose log2, yielding information measurements in ‘bits.’ Among all possible features, we found that some of them carry more information than others [16]. For example, with respect to distributions on the maze state space, force distributions reveals most information about a user’s identity (15 bits); whereas, 2D torque reveals the least amount information (3 bits). Although relative entropy is not a true metric (due to asymmetry), we interpret its measure as defining the distance between a specific user’s state vector probability distribution, and the general distribution modeled after all users. In this sense, we find force and toque distributions diverge by the greatest amount. Therefore, as we discuss next, features are extracted exclusively from the distribution of these features. Such an approach is taken towards feature selection for the maze application only. Our feature selection process for virtual cheque data is indeed different: we employ Principal Component Analysis. 4.2. Feature Extraction Relative entropy focuses our attention towards the force and torque distributions, leading to the next step in the development of pattern vectors that describe the way in which users interact with virtual objects: Feature Extraction. We consider the force and torque distributions projected onto each 3D world axis defied within the haptic interface; thus, features are derived from six separate distributions (i.e. force in the z-direction.) We use common statistical parameters to define these distributions mean ( µ ) and standard variance ( σ2 ); furthermore, we sum all instances of the distribution in order to produce another parameter ( Σ ). We also consider the length array containing the instances of the distributions, ( t ). With these parameters, we define the properties of each haptic users’ interactions. Consequently, we choose those of the greatest user-classificatory worth as features for use in the authentication task. To evaluate the user classificatory worth of different features, we use the well-known chi-squared test for independence. Such a test evaluates the strength of the relationship between identity and parameter. Using the test, we select parameters with the strongest dependence on their associated creator (user). These parameters are: the sum of the instances recording x-direction force and z-direction torque, the mean values of y-direction force and z-direction torque, and the variance of z-direction torque, denoted by ( Σfx ), ( Σtz ), ( µfy ), ( µtz ), and (σ2

tz) respectively. With these parameters, we form the pattern vectors (r) that describe psychomotor patterns of virtual maze users: r = (Σfx, µfy, Σtz, µtz, σ2

tz).

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

5. METHODOLOGY : CLASSIFIER DESIGN The fundamental assumption of our study is that there exist natural differences between the psychomotor patterns exhibited by individuals in virtual domains. Based on this assumption, we have proposed the use of haptics use in the authentication of human identity. The available haptic interface provides us with a rich description of the observable physical parameters that define human-machine interaction. In a previous endeavor [16], we have found the user-classificatory worth of parameters is variant amongst the set of all observables. Indeed, force and torque measurements contain the most information pertaining to a user’s identity. This conclusion is drawn by calculating the relative entropy between intra-personal and inter-personal time-based physical parameter distribution functions. We interpret relative entropy as a pseudo-metric that defines the distance between an individual’s features and those of the entire population of users. Therefore relative entropy provides a focus for the extraction of features to employ in the user authentication task. In the latter portions of this chapter, we present classification schemes for use in the authentication task, but first we present the results of an experiment which encouraged the pursuit of these schemes. 5.1. Clustering Psychomotor Patterns Standard pattern recognition methodology requires an algorithm to compute the similarity between a given pattern and pre-stored reference patterns. This approach suits our purpose only if there exists a sufficient grade of distinction between the psychomotor patterns of different haptic users. Poor performance of pattern recognition systems must receive careful consideration. Specifically, weakness might not be due to the core classification algorithm. Instead, poor performance could result from strong similarity amongst the set of reference patterns. In this case, subsequent to given modifications of the core algorithm, the resulting performance improvements would be bound by a relatively low upper bound. Put otherwise, the system could never achieve perfection, due to strong similarities in the reference pattern collection. Taking this into consideration, the development of support for our fundamental assumption is necessary. 5.1.1 The K-Means Algorithm Unsupervised machine learning is related with the concept clustering a set of patterns on the basis of similarity[19]. Moreover, unsupervised learning takes the approach of disregarding class in the clustering of patterns. Unsupervised clustering provides us with a way to measure haptic user psychomotor pattern similarity, without labeling each pattern with its associated identity. Such an approach allows us to assess the fundamental assumption of our study without having to also assess the suitability of a certain pattern recognition scheme, for the haptic authentication task. The k-means algorithm is the unsupervised clustering procedure that we utilize. The algorithm clusters a set of patterns into k disjoint subsets. Geometrically, the k-means algorithm can be thought of as the careful relocation of cluster centriods in n-space, whereby n we denote the dimension of the vectors used in pattern representation. More specifically, the k-means algorithm begins by placing k centriods (points) in n-space and assigning each pattern (vector) to the closest of these k points. Next, the locations of the centriods are adjusted so that they coincide with the barycenters of their associated cluster [20]. Following this relocation, patterns are again reassigned to the closest cluster. The process continues until the minimization of J (see Eq.1). In Eq. 1, xn is the nth pattern vector, and µj is the geometric centriod of the vectors in cluster Sj.

Eq.1: J = ||x∑∑= ∈

k

j Sjn1n – µj||2

5.1.2 Results of the K-Means Experiment

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

The development of pattern vectors follows is discussed in section 4.1. The psychomotor patterns of our virtual maze users are represented by vectors r = (Σfx, µfy, Σtz, µtz, σ2

tz). As input to the k-means algorithm, we use patterns generated by forty-five different virtual maze users, where each user contributed three patterns to the total collection. These 135 5d vectors are clustered into k = 45 disjoint subsets using the k-means algorithm. By way of a cluster-class (user) comparison, we find that 86.6% of patterns are correctly clustered. Certainly, these results provide only mixed support for the fundamental assumption of our study, leading us to believe the psychomotor patterns of different haptic users are only somewhat unique. Given this result, we are motivated to pursue the development of more sophisticated classification schemes for implementation in the haptic user authentication task. 5.1.3 Clustering Keystroke Pattern Vectors Using a k-means approach towards user authentication based on the keystroke dynamics of virtual phone dialing, we experiment with a variety of different pattern vectors. We find the subsequent user-classificatory results to be variant with the features chosen to form the pattern vectors, as is indeed expected given our relative entropy analysis of similar data generated from the haptic interface. Contrary to the data generated directly from the virtual maze application, rarely are user-specific patterns found in force data of the virtual phone users. Instead, position data contains the most information pertaining to a user’s identity. In accordance with this principle, we form pattern vectors based on time-series position data. Remarkably, individuals follow significantly unique trajectories as they move from one telephone button to the next; however, these trajectories appear to be only somewhat unique. Using the k-means clustering approach, approximately 80% of users’ patterns were correctly clustered when a cluster-class comparison was performed [28].

Figure 6. The output of the clustering analysis identify two classes of errors False Acceptance Rate (FAR) and False Rejection Rate (FRR) obtained by the performance with 10 of the best features

Empirically, we assume that in this experiment, the way that user’s stylus is handled in terms of position is considered as a good measure for the sensibility of classification of the information. 5.2 Establishment of Human Identity using Haptics In [16], we explain that haptic-based biometrics is perhaps best suited to verification applications, as the identification-mode performance of these systems is unreasonable for any practical applications. At best, using digital signal processing techniques, signals generated from the maze-based haptic interface were used successfully for the identification of humans only 75.5% of the time. Identification can only be accomplished using biometrics. In this section, we pursue the idea of using artificial neural networks for the identification task. Artificial neural networks (ANNs) are well-known for their excellent performance when given complex classification tasks [21]; thus, we surmise that ANNs are capable of haptic-user identification. A neural network consists of simple processing elements (neurons), which can exhibit complex behaviour depending on the connections between them [22]. By the late 1950s, artificial neural networks had begun to gain ground in terms of their usage. In 1958, psychologist Frank Rosenblatt

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

implemented to the first “Perceptron”, whose purpose was to model the human brain’s processing and recognition of visual stimuli [23]. Since then, the use of ANNs has become intimately related with the discipline of machine intelligence. There exists many varieties of ANN structure, among which is the classic three layer feed-forward architecture, often used for pattern recognition tasks. We make use of this architecture, and train a neural network to associate pattern vectors with the user who generated them. 5.2.1 Artificial Neural Network Architecture We construct an artificial neural network with 5 input, 13 hidden, and 22 output neurons for the user identification task. The general network architecture is depicted in Figure 7.

Figure 7: The 3-layer feed-forward architecture chosen for ANN implementation as seen in the Weka GUI [ 26].

The input neurons operate on a numerical values from different dimensions of the 5D pattern vectors. Each ‘neuron’ is a sigmoid function, which receives input p and outputs {q ∈ (0,1)} according to Eq.2. Neural connections are essentially characterized by weights, denoted by w. To explain, consider the passing of output qj, from neuron ‘j’, to neuron ‘k’: then, neuron k operates on input pk according to Eq.3. The weights and biases are adjusted using stochastic backpropagation. This process is referred to as ‘training’, and, as we discuss next, makes adjustments beginning from the output layer and working backwards.

Eq.2: q = )exp(1

1p−+

Eq.3: pk = wkjqj

The training process involves a set of reference pattern vectors, R = { r : r = (Σfx, µfy, Σtz, µtz, σ2

tz, I), I ∈ {1,2,3,….,22} }. The last (sixth) dimension of the pattern vectors is an integer used to store the identity of the user who generated the vector; hence, we train our ANN to identify twenty-two different haptic users. Training begins via the random initialization all weights and biases. Then, until a termination condition is met, we adjust the weights and biases so as to minimize the error between the ANN’s user-classification decision, and the correct decision. More specifically, for each pattern vector, we take the error on output values to be to be δz (see Eq.4), where tk is the correct output, and ok is the actual output. Next, we calculate the corresponding error on the values passed to the output from the hidden layer, δh , according to Eq.5; wherein, oh is the output of hidden neuron h, and wvh is the weight of the connection between ‘h’ and the output neuron whose error is δv. Finally, the weights are updated according to Eq.6, where we introduce new notation: ‘learning rate’ and ‘momentum’, denoted by η and τ respectively. Also, in Eq.6, ∆wji denotes

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

the value of the change made to wji during the previous training cycle (epoch). The learning rate is used to control the magnitude of the changes, and the momentum is a value used to prevent unwanted oscillations of the weight values.

Eq.4: δz = oz(1- oz)(tk – ok)

Eq.5: δh = oh(1- oh) w∑=

22

1vvh δv

Eq.6: wji wji + ηδjpj + τ(∆wji)

5.2.2 ANN Performance in the Identification Task In [16], signal processing techniques were combined with pattern recognition methodology to successfully identify haptic users 75.5% of the time. Using the architecture described in 5.2.1, after learning to associate 5D psychomotor pattern vectors with usernames, our artificial neural network correctly identified 90.9% of users. The network was trained for 500 epochs, and we set η = 0.4 and τ = 0.2. The values of these parameters were determined in an ad hoc manner, and we leave a statistical procedure seeking ‘optimum’ parameter selection for future research. As a figure of merit, the false accept rate of our ANN, when operating in verification-mode, is 0.4%; a remarkable improvement on previous haptic-based biometric false accept rates, to date, on the order of 10%. The low FAR of the ANN is due to the fact that only one output neuron is excited when the input layer receives a pattern. Unlike other systems, a pattern is never decided to belong to more than one user. On the other hand, a pattern is always associated with at least one user; hence, the lowest possible bound on the probability of a successful attack by a non-registered user is 4.5%. This probability can be reduced by increasing the size of the output layer, and thus the number of enrolled users. 5.3 Signature Verification Using Haptics With the rapid advancement of internet commence, reliable on-line signature verification techniques are in high demand. In many current implementations, the digital characterization of one’s signature is captured using physical signature pads. The use of Haptics in signature verification allows a user to receive force-feedback signals, adding a sense of texture to the signing process that cannot be achieved using traditional signing pads. In a previous study [16], we tested the possibility of using Haptics for practical signature verification applications. Using dynamic programming techniques, we implemented an algorithm which correctly verified 61/64 signatures collected from a virtual environment. One significant drawback of our approach in [16] is the size (memory) needed to store our digital characterization of virtual signatures. Another problem is the high computational cost associated with the dynamic alignment of these large datasets- which makes real-time verification a difficulty. In this section, we adopt a different approach to the verification task. Our goal is to dramatically reduce the dimensionality of quality digital signature characterizations by embedding their most informative aspects into a small ‘feature’ vector. In the past, such an approach has proven to be an elusive task [25]. Using principle component analysis, quantization, and dynamic time warping, we evaluate the information loss associated with dimensional compression and subsequent verification of virtual signatures. Fundamentally a change of basis transformation, principal component analysis (PCA) detects the structure of multivariate relationships [24]. We hypothesize that this structure is a of function of the user whose interactions are quantitatively characterized by A. To test this idea, we transform physical parameter matrices describing different haptic users, and we compare their reduced forms. We use dynamic time warping, (DTW), to make these quantitative comparisons. The DTW algorithm establishes a non-linear

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

mapping between two streams of time-series data. This mapping is formed so as to minimize the difference between corresponding instances of the input data. The method is especially useful in our study, allowing for a robust similarity measure between time-series measurements of variant length. For us, the DTW algorithm output is interpreted as a similarity measure that describes the difference between the structure of two users’ multivariate (physical parameter) relationships. 5.3.1 Proposed Approach The available haptic interface allows us to capture time-series measurements of many physical parameters, as discussed in chapter 3. By A we denote the matrix whose columns each correspond to a unique physical parameter. Each row of A corresponds to an instantaneous ‘snapshot’ of the state of the haptic system. The dimensionality of A is quite large, making the interpretation (and transmission)of its information content a cumbersome task. We use Principle Component Analysis (PCA) to reduce the dimensionality of A. PCA allows us to retain most of the information content of A. Therefore, this technique is used to compress the feature space such that the important information is not lost. To describe the PCA algorithm, we let the size of A be denoted m x n. The first step of the algorithm is to calculate the mean values of each column of A, which we denote by µj. Next, i 1,…m, & ∀ ∈ ∀ j

1,…n, A∈ ij Aij - µj; hence, µj 0 ∀ j ∈1,…n. With the empirical mean of each column now zero, the next step is to find the covariance matrix of A, which is denoted cov. From cov, we calculate its eigenvalues {λ1,…, λn} and corresponding eigenvectors. The eigenvectors are placed in the rows of a matrix denoted by F. This is done in such a way that the first row of F contains the eigenvector that corresponds to max{ λ1,…, λn }. We call this first row of F the ‘1st principal component’ of A. The second last step of this algorithm is where we must choose the number of principal components to use in the compression of A. In our analysis, we choose to use only the 1st principle component. This choice is made so that we can access the worst-case cost of employing PCA; put otherwise, we analyze how much information will be lost if we maximally compress the dimensionality of A. Choosing to use only the 1st principal component, A FAT. Therefore, we use PCA to reduce the size of A to 1 x m. We quantize the values of FAT in such a manner that FAT Q, whereby Q {0, 0.1, 0.2, …, 1}. From Q, we obtain a sequential representation. This sequential representation is of the form s = (q1, … , qn); whereby qj stores the jth discrete value appearing in Q. For example, in the case of a flat Q, s would contain only one element. It is these s vectors which we take as the final compressed version of physical parameter matrices characterizing virtual signatures. The s vectors are compared using a dynamic time warping, producing a similarity score: MS, where a low MS indicates strong similarity. 5.3.2 Experimental Results As we choose to project physical parameter matrices onto only their first principal component axis, information loss is inevitable in the general case. Moreover, as these projections undergo quantization and discrete sequence extraction, we expect the information content to degrade further. Such transformations are depicted in Figure 8, where we see the virtual ‘size’ of the signature representation is reduced to 4.1% of its initial value. This data compression suits the goal of our present experiment: to reduce both the dimensionality and size of a virtual signature. In order to test the magnitude of information loss, we used the DTW technique to generate a total of 5120 match scores. These scores are drawn between 144 signatures created by 16 different haptic users in the virtual domain. The genuine and imposter distributions of these scores are shown in Figure 9. If we impose the condition that a signature must match with at least two ‘reference’ signatures, the probability of successful virtual signature verification is 92% @ 25% FAR. Clearly, this result shows that our proposed approach removes a significant portion of useful information from the large datasets which fully characterize the virtual signatures; yet, due to the extreme compression we impose on signature features, a PV of 92% seems somewhat surprising. Therefore, we believe that there

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

exists an optimal balance between the magnitude of feature compression and corresponding reductions of the computational cost of dynamic alignments.

Figure 8:

Top Left: Original 2D signature (4.87 kB); Top Right: 1D proBottom Left: Quantized Version of the PC-Projection (0.238 KB); Bo

Figure 9:

Left: Distribution of Imposter Match Scores; Right: D

0.9, 0.8, 0.7, 0.6, 0.5, 0.9, 0.8, 0.7, 0.6, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.6, 0.5, 0.3, 0.2, 0.1, 0.2, 0.3, 0.4, 0.3, 0.2, 0.1,

0

jection on the 1st principal component axis; ttom Right: Final Sequential Representation(0.2 kB)

istribution of Genuine Match Scores

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

6. CONCLUSIONS and FUTURE WORK

A series of novel experiments have allowed us to explore the use of haptics for the authentication of human identity. Given raw data generated directly from the haptic interface, we have explored the idea of haptic-biometrics. Using relative entropy as a pseudo-metric and the chi-squared test for independence, we extracted unique-to-the-individual parameters describing patterns exhibited by users in the virtual domain. Forming quantified vector characterizations of these patterns, we used unsupervised machine learning to partition the vectors based on their 5D geometrical closeness. Clustering the vectors without any knowledge of the user who created them, the k-means algorithm correctly portioned 86.6% of 135 vectors generated by 45 different users’ patterns. This result motivated us to develop an artificial neural network to identify haptic users. With this network, the probability of successful identification advanced to 90.9%; furthermore, the probability of falsely verifying a registered user claiming to be someone else dropped to 0.4%. In addition to the development of an artificial neural network, we took a innovative approach towards signature verification. Using the haptic interface to generate physical parameter matrices, a combination of PCA and DTW methodologies allowed for enormous reductions of the size of digital signature. Specifically, the memory needed to store the signatures was reduced to 4.1% of its original size, thus dramatically increasing the speed of dynamic time alignments. Despite the dimensionality compression imposed on the physical parameter matrices, our “Quantized PCADTW” algorithm correctly verified 92% of the signatures from our database.

We have presented several classification schemes for implementation within such systems, and their performance given data of ‘first-time’ haptic users is remarkable. In fact the above-mentioned results are based on the behaviour of users who had never interacted with our specific set of haptic-based applications before; moreover, the psychomotor patterns of these users were captured over a relatively short period of time (on the order of minutes). Nevertheless, these users exhibited remarkably consistent distinguishing characteristics. We know that the variation of a user’s patterns diminishes throughout the hapto-task session [16]. Therefore, we believe that our results place a lower bound on the possible performance of haptic-biometrics. In the future, there will most probably exist environments where users interact with haptics on a daily basis. Within such environments, the uniqueness and consistency of user patterns would be significantly greater than ‘first-time’ haptic users. In these environments, continuous verification using haptic-biometrics would be commonplace. The advantages of haptic-biometrics over traditional methods are vast, allowing for the continuous automatic authentication of human identity through the analysis of psychomotor patterns exhibited in virtual domains Our results indicate haptic interfaces provide data rich in user-classificatory worth; hence, we conclude haptic-biometrics will become commonplace within high-security virtual environments in the near feature. In addition our haptic-biometrics system should be able to manage an intelligent feedback between the feature selection and profile construction processes to improve the performance. In our system, we did not impose constraints on reference pattern set similarity, nor did we filter the noise inevitably associated with measurements generated from the haptic-interface. Additionally, future research should probe the system’s vulnerability to attack, assessing weaknesses and implementing corrective modifications such as parallel processing units. Also, feature work must seek a statistical optimization of the parameters used to train these units. References

International Journal of Multimedia Tools and Applications - Springer Science+Business Media B.V., (To Appear)

[1] M.Orozco, Y.Asfaw, A.Adler, S.Shimohammandi and A. El Saddik. “Automatic Identification of Participants in Haptic Systems” in Proceedings of IEEE IMTC 2005- Instrumentation and Measurements Technology Conference Ottawa, Ontario, Canada, 17-19 May 2005 [2] J.L. Wayman, “Fundamentals of Biometric Authentication Technologies”, Proc. Card Tech/Secure Tech, San Jose State University, 2000 [3] FreeForm® Concept™ from SensAble technologies. 3D Design and Development http://www.sensable.com/products/3ddesign/concept/index.asp [4] Ziegler, R., Müller, W., Fischer, G. and Göbel, M., "A Virtual Reality Medical Training System". Int. Conf. on Comp. Vision, Virtual Reality and Robotics in Medicine, CVRMed'95, Nice, Lecture Notes in Comp. Sci., 905. pp.282-286, Springer-Verlag, 1995. http://www.vision.ee.ethz.ch/projects/Lasso/start.html[5] Immersion, I., "Laparoscopic impulse engine". Company brochure, Santa Clara, CA, 1995. [6] Antonio Zorcolo, Enrico Gobbetti, Gianluigi Zanetti, and Massimiliano Tuveri. Catheter insertion simulation with co-registered direct volume rendering and haptic feedback. In J. D. Westwood, H. M. Hoffman, G. T. Mogel, R. A. Robb, and D. Stredney, editors, Medicine Meets Virtual Reality 2000 - Envisioning Healing: Interactive Technology and the Patient-Practioner Dialogue. Pages 96-98, IOS, Amsterdam, The Netherlands, January 2000. [7] Bromme, A.; A classification of biometric signatures Multimedia and Expo, 2003. ICME '03. Proceedings. 2003 International Conference on Volume 3, 6-9 July 2003 Page(s):III - 17-20 vol.3 [8] Hale, K. S. and Stanney, K. M. 2004. Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. 24, 2, 33. 0272-1716. [9] Kahol, K., Tripathi, T.,MacDaniel, L.,Braton, P., Panchanathan, ACM Transactions on Computational Logic, Vol. V, No. N, December 2005, Pages 1–25. [10] Joyce,R., and Gupta, G. Identity Authentication Based on Keystroke Latencies. Communication of ACM,Vol 33 Issue 2, February 1990. [11] Monrose, F. and Rubin, A. Authentication via Keystroke Dynamics. In Proceedings of the 4th ACM conference on Computer and communication security. Zurich, Switzerland, 1997, 48-56. [12] T. Qu, A, El Saddik, A. Adler, "Dynamic Signature Verification System Using Stroke Based Features", IEEE Int. Workshop on Haptic Virtual Environments and their Applications, Ottawa, Canada, Sept. 2003. pp. 83-88 [13] R. Plamondon and S.N. Srihari, "On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 63-84, 2000. [14] Reachin Technologies http://www.reachin.se/[15] Jain, A.K.; Ross, A.; Prabhakar, S. IEEE Transactions on Circuits and Systems for Video Technology. Vol 14, No.1 January 2004 Page(s): 4- 20 [16] Abdulmotaleb El Saddik, Mauricio Orozco, Yednek Asfaw, Shervin Shirmohammadi and Andy Adler, “A Novel Biometric System for Identification and Verification of Haptic Users”, _IEEE Transactions on Instrumentation and Measurement_ (accepted, to appear) [17] A. Adler, R. Youmaran, S. Loyka, “Information Content of Biometric Features”, Ottawa, Canada, 2005 [18] C. E. Shannon, “A Mathematical Theory of Communication”, 194 [19] http://www.ucl.ac.uk/oncology/MicroCore/HTML_resource/unsup_menu.htm[20] http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/kmeans.html[21] D. Siganos "Why Neural Networks”, SURPRISE Journal, Imperial College of Science Technology and Medicine, London, UK, 1996. [22] Wikipeddia, http://en.wikipedia.org/wiki/Artificial_neural_network[23]http://www.computerworld.com/softwaretopics/software/appdev/story/0,10801,57545,00.html [24] ‘Principal Components and Factor Analysis’, http://www.statsoft.com/textbook/stfacan.html[25] Y. Gdalyahu, D. Weinshall, “Flexible Syntactic Matching of Curves and Its Application to Automatic Hierarchical Classification of Silhouettes”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21, No. 12, December 1999. [26] Ian H. Witten and Eibe Frank (2005) "Data Mining: Practical machine learning tools and techniques", 2nd Edition, Morgan Kaufmann, San Francisco, 2005 [27] M. Lexa, “Useful Facts about the Kullback-Leibler discrimination distance”, Houston, Texas, 2004. [28] M., Orozco, I., Shakra, A., El Saddik Haptic: the new biometrics-embedded media to recognizing and quantifying human patterns. November 2005 Proceedings of the 13th annual ACM international conference on Multimedia MULTIMEDIA '05