Procedia Technology 5 ( 2012 ) 895 – 902

2212-0173 © 2012 Published by Elsevier Ltd. Selection and/or peer review under responsibility of CENTERIS/SCIKA - Association for Promotion and Dissemination of Scientific Knowledgedoi: 10.1016/j.protcy.2012.09.099

CENTERIS 2012 - Conference on ENTERprise Information Systems / HCIST 2012 - International Conference on Health and Social Care Information Systems and Technologies

A gaze-based interaction system for people with cerebral palsy Adriano Galante, Paulo Menezes*

Institute of Systems and Robotics, Department of Electrical and Computer Engineering, University of Coimbra – Pólo II, 3030-790 Coimbra, Portugal

Abstract

In this paper we present the development of an augmentative system for people with movement disabilities (mostly cerebral palsy people) to communicate with the people that surround them, through a human-computer interaction mechanism. We developed an assistive technology application based on gaze tracking in order to select symbols in communication boards, which represent words or ideas, so that they could easily create phrases for the patient's daily needs. This kind of communication boards is already used by cerebral palsy people, here we intend to extend their use for people with absolute no motor coordination. An important improvement of the proposed system with respect to the existing solutions is the ability to work in the presence of voluntary or involuntary head movements.

© 2012 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of CENTERIS/HCIST.

Keywords: Cerebral palsy; physical disabilities; eye tracker; gaze tracking; communication; human-computer interaction; board communication

* Paulo Menezes. Tel.: +351 239 796291; fax: +351 239 796247. E-mail address: pm@deec.uc.pt. Adriano Galante. E-mail address: agalante@isr.uc.pt.

Available online at www.sciencedirect.com

© 2012 Published by Elsevier Ltd. Selection and/or peer review under responsibility of CENTERIS/SCIKA - Association for Promotion and Dissemination of Scientific Knowledge

896 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

1. Introduction

Cerebral palsy is a disease that affects many people in all over the world. Patients affected by cerebral palsy (CP) do not have the ability to communicate with the ones that surround them in the same way as healthy people do. Only in Europe, there are about 1,140,000 people who suffer of cerebral palsy [1]. For a person suffering from cerebral palsy, it is extremely complicated to communicate with the others because the disease describes a group of disorders of the development of movement and posture causing activity limitation. The motor disorders of cerebral palsy are often accompanied by disturbances in senses, cognition, communication, perception, behaviour, and/or seizure. These disorders give raise to different handicap levels affecting mobility, senses, language, reasoning, attention, and so forth [2]. When CP results in strong mobility impairment the solution is to use electric wheelchairs controlled by adapted joysticks. Speech capabilities are also frequently affected, even in people with full cognitive faculties. For the latter case a successful method already used by several people are the communication boards. These boards consist in a grid with symbols representing verbs, actions, objects, places, etc. [9], and are customized for each user, depending on his/her cognitive and/or pointing capabilities (see figure 1 for an example). These boards are normally printed in paper, and their use is done by pointing the symbols in sequence by using a finger, or a hand or head attached pointer. Although requiring a special training in the use of these communication means, users can reach a good communication level with the help of special education teachers, technicians and speech therapists.

Fig. 1. Example of a paper printed communication board used by cerebral palsy people.

There are indeed people suffering from CP do not have any control on the movements of their limbs or head. These are totally dependent on caregivers, and cannot use communication boards or control wheelchairs. In addition they do not possess any capability to select their favourite channels on a TV set, or to choose a given preset radio station on a tuner. We have observed that at least some of these people can fixate their gaze onto objects, even during involuntary movements. This motivated us to develop a gaze-based communication board system. If successful it would enable these people not only to communicate with other people, but also interact with devices like Hi-Fi systems or TV sets. In addition this would enable the user to give high-level commands to wheelchairs, as long as it can interpret them and navigate autonomously.

897 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

1.1. Related works

The eye gaze estimation and analysis has received a considerable attention in the recent years, motivated by the benefits for empowering motor disabled people, but also for its interest for the human-computer interaction area. This has given raise to several research projects that aim at improving the quality of life of people suffering from impairments related with motor-control disorders, such as amyotrophic lateral sclerosis (ALS) or cerebral palsy (CP). Among these projects we may cite the COGAIN project [1], which is a good example of the state of the research in this particular area. In this project, some known communication methods for cerebral palsy patients were brought to computer-controlled systems in order to develop some assistive technology devices. The applications were developed based on the gaze tracking, eyebrow movements and some word and letter keyboards to enable communication.

On the company side, SensoMotoric Instruments (SMI) and Tobii are the main manufacturers of eye-tracking devices. While SMI focus their production on eye tracker analysis systems, Tobii also produces some augmentative communication devices especially suitable to individuals with cerebral palsy. With these devices, disabled people can easily engage in daily communications via text or symbols, with word and phrase prediction to enhance rate of communication [6]. Another project, called “MagicEye”, focused on the same premise which is to help disabled people to communicate and interact with a computer. The basic principle of this eye tracker system is the same of the most known commercialized remote eye trackers and the developed algorithm enables the detection of the pupil and then the gaze coordinates are mapped in the screen coordinates. This enables the user to control de mouse cursor to interact with the computer [12]. Prentke Romich Company [7] is another company whose devices use semantic compaction, a language method that sequences a small number of multi-meaning icons to form words, phrases or sentences. In general, these devices range from those that are designed for emerging communicators to word-based complex communication systems [8].

Although these technologies have great utility for these groups of people, most of the existing products have prohibitive prices for most of the potential users. Another limitation of these projects and products is that they do not comply with head movement. In some forms of CP, involuntary movements also affect head pose constantly, but the person can still fixate voluntarily his/her gaze on any object. This motivated the development of a gaze-based communication system that is compliant with involuntary or voluntary user head movements. The related studies and implementation will be detailed in the next sections

2. Problem Formulation

The problem we want to address is that of having a communication board as the one presented in figure 1 in front of a user, and determine to which symbol the user is looking at an each instant in time.

The use of an eye tracker comes naturally as part of the solution for this problem. In the current application we have chosen the SMI iView X HED. This head mounted device, as shown on figure 2, consists on a camera that observed the user's eye through an infrared (IR) mirror (4), and a second camera that points forward (1) [4]. This IR-based dark-pupil eye tracker, maps the pupil position to the image coming from the forward pointing camera.

From this we can understand that the eye tracker relates the gaze direction with the viewed scene image, but not with the observed objects that make part of that scene. For the case of current application it is required to know which symbol on the communication board the user is looking at. Therefore, independently of the relative position and orientation of the user head and the communication board, we need to establish a relation between the gaze direction and the board itself.

898 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

Fig. 2. Eye tracker scheme: 1) scene camera 2) infrared camera with infrared emitter 3) image plane 4) infrared mirror.

Another problem to address is how to enable the user to select each of the symbols. Considering that the first problem is already solved, we can use one of two possibilities: 1) detect gaze fixation, and 2) use an adapted switch.

3. Proposed Solution

As, stated above, the SMI eye tracker outputs are the coordinates of the point the user is looking at in the image coming from the forward camera. To solve the problem of identifying the symbol that a user is looking at we need to map the coordinates given by the eye tracker onto the communication board coordinates. Being the board a planar object we can relate coordinates of its referential and image coordinates by a homography [10]. As long as we know the homography parameters, we can transform uniquely any image point into the corresponding board point, and vice versa. This relationship is expressed by

where and are the homogeneous coordinates of, respectively, an image point and the corresponding board point. is a matrix representing the homography transformation.

There is in fact one important detail that we need to address, is that this homograph depends on the relative position of the board w.r.t. the camera. This means that we need to compute this homography every time the user moves his/her head. To estimate the homography matrix we need to establish a relationship between known board points and corresponding image points. As this matrix is defined up to a scale, from its 9 coefficients we can fixate one of them resulting in 8 unknowns. Thus we can write the necessary 8 equations using the known relationships between 4 board points and their images [10]. If more than 4 points are used, we can compute a least squares solution that minimizes the effects of noise introduced in imaging and image processing.

Being this estimation a fast process, we can compute the homography for each image received from the camera. This means that even if the user moves his/her head, the correct mapping between eye-tracker coordinates and communication board is always computed (see figure 3). For this we need to be able to identify the known set of points in each image of the board. We have introduced a simple marker for these propose that we describe next.

899 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

Fig. 3. Schematic block diagram of the application.

3.1. Binary position markers

For the sake of simple and fast detection of these points a type of markers was designed that not only enable fast and robust identification of these points but also contain a binary code that can be used to distinguish a particular marker from a set. Figure 4 represents one of these markers. They are composed of a square black frame on a white background, a binary identification code, formed by black circles, and an orientation point. This marker is robustly detected together with its eight frame corners, and their order is defined using the orientation dot [5].

Fig. 4. Binary position marker.

4. Implementation

Speech therapists, which work with children with CP, have informed us that people that use this kind of communication walk through a learning process that starts with simple card communication boards, going progressively toward more complex ones. Also the use of this gaze-based communication requires an adaptation period so the CP patient could understand its functioning. Given this, we have designed a system that may be customized for each particular user. This customization can be made on the number of symbols due to cognitive limitations, or on the symbol size due to physical limitations.

A gaze based communication system could be built using the printed communication boards, but

900 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

interaction design principle made us reject this option. Interaction design experts state that feedback is very important in any interactive system. The printed board does not give any kind of feedback by itself, thus the user does not know if the intended symbol is already selected or not, after looking at it for a while. This will deteriorate largely the user’s performance, as it will require a greater concentration. This feedback can be produced using audio, but this may interfere with the conversation with other people.

Given this, the proposed solution consists in displaying the communication board on a computer screen. This solution enables us to use visual feedback by high lightening the symbol towards which the user is looking at. The symbol selection can be made in two ways: 1) detecting when the user looks towards a symbol more than a given period of time, or 2) using some form of adapted push-button. We have opted for the latter options as it will enable a faster use of the system (depending on the user capabilities), and will induce a lower tiredness level than the former. Selecting sequences of word symbols, the user can build simple phrases.

5. Experiments and Results

The main purpose of the experiment was to validate the usefulness of the developed system and observe the interaction between the user and the device in actual use conditions. The application was subjected to usability tests by the researchers of the Augmented and Interactive Realities Laboratory [11] and then with some other voluntary people, in a total of 9 different experiences.

5.1. Technical setup

In this phase, we used a computer with a 21'' flat screen and the interaction is done about 80 cm from the screen, being the symbol size 4x4 cm. As shown in figure 5, all the interaction area is composed with 27 word symbols and 3 action buttons (forward, backward and reload). The reload button gives the user the opportunity to generate a new phrase. The backward and forward buttons allows the user to navigate to another set of word symbols. For the sake of usability, the generated phrase would appear in the bottom of the interaction area.

Fig. 5. Graphical user interface: 27 word symbols and 3 action symbols (forward, backward and reload). The generated phrase appears in the bottom.

901 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

The material used to test the application consists in a desktop computer, a SMI iView X HED eye tracker equipment (helmet with cameras and software computer) and an adapted switch (to enable the selection of icon words). In figure 6 we can see the full testing setup.

Fig. 6. Application setup: 1) Eye tracker 2) Adapted switch 3) Application graphical interface 4) Eye tracker software computer.

5.2. Testing procedure

A qualitative approach was the right methodology chosen for the evaluation of the experience and this took into account the accessibility and utility advantages given by this developed application of assistive communication. The first goal was to familiarize the user with the entire working environment (mostly the eye tracker helmet to see if it is uncomfortable or not), then adjust all the equipment to proceed to the next step.

The next step was to calibrate the eye tracker for the present user. This calibration uses the default calibration system from the eye tracker and this step must be done with help from another person that validates the calibration results. After getting a good calibration, we could proceed to the real utilization of the assistive communication application and evaluate the results of a high level interaction based on a common computer desktop. These sessions were useful to evaluate the usability and functionality of the developed application and to see if has good results as substitute of the already known board communication methods.

5.3. Results

According to the observation of the experimental procedure, if the calibration process results were not satisfactory, some errors of gaze position could arise that affect the functionality of the board communication system. Because of this bad calibration is difficult for the user to select the correct symbol and build the wanted phrase. In spite of this, all the marker orientation process works perfectly without the effect of involuntary or voluntary head movements in the usability of the application. The interaction with the home made switch was perfectly normal and the selection of the desired word icons occurs without any problems. In

902 Adriano Galante and Paulo Menezes / Procedia Technology 5 ( 2012 ) 895 – 902

sum, all the voluntary people used perfectly the application and wrote simple communication phrases with the board communication setup, such as “eu quero falar computador” (“I want talk computer”), “eu quero beber sumo” (“I want drink juice”), “eu quero brincar brinquedos” (“I want play toys”), etc.

6. Conclusion and Future Work

The possibility of using assistive technology mechanisms as long as we move our head is the added value of this research. This is the main difference between our project and other projects already developed. By this way, the application will be much more comfortable, less intrusive, allows freedom of movement and it is much easier to use. Despite the fact that our work has just started, preliminary results shows that, in general, eye tracker based computer interaction systems are very useful, giving the user some advantages over other traditional assistive technologies, such as flexibility and low cost. This research is still in course and other improvements will be added hereafter, like more symbolic items, syntactic and semantic brokers who may be used in the project to generate a more accurate output, sound phrases generation in order to make communication even more real. Although still lacking some of these features, the developed application can be considered an asset in the communication of cerebral palsy people.

The application was mainly developed for cerebral palsy people but may be easily exportable to other impaired people suffering similar disorders (such as spinal cord injured quadriplegic people), but in this case the system will be used to simple computer interaction and not for communication. The impossibility of retaining the binary position marker in every active window force us to place the marker outside the screen and then the user has the possibility to control the computer mouse cursor with his own eyes.

Future work includes the integration of this system on a mobile system suitable to be adapted to a wheelchair. Tests with people with CP are planned to analyse the usability of this system with real potential users.

Acknowledgements

The authors thank the Coimbra Delegation of the Portuguese Association of Cerebral Palsy (APCC) [3], especially his director Dr. Carlos Barata, and the speech therapists for the cooperation and valuable advices.

References

[1] Bates R, Donegan M, Istance HO, Hansen JP, Räihä K-J. Introducing COGAIN: communication by gaze interaction. Springer-Verlag 2007.

[2] Mauri C, Granollers T, Lorés J, García M. Computer vision interaction for people with severe movement restrictions. Human Technology – An Interdisciplinary Journal on Humans in ICT Environments 2006.

[3] Associação de Paralisia Cerebral de Coimbra, http://www.apc-coimbra.org.pt/; May 2012. [4] iView XTM System Manual, Version 2.4; December 2009. [5] Patrão B. Biblioteca para desenvolvimento de aplicações de Realidade Aumentada em marcadores binários. DEEC – FCT,

Universidade de Coimbra; 2011. [6] Tobbi ATI – Tobii C15 Augmentative and Alternative Communication Device, http://www.tobii.com/assistive-technology/north-

america/products/hardware/tobii-C15/; May 2012. [7] Prentke Romich Company, http://www.prentrom.com/; May 2012. [8] Cumley J, Maro, J, Stanek M. Assessing Students’ Needs for Assistive Technology (chapter 3). Wisconsin Assistive Technology

Initiative; 2009. [9] Cerebral Palsy Source, http://www.cerebralpalsysource.com/, May 2012. [10] Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. Cambridge University Press; 2004. ISBN: 0521540518. [11] Augmented and Interactive Realities Laboratory, http://orion.isr.uc.pt/, May 2012. [12] Projecto MagicEye, http://www.magickey.ipg.pt/magiceye.aspx/, May 2012.