A case study of using eye tracking techniques to evaluate the usability of e-learning courses

Int. J. Learning Technology, Vol. X, No. Y, xxxx 1

Copyright © 200x Inderscience Enterprises Ltd.

A case study of using eye tracking techniques to evaluate the usability of e-learning courses

Mingzhuo Liu* Distance Education College, East China Normal University, Shanghai 200062, China E-mail: [email protected] *Corresponding author

Zhiting Zhu Shanghai Engineering Research Center of Digital Education Equipment, East China Normal University, Shanghai 200062, China E-mail: [email protected]

Abstract: Usability is an important quality indicator for interactive IT products, such as e-learning courses, and can directly impact learning effectiveness, efficiency and satisfaction. In this study, we use eye tracking method together with traditional interviewing and observing methods to evaluate and examine the functionality and usability of the homepage of an e-learning course. Through the use of eye tracking in the usability experiment, we learned finer points and developed confidence in our understanding about what drew attention and why on the course homepage, and by studying what people looked at and did not look at, we were able to glean more detailed insights about how the web user worked his or her way through usable and unusable designs. The integration usage of eye tracking with interviewing method produced strong corroborative results. It provided us with a richer interpretation than either technique alone. The findings from this report contribute to a broader process of iterative design and continuous development of e-learning course or e-learning applications functionality to be effective, efficient and satisfying for students.

Keywords: e-learning; usability; eye tracking; e-course.

Reference to this paper should be made as follows: Liu, M. and Zhu, Z. (xxxx) ‘A case study of using eye tracking techniques to evaluate the usability of e-learning courses’, Int. J. Learning Technology, Vol. X, No. Y, pp.000–000.

Biographical notes: Mingzhuo Liu is an Associate Professor and the Director of Online Learning Course Division, Distance Education College, East China Normal University. Her research directions are online learning design, development and evaluation, online learning course standards and usability engineering in online learning.

Zhiting Zhu is a Professor and the Acting Director of Shanghai Engineering Research Center of Digital Education Equipment, East China Normal University. His research interests are theories of e-education, e-learning technology standards, teacher training and technology culture.

2 M. Liu and Z. Zhu

1 Background

Jay Cross is recognised as one of the first to use the term e-learning on the web in 1999 (Internet Time Wiki, 2009). Since that time the term e-learning has taken on different meanings in different contexts (Nicholson, 2007). Rosenberg (2001) defined e-learning as the use of internet technologies to deliver a broad array of solutions that enhance knowledge and performance. Masie (2008) similarly defined e-learning as the use of network technology to design, deliver, select, administer, and extend learning. Govindasamy (2002) stated that e-learning included instruction delivered via all electronic media including the internet, intranets, extranets, satellite broadcasts, audio/video tape, interactive TV, and CD-ROM. According to Hung (2012), definitions of e-learning can be classified into two categories, the first focusing on delivery over a network, such as Rosenberg’s and Masie’s; the second category includes all electronic media, such as Govindasamy’s. According to American Society for Training & Development (ASTD, 2012), with a more general definition, e-learning covers a wide set of applications and processes, such as web-based learning, computer-based learning, virtual classrooms, and digital collaboration. It includes the delivery of content via internet, intranet/extranet (LAN/WAN), audio- and videotape, satellite broadcast, interactive TV, CD-ROM, and more. In this paper, e-learning refers to web-based learning, which means that educational content is delivered via a web browser over the public internet, a private intranet, or an extranet. Today web-based e-learning plays an important role in education and this role will be increasingly more significant across all education segments, especially given the trend toward lifelong learning (Alexander, 2001; Bates, 2005; Gulati, 2008; Wang et al., 2009).

E-learning involves taking a course via a computer through the internet. E-learning course, known as online course (abbreviated as e-course in the following) occupy an important status in e-learning. The definition of online courses varies in terms of how and how much of the course content is presented online, the nature of the interaction among class members, and the overall learning experience. According to Allen and Seaman in SLOAN’s 2003 report, online courses are those in which at least 80 percent of the course content is delivered online. This definition has been taken as the rubric by some institutions to select or evaluate education programmes, such as US News (2012) and World Report for its 2012 rankings. However, Shimabukuro (2012) argued that SLOAN-C’s nearly decade old ‘at least 80%’ definition may be out of sync with reality. He mentioned that in fact, some of the top universities in USA., such as Harvard, Yale and UC Berkeley just use simple dichotomy to differentiate between online and blended. A course is either fully online or it is not. If it is, then it’s online. If it is not, then it is blended. Bourne (2010) defined online course as all course activity is done online, and there are no required face-to-face sessions within the course, and totally eliminate geography as a factor in the relationship between the student and the institution.

Following this line of thought, the webpage interface of e-courses is so important that it can influence learners’ learning effectiveness, efficiency and satisfaction, as the learners will not want to read web pages that are not user-friendly. In addition, learners will not want to navigate through difficult pages in order to explore or understand them, and they will not interact with pages that respond differently from their expectations (Goto and Cotler, 2005). To develop user-friendly e-course, it is necessary to perform usability evaluation during different phases throughout e-course’s design and development process. Usability testing was developed in part to address those problems

A case study of using eye tracking techniques 3

of feature shock and unusable functionality (Dumas and Redish, 1993). Different approaches and a set of methods have been proposed to evaluate interactive systems, such as task analysis, card sorting, thinking aloud, and so on. The objective of this complex methodological apparatus was to propose effective solutions to measure the quality of interactive systems, identify problems, and suggest remedial actions (Lanzilotti and Ardito, 2010). Each one of the methods has its own benefits and drawbacks, and as such, they are generally applied in tandem in different phases of an interface design (Desurvire et al., 1992; Jeffries et al., 1991; Tan et al., 2009). This report examines the potential of integrating eye-tracking techniques with traditional interviewing methodology. The study examines the functionality and usability of the homepage of an e-course. It aims to obtain insights which could be used to improve usability for the e-learning course. The results contribute to a broader process of iterative design and continuous development of e-course functionality to be effective, efficient and satisfying for students. The study also examines the additional contribution of using eye tracking to traditional usability methods.

2 Theoretical considerations

Usability is an important indicator for the quality of interactive IT products. According to ISO 9241-11 (ISO/IEC, 1998), usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. In addition, the effectiveness, efficiency, satisfaction and context of use in the description are defined as follows.

• effectiveness: accuracy and completeness with which users achieve specified goals

• efficiency: resources expended in relation to the accuracy and completeness with which users achieve goals

• satisfaction: freedom from discomfort, and positive attitudes towards the use of the product

• context of use: users, goals, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used.

People are coming to understand that usability is a key attribute of any product used by people (Wiklund, 1994). When a product development team designs a product, it tries to include the features people need to accomplish tasks, and present those features in a manner that people intuitively grasp and find efficient to use in the long term. Developers also attempt to eliminate the potential for critical, design-induced mistakes and to include design qualities that make people feel good about using the product. Almost every aspect of a product’s appearance, feature set, and interaction scheme affects these goals (Wiklund, 1994). If a product is to have a high level of usability, the product development team must approach the design of the user interface – the elements of the product that people interact with – at a broader conceptual level as well so that they fit the context of use. Since the term usability entered common usage in the early 1980s, different approaches and a set of methods have been proposed to do usability testing. Usability, testing is a systematic way of observing actual users trying out a product and collecting information about the specific ways in which the product is easy or difficult for

4 M. Liu and Z. Zhu

them (Dumas and Redish, 1993). Usability testing with potential users is the best way to find problems related to users’ tasks and experiences (Riihiaho, 2000). A common way is to ask some participants to accomplish a set of realistic tasks and collect data on performance measures as they do so. Usability can be gauged by both objective and subjective variables. Objective measures may include task-completion time, the number of queries and the error rate. Subjective measures, such as satisfaction, are often collected by questionnaires. In this study, we use eye tracking method together with traditional interviewing and observing methods to evaluate the homepage screen layouts of an e-course.

Eye tracking is a technology that monitors eye movements as a means of detecting abnormalities or of studying how people interact with text or online documents. It is based on the idea that the direction of a person’s gaze is directly related to the relative positions of the pupil and the reflection of an object off the cornea (Namahn, 2003). In the early days, eye tracking was mainly used in the field of psychological research (for example, studies of reading strategies), then it was used in the human-computer interaction and usability testing. In 1998, Ellis employed eye tracking as an evaluation technology to evaluate the usability of a series of websites. Typically eye tracking requires the participants to wear special gear (eye-tracker) while undertaking a set of tasks according to an experimental protocol. According to the eye movements recorded by the eye tracker, designers or researchers can describe and analyse how participants process the visual information of the interface using spatial position, interest position and the gaze plot. The eye behaviours can disclose the spatial positions on the interface of which the users are interested in or paying attention to (Loftus and Mackworth, 1978).

This method has potential to beneficially complement traditional usability testing methods (Ellis et al., 1998). During a CHI 99 workshop “The hunt for usability: tracking eye movements” (Karn et al., 2000) a team of workshop participants focused on incorporating eye tracking in usability testing, and they identified a list of reasons for using eye tracking during usability tests, such as ‘evaluate a specific interface’, ‘measure how long a user looked at an area of interest’, ‘capture a sequential scan path’, ‘help discriminate ‘dead time’’, ‘support other types of data’, and so on. Eye movement recordings can provide a record of the pattern of fixations (as gaze plot shows), the time spent looking at various display elements (as heat map shows), and insight into the deployment of visual attention (as AOI-area of interest shows). Usually having the eye fixate on an object for over 100s is called ‘fixation’, and the jumping from one fixation to another is called ‘saccade’ (Zhao and Zuo, 2006; Li et al., 2007; Shen and Zhang, 2000; Zhang and Shen, 2001). Eye movements are made up of two components of alternating sequences of fixations and saccades (Shen and Zhang, 2000; Zhang and Shen, 2001). Spatial and temporal arrangement of a sequence of fixations is called ‘gaze plot’. Details about those variables, such as gaze plot, heat map and so on, will be discussed in data collection part.

3 Goals, tasks and participants

This study undertakes usability testing on an e-course named ‘history of foreign literature’ by integrating eye-tracking techniques with traditional interviewing methodology. It focuses on the functionality and usability of the homepage of this course.


Next we will provide a brief introduction to the course and then describe the testing goals, tasks and participants.

4 Course description

This course is a compulsory undergraduate course for Chinese literature major students and also an elective course for those who are from other subject areas, such as English, History and Education majors. Hundreds of students have been enrolled to the course from its first opening in 2001 fall. In 2007, we planned to rebuild it to refresh the content and meet technical requirements. In preparation for the rebuild, students were given a questionnaire, and teaching assistants and teachers were interviewed about their experiences with the class and online resources. The responses identified some problems related to the usability of the course. Firstly, the layout and navigation of the web pages are not good enough for students to use. For example, students asked how could they return to the expected content webpage, where could they get help when they encountered difficulties during study and where could they find basic information about the course? Secondly, students commented that the multimedia was monotonous and dull. The students complained that the whole course was like a textbook moved online. There was too much text without enough interesting media or interaction, and the processes were confusing. Thirdly, the supporting tools and basic information for the course such as syllabus, learning guidance and so on were not provided in the old version of this course.

The designers concluded that the new version of course should be easier to use and easier to learn. Based on these findings, the goals for the redesign were to create a student-centred learning environment (an independent website), in which there existed lots of learning modules (to include not more than a half hour of lectures for each module), cases, supplemental materials, exercises and kinds of interactive activities. Besides the improvements to learning materials and activities, this new course would also provide improved course basics and learning tools for the students. The course basics include the syllabus, the learning guidance, an introduction to the teacher, and assessment requirements. Learning tools consist of note-taking, bookmarking, calendar and search tools. For instance, when the students are reading, they can use the note-taking tool to mark down what they are hearing and thinking.

The homepage plays a key position in the information architecture of the course and the students’ ability to understand and use that architecture efficiently. The redesign process started with developing six alternative versions of the homepage and using a focus group for review. Following the focus group two versions (shown as Figure 1 and Figure 2, hereby called Version 1 and Version 2) were identified as having most promise. From a visual interface perspective, the layouts of the two versions are very similar. Collectively there existed two main differences in them:

1 in the middle part, in Version 1, is an image representing a television, and in Version 2, it is a rectangle elaborated with two big colourful balloons

2 the ‘course learning label’ in Version 1 stands out, but in Version 2, it is embedded in the upper navigation bar parallel with other basic information labels.

The next step in the redesign process was to determine which version has better usability and why?

6 M. Liu and Z. Zhu

Figure 1 Version 1 for ‘history of foreign literature’ homepage (see online version for colours)

THE TARGET OF TASK 1

THE TARGET OF TASK 2 THE

TARGET OF TASK 3

Figure 2 Version 2 for ‘history of foreign literature’ homepage (see online version for colours)





4.1 Testing goals

There are four main testing goals to be reported for this study:

• First, could the participants find the learning entrance easily? That is to say, will the participants find it easy to get started on the learning activities?

• Second, could the participants easily find the supporting tools? That is to say, when they encounter difficulties during learning, will they know how to get help from the course resources?

• Third, are the ‘course learning’ label easy to find and its meaning clear enough for the participants?

• Fourth, the two colourful balloons are aesthetically pleasing but do they unduly distract the participants’ attention from their work?

4.2 Testing tasks

In this study, the participants were asked to perform three tasks on the homepages of the two versions. The tasks are as follows:

1 Task 1: find ‘the entrance of chapter one’ and click. The usability goal of Task 1 is to determine if the entrance to chapter one is clear and easy to find.

2 Task 2: find the ‘Q&A’ label and click. The usability goal of Task 2 is to determine if the layout of ‘learning support’ area on the right side of the interface is reasonable or not and if the items are easily found, recognised, and understood.

3 Task 3: find the ‘course learning’ label and click. The usability goal of Task 3 is to let facilitator know if ‘course learning’ label makes sense to the participants and if the position of its icon is clear.

4.3 Participants

According to Pernice and Nielsen (2009), in a qualitative eye-tracking study, about six users is typically enough for an accurate assessment of the main usability issues in a design. The study was carried out with eleven participants who were divided into two groups. One group was asked to test Version 1 and the other was asked to test Version 2. Every participant was asked to complete three tasks. A usability professional acted as test facilitator and also debriefer.

From Table 1, we can see that there are five male participants and six female participants; the participants occupy three periods of ages; six participants have online learning experience; six participants were asked to test Version 1 whereas the others were asked to test Version 2.

8 M. Liu and Z. Zhu

Table 1 Participant demographic information

Group number

Age (1: 20-30; 2: 31–40; 3: 40+)

Gender (1 – female, 2 – male)

Online learning experience

(1 – no, 2 – yes)

Testing version

Group 1 2 2 1 Version 1 1 1 1 3 1 2 1 1 2 1 1 1 1 2 2 Group 2 2 2 1 Version 2 3 1 1 2 1 2 2 2 2 3 2 2

5 Tools, procedure and data collection

5.1 Testing and analysing tools

A Tobii T60 eye tracker was used in this study at East China Normal University (ECNU)-University of Missouri, USA (MU) joint lab of usability testing. This apparatus is embedded in a computer monitor, not a head mounted. The apparatus is not apparent to the participant and monitor appears no difference than others that the participant may have used. At the time of testing, this special monitor (eye tracker) was connected with a computer which was installed with the tracing and analysing software of Tobbi T60 named Tobii Studio. The eye tracker samples the gaze position once every 16–17 millisecond (i.e., frequency of 60 Hz) to detect and collect the data of eye movements. Eye tracker is used usually with the following key steps:

• The first step, to connect the eye tracker and the computer together with related equipments, open the Tobbi Studio software, and preset up the testing project.

• The second step, to ask the participant to sit in front of the eye tracker, ensuring that the participants sit comfortably at a distance 70 cm location from the eye tracker (distance depends on the type of eye tracker, 70 cm is appropriate for the Tobii T series).

• The third step, to click the start recording button on the design and record interface of Tobii Studio software, filling in the information of the participant.

• The fourth step is calibration. Calibration is to let the eye tracker know the features of the participant’s eye movements. In other words, when the eyes movements are captured, the eye tracker needs to establish a link between the eye movements and the screen. This is the premise to collect the accurate eye movement data. In order to get the best collection effect, eyes should be in the window central region. Adjust the


seat, table or the eye tracker bracket until the screen indicator can reliably show the eyes and the indicator state strip appears green. If the calibration result is good, click accept button.

• The fifth step, to click start recording button, those preset stimulus materials will be presented one by one, and the eye tracker will record.

• The sixth step, after recording, to click visualisation or replay tab for data analysis.

Tobii Studio can support the following stimulus presentation materials, in other words, the eye movements data of the following formats of materials can be recorded and analysed by Tobii Studio:

1 Images: The tested materials can be an image or a picture and the file format must be .JPG.

2 Movies: Movies usability can be tested by eye tracker too, but the video file must be in AVI format.

3 Web: You can enter a URL or URL webpage presentation. The current Tobii Studio only supports Internet Explorer, all of the webpage presentation materials will be automatically opened by Internet Explorer.

4 External video: The external videos from television, video recorder or another computer can be tested too.

5 Screen recording: Tobii Studio can record the usage data of the third party software. It is commonly used for a simple graphical interface application of eye movement recording, such as word processing software, PowerPoint, etc. In this study, the stimulus presentation materials are the webpage URLs of the three tasks.

After starting, Tobii Studio will be hidden, but the computer desktop will show up. Participants operate the computer in a natural state. At the same time, all of the screen contents, the participant’s eye-movements, the keyboard operations and mouse clicks will be recorded.

5.2 Testing procedures

The testing is done with the participants individually working through the following protocols:

• Firstly, the participant was asked to sit in front of the eye tracker at a distance 70 cm location from the eye tracker, and the facilitator gave brief introduction about this testing to the participant, including testing goal, attentions, the eye tracker operating and other related items that the participant needs to know.

• Secondly, the participant was asked to do a calibration. If the calibration is successful, the participant would be asked to complete the three predefined tasks, during the testing, the eye tracker traced the participant’s eye movements and fixations.

• Finally, at the end of the testing, the participant was interviewed by the facilitator. The participant was required to describe his/her opinions for the task related functions.

10 M. Liu and Z. Zhu

5.3 Data collection

In this study, both qualitative and quantitative data were collected. The qualitative data were interview responses from each participant at the end of his/her testing and the observations made by the facilitator during the test. The quantitative data were the fixation points and eye movements captured and presented by Tobii Studio as graphs and statistics. The dependent variables of the study and their operational definitions are shown as Table 2. Table 2 Dependent variables for analysis

Dependent variables Operational definition Hotspots (heat map) Using different colours to show which area attracts more eye fixations.

The deeper the colour is, the more attention the area gets. Completion percentage

The number of participants who complete each task divided by the total number of participants. The higher the percentage the better is the usability.

Average time to first fixation on the target area

Summing the time that each participant spends in having his/her first fixation on the target area, then divided by the total number of participants. The lower the value of this variable the better is the usability.

Average completion time

The total time spent by all of the participants in finishing task is divided by the number of participants. The lower the value of this variable the better is the usability.

Average fixation number in target area

The average number of fixations that the participants have in target area. The meaning depends on context.

Average time for the first fixation

It refers to the average time spent for the first fixation on the interface by all of the participants. It is usually compared with ‘average time to first fixation on the target area’ to see the time gap between first fixation on the interface and the first fixation on the target area. A lower time gap means the target area was easier to find.

Average times of entering the target area

Its explanation depends on context. It should be explained with other variables. Sometimes the high value of this variable means that the meaning of the target is a blur and confused, so the participants have to be out and into the target area again and again to make sure. Sometimes, the high value means the target is attractive so the participants like to visit it more and more.

6 Task analysis and findings

6.1 Task analysis

6.1.1 Task 1: please find the entrance of chapter one and click on it

Figure 3 displays the hotspots (heat map) of Task 1 on two versions of the interface, and Table 3 shows the average value of each dependent variable.


Figure 3 Hotspots of Task 1 on Version 1 and Version 2 (see online version for colours)

Table 3 Average number of each variable on two web pages

Variables Version 1 (seconds) Version 2 (seconds)

Completion percentage 100% 100% Average time to first fixation on the target area 3.378 2.755 Average completion time 5.082 6.963 Average fixation number in target area 1.5 2.2 Average time for the first fixation 0.707 0.958 Average times of entering the target area 1.33 1.18

From Figure 3 and Table 3, we can see:

1 The middle information box attracted more attention, the target area (where chapter one is located) is obvious and it is easy to find.

2 All of the participants took only a short time to enter the target area. The average time to first fixation on the target area of Version 1 was 3.378 seconds, and on Version 2 it was only 2.755 seconds.

3 All of the participants completed Task 1 easily and they finished it in a short time. In Version 1, all of the participants completed Task 1 in less than 9 seconds and the average completion time was 5.082 seconds. In Version 2, all of the participants completed Task 1 in less than 16 seconds and the average completion time was 6.963 seconds. Among those participants who were asked to do tasks in Version 2, there was a person whose computer operating skill was poor, if he is excluded, the average completion time would be less than 5 seconds.

In conclusion, for both versions, all of the participants completed Task 1 easily and quickly. Comparatively speaking, the design of Version 2 is better than Version 1 based on the following comparisons:

1 Participants spend just 2.755 seconds for the first fixation on the target area in Version 2, while the participants spend 3.378 seconds on Version 1 even though the layout of the two versions are almost the same.


2 From Figure 3, we can see that the hotspots in Version 2 are more concentrated (shown as deep colour), while in Version 1, the hotspots scatters a little.

3 The time gap between the variable ‘average time for the first fixation’ and the variable ‘average time to first fixation on the target area’ in Version 1 is 2.671 seconds whereas in Version 2 is 1.797 seconds. A lower time gap means the target area was easier to find.

4 In the interview, most of participants said the entrance of chapter one was easy to be found for both versions, and most of them said the first attraction was the ‘information box’ in the upper middle of the interface. The interview results were consistent with the eye tracking data. When the debriefer asked the participants if the two colourful balloons on Version 2 attracted their eyesight, everybody said no, and some of them said they even did not notice the two balloons. The hotspots of Figure 3 confirm the interview responses by not showing any fixation on the two colourful balloons on Version 2. Thus, it could be inferred that the two colourful balloons on Version 2 did not play a negative role in the usability of the interface.

6.1.2 Task 2: find the ‘Q&A’ label and click on it

Figure 4 displays the hotspots of Task 2 on Version 1 and Version 2, and Table 4 shows the average value of each dependent variable.



Variables Version 1 (seconds) Version 2 (seconds)

Completion percentage 100% 100% Average time to first fixation on the target area 2.61 2.961 Average completion time 1.28 1.5 Average fixation number in target area 1.3 1.2 Average time for the first fixation 0.9 1.26 Average times of entering the target area 1.3 1.2



1 The hotspots in two versions were all clear, and in Version 2, the only one hotspot was the target area.

2 The participants used less time to enter the target area, in Version 1, the average time is only 2.61 seconds, and in Version 2, the average time is 2.961 seconds.

3 All of the participants completed Task 2 easily. In Version 1, the average completion time is 1.28 seconds, and in Version 2, the average completion time is 1.5 seconds.

Based on the analysis above, we can see that all of the participants completed Task 2 easily and quickly. The testing results indicated that the ‘learning support area’ was clear and easy to find.

6.1.3 Task 3: find out the ‘course learning’ label and click on it

Figure 5 displays the hotspots of Task 3 on Version 1 and Version 2, and Table 5 shows the average value of each dependent variable.



Variables Version 1 (seconds) Version 2 (seconds) Completion percentage 100% 20% Average time to first fixation on the target area 4.859 7.629 Average completion time 10.648 9.054 Average fixation number in target area 2.167 0.4 Average time for the first fixation 0.197 0.04 Average times of entering the target area 2.0 0.2


1 In Version 1, the hotspots were widely distributed over the top navigation bar and the right ‘learning support area’. There are many hotspots and the hotspots had wide dispersion, but the hotspot of the target area was not very prominent. In Version 2, the hotspots were widely distributed over the right ‘learning support area’ and the


middle ‘information guiding box’. In contrast, the top navigation bar where the target area was located was barely noticed.

2 The participants used a longer time to enter the target area than in the prior two tasks, in Version 1, the average time is 4.859, and in Version 2, the average time is 7.629.

3 The participants used a longer time to finish Task 3. In Version 1, the average completion time is 10.648, and in Version 2, the average time is 9.054.

4 The average time for the first fixation is very short, for Version 1, it is only 0.197, and for Version 2, it is only 0.04, but the time to first fixation on the target area is very long. It indicates that the target area is not the first fixation of the participants.

6.2 Findings

Based on the analysis above, participants’ debriefing after each task, and also observations during testing, we developed the following insights:

6.2.1 Ease of use

The layout of the homepage is effective and easy to use. This can be concluded from the results of Task 1 and Task 2. For the two tasks, in both versions, every participant completed the tasks easily and quickly, and also the hotspots area are concentrated, which means the layout is efficient for the participants to find the learning entrance and seek help when they encounter difficulties during learning.

6.2.2 Learnability

That the homepage is easy to learn can be concluded from comparison between Task 1 and Task 2. In Task 2, the participants used less time to find the target, which means while completing Task 1, the participants got an impression about the layout, so it is easier for them to finish Task 2. It can also be concluded from the variable ‘average time to first fixation on the target area’ and ‘average time for first fixation’ that in Version 1, for Task 1, the time gap is 2.671 (3.378 minus 0.707), for Task 2, the time gap is 1.71 (2.61–0.9), whereas in Version 2, for Task 1, the time gap is 1.797 (1.755 minus 0.958), for Task 2, the time gap is 1.701 (2.961 minus 1.26). These values for Version 1 and Version 2 show that the time gap for Task 2 is much less than that for Task 1 and indicate that the interface can be learned easily and that learning the interface benefits usability.

6.2.3 Bad design of ‘course learning’ label

It was difficult for the participants to finish Task 3. In Version 2, only two people finish Task 3 (as shown in Table 5). The location of the target area was not easy to perceive. Most participants were blind to the target icon. In post-testing interviewing participants indicated that in Version 2, the two balloons attracted attention to the middle information block more and led to the navigation bar being relatively ignored. In Version 1, the task completion rate is higher than in Version 2, but the graphic design of the ‘course learning’ icon seemed to increase the difficulty for the participants to find it, because most participants thought it was just a graph for decoration instead of a navigation label. In fact, the ‘course learning’ icon plays the same role as the entrances of every chapter, so


if it was deleted, it would not affect the usage of the course. The participants would then navigate to the course content through the chapter navigation panel.

6.2.4 Impact of using two colourful balloons

In Version 2, designers were concerned that the two balloons might interfere with the participants’ reading, however, the eye tracking data show that they were not the focus of attention. The eye tracking data allow this type of question to be addressed with objective data which would not be available from other usability testing methods, such as task analysis, thinking aloud and so on. Our impression has been that similarly to the use of eye tracking in examining the impact of the balloons, the eye tracking method offered valuable supplemental data for addressing usability issues.

6.2.5 Version 2 is slightly better than Version 1

Based on the analysis of the hotspots, statistic for the three tasks, and interviewing the participants after testing, we found that each version had good navigation and layout. Comparatively speaking, the design of Version 2 was a little better than Version 1. In Version 2, the area of interest (AOI) was more concentrated (see Figure 6). The participants could find the learning entrance and learning supporting area more easily and quickly.

Figure 6 Area of interest (see online version for colours)


7 Conclusions and suggestions

Through the use of eye tracking in the usability experiment, we concluded that eye tracking data identifies the potential focus of attention (shown as the heat map); which regions are easy to be overlooked (shown as the heat map and gaze plots); or what things are easy to distract users’ attention, which can be drawn from the heat map, gaze plots and the related statistics. Through the eye tracking process, we were able to verify that what we interpreted as ‘confusions’ were indeed confusions, and we were able to probe why these confusions occurred, such as the situation in Task 3. In a word, we learned finer points and developed confidence in our understanding about what draws attention and why on one web page, and by studying what people look at and do not look at, we are able to glean more detailed insights about how the web user works his or her way through usable and unusable designs. According to Pernice and Nielsen (2009), eye tracking adds another dimension to the usability testing process: watching what people are looking at adds a level of interest for the designers to usability studies. Following the eye at work makes you feel as though you are in the user’s head, thinking with him. If nothing else, the eye tracking studies are more interesting and hold observers’ attention more easily. Eye tracking technology also helps test facilitators avoid a major blunder: interrupting the user when he is quiet for a long time. This is a huge benefit since interrupting the participant too much or inappropriately is one of the greatest challenges for even seasoned usability professionals.

Other conclusions are also drawn from the case study. One is that combining eye tracking with interviewing produces strong corroborative results. This integration provided the research team with a richer interpretation than either technique alone. The result is a set of strong, detailed recommendations for homepage improvement that have both statistical and qualitative bases. The other was that eye tracking was a beneficial complement to traditional usability testing methods. Certain types of questions are difficult to answer efficiently with traditional usability testing methods (Dumas and Redish, 1993; Nielsen and Mack, 1994). Take this case study as an example, the two versions are similar, it is difficult for people, even design specialists and usability professionals to tell which one is better and why with only time to complete a task, percentage of participants succeeding, type and number of errors, and subjective ratings of ease of use. According to Dumas and Redish (1993), usability testing methods have not changed significantly since the origins of the practice. Usability studies typically address human performance at a readily observable task-level, including measures like time to complete a task, percentage of participants succeeding, type and number of errors, and subjective ratings of ease of use.

However, using eye tracking technology also adds costs and requires trained personnel. The addition of eye tracking takes a traditional usability test and makes it more time-consuming, and more expensive (Pernice and Nielsen, 2009). The findings from our report show that eye tracking can be used in the design and development process of e-courses or other e-learning applications effectively with other usability testing methods to provide measurable improvements in the effectiveness, efficiency and users’ satisfaction of the products. Further research is needed to improve our understanding of when eye tracking should be used in a process of usability testing and how these findings could be nicely integrated so as to make optimal design decisions.


References Alexander, S. (2001) ‘E-learning developments and experiences’, The Journal of Education and

Training, Vol. 43, No. 1, pp.240–248. ASTD E-Learning Glossary (2012) available at http://www.astd.org/LC/glossary.htm (accessed on

22 January 2012). Bates, A.W. (2005) Technology, E-learning and Distance Education, Routledge, London. Bourne, J. (2010) available at http://commons.sloanconsortium.org/wiki/online-course-definition-

edit-and-comment-here (accessed on 22 January 2012). Desurvire, H.W., Kondziela, J.M. and Atwood, M.E. (1992) ‘What is gained and lost when using

evaluation methods other than empirical testing’, in Monk, A., Diaper, D. and Harrison, M.D. (Eds.): People and Computers VII, pp.89–102, Cambridge University Press, Cambridge.

Dumas, J.S. and Redish, J.C. (1993) A Practical Guide to Usability Testing, pp.12, 22–38, Ablex Publishing Corporation, Norwood, New Jersey.

Ellis, S., Candera, R., Misner, J. et al. (1998) ‘Window to the soul? What eye movements tell us about software usability?’, Proceedings of the Usability Professionals’ Association, pp.151–178, UPA Press, Washington, DC.

Goto, K. and Cotler, E. (2005) Web Design: Workflow That Works, New Riders, Berkeley, CA. Govindasamy, T. (2002) ‘Successful implementation of e-learning pedagogical considerations’,

The Internet and Higher Education, Vol. 4, No. 3, pp.287–299. Gulati, S. (2008) ‘Technology-enhanced learning in developing nations: a review’, International

Review of Research in Open and Distance Learning, Vol. 9, No. 1, pp.1–16. Hung, J-l. (2012) ‘Trends of e-learning research from 2000 to 2008: use of text mining and

bibliometrics’, British Journal of Educational Technology, Vol. 43, No. 1, pp.5–16. Internet Time Wiki (2009) available at http://internettime.pbworks.com/w/page/20095794/

FrontPage (accessed on 22 January 2012). ISO/IEC (1998) 9241-11 Ergonomic Requirements for Office Work with Visual Display Terminals

(VDT)s – Part 11 Guidance on Usability, ISO/IEC 9241-11:1998(E). Jeffries, R.J., Miller, J.R., Wharton, C. and Uyeda, K.M. (1991) ‘User interface evaluation in the

real world: a comparison of four techniques’, P. ACM CHI’91 Conference, New Orleans, LA, pp.119–124.

Karn, K.S., Ellis, S. and Juliano, C. (2000) ‘The hunt for usability: tracking eye movements’, available at http://old.sigchi.org/bulletin/2000.5/eye.html (accessed on 22 January 2012).

Lanzilotti, R. and Ardito, C. (2010) ‘Do patterns help novice evaluators? A comparative study’, International Journal of Human-Computer Studies, Vol. 69, No. 1, pp.52–69.

Li, H., Wang, Q. and Ge, L. (2007) ‘Webpage usability evaluation study on eye gaze tracking technology’, Journal of Ergonomics, Vol. 13, No. 2,:p.57.

Loftus, G.R. and Mackworth, N.H. (1978) ‘Cognitive determinants of fixation location during picture viewing’, Journal of Experimental Psychology: Human Perception and Performance, Vol. 4, No. 4, pp.565–572.

Masie, E. (2008) ‘Luminary perspective: what is the meaning of the e in e-learning’, in Biech, E. (Ed.): ASTD Handbook for Workplace Learning Professionals, pp.377–381, ASTD Press, Alexandria, VA.

Namahn (2003) available at http://www.namahn.com/resources/documents/note-eyetracking.pdf (accessed on 22 January 2012).

Nicholson, P. (2007) ‘A history of e-learning: echoes of the pioneers’, in Fernández-Manjón, B. (Ed.): Computers and Education: E-learning, from Theory to Practice, pp.1–11, Springer, Dordrecht.

Nielsen, J. and Mack, R. (1994) Usability Inspection Methods, pp.1–23, John Wiley & Sons, Inc., New York, Chichester, Brisbane, Toronto, Singapore.


Pernice, K. and Nielsen, J. (2009) ‘Eye tracking methodology: 65 guidelines for how to conduct and evaluate usability studies using eye tracking’, available at http://www.useit.com/ eyetracking/methodology/ (accessed on 29 January 2012).

Riihiaho, S. (2000) ‘Experiences with usability evaluation methods’, thesis at the Helsinki. Rosenberg, M. (2001) E-Learning: Strategies for Delivering Knowledge in the Digital Age, p.28,

McGraw-Hill, New York. Shen, M. and Zhang, G. (2000) ‘Retina and reading: eye movement control in ideal observer

model’, Journal of East China Normal University, Vol. 18, No. 4, pp.59–65. Shimabukuro, J. (2012) ‘Sloan-C’s definition of ‘online course’ may be out of sync with reality’,

available at http://commons.sloanconsortium.org/wiki/online-course-definition-edit-and-comment-here (accessed on 22 March 2012).

Tan, W., Liu, D. and Bishu, R. (2009) ‘Web evaluation: heuristic evaluation vs. user testing’, International Journal of Industrial Ergonomics, Vol. 39, No. 4, pp.621–627.

US News (2012) ‘Top online bachelor’s programs’, available at http://www.usnews.com/education/ online-education (accessed on 22 March 2012).

Wang, Q., Zhu, Z., Chen, L. and Yan, H. (2009) ‘E-learning in China’, Campus-Wide Information Systems, Vol. 26, No. 2, pp.77–81.

Wiklund, M.E. (1994) Usability in Practice – How Companies Develop User-Friendly Products, pp.6–9, AP Professional, Boston.

Zhang, G. and Shen, M. (2001) ‘Eye tracking techniques in usability testing’, Journal of Ergonomics, Vol. 7, No. 4, pp.9–14.

Zhao, X. and Zuo, H. (2006) ‘Eye tracker and eye tracking techniques’, Computer Engineering and Applications, Vol. 12, No. 12,:p.55.

A case study of using eye tracking techniques to evaluate the usability of e-learning courses

Documents

Transcript of A case study of using eye tracking techniques to evaluate the usability of e-learning courses