An Experimental Usability Study for Augmented Reality Technologies in the Tourist Sector

Aurkene Alzua-Sorzabala, María Teresa Linazab, and

Marina Abada

a Faculty of Humanities, San Sebastian Campus Deusto University, Spain

{aurkenea}@ud-ss.deusto.es

b Department of Tourism, Heritage and Creativity, Visual Communication Technologies (VICOMTech), Spain

mtlinaza@vicomtech.es

Abstract

This paper presents the usability study of a new prototype based on Augmented Reality technologies, which has been tested on a tourist site in the North of Spain. The overall aim of the usability study of the prototype has been the development of typologies of users and standards, so that new interactive tourist experiences could be personalized in the future taking into account concepts such as usefulness and ease of use. Observation has been a key issue within the usability study so that the possibilities and limitations of the design of the prototype have been analysed on the basis of the behaviour of users. Most of the participants agreed that the approach of using Augmented Reality technologies enhances the interactive experience with tourist content. Moreover, they have positively assessed the usefulness of the prototype as a tourist information point due to its capability of providing Location-Based contents.

Keywords: usability studies; Augmented Reality; tourism; interactive experience.

1 Introduction

Added-value services based on innovative Information and Communication Technologies (ICT) have launched new research challenges related to content personalization. These services should enhance the experience of users and skip what they consider irrelevant content for them. Thus, behavioural models which relate acceptance and usability of ICT with user profiles are crucial, as they provide further information for demand identification. From the point of view of the theoretical analysis, innovative methodological approaches should include two main aspects in order to take into account all aspects related to the usability of ICT-based products and services: a quantitative analysis of the numerically measurable aspects

influencing the acceptance of the technology and a qualitative analysis of the soft factors that determine its usability.

Several disciplines within the domain of Social Sciences have traditionally distinguished between quantitative and qualitative techniques. However, new trends have shown the convenience of combining both methodologies in a triangulation method, where both qualitative and quantitative methods are used in varying ways and degrees. The aim of quantitative analysis is the generation of standardized data in a statistically representative way so that methodologies can be replicated and results can be compared in further research. The qualitative approach includes new factors such as the “level of interest” and “suggestion”. It is not enough to know whether technology is usable but to analyse whether users are willing to use it and under which conditions. It must be mentioned that the degree up to which these factors influence the attitudes of the users with regard to technology usage in tourism settings is difficult to measure. User acceptance of technology is typically measured by looking at the intentions. The stronger the intention to use a technology, the more that user is supposed to accept the technology.

This paper presents the usability study of a new prototype, combines the commonly known concept of tourist binoculars and Augmented Reality technologies. The usability study and evaluation of the prototype have been based on both qualitative and quantitative analysis. Section 2 presents a brief state of the art regarding concepts related to methodological approaches for usability analysis. The new prototype based on Augmented Reality Technologies called PRISMA is described in Section 3 from a technical point of view, including its main features and the usage of the prototype. Section 4 focuses on the developed methodology, which has been separated in qualitative and quantitative methods. Finally, some general conclusions are provided in Section 5.

2 Literature review

Several theoretical models have been developed to study user acceptance and usage of emerging Information Technologies. The Technology Acceptance Model (TAM) has become a well-established robust model for predicting user acceptance (Davis, 1989; Davis et al, 1989). The goal of TAM is “to provide an explanation of the determinants of computer acceptance that is general, capable of explaining user behaviour across a broad range of end-user computing technologies and user populations, while at the same time being both parsimonious and theoretically justified”. Specifically tailored for modelling user acceptance of Information Technology, TAM is a valuable tool for predicting attitudes, satisfaction and usage from beliefs and external variables.

There are some examples regarding the application of the TAM to the tourism sector. Noor et al (2005) present a survey conducted to determine whether the perception of trust, risk and sharing has an effect on the sharing of information at a tourism website. The study was conducted using the tourism website www.virtualtour.com which has 400000 members. Rasinger (2006) describes the research results comprising measures for discovering accepted mobile information services in tourism and defining the process for customer-based mobile information services development. Dilworth & Shafer (2004) examine visitor attitudes towards and intent to use Intelligent Transportation Systems (ITS) in two of the National Park units in California.

Another approach different from the TAM is the Concept of Flow (Novak, Hoffman and Yung, 2000), including factors such as skill and control, challenge and arousal, or focused attention. These factors contribute to flow, a state of mind where the user is completely devoted to the user of a system and forgets everything else around him/her.

Another theoretical area relevant for this contribution is usability testing and system evaluation. Usability study has proven to be an invaluable tool for ensuring the quality of computerised systems. The studies of stationary computer systems are an established discipline within Human-Computer interaction with widely acknowledge techniques and methods. Most research investigating the design of tourism technologies has focused on the concept of guides. These devices help travellers and others unfamiliar with an environment to navigate, learn and perform situated actions.

Research into developing technologies to support tourism has been popular with those exploring contextual awareness issues, and virtual or augmented reality (VR/AR). Some examples could include the GUIDE system developed at Lancaster University (Cheverst, et al., 2000) or a study of tourist requirements for the design of a city guide conducted by Brown and Chalmers (2003).

3 Description of the prototype

The key objective of the PRISMA prototype is the integration of Augmented Reality technologies into the widely known concept of tourist binoculars. By means of these technologies, the real scene can be enhanced by virtual information to increase users’ experience. PRISMA is basically a video see through visualization system composed of a camera to record the real time video stream, binoculars as the visualization device and an inertial sensor mounted on the structure to track the point of view and rotation of the binoculars (Fritz, Susperregui & Linaza, 2005; Alzua-Sorzabal, Linaza & Susperregui, 2006).

The field of view of the spectator is captured by the camera mounted on the binoculars and sent to the processing unit, where the graphical data is added and the augmented stream is sent back to the binoculars. The tracking of the system captures and records the current location and orientation of the binoculars. The central processing unit converts these data into an orientation vector and the system displays the correct information on the binoculars. The graphical objects, which represent highlighted tourist assets or other abstract information, are drawn to enhance and complement the perception of the user of the natural environment.

Fig. 1. PRISMA prototype in the real environment.

PRISMA provides different added-value multimedia content As soon as users begin the interaction, the system personalises the content regarding the language (Basque, Spanish, English and French). Each language has an associated target group of users. Then, users choose what kind of information they are interested in among previously personalized contents.

Fig. 2. Interaction interfaces for audiovisual contents.

An overview from one of the hills of Donostia-San Sebastian city in the North of Spain has been chosen as the real validation environment. The binoculars have been placed at the middle height of the hill to allow tourists and also local citizens to "visit" some of the interesting cultural and tourist attractions of the city, such as the Cathedral and Santa Clara Island in the middle of the bay.

Visualizedarea

�������� ������������������������

���� ������ ������ ������ ������ ������������������������������������������������������������������������������������

�������� � �� �� � �� �� �� � �� �� �� � �� �� �� � ���� ����� ����� ����� ��

�������� ��� ���� ��� ���� ��� ���� ��� ���� �� ���� ���� ���� ��

�������� ��� ����� ����� ����� ��� � ��� � ��� � ��� � ��

!�!�!�!� " ����� �" ����� �" ����� �" ����� ��� ���� ���� ���� �� �� �� �� �� " ����� " ����� " ����� " �����

#�#�#�#� ����� � ����� � ����� � ����� �

$�$�$�$� %�&�%�&�%�&�%�&� ' ��' ��' ��' �� �� �� �� �� (������(������(������(������)���)���)���)���

*�*�*�*� ���������������� �� �� �� �� +�, ��+�, ��+�, ��+�, ��- .���- .���- .���- .����.��.��.��.�

Visualizedarea

�������� ������������������������

���� ������ ������ ������ ������ ������������������������������������������������������������������������������������

�������� � �� �� � �� �� �� � �� �� �� � �� �� �� � ���� ����� ����� ����� ��

�������� ��� ���� ��� ���� ��� ���� ��� ���� �� ���� ���� ���� ��

�������� ��� ����� ����� ����� ��� � ��� � ��� � ��� � ��

!�!�!�!� " ����� �" ����� �" ����� �" ����� ��� ���� ���� ���� �� �� �� �� �� " ����� " ����� " ����� " �����

#�#�#�#� ����� � ����� � ����� � ����� �

$�$�$�$� %�&�%�&�%�&�%�&� ' ��' ��' ��' �� �� �� �� �� (������(������(������(������)���)���)���)���

*�*�*�*� ���������������� �� �� �� �� +�, ��+�, ��+�, ��+�, ��- .���- .���- .���- .����.��.��.��.� Fig. 3. Graphical augmentation of the point of view.

When users discover a tourist asset about which they are interested in getting additional information, it is enough to click on the marker of the object and a menu with the available choices for the content. PRISMA allows tourists to gain multimedia personalized information about these tourist assets, including textual information (timetables, directions, telephone numbers), 2D photographs and maps, or even videos and 3D reconstructions.

4 Evaluation methodology for the usability study

The proposed evaluation methodology fits into the definition provided by Steele and Moss (1970), who defined the evaluation as “the systematic process of analysing what is better, more desirable, effective or suitable regarding some pre-defined criteria and objectives based on precise comparisons among several observation data with standardised criteria”. Thus, objectivity is a must.

The overall aim of the usability study of PRISMA has been the development of typologies of users and standards, so that new interactive tourist experiences could be personalized in the future taking into account concepts such as usefulness and ease of use.

Therefore, the behaviour of the user when dealing with new technologies as a means of communication of tourist contents has been assessed. In order to achieve this aim, the quality of the communication between the prototype and the user has also been measured. Additionally, the usability study has identified the typology of users that may accept, use and even enjoy Augmented Reality technologies in tourism, leisure and cultural environments. Therefore, it is crucial to assess the impact of PRISMA within the added-value experience with the innovative 3D visualization prototype.

Concerning the sample, about 100 people took part in the usability study, from which 47 answered in-depth questionnaires, at the real scenario at the Monte Urgull in San Sebastian (Spain) in March 2006. The average profile of the participant is a man between 20 and 35 years old (61% of the sample) and quite familiar with the use of new technologies (mobile devices, laptops, digital cameras, Internet). Some of the participants worked in the tourism sector, including destination managers and content providers, because the usability study was also devoted to have feedback for the business exploitation plan of the prototype.

4.1 Quantitative analysis

The main objectives of the quantitative analysis of PRISMA are the assessment of the behaviour of the sample regarding the integration of ICT in tourist environments and the validation of the added value of Augmented Reality technologies to tourist attractions. Data has been collected on the basis of one specific questionnaire developed for the usability study of PRISMA, including aspects related to different constructors such as the perceived usefulness (PU) of the device, its perceived ease of use (PEOU) or the perceived added-value. The main difficulty has been related to the design of the questions which could bias the answers.

This questionnaire included 30 questions and has been filled by the participants of the survey just after experiencing the device. The answers were coded and processed using statistical software. The parameters that have been analysed included the profile of the user (socio-demographic data of the sample and familiarity with new technologies), parameters of the usability study (basically usefulness and ease of use), assessment of the contents (data about usefulness, scientific and technical quality of multimedia contents) and assessment of the prototype (perceived added-value of the prototype and willingness to pay).

Before experiencing the prototype, the evaluation team presented the project briefly an gave some basic instructions about the use of the prototype. Most of the participants (55%) agreed on the suitability of the explanations provided and found themselves confident when experiencing the prototype afterwards.

Table 1. Some of the statements for the quantitative analysis.

H1: I know the basic technological issues in order to experience the prototype. H2: The provided explanations were enough to use the device correctly. H3: The use of new technologies enhances the experience with cultural and natural resources. H4: PRISMA provides new, interesting and useful information about the city. H5: PRISMA is a new tourist information point. H6: The experience of interacting with the prototype has been amazing and enriching. H7: Interacting with PRISMA is quite simple. H8: I have been able to navigate through and access easily the provided contents. H9: I have learnt new subjects about the city.

Firstly, the usefulness of advanced technologies such as PRISMA within tourist environments was assessed. Two thirds of the sample agreed with the statement that PRISMA enhances the access to cultural goods and tourist assets. The same percentage affirmed that PRISMA is a useful tourist information point. They also described the overall interactive experience with the prototype as amazing and enriching. The values of the survey also showed a positive trend (more than half of the sample) when studying the usefulness of the prototype in order to communicate contents and develop affective relations.

The following statements dealt with the ease of use of the prototype and with the searching for additional contents. The usage of the prototype was quite simple for half of the sample, although it must be mentioned that 25% of the sample found it very difficult. However, this pattern was not extensible to the perception of users regarding the intuitiveness of the prototype. Therefore, it can be concluded that users found it easier to learn the usage of the prototype rather than to practically use it. Regarding the navigation and the ease of access to the desired contents, more than half of the participants found the desired contents easily. None of the participants was intimated by the prototype or worried about damaging the prototype due to their interaction.

The usefulness and the quality of the contents, and the technical quality of the multimedia material have also been evaluated. As an overall conclusion, it can be said that contents have been positively assessed. Most of the participants agreed that they have learned something new from their experience, although they found that historical information about the city was too hard for a general profile of a user. Two thirds of participants were surprised by the displayed audiovisual material about the city.

Finally, the questionnaire focused on issues related to the perceived added-value of the prototype and the willingness to pay for the experience. Regarding the former, more than 85% of the sample agreed on the added-value of PRISMA compared to

more traditional tourist experiences. Moreover, 75% of the users would recommend the system to other users. The same trend appeared when asking about the willingness to pay, with more than 80% of positive answers. The average expected price would be between one and three euros.

4.2 Qualitative analysis

The evaluation of tourist information services based on ICT such as PRISMA should take into account not only the added-value of the displayed contents, but also the fulfil of the requirements and desires of users when interacting with such services. Studying the reactions of users is the preferred way of getting this information within qualitative analysis. Generally speaking, qualitative analysis collects data using three main approaches: direct observation, interviews and analysis of written documents.

Within Social Sciences, observation can be defined both as the recognition process and the behaviour registration of people, objects and events. Although there are several observation techniques, all of them are based on the same principle: behaviours are a measurable image of thoughts, expectations and emotions concerning social interactions. Systematic register and its analysis try to explain human behaviour, defining relationships to standardise behavioural patterns and to extend those models to other contexts on the basis of the quality and representation of the sample.

Observation has been a key issue within the usability study of PRISMA. Therefore, it was crucial to analyse the behaviour of users in order to assess the possibilities and limitations of the design of the prototype. Observation has focused on three main aspects regarding Human-Device Interaction: body position (ergonomics, comfort); dynamic interaction user-device-environment; and interaction choices during the navigation. Body position was the first factor for the assessment of adaptability and usability of the prototype. The height and the position of legs, arms and back have been considered as indicators of the comfort of the users during the experience.

The height of the prototype was a big limitation for people smaller than the position of the visualization device (1.6 meter from the ground). In some cases as shown in Fig. 4, the user stood on a step in order to reach the binoculars. People much taller had to adjust their body position as well.

Fig. 4. Body position for some of the participants.

The position of the hands during the interaction with the buttons was quite diverse due to the different number of buttons on each side of the prototype. Although zoom buttons on the left side of the prototype were easily used, several difficulties arose when interacting with the navigation buttons on the right side. Only few participants have maintained the hands on these buttons during the whole experience, but most of them took them away and only placed them again during the interaction. Thus, the diversity of behaviours made it very difficult to find profiles and patterns of usability.

Fig. 5. Position of the fingers regarding the interaction buttons.

Although observation of the reactions of participants when interacting with contents was quite limited, it provided very interesting material for the usability study. As expected, tourist assets placed on the borders of the scenario were less visited than the central ones (Fig. 6). Moreover, no pattern could be defined regarding the selection between historical and practical contents.

Apart from direct observation, qualitative analysis has also included in-depth interviews. Due to field limitations, the designed sample was selected on the basis of its professional profile: a representative group of technicians, professionals from the tourist and cultural sector, and a group of pre- and post-graduated students.

Most of the participants live in San Sebastian, so they have a deep knowledge of the validation scenario.

������������� ����

�����������

� ����

����������

��������

����������������

����

� ����������

��������������

��������

!����

�� ��"��� ����

# ����

!$���

�$�$�$�$ �!�!�!�! �������� ���� ���� �!�!�!�! ���� ���� !!!!

������������� ����

�����������

� ����

����������

��������

����������������

����

� ����������

��������������

��������

!����

�� ��"��� ����

# ����

!$���

�$�$�$�$ �!�!�!�! �������� ���� ���� �!�!�!�! ���� ���� !!!!

������������� ����

�����������

� ����

����������

��������

����������������

����

� ����������

��������������

��������

!����

�� ��"��� ����

# ����

!$���������������� ����

�����������

� ����

����������

��������

����������������

����

� ����������

��������������

��������

!����

�� ��"��� ����

# ����

!$���

�$�$�$�$ �!�!�!�! �������� ���� ���� �!�!�!�! ���� ���� !!!!

Fig. 6. Frequency of visits to the different contents.

In order to fulfil the requirements of a semi-structure interview, the four conceptual areas of the TAM were included in the evaluation, namely performance expectancy (perceived usefulness, extrinsic motivations and added-value), effort expectancy (complexity of the system and ease of use), social influence and facilitating conditions (perceived performance, facilitating conditions). All the interviews have been recorded for further assessment.

Regarding the physical usefulness and ergonomics, the interviews reinforced the results of the observation. Many participants mentioned the associated problems related to the physical design of the prototype. The visualization device is about 1.60 meter high, which may be too high for general public and even inaccessible for children. Another aspect mentioned was the uncomfortable position of the arms during the experience, most of the time lying over the speakers. The navigation buttons were identified as one of the main difficulties concerning the comfort and ergonomics due to the big number of buttons and the lack of correspondence between the navigation direction and the physical location of the buttons on the right side.

Concerning the perceived added-value, PRISMA has been catalogued as an added-value tourist tool. Compared to books and brochures, there is a reduction on the effort that should be done by tourists: information is already there. Many of the participants agreed that the prototype is a very useful way of getting in contact with the city: it offers a panoramic view of the city, provides the user with in situ information and no tourist guide is needed.

Regarding the business model, most of the sample agreed on a pay-per-view model, although the prototype could be free of charge as part of the promotion strategy of the city.

5 Conclusions

In order to measure the scope of the results and conclusions, it must be mentioned that the profile of the sample has been a little bit biased, with some of the participants working in the tourism sector. It must be clarified their double role within the study: a common tourist role for the quantitative analysis and a professional of the tourist sector role for the qualitative one. The participants provided a critical vision taking into account not only their personal point of view, but also an extended discussion about further developments of the prototype. The conclusions have been grouped on the basis of the issues that have been highlighted within the evaluation process.

Most of the participants agreed that the approach of using advanced visualization technologies such as Augmented Reality enhances the interactive experience with tourist content. The interviews concluded that PRISMA provides added-value multimedia contents in an interactive way. This way, information is more accessible with new visualization means and innovative interaction metaphors. Moreover, more than 80% of the participants were willing to pay up to three euros for the experience. As a conclusion, the participants have positively assessed the usefulness of the prototype as a tourist information point for urban environments due to its capability of providing Location-Based contents to allow better route planning.

Concerning the ease of use, the results of the quantitative analysis were mostly positive, although issues related to interaction, comfort and simplicity in the access to contents have to be improved. Many limitations or usability problems were related to the fixed height and the restricted physical adaptability of the prototype to the physical conditions of the user. Position of legs, arms and hands of the user was influenced by the binoculars, the interaction devices, the mechanical rotation of the visualization device and the design of the device as a whole. Further improvements in the physical design of the device will have a great impact not only on a more comfortable experience for the user, but also on a smaller damage when using the device. A final recommendation was the need of the contextualization of the prototype in order to facilitate its usage.

Regarding the contents, more than half of the sample found them usable, interesting and innovative. Moreover, two thirds of the users were impressed by the historical images that were displayed. Interactive contents based on QuickTime panoramas had the greatest impact, specially the navigation and zooming inside the Cathedral of the

city. However, the potential of the prototype could be extended by panoramic approaches, holograms or non-linear story-telling techniques. As a final conclusion, it can be said that PRISMA can be considered as a prototype with a great potential impact over the tourist sector. Further application scenarios such as environmental impact of architecture planning projects have been suggested.

References

Ajzen, I. & Fishbein, M. (1980). Understanding Attitudes and Predicting Social Behavior. Prentice-Hall, Inc.

Alzua-Sorzabal, A., M.T. Linaza & A. Susperregui (2006). Providing On-Site Augmented Information to Tourists. Information and Communication Technologies in Tourism 2006, M. Hitz, M. Sigala and J. Murphy (eds.), SpringlerWien New-York.

Brown, B., & Chalmers, M. (2003). Tourism and mobile technology. In: Kari Kuutti and Eija Helena Karsten (editors). ECSCW 2003: Proceedings of the Eighth European Conference on Computer Supported Cooperative Work, Helsinki, Finland, 14–18 September 2003. Dordrecht: Kluwer Academic Press.

Cheverst, K., Davies, N., Mitchell, K., & Friday, A. (2000). Experiences of developing and deploying a context–aware tourist guide: The GUIDE project. MobiCom 2000: Proceedings of the Sixth Annual International Conference on Mobile Computing and Networking, August 6–11, 2000, Boston, Massachusetts.

Davis, F.D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarerly (September): 319-340.

Davis, F.D., R.P. Bagozzi & P.R. Warshaw (1989). User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Management Science 35: 982-1003.

Dilworth, G. & Shafer, S. (2004). Visitor Preceptions of Intelligent Transportation Systems in a National Park. In: Proc. of the 2004 Northeastern Recreation Research Symposium.

Fishbein, M. & Ajzen, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Reading, Addison-Wesley.

Fritz, F., A. Susperregui & M.T. Linaza (2005), Enhancing Cultural Tourism Experiences with Augmented Reality Technologies. In Proc. of the 6th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage VAST2006.

Noor, N.L., M. Hashim, H. Haron, & S. Ariffin (2005). Community Acceptance of Knowledge Sharing System in Travel and Tourism Websites: an Application of an Extension of TAM. In: Proc. of the European Conference on Information Systems.

Novak, T.P., Hoffman, D.L. & Y.-F.- (2000). Measuring the Customer Experience in Online Environments: A Structural Modeling Approach. Marketing Science 19(1): 22-42.

Rasinger, J., Fuchs, M., Höpken, W. & Tuta, M. (2006). A Customer Based Approach to discover Accepted Mobile Information Services in Tourism. In: Research@EyeforTravel, London, 5-6 June, pp. 31-46.�

Acknowledgements

This paper is part of the project "PRISMA- Augmented vision for Cultural Tourism Applications", financed by The Movie Virtual and Ereiten Kultur Zerbitzuak within the INTEK program of the Department of Industry of the Basque Government.