Formative Evaluation and Preliminary Findings of a Virtual Reality Telerehabilitation System for the...

Judith A DeutschResearch in Virtual Environment AndRehabilitation Sciences (RiVERS) LabDepartment of Developmental andRehabilitative SciencesUniversity of Medicine and Dentistryof New JerseySchool of Health Related ProfessionsNewark New Jerseydeutschumdnjedu

Jeffrey A LewisResearch in Virtual Environment andRehabilitation Sciences (RiVERS) LabDepartment of Developmental andRehabilitative SciencesUniversity of Medicine and Dentistryof New JerseySchool of Health Related ProfessionsNewark New JerseyandVR LABCenter for Advanced InformationProcessingRutgers UniversityPiscataway New Jersey

Elizabeth WhitworthDepartment of Information SystemsNew Jersey Institute of TechnologyNewark New Jersey

Rares BoianVR LABCenter for Advanced InformationProcessingRutgers UniversityPiscataway New Jersey

Grigore BurdeaVR LABCenter for Advanced InformationProcessingRutgers UniversityPiscataway New Jersey

Marilyn TremaineDepartment of Information SystemsNew Jersey Institute of TechnologyNewark New Jersey

Presence Vol 14 No 2 April 2005 198ndash213

copy 2005 by the Massachusetts Institute of Technology

Formative Evaluation andPreliminary Findings of a VirtualReality Telerehabilitation Systemfor the Lower Extremity

Abstract

Usability studies are an essential and iterative component of technology develop-ment and ease its transfer from the laboratory to the clinic Although such studiesare standard methodology in todayrsquos graphical user-interface applications it is notclear that current methods apply to new technologies such as virtual reality Thusexperimentation is needed to examine what existing methods can be viably trans-ferred to the new user-interaction situations In this paper 5 integrated interfaceswith 3 simultaneous users are evaluated via a set of usability studies which adapttraditional methods for assessing the ease of use of the interface design A singleexpert domain user was run in an intensive study that examined the therapist man-ual and interfaces of the Rutgers Ankle Rehabilitation System (RARS) The interfaceand manual were extensively modified based on this evaluation A second studyinvolving 5 therapists was then conducted to evaluate the telerehabilitation compo-nent of the RARS system In both studies the tester and developerrsquos observationsalong with the session videotapes and therapist-user questionnaires were triangu-lated to identify user problems and suggest design changes expected to increasethe usability of the system Changes that resulted from the analysis with the domainexpert are described and recommendations for how to conduct usability studies insuch multiuser remote virtual reality situations are proposed Results from the pilotusability telemonitoring studies are also presented The validity of usability studies inthe development and refinement of rehabilitation technology is highlighted

1 Introduction

A key usability engineering methodology used to evaluate how easily theuser understands and employs the application is called a usability study Theiterative application of usability studies to the design process serves as an inte-gral component to the generation of easier-to-use interfaces (Mayhew 1992Nielsen 1993)

Standard methods for conducting usability studies are well documented anddescribed in basic human-computer interaction textbooks (Preece Rogers ampSharp 2002) Typical usability studies involve giving a user a real task to per-form while recording the process in some fashion (usually videotaping) (Man-tei amp Teorey 1988) Literature on usability-study operation has focused onthe evaluation of web-page usage (Cato 2001) and the evaluation of single-user interfaces (Mayhew 2002) Assessing the usability of rehabilitation inter-

198 PRESENCE VOLUME 14 NUMBER 2

faces involving multiple users virtual reality exercisesremote connections and integrated processes still lacksstandard approaches A usability methodology involvingvirtual reality navigation tasks was recently proposed(Hix amp Gabbard 2002) which involved the traditionalfour stages User task analysis expert-guided-based eval-uation formative usability evaluation and finally sum-mative comparative evaluation Elements of Hixrsquos meth-odology are applied in this paper

The design of any rehabilitation technology requiresan assessment and understanding of how the technologywill be used and deployed This is especially criticalwhen transferring technology from a research laboratoryto clinical use where the technology must be robustand easy to learn and use since the focus is on the careof the patient To achieve this goal in the developmentof the Rutgers Ankle Rehabilitation System (RARS) aset of specific methodologies called usability engineeringwas applied This paper describes how usability testingand software design iteration were performed collabora-tively by a group of engineers and clinician scientistsKey issues in this description are the adaptation of stan-dard usability methodology to assess the ease of use of avirtual reality and related telerehabilitation system Find-ings from a formative evaluation consisting of an early-stage usability evaluation with an expert domain userfollowed by pilot usability studies run on 5 physicaltherapists using the remote-monitoring system and theresulting modifications to the system are reported here

2 Brief Description and Application ofthe Technology

The Rutgers Ankle Rehabilitation System (RARS)is a VR-based rehabilitation environment for patientshaving lower-extremity dysfunction As shown in Figure1a the patient sits in the chair and places the affectedlower extremity in a boot mounted on a platform thathas six degrees of freedom By moving their feet pa-tients manipulate a virtual object presented on the com-puter screen In the particular VR rehabilitation applica-tion described here the platform serves as a ldquojoystickrdquoused for flying an airplane through 3D targets The

therapist can set a variety of parameters that match thepatientrsquos therapy needs In addition the therapist canview these parameters remotely and interact with thepatient through a telemonitoring system that includes agraphics display a camera for observing the patient andan audio channel for communication

The RARS consists of local hardware and softwarecomponents as well as a remote-monitoring subsystem

Figure 1 RARS system overview (a) patientrsquos station (b) remote-

monitoring station copy UMDNJ Rutgers 2003

Deutsch et al 199

The key hardware element is a small parallel-kinematicsrobot in which patients place their feet The main localsoftware component is a library of virtual reality simula-tions written in WorldToolKit (Sense8 Co 1998) Theremote-monitoring station is a standard PC with avideo audio and network connection with the local PCdriving the simulation seen by the patient The RARSincludes a pan-tilt-zoom (PTZ) camera and softwareused for remote monitoring The remote-monitoringscreen runs a Java 3D (Sun Microsystems 2002) graph-ics applet which follows in real time the local VR exer-cise

A therapist in a remote location can manipulate thecamera to see the patient or the patientrsquos lower extrem-ity (Figure 1b) The current exercise configurations andperformances are detailed on the monitoring applet andmimic the information that the local therapist can setand examine both numerically and graphically The cen-ter of the Java 3D applet shows several views of the an-klersquos relative motion to the leg as well as a 3D mock-upof the actual exercise simulation The RARS and telere-habilitation subsystem are described in detail elsewhere(Girone Burdea Bouzit Popescu amp Deutsch 2000Girone Burdea Bouzit Popescu amp Deutsch 2001Boian Lee Deutsch Burdea amp Lewis 2002 BoianDeutsch Lee Burdea amp Lewis 2003 Lewis BoianBurdea amp Deutsch 2003)

The system was developed collaboratively by a clini-cian scientist (JED) and several engineers (GB MGRB JAL) It was designed for physical therapists to al-low them to apply principles of exercise in designing atreatment session Thus with basic knowledge of thesystem and the parameters that can be modified to cre-ate exercises of varying difficulty levels any physicaltherapist can use the system with a variety of patientsThe system is currently designed for individuals whohave lower-extremity dysfunctions that interfere withtheir functional mobility for use in a sitting position Ithas been used to rehabilitate individuals who have hadankle sprains or fractures (Girone et al 2000 DeutschLatonio Burdea Boian 2001a) and poststroke individ-uals to address neuromuscular deficits that have inter-fered with ambulation (Boian et al 2002 DeutschLatonio Burdea Boian 2001b)

The software evaluated in this study consists of fourinterfaces used sequentially to set a patient baseline(Figure 2a) configure an exercise session (Figure 2b)supervise and modify the exercise task in the virtual en-vironment (Figure 2c) and monitor the patient re-motely (Figure 2d) The user interfaces are listed belowas they appeared in the therapistrsquos training manualwhich was the fifth interface evaluated in this study

3 Testing Usability

Usability was tested in two phases first in a forma-tive evaluation by a domain expert and subsequently ina pilot study of the telemonitoring system using 5 physi-cal therapists The methods used in both phases of thestudy were the same In this section of the paper wepresent (a) an overview of the study design (b) chal-lenges faced in testing usability (c) detailed descriptionof the study protocol and (d) how this study differsfrom other usability studies

31 Overview of Study

Testing during the formative evaluation and re-mote telemonitoring pilot study was conducted overtwo sessions The setting and terminology describingthe study design are presented in Figure 3

In the first session therapist-users learned how to usethe RARS at the rehabilitation site They alternated be-tween reading the manual (one of the interfaces) usingthe hardware and practicing with the other four inter-faces (shown in Figure 2) Training included (a) firstpositioning a patient in the chair and establishing abaseline (shown in Figure 2a) (b) configuring an exer-cise (using the screen shown in Figure 2b) requiringsetting parameters which include platform range ofmotion platform resistance exercise time airplanespeed air turbulence visibility and presence or absenceof haptic effects and (c) monitoring exercise perfor-mance in real time (see Figure 2c) Therapists practicedon themselves first and then had an opportunity to trythe RARS with an experienced user The session wasoverseen at the rehabilitation site (also called the local


site) by a usability engineer and at the remote site by asystem developer (see table in Figure 3)

In the second session therapists returned and weretested on their retention of knowledge about the RARSTesting was done using a naıve patient at the rehabilita-tion (or local) site Therapists were then asked to moveto the remote site An experienced patient and local

therapist remained at the rehabilitation site while thetherapist-user and the usability engineer moved to theremote site At the remote site therapists were in-structed in the use of the telemonitoring system andhad an opportunity to remotely monitor the patient-user and local therapist at the rehabilitation site (seeFigure 1b) Features of the system related to storing

Figure 2 The RARS software components (prepilot) (a) Baseline screen (b) Configuration screen (c) VR exercise simulation seen by the

patient and local therapist (d) screen view of the remote-monitoring station copy UMDNJ Rutgers 2003

Deutsch et al 201

patient-exercise data in online databases and interpret-ing these data were not included in this study

During both sessions a usability engineer interactedwith the patient and therapist-user Both sessions werevideotaped Questionnaires were administered at theend of each session to capture the therapistrsquos impres-sions about ease of use of the system

32 Specific Challenges Faced inTesting Usability

Typically a usability study involves users who areasked to interact with software and to perform a set oftasks that they would normally carry out using the sys-tem (Nielsen 1993) A record is made of the usersrsquo in-teraction either via tester coding audio recording

transaction logging or videotaping Often more thanone such data-collection method is used In the presentstudy the same procedures were used except that testingwas performed simultaneously or in sequence on fiveintegrated user interfaces (a training manual baselinetesting exercise configuration exercise testing and re-mote monitoring) and with multiple users (a local ther-apist a remote therapist and a patient) Normally eachof the interfaces (except for the training manual) istested in an unrelated usability study However giventhe manner in which rehabilitation is practiced it wasimportant to maintain the tight coupling between eachof the interfaces and the patientrsquos therapy Therefore aset of integrated tasks had to be designed These tasksrelated the user interfaces to each other This created aproblem of separating the effects of one interface study

Figure 3 The usability model for the rehabilitation and remotes sites in Session I instruction and practice with RARS at the rehabilitation site

and in Session Two retention testing on RARS at the rehabilitation site and instruction and practice with remote monitoring at the remote site

copy UMDNJ 2003


on the next interface evaluation Because of this meth-ods were needed to sort out the interactions betweenthe integrated user interfaces

To run a realistic usability study one needs to use realpatients Normally usability engineers conduct usabilitystudies but such individuals would not have experiencein patient therapy Thus they would not be able to makecorrective suggestions if the therapist-user in the studygenerated an error due to a system-interface misunder-standing Having the therapist-designer of the applica-tion present solved this problem but this approach isnot normally accepted in usability evaluations because ofthe possible unintended effect on the user in the study

A challenge also unique to this study was how toevaluate a remote-monitoring interface Specifically itwas important to ascertain whether the remote therapisthad enough information about the patient to accuratelydirect the remote therapy session Part of the difficultyin evaluating the effectiveness and usefulness of the pa-tient information provided through remote monitoringwas that the therapist was new to a complex systemHere the effect of being a novice RARS user is con-founded with the usefulness of the remote informationprovided Thus the data collected on this user may notreflect the experiences of more experienced users

A final challenge was the number of users involved inthe interaction The local sessions have two users atherapist-user and a patient-user The remote monitor-ing involves three users the therapist-user at a remotelocation a local therapist (or therapist assistant) and thepatient-user Although multiuser evaluation sessionsencourage users to talk to each other about the prob-lems they are having with the interface some data arelost as the number of users increases as one user maysolve a problem before it can be observed and solved bythe other two users (Sullivan 1991)

33 Methods Used in the Present Study

331 Sessions The two usability studies wereconducted on the interface over two days and in severalstages The protocol was the same for both studies Inthe first usability study there was an expert domainuser in the second usability study there were 5 physical

therapists A training manual was written by two systemcodesigners (JAL and JED) in which the RARS systemand its telerehabilitation component are described Themanual was designed as a teaching tool to be used overtwo sessions It was 30 pages long with figures on theleft side and text in bullet form on the right side It con-tained questions and exercises to stimulate learning

In the first session the therapist-users read the man-ual and practiced on the system Therapists completedthe following sections System Overview Haptic Inter-face Patient Set Up Session Configuration and Exer-cise Training First therapists were given an overview ofthe system and an explanation of the main exercisewhich consisted of using the foot as a joystick to ldquoflyrdquoan airplane through successive targets (such as the lightsquare shown in Figure 2c) Therapists were given theopportunity to ldquoflyrdquo the airplane to experience the exer-cise being administered to the patient Second thera-pists were instructed on how to set the system parame-ters A pseudopatient who knew the VR system wasused in this stage of the study Therapists learned to usethe system by first positioning the experienced patient inthe chair and then establishing the baseline parameters(the screen for setting these parameters is shown in Fig-ure 2a) The baseline parameters were captured by ask-ing the patients to move their feet into plantardorsi-flexion and inversioneversion as far and as fast aspossible in order to establish the patientsrsquo torque andrange-of-motion capabilities This is needed so that thesubsequent exercises configured for the patient neitherexceed the patientrsquos capabilities nor fail to challenge thepatient Third therapists then learned how to configurean exercise for the patient (the screen for this is shownin Figure 2b) This task included setting the platformrange of motion platform resistance placement of tar-gets alignment in space (horizontal vertical or combi-nation) exercise duration airplane speed amount of airturbulence amount of visibility and presence or ab-sence of haptic effects for example the sensation of ajolt when a target frame is inadvertently hit by the air-plane Varying these parameters allows the therapist toconfigure different types of exercises such as strength-ening or endurance and varying levels of difficultyTherapists were then given several clinical scenarios and

Deutsch et al 203

asked to oversee the experienced patient using the simu-lation At the end of the session the therapist-userscompleted two questionnaires a standard usability ques-tionnaire (IBM Ease of Use Survey 2003) and an in-strument-specific questionnaire that tested knowledgeabout the RARS system

In a second session therapists were tested on theirretention of knowledge by using a naıve patient Theywere required to set the patient up perform a baselineand then configure exercises based on specific criteria(eg increase strength or endurance) Therapist-usersreviewed the training manual for 15 minutes and wereallowed to ask questions of the usability engineer aboutthe manual content Therapist-users then proceeded toposition and train a naıve patient This required settinga baseline and configuring several exercises For parttwo of the second session therapist-users worked at theremote site Guided by the manual therapist-users thenfollowed the tutorial on the remote-monitoring systemand had an opportunity to direct a session remotely Inthe remote-monitoring session therapist-users wereinstructed on how to use the PTZ camera to view differ-ent aspects of the local scene They learned as well tocommunicate with the local therapist and the patient byusing the audio system and the web-chat window Thelocal therapist established the patientrsquos baseline and setup an exercise while the remote therapist monitored thesession Deliberate errors were made by the local thera-pist to probe the remote therapistrsquos attention A 10-minute break while the camera was set and communica-tion was checked was provided before the remotetherapists began a treatment session The local therapistconfigured the baseline and the remote therapist in-structed the patient on how to use the system and whatto do during a strengthening and coordination exerciseThe remote therapist was required to adjust the exerciseparameters (by communicating with the local therapist)during the simulation to maximize the patientrsquos perfor-mance At the conclusion of the session a usabilityquestionnaire was administered for the remote-monitoringcomponent of the system

During both sessions a usability tester (EW) inter-acted with the therapist and the patient while collectingvideo data from the screens and making notes of her

observations One of the system developers (JED) re-motely monitored the sessions also taking notes andserving as a resource to the tester if questions about thesystem and its use arose during testing Throughoutboth sessions users were required to talk aloud aboutwhat they were experiencing within the system Talkingaloud is a well-established method in usability testingthat gives experimenters a better understanding of theuserrsquos thought processes This method allows the cap-ture of cognitive difficulties that occur in real time thatmay be forgotten or even pass unnoticed by the user

332 Formative Evaluation Only a single ex-pert domain therapist was used in this first usabilitystudy designated as the formative evaluation This per-son represented someone who was familiar with thetypes of therapy that RARS was designed to provideand who could therefore give us feedback on not justhow usable the interface was for her but also on howusable the interface was for the patient and for the inte-gral task of the therapist working with the patient Inaddition this person because of her knowledge of thefield could give us viable feedback on how well the us-errsquos manual would be received and understood by othertherapists

333 Remote-Telemonitoring Pilot Study Inthe next set of usability studies that were run 5 addi-tional physical therapists were trained in the use ofRARS following the protocol described above The re-sults of pilot usability studies conducted on the telereha-bilitation part of the study are reported here In thisfinal stage that was run at a separate time from the ini-tial training the remote-monitoring interface was de-scribed to the therapist and used in a second exercisesession with the pseudopatient The remote-monitoringscreen is shown in Figure 2d Features and interfaces ofRARS related to storing and interpreting the data fromthe exercises were not included in this study The video-tape of the expert user was reviewed for a gestalt im-pression of the therapist-usersrsquo problems and data fromthe testerrsquos notes developerrsquos notes and user question-naires were triangulated to identify the strengths and


limitations of the five integrated user interfaces for theformative evaluation

34 How This Studyrsquos Methods DifferFrom Existing Usability Studies

Although common usability methods were used inthis study as indicated earlier the study involved severaldifficulties because of its multiuser multi-interface andremote-testing aspects The common methods were theuse of videotaping experimenter coding during thestudy and the completion of user-satisfaction question-naires following the study In addition to this approachthe following methods were applied First the entire setof user studies was embedded into a single system man-ual which was used as a guide for the therapist-userThus the therapist-user could review other parts of thesystem during the process of learning the new featuresThe entire manual was used because this best simulatedthe real-life situation in a clinic where therapists have asingle complete manual

Second the studies involving the local RARS compo-nents were run with one of the interface designers(JED) remotely monitoring the subject Although re-mote monitoring has been conducted in other usabilitywork this is usually done when no experimenter is inthe room A key problem with application designersbeing present during usability studies is the possibilityfor their nonverbal and subverbal behavior to influencethe user being tested Nevertheless in a study wheretherapy is being applied to another human through theinterface it is essential to have the advice of the therapist-designer who has a deeper understanding of the ther-apy than a usability engineer Having the remote-observation capability of RARS permitted the therapist-designer to be ldquopresentrdquo without being in the sameroom and thus unduly influencing the subject in thestudy An added advantage was the ability to have a sec-ond remote note taker

Third a set of true-false questions to assess howmuch the therapist in the study understood about theoperations of the system was administered to the userA verbal report of the problems with the system wasrecorded on video and through experimenter coding

However this report did not ascertain what problemswere due to a misunderstanding of the interface by theuser or due to the cognitive complexity of the task thatis handling a patient and in the remote session com-municating to an assistant therapist Thus althoughcognitive-comprehension questionnaires are used in us-ability work this set was used specifically to evaluate thedepth of problems found in the video and coded datacapture

Finally the therapist-user was required to explain theusage of the system to both the patient and to the localtherapist in the study This is the most unique of thenew approaches that were taken in this study Having auser explain what this person has just learned can give ausability engineer much more detail about what is mis-understood omitted or misrepresented by the userThe data captured with this technique was especiallyuseful for the evaluation of the remote session Becausethe remote therapist-user not only explained to the localtherapist what steps to take in adjusting therapy butalso gave reasons for the system adjustments it could bedetermined how well the therapist-user understood thepatient-user problems via the remote monitoring

4 Results

41 User Questionnaires

411 Formative Evaluation Table 1 summa-rizes the expert domainrsquos ratings of the systemrsquos ease ofuse during the formative evaluation The therapist-usergenerally rated the RARS system positively on ldquocomfortof userdquo ldquoease of learningrdquo ldquoability to effectively com-plete workrdquo ldquoorganization of information on the sys-temrdquo ldquoenjoyment and pleasantness of interfacesrdquo andldquooverall satisfactionrdquo These items received scores be-tween 6 and 7 with 7 being the highest rating on thescale Items that received lower ratings were ldquoability tocomplete work efficiently and quicklyrdquo ldquothe systemrsquosability to signal the user when they have made an er-rorrdquo and the ldquoinformation provided for the system be-ing easy to understandrdquo These features received scoresbetween 3 and 4

The therapist-userrsquos response to open-ended ques-

Deutsch et al 205

tions about the positive aspects of the system includedldquoenjoyment of the exercisesrdquo ldquoflexibility of the systemthat allowed customizing the exercises for the patientrdquoand ldquothe ease with which [heshe] learned the systemrdquoAccuracy for responses to the RARS true false ques-tions was 87 The results for the telemonitoring com-ponent were slightly lower than those for RARS as awhole with 6 being the highest rating The therapist-user rated the organization enjoyment and functional-ity of the remote-monitoring screen as a 5 Positivecomments were made about the ldquoopportunity to directtreatment from a remote locationrdquo and the ldquoability tomonitor the patient from various viewpointsrdquo includingfocusing on the area of treatment in this case the ankleIn addition the therapist stated ldquoWith some clarifica-tion the system is easy to figure out especially with aprior training sessionrdquo Specific comments on the nega-tive aspects of the system were regarding the time lag ofthe voice feedback (microphones) the inconsistency ofbuttons between the local and the remote interface andthe difficulty of multitasking especially during remotemonitoring where the therapist-user would be typing in

the chat box engaging in verbal interactions with thelocal therapist and monitoring patient performance atthe same time

412 Remote-Telemonitoring Pilot StudyTable 2 summarizes the 5 therapistsrsquo responses to theusability of the telerehabilitation system Responseswere recorded using a 7-point Likert scale withldquostrongly disagreerdquo rated as a 1 to ldquostrongly agreerdquorated as 7 The lowest ratings were for the systemrsquos abil-ity to provide information on errors The highest ratingswere for liking the use of the interface There was vari-ability in the responses with ratings ranging from 2 to 7for comfort and ease of use of the system

Feedback was also elicited from the therapists usingopen-ended questions such as ldquoWhat were the mostpositive and negative aspects of the systemrdquo Therapistscommented on issues related to communication Theywere distracted by the lag between the remote and thelocal site and complained about the quality of the audiocommunication They stated as well that the web-chatfeature was not responsive to real-time needs One ther-apist commented that the multiple-screen layout wastoo complicated to manipulate and another similarlysaid that a larger screen would be beneficial Requestswere made to have information available from the base-line configuration which could not be monitored re-motely as well as to be able to remotely control thesettings for the session

42 Tester and Developer ObservationsResulting in Interface Changes

This section describes the observations of thetester and developer coupled with support from thetherapist-user comments made during the video ses-sions These observations were used to make changes tothe therapist-training manual and the interfaces Themanual lacked clarity and its construction producedcognitive overload for the user To address these issuesthe manual format was changed reducing the informa-tion on each page and labeling and numbering figuresto make it easier for the reader to follow sequential in-formation (see Figures 4a 5 6b)

Table 1 Formative Evaluation Questionnaire Scoresby Category

Categories RARS Monitor

Comfort of use 6 6Ease of learning 6 6Ability to effectively complete work 7 6Organization of information on the

system 7 5Enjoyment and pleasantness of

interfaces 7 5Ability to complete work efficiently

and quickly 4 5The systemrsquos ability to signal the

user when they have made anerror 3 NA

Information provided for thesystem being easy to understand 4 5

Overall satisfaction 7 6


Terminology was inconsistent across four of the inter-faces and was not the preferred terminology of a physi-cal therapist For example ldquoDisplacementrdquo was usedwhen the term ldquoRange of Motionrdquo was preferable as thecommon term used in physical therapy (Figures 4a 56ab) Inconsistent placement of the same terms in dif-ferent parts of the interface also caused confusion Forexample the term ldquopitchrdquo preceded the term ldquorollrdquo ontwo of the screens and then the order was reversed to

ldquorollrdquo and ldquopitchrdquo This proved to be distracting andwas corrected (see Figure 6c)

The ldquomappingrdquo between the buttons and the tasksthey performed (see Figure 4b) for the buttons labeledldquoStartrdquo ldquoSaverdquo ldquoConfigurerdquo and ldquoExitrdquo was unclear tothe users It was unclear partially because the buttonswere designed with words that involved large concep-tual meanings for example ldquoStartrdquo seemed to have thesame function as ldquoConfigurerdquo and ldquoSaverdquo seemed to

Table 2 Remote-Monitoring Questionnaire Results (each initial corresponds to one of the 5 therapists participatingin the study)

Usability of remote-monitoring items and ratings 1 2 3 4 5 6 7

Overall I am satisfied with how easy it is to use this system C A E D BIt was simple to use this system C A E D BI can effectively complete my work using this system A C B E DI am able to complete my work quickly using this system A B E C DI am able to efficiently complete my work using this

system A B C E DI feel comfortable using this system C A E D BIt was easy to learn to use this system C A E D BI believe I became productive quickly using this system A C E B DThe system gives error messages that clearly tell me how to

fix problems A D EWhenever I make a mistake using the system I recover

easily and quickly A C B E DThe information (such as on-screen messages and other

documentation) provided with this system is clear A E B C DIt is easy to find the information I needed E A B C DThe information provided for the system is easy to

understand E A B C DThe information is effective in helping me complete the

tasks and scenarios A B C E DThe organization of information on the system screen is

clear A C E B DThe interface of this system is pleasant E B C A DI like using the interface of this system C E B D AThis system has all the functions and capabilities I expect it

to have A B E C DOverall I am satisfied with this system B E C D A

Deutsch et al 207

have similar functionality to ldquoExitrdquo These buttons rep-resented a sequence of tasks that were to be performedby the user to establish the baseline parameters for thepatient Some of the buttons had no meaning until aportion of the task was accomplished Thus it wouldhave been more appropriate to design the buttons sothat they matched the sequence of procedures the userwas to take when setting the baseline and even tochange their size and accessibility to make the processvisible to the user

Making procedures more apparent within the inter-face by reorganizing the presentation of the functionssolved this command-structure problem The new de-sign involved three large buttons ldquoStartrdquo ldquoSaverdquo andldquoNextrdquo in a line and a small ldquoXrdquo button to quit thescreen (see Figure 4b) The order and direction of the

buttons in the revised interface correspond to the ac-tions that the therapist-user is supposed to take to com-plete a patient baseline The ldquoConfigurerdquo button whichwould take the user to the next screen to configure anexercise was changed to ldquoNextrdquo being an easily under-stood term and a more obvious indicator that the userwould be taken to a new screen The button that wouldexit the user from the screen was made smaller and sep-arated from the other buttons making it less likely thatusers would click on the exit button by mistake

Another problem involved the toggle switch in theexercise screen where the system started off paused sothat the patients could orient themselves with the air-plane they were supposed to pilot with their ankles andbe ready for the exercise In the initial design the userwould have to press the ldquoPauserdquo button again in orderto start the system which is not very intuitive In orderto make the process more intuitive the previously largeldquoQuitrdquo button was reduced to a small ldquoXrdquo at the bot-tom right of the screen and the space created used toinclude a large ldquoStartrdquo button (see Figure 6d)

Observations of the remote-monitoring componentof RARS highlighted several of its limitations Therewas a reliability issue with the audio communication andaudio feedback The order of the elements on the re-mote screen did not match that on the local interfacesmaking it more difficult for the user to learn Thetherapist-user did not consistently read the messagespresented on the chat window which is attributed tohaving to attend to three elements the VR exercise sim-ulation chat window and camera view of the patientAs a result of these difficulties the terms used on theremote-monitoring station were altered and reorderedbefore the 5 therapists were introduced to the system

Modifications made after the formative evaluationwere successful in improving use of the remote-monitoringsystem however other areas required change To re-duce confusion a foot side view was added to each re-mote screen The parameters were color-coded to iden-tify those that could be changed (gray background) andthose that could not (blue background) (Figures 7a and7b) To increase the capabilities of the remote thera-pists screens will be developed to allow for baseline andexercise configurations to be viewed and controlled re-

Figure 4 Interface modifications (a) baseline diagram in manual

clarifies the screenrsquos elements (b) command modification of baseline

screen (the ldquoConfigurerdquo button was changed to ldquoNextrdquo and the

ldquoEXITrdquo button was minimized) copy UMDNJ Rutgers 2003


motely The red letters on the black background will berevised for visual clarity This change as well as the newremote capabilities of the system will be evaluated infuture usability studies

5 Discussion

A formative evaluation of the RARS was con-ducted adapting existing usability-study methodologyThe findings from the testerrsquos notes videos user ques-tionnaires and developerrsquos notes found key problemswhich required modifications Changes were made tothe user interfaces to be consistent with the therapistsrsquolanguage and use (Preece et al 2002) Small orderchanges on the screen were made to reduce the diffi-culty the user-therapists had in finding informationtherefore consistency in layout was implemented (Teitel-

baum amp Granda 1983) The command structure wasaltered to better reflect the clinical decision-making pro-cess The toggle switch which started the machine in astate that the user did not specify and then required atoggle switch to alternate between states was also mod-ified (Norman 1988) The most substantial changeswere made to the therapist-training manual to reducethe confusion experienced by the reader

While all these changes served to refine and easethe use of the system it is noteworthy that the expertdomain user rated her experience with the ldquocomfortof userdquo ldquoease of learningrdquo ldquoenjoyment of the inter-facesrdquo and ldquooverall satisfactionrdquo as highly satisfyingThe aspects of the system that received lower ratingswere related to the training manual and its use Im-portantly the therapist-user was able to correctly ap-ply principles of exercise to create effective exercisesessions for an experienced patient-user and knowl-

Figure 5 Configuration diagram in manual clarifies the screenrsquos elements copy UMDNJ 2003

Deutsch et al 209

edge retention of the system was 87 after using thesystem only once

Substantial changes to the manual and user-integratedinterfaces were made after the formative usability evalua-tion Modifications to allow for language consistencymade to the telerehabilitation component after the for-mative evaluation were successful Subsequently the 5therapists in the pilot study of the remote-monitoringsubsystem did not have complaints about languageThis reinforces the value of an iterative process in devel-oping interfaces and the usefulness of a formative evalu-ation

Visualization of the multiple screens used to monitora remote session will require improvements The currentsystem is not integrated as the remote user has to openand monitor three independent screens Possible solu-tions are synchronizing these screens as a single applica-tion that could be opened and resized simultaneouslyIn addition more time allocated for therapist-user train-ing may be required than that provided in this study

Communication with the local therapist and patientalso remains an area for improvement The system thatwas developed for these studies also relied on a chatmechanism that did not elicit the local therapistrsquos atten-tion and thus caused delays in communication Al-though improvements to the audio communicationwere made following the formative evaluation (use ofheadset by remote therapist) the pilot study demon-strated the need for further refinement Delays in thetransmission of the audio communication could be at-tributed in part to the Local Area Network (LAN)which mediated the communication The lack of reliableand fast communication infrastructure plagues teleworksystems in general and it is expected that delays willdiminish with the (future) introduction of faster andmore reliable networks as part of medical informaticsinfrastructure

51 Limitations of the Current Study

This study was a single formative-evaluation studyin which usability of the RARS was tested using the ad-vice of a single expert domain user and usability of thetelerehabilitation component was tested using 5 physicaltherapists Only the key features of the system weretested because the focus of the study was in understand-ing how the interface would work in the multiattentivetask of patient therapy and monitoring During the tele-rehabilitation tasks video information was gatheredonly at the remote-monitoring site because of the short-age of qualified experiment personnel and equipmentCollecting data from the local therapist would involvecollecting video data and having a second camera per-son present on the local therapistrsquos side of the interac-tion Collecting data from the patient-user within thesame interaction would prove even more difficult As

Figure 6 Interface changes to the exercise screen following the

study (a) terminology modification (b) simulation diagram in manual

clarifies the screenrsquos elements (c) ordering modification for

consistency (d) command modification (ldquoQuitrdquo button was minimized

and replaced with a ldquoStartrdquo button) copy UMDNJ 2003


Figure 7 Remote-monitoring screen modifications (a) original monitoring window (b) modified monitoring window the font size of the display

on the left-hand side was made larger The parameters that could not be changed were highlighted in blue and those that could be changed

were highlighted in gray The order of the parameters was made consistent with the exercise screen (see Figure 6) copy UMDNJ Rutgers 2003

Deutsch et al 211

described earlier it is important to hear users talkthrough their ideas However with both the patient-user and the local therapist in the same room havingboth talk aloud would be cumbersome if not entirelyunnatural Lack of sophisticated video capture equip-ment and trained users (eg the assistant therapist) toserve in secondary roles in the study restricted some ofthe control of the study and the information that couldbe captured

An additional limitation is the design of the telereha-bilitation study which was conducted in a single ses-sion The 5 therapists may not have had adequate timeto learn the new system completely and there was noopportunity to measure retention of knowledge for eachindividual stage of the system learning

6 Conclusions and Future Work

The five integrated interfaces of the RARS wereevaluated first by an expert domain user Based on theexpert therapistrsquos report the RARS and its telerehabili-tation component were easy to use and consistent withher style of therapy practice Many modifications wereprompted by this formative evaluation validating usabil-ity studies as an integral component of design and im-plementation of rehabilitation technology In additionthe modifications made to traditional usability methodsaided in capturing data necessary for the interface evalu-ation and helped to explain the meaning of data thatwere confounded by the complexity of the study

The remote-monitoring system was subsequentlytested on 5 therapists from different practice areas andwith varying levels of experience In anticipation of thisstudy the training manual was modified for clarityModifications from the initial formative evaluation easeduse during the subsequent part of the study Areas thatstill need to be improved in the telerehabilitation com-ponent related to communication with the local thera-pist and patient as well as visualization of the remote-monitoring scene Controlling all aspects of the sessionremotely will reduce the need for a local therapist andperhaps increase the capabilities of the system

There remain other aspects of the RARS system and

its remote-monitoring component that warrant furtherstudy Patient data is stored by the system transparentlyin an Oracle database which is then accessed by thetreating physician or therapist using a web interface por-tal (Boian et al 2002) The database portal and its easeof use to generate graphs and to follow patient thera-peutic history remains to be studied Another aspectthat requires more attention is the patientrsquos subjectiveevaluation of a telerehabilitation intervention in generaland one using the RARS in particular Finally as theremote-monitoring station is placed farther from thelocal site the influence of Internet-based communicationwill become an important focus area for further research

Acknowledgments

Research reported here was supported by grants from the Na-tional Science Foundation (BES-0201687) and the New Jer-sey Commission on Science and Technology (RampD Excel-lence Grant)

References

Boian R F Deutsch J E Lee C S Burdea G amp LewisJ A (2003) Haptic effects for virtual reality-based post-stroke rehabilitation Proceedings of the Eleventh Symposiumon Haptic Interfaces For Virtual Environment and Teleop-erator Systems 247ndash253

Boian R F Lee C S Deutsch J E Burdea G amp LewisJ A (2002) Virtual reality-based system for ankle rehabili-tation post stroke Proceedings of the First InternationalWorkshop on Virtual Reality Rehabilitation (Mental HealthNeurological Physical Vocational) 77ndash86

Cato J (2001) User-centered web design London Addison-Wesley

Deutsch J E Latonio J Burdea G amp Boian R (2001a)Rehabilitation of musculoskeletal injuries using the Rutgersankle haptic interface Three case reports Proceedings of Eu-rohaptics 11ndash16

Deutsch J E Latonio J Burdea G amp Boian R (2001b)Post-stroke rehabilitation with the Rutgers ankle systemmdashAcase study Presence Teleoperators and Virtual Environ-ments 10(4) 416ndash430


Girone M Burdea G Bouzit M Popescu V G ampDeutsch J (2000) Orthopedic rehabilitation using thelsquoRutgers anklersquo interface Proceedings of Virtual RealityMeets Medicine 2000 89ndash95

Girone M Burdea G Bouzit M Popescu V G ampDeutsch J (2001) A stewart platform-based system forankle telerehabilitation Autonomous Robots 10 203ndash212

Hix D amp Gabbard J (2002) Usability engineering of vir-tual environments In K Stanney (Ed) Handbook of virtualenvironments Design implementation and applicationsMahwah (pp 681ndash699) NJ Erlbaum

IBM (2003) IBM ease of use survey Available from httpwww-3ibmcomibmeasyeou_extnsf Publish2003

Lewis J A Boian R F Burdea G C amp Deutsch J E(2003) Real-time web-based telerehabilitation monitoringProceedings of Medicine Meets Virtual Reality 11 190ndash192

Mantei M amp Teorey T J (1988) Costbenefit analysis forincorporating human factors in the software lifecycle Com-munications of the ACM 31 4 428ndash439 Paper repub-lished in B Boeinm (Ed) Software Engineering Los Ange-les Computer Society Press 1989

Mayhew D J (1992) Principles and guidelines in softwareuser interface design Englewood Cliffs NJ Prentice-Hall

Mayhew D J (2002) The usability engineering lifecycle Apractitionerrsquos handbook for user interface design San Fran-cisco Morgan Kaufman

Nielsen J (1993) Usability engineering New York AcademicNorman D A (1988) The psychology of everyday things New

York Basic BooksPreece J Rogers Y amp Sharp H (2002) Interaction design

Beyond human-computer interaction New York WileySense8 Co (1998) WorldToolKit release 8 technical overviewSullivan P (1991) Multiple methods and the usability of in-

terface prototypes The complementarity of laboratory andfocus groups Proceedings of the ACM Conference on SystemsDocumentation 106ndash118

Sun Microsystems (2002) The Java 3D API SpecificationJava 3D vl 3 June 2002

Teitelbaum R C amp Granda R E (1983) The effects ofpositional constancy on searching menus for informationProceedings of the ACM CHIrsquo83 Conference on Human Fac-tors in Computing Systems 150ndash153

Deutsch et al 213

faces involving multiple users virtual reality exercisesremote connections and integrated processes still lacksstandard approaches A usability methodology involvingvirtual reality navigation tasks was recently proposed(Hix amp Gabbard 2002) which involved the traditionalfour stages User task analysis expert-guided-based eval-uation formative usability evaluation and finally sum-mative comparative evaluation Elements of Hixrsquos meth-odology are applied in this paper

The design of any rehabilitation technology requiresan assessment and understanding of how the technologywill be used and deployed This is especially criticalwhen transferring technology from a research laboratoryto clinical use where the technology must be robustand easy to learn and use since the focus is on the careof the patient To achieve this goal in the developmentof the Rutgers Ankle Rehabilitation System (RARS) aset of specific methodologies called usability engineeringwas applied This paper describes how usability testingand software design iteration were performed collabora-tively by a group of engineers and clinician scientistsKey issues in this description are the adaptation of stan-dard usability methodology to assess the ease of use of avirtual reality and related telerehabilitation system Find-ings from a formative evaluation consisting of an early-stage usability evaluation with an expert domain userfollowed by pilot usability studies run on 5 physicaltherapists using the remote-monitoring system and theresulting modifications to the system are reported here

2 Brief Description and Application ofthe Technology

The Rutgers Ankle Rehabilitation System (RARS)is a VR-based rehabilitation environment for patientshaving lower-extremity dysfunction As shown in Figure1a the patient sits in the chair and places the affectedlower extremity in a boot mounted on a platform thathas six degrees of freedom By moving their feet pa-tients manipulate a virtual object presented on the com-puter screen In the particular VR rehabilitation applica-tion described here the platform serves as a ldquojoystickrdquoused for flying an airplane through 3D targets The

therapist can set a variety of parameters that match thepatientrsquos therapy needs In addition the therapist canview these parameters remotely and interact with thepatient through a telemonitoring system that includes agraphics display a camera for observing the patient andan audio channel for communication

The RARS consists of local hardware and softwarecomponents as well as a remote-monitoring subsystem

Figure 1 RARS system overview (a) patientrsquos station (b) remote-

monitoring station copy UMDNJ Rutgers 2003

Deutsch et al 199





3 Testing Usability











Deutsch et al 201








copy UMDNJ 2003










Deutsch et al 203















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213





3 Testing Usability











Deutsch et al 201








copy UMDNJ 2003










Deutsch et al 203















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213






Deutsch et al 201








copy UMDNJ 2003










Deutsch et al 203















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213








copy UMDNJ 2003










Deutsch et al 203















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213









Deutsch et al 203















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213















4 Results




Deutsch et al 205































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213































Deutsch et al 207













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213













5 Discussion





Deutsch et al 209
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213
















Deutsch et al 211








Acknowledgments


References





















Deutsch et al 213








Acknowledgments


References





















Deutsch et al 213















Deutsch et al 213

Formative Evaluation and Preliminary Findings of a Virtual Reality Telerehabilitation System for the...

Documents

Transcript of Formative Evaluation and Preliminary Findings of a Virtual Reality Telerehabilitation System for the...