Journal of Surgical Research -, 1–6 (2010)doi:10.1016/j.jss.2010.04.052

ARTICLE IN PRESS

Description of Web-Enhanced Virtual Character Simulation System to

Standardize Patient Hand-Offs

Lori Filichia, M.D.,1 Shivashankar Halan, B.Tech., Ethan Blackwelder, M.S., Brent Rossen, M.S.,Benjamin Lok, Ph.D., James Korndorffer, M.D., and Juan Cendan, M.D.

Department of Surgery, Health Science Center, University of Florida, Gainesville, Florida

Submitted for publication January 13, 2010

Introduction. The 80-h work week has increaseddiscontinuity of patient care resulting in reports of in-creased medication errors and preventable adverseevents. Graduate medical programs are addressingthese shortcomings in a number of ways.

Methods. We have developed a computer simulationplatform called the Virtual People Factory (VPF),which allows us to capture and simulate the dialoguebetween a real user and a virtual character. We haveconverted the system to reflect a physician in the pro-cess of ‘‘checking-out’’ a patient to a covering physi-cian. The responses are tracked and matched toeducator-defined information termed ‘‘discoveries.’’Our proof of concept represented a typical post-operative patient with tachycardia. The system isweb enabled.

Results. So far, 26 resident users at two institutionshave completed the module. The critical discovery oftachycardia was identified by 62% of users. Residentsspend 85% of the time asking intraoperative, postoper-ative, and past medical history questions. The systemimproves over time such that there is a near-doublingof questions that yield appropriate answers betweenusers 13 and 22. Users who identified the virtualpatient’s underlying tachycardia expressed more con-cern and were more likely to order further testing forthe patient in a post-module questionnaire (P [ 0.13and 0.08, respectively, NS).

Conclusions. The VPF system can capture uniquedetails about the hand-off interchange. The system im-proves with sequential users such that better match-ing of questions and answers occurs within the initial25 users allowing rapid development of new modules.

1 To whom correspondence and reprint requests should beaddressed at Department of Surgery, Health Science Center, Univer-sity of Florida, P.O. Box 100-286, Gainesville, FL 32610-0826. E-mail:lori.filichia@surgery.ufl.edu.

1

A catalog of hand-off modules could be easily devel-oped. Wide-scale web-based deployment was uncom-plicated. Identification of the critical findingsappropriately translated to user concern for the pa-tient though our series was too small to reach signifi-cance. Performance metrics based on theidentification of critical discoveries could be used toassess readiness of the user to carry off a successfulhand-off. � 2010 Elsevier Inc. All rights reserved.

Key Words: resident hand-off; resident sign-out;simulation; resident education.

INTRODUCTION

Since the ACGME introduced the 80-h work week inJuly 2003 residency programs rely on a model wheremultiple teams of residents are responsible for a givenpatient’s care. ‘‘Night-float’’ residents or teams takecare of patients overnight and on weekends, increasingthe need for patient ‘‘hand-offs’’ to transfer the care ofpatients from one resident team to another [1]. Individ-ual residents and groups of residents often transitionpatient care between one another using a process com-monly referred to as a ‘‘sign-out’’ or a patient ‘‘hand-off’’ [2].

A hand-off is the transfer of information and profes-sional responsibility from one provider to another whilepatient is in hospital. This type of transfer of patientcare information has been most commonly performedand studied in the nursing profession, however, morerecently, attention is being shifted toward hospitalistand resident physician hand-offs [2–4]. With the shiftaway from the traditional surgical residency structureof in-house call every second to every third night, whichprovided excellent continuity of care, to the paradigm ofmultiple teams of residents taking care of a single

0022-4804/$36.00� 2010 Elsevier Inc. All rights reserved.

JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 20102

ARTICLE IN PRESS

patient, the discontinuity of patient care is greater thanever before. To this end the Joint Commission has de-veloped expectations that hospital systems would ‘‘im-plement standardized approaches to ‘hand-off’communications’’ in their National Patient SafetyGoals, 2009 [5].

Suboptimal communication between physicians dur-ing sign-out has the potential to increase the incidenceof preventable adverse events [6]. A case-control studyperformed at Brigham and Women’s Hospital foundthat preventable adverse events were strongly associ-ated with coverage by a physician from a team otherthan the patient’s primary team, which nearly doubledwhen the cross-covering physician was an intern [7].

A literature review on the subject of physician patienthandoffs reveals that most reports were performed atsingle institutions and there were few controlled inter-ventions. These papers most often utilized subjectivedata such as observations and/or participant surveys[2–4]. At the conclusion of these reports, there arerecommendations for improvements in the process ofpatient hand-off; however, little has been done to sys-tematize an educational curriculum or series of metricsin order to prepare residents for this responsibility [3,5]. Many programs are currently addressing the issueof patient hand-offs, and this was one of the primary fo-cus points at the 2009 AAMC Annual Congress.

METHODS

Our team has developed a web-based computer application calledthe Virtual People Factory (VPF) for creating modeled conversationsbetween real and virtual humans. This system allows us to capturea dialogue between a real user and a virtual character; the systemthen learns from each user interaction thereby improving the virtualcharacter’s ‘‘intelligence.’’ We have developed a hand-off case scenariobetween two resident physicians in the process of ‘‘checking-out’’ a pa-tient. For proof of concept, we developed a scenario where a patientunderwent a laparoscopic adjustable gastric banding earlier in theday and is having a postoperative course remarkable only for tachy-cardia. The user interacts with the system in their own languageand without any prompts by typing in a box, similar to instant mes-saging, the system responds by matching stimuli provided by theuser to stimuli from a previous user, and a response entered by thescript author.

Statistical analysis of the likelihood that participants were con-cerned or ordered further testing for the patient was conducted usinga goodness of fit c2 evaluation using the expectation that all users thatidentified tachycardia would be concerned and that further testingwould be warranted for the patient. The statistical package http://www.people.ku.edu/wpreacher/chisq/chisq.htm was used for the cal-culation; P value was set at 0.05.

Scenario Development

The educator begins to create a script by choosing broad basedtopics and ‘‘discoveries,’’ which are important concepts that shouldbe gleaned from the conversation. Ideally, the discoveries would beevidence-based in nature. Next, the scenario is created by adding sev-eral human conversational stimuli, ‘‘triggers,’’ and a number of virtualhuman responses, ‘‘speeches.’’ These are the educator’s anticipation as

to the most frequently asked questions and statements that real userswould provide in an actual conversation [8]. Next, the educator hasa number of users test the scenario, who interact with the systemand provide feedback, suggestions, and mark any incorrect responsesby the virtual character. The system also records which stimuli failedto match any responses. Generally, it can be noted that as the numberof interactions progresses the number of unanswered questions de-cline for the average user. Accordingly, the number of suggestionsthat a user leaves also correlates with interaction duration and ques-tioning style of the user.

As the system and other users provide feedback, the author of themodule reviews and validates the suggested stimuli and edits the re-sponses appropriately. This helps expand the scenario and addsa wider variety of speeches and triggers that the average user wouldprovide. The educator can spend as much or as little time as needed toedit the script. For example, some users may leave 10 suggestions,others may leave as little as one or none. For the script described inthis paper, the educator typically logged on to edit the script once ev-ery several days and would spend between 5 min to half an hour ata time editing and improving the script.

The author then continues to test the scenario with other users un-til the level of error in the virtual human’s responses is low; practi-cally, this is when most posed questions from the user are met witha valid answer from the virtual physician. The end result is a virtualhuman that appears to converse about a given subject in a naturalway.

Specific Module

The module user first encounters the following description: ‘‘Therehas just been a shift change and you are an incoming physician. Dr.Johnson is turning his patient over to your care. You have 3 min tocomplete the process. Simply type in the box what you would normallyask.’’ The following prompt then appears: ‘‘The patient is Mr. Harris,a 32 y old with a body mass index of 42 who underwent laparoscopicplace of a ‘‘Realize’’ adjustable gastric band today. He was the firstcase of the day.’’ After that, the user engages the system with thegoal of eliciting the necessary information to take care of the patientovernight (Fig. 1). The system tracks the information exchange andan educator defines critical information that must be gleaned; theseare termed ‘‘discoveries.’’ The resident receives feedback via a barthat is located in the top right hand corner of the screen, which revealsthe number of discoveries as the user obtains them. When all of thediscoveries are made, for example, the bar is completely colored inand 6/6 is shown. This information can then be used as both a feedbackand educational tool for the participating resident. Individuals can in-teract with the system multiple times and their individual perfor-mance can be tracked in order to assess for improvements overtime. At the end of the encounter, several follow-up questions are pro-vided to assess the user’s understanding of the encounter and includequestions assessing the user’s level of concern for the patient and fur-ther actions to be taken.

RESULTS

Phase 1 included the development of a scenario ofa resident checking out a post-operative patient whohad a laparoscopic gastric band placed the same morn-ing. Tachycardia was chosen as a frequent but concern-ing presentation. We began the initial scriptdevelopment and testing using surgical residents atthe University of Florida. When the script was ade-quate, it was distributed to a group of surgical residentsat Tulane University for feedback. We then analyzedthe transcripts and tabulated the time users spent on

FIG. 1. Hand off screen shot. Screen shot of the hand-off interaction between the VPF system and the user.

FILICHIA ET AL.: WEB-ENHANCED VIRTUAL CHARACTER SIMULATION SYSTEM 3

ARTICLE IN PRESS

the task, as well as the number of discoveries each usermade. There was a post-interaction questionnaireregarding the level of concern for the patient and anyfurther interventions planned.

So far, 26 residents have logged into the system(Fig. 2) with 18 residents interacting for more than 2min on the exchange. The median time spent on the sys-tem for users spending more than 2 min was 6:06 (range2:21–19:24). Residents spend 85% of the time asking in-traoperative, postoperative, and past medical historyquestions, with minimal time dedicated to other criticalquestions (Fig. 3). The amount of time spent interactingwith the system correlated linearly with the number ofdiscoveries (Fig. 4). Accordingly, it can be seen thatsome users attain a higher number of discoveries ina much shorter time frame than others.

The critical discovery of tachycardia was identified by62% of users (Table 1). All users who identified tachy-cardia expressed more concern for the patient com-pared with 25% of those that did not identifytachycardia (P¼ 0.13). Sixty percent of users that iden-tified tachycardia ordered either testing or a transfer tohigher care status compared with 25% of those that didnot identify the problem (P ¼ 0.08) (Table 2).

The script was tested with different groups of usersand then improved upon by updating new speechesand trigger questions, as well as incorporating usersuggestions and identifying missed or incorrect re-sponses recognized by previous users. We have found

that it takes about 20 users to reach a plateau wherethe level of error in the virtual human’s responses areconsidered low enough. Practically, this is when almostall of the questions from the user are met with a validresponse from the virtual physician. The system im-proves over time such that there is a near-doubling ofeffective questions between users 13 and 22. After threerounds of user testing, there was an increase in ‘‘charac-ter intelligence’’ as the number of triggers increased by120 (449 to 569) and the number of speeches increasedby 29. The number of discoveries increased as the sys-tem learned from prior users: The median number ofdiscoveries identified increased from 4 (range 2–11) ingroup 1 to 7.5 (range 4–11) in group 2.

The system offers the ability to track a number ofissues of potential educational research interest. Inparticular, the system allows the educator to linkthe question asked with a particular sub grouping(e.g., questions about prior surgeries are marked un-der a heading of past medical history) allowing forthe graphical analysis of the topic flow during a dis-cussion (see dot-graph, need to remove names). Alter-natively, the rate of critical discoveries can be used ina manner that informs the educational curriculum.The graphic (bar graph of discoveries) reveals thatalthough most users asked about tachycardia;however, few asked about the patient’s use ofinsulin—this could be seen as a systematic concernand addressed appropriately.

FIG. 2. Topic flow diagram. Diagram shows each of the 26 users who engaged with the system, the time each user spent, and the numberand category of topics covered in the user’s interaction.

FIG. 3. Time spent on each topic. Pie chart shows each of the topics and the percentage of time users spent asking questions falling into eachcategory. Residents spent the most time asking questions about the postoperative course and the least time on preoperative issues.

JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 20104

ARTICLE IN PRESS

FIG. 4. Discoveries versus time graph. Graph shows the number of discoveries versus time. Users who spent more time interacting with thesystem achieved a higher number of critical discoveries.

TABLE 1

Frequency of Discoveries. Table Lists the Critical Discoveries and Shows the Number of Times Each DiscoveryWas Made. The Critical Discovery Was Made by 16 Users (62%)

FILICHIA ET AL.: WEB-ENHANCED VIRTUAL CHARACTER SIMULATION SYSTEM 5

ARTICLE IN PRESS

TABLE 2

The Table Represents the Likelihood of PursuingAdditional Testing Following Interaction with the

Virtual Patient Hand-Off System. System Users WhoIdentified that the Patient Was Tachycardic OrderedTesting More Frequently Though Not Statistically So

(P [ 0.08)

Which of the following actions would you recommend? (Please checkall that apply)

Answer options Response percent Response countNone 34.8% 8X-rays 26.1% 6Lab work 26.1% 6Transfer to higher care unit 4.3% 1Other (please specify) 2Answered question 23Skipped question 0

JOURNAL OF SURGICAL RESEARCH: VOL. -, NO. -, - 20106

ARTICLE IN PRESS

DISCUSSION

As many residency programs have to switch to a ‘‘cross-cover’’ system of patient coverage in order to comply withresident work hour restrictions, there has been an in-creasing focus on improving patient hand-offs betweenresidents. Common conclusions that are drawn through-out the literature are that in addition to providing writ-ten or computerized sign-out information on thepatients, verbal communication between practitioners,preferably a face-to-face, interactive hand-off process isessential [3, 9, 10]. Ideally, residents will engage in anactive dialogue regarding essential patient careactivities, and includes the opportunity to ask andrespond to questions [3, 10]. Most current systems ofhand-offs focus on the physician who is handing off thepatient, rather than the physician who will be takingover the patient’s care [2]. Recognition that this shouldbe an interactive process between both physicians andthat the receiving physician has a responsibility to ex-tract critical information is essential.

This is a proof of concept study developed to assessthe feasibility of developing a scenario, testing it, andthen distributing it to a number of users. With theVPF, we can capture unique details about the hand-off interchange. The system can learn and approacha peak within 25 users, allowing rapid wide-scaleweb-based deployment. Web-based deployment wasused without difficulty in the trial. The developmentphase of the system is near completion and efforts arenow being turned to developing a catalogue of hand-offs scenarios. These can be unique to each specialty ro-tation, and a catalog of the most common postoperativesituations for a given specialty could be easily devel-oped. The system allows the educator to develop thesescenarios without the need for a computer scientistand has the potential to be deployed to multiple

institutions via the internet. There is no need foradditional interactive tools or interfaces beyond thetyping and point-click technologies.

Residents who identified tachycardia were more aptto develop concern for the patient and intervene. Thestudy group was small and the statistical analysis ofthese subgroups was, therefore, insufficient. As we pro-ceed with this research we will investigate the ability toreach statistical conclusions about patient care con-cerns using this educational methodology.

The program could be used as a training and evaluationtool forall levels of residents. It couldbeused tohelpbegin-ning residents develop the ability to be able to elicit the ap-propriate information from the primary physician aboutvarious patient situations when assuming the care ofa patient in a cross-covering role. Performance metricsbased on the identification of critical discoveries could beused to assess readiness of the user to carry off a successfulhand-off. The ultimate goal is to provide a reproduciblemethod that can track performance metrics and can beused to help standardize a mechanism for patienthand-offs between residents, to highlight specificeducator-driven teaching points and to develop educa-tional level-specific training materials that can be distrib-uted widely over the web without difficulty.

SUPPLEMENTARY DATA

Supplementary data associated with the article can befound in the online version, at doi:10.1016/j.jss.2010.04.052.

REFERENCES

1. Okie S. An elusive balance: Residents’work hours and the conti-nuity of care. N EnglJ Med 2007;356:2665.

2. Solet DJ, Norvell JM, Rutan GH, et al. Lost in translation: Chal-lenges and opportunities in physician to physician communica-tion during patient handoffs. Acad Med 2005;80:1094.

3. Vidyarthi AR, Arora V, Schnipper JL, et al. Managing disconti-nuity in academic medical centers: Strategies for a safe andeffective resident sign-out. J Hosp Med 2006;1:257.

4. Sharit J, McCane L, Thevenin D, et al. Examining links betweensign-out reporting during shift changeovers and patient man-agement risks. Risk Anal 2008;28:969.

5. National Patient Safety Goals. Available at: www.JCAHO.com.Accessed December 2009.

6. Arora V, Johnson J, Lovinger D, et al. Communication failures inpatient sign-out and suggestions for improvement: a criticalanalysis. Qual Saf Health Care 2005;14:401.

7. Petersen LA, Brennan TA, O’Neil AC, et al. Does house staff dis-continuity of care increase the risk for preventable adverseevents? Ann Intern Med 1994;121:866.

8. Rossen B, Lind DS, Lok B. Human-centered distributed conver-sational modeling: Efficient modeling of robust virtual humanconversations. Proceedings of the 9th International Conferenceon Intelligent Virtual Agents–2009. Amsterdam, The Nether-lands, Sept. 14–16, 2009.

9. Kemp CD, Bath JM, Berger J, et al. The top 10 list for a safe andeffective sign-out. Arch Surg 2008;143:1008.

10. Van Eaton EG, Horvath KD, Lober WB, et al. Organizing thetransfer of patient care information: The development ofa computerized resident sign out system. Surgery 2004;136:5.