Computers & Education 59 (2012) 1001–1008

Contents lists available at SciVerse ScienceDirect

Computers & Education

journal homepage: www.elsevier .com/locate/compedu

Efficacy of virtual patients in medical education: A meta-analysis of randomizedstudies

Fabrizio Consorti*, Rosaria Mancuso, Martina Nocioni, Annalisa PiccoloDepartment of Cardiocirculatory Pathophysiology, Anestesiology and General Surgery – 1st Faculty of Medicine University “Sapienza” of Rome, viale del Policlinico, 00161 Rome, Italy

a r t i c l e i n f o

Article history:Received 3 September 2010Received in revised form11 April 2012Accepted 25 April 2012

Keywords:Distance education and telelearningEvaluation of CAL systemsSimulationsMultimedia/hypermedia systems

* Corresponding author. Dip. di Fisiopatologia Cardþ39 06491695.

E-mail addresses: fabrizio.consorti@uniroma1.it (yahoo.it (A. Piccolo).

0360-1315/$ – see front matter � 2012 Elsevier Ltd. Adoi:10.1016/j.compedu.2012.04.017

a b s t r a c t

A meta-analysis was performed to assess the Effect Size (ES) from randomized studies comparing theeffect of educational interventions in which Virtual patients (VPs) were used either as an alternativemethod or additive to usual curriculum versus interventions based on more traditional methods.Meta-analysis was designed, conducted and reported according to QUORUM statement on quality ofreporting of meta-analyses. Twelve randomized controlled studies were retrieved, assessing a total of 25different outcomes.Under a random-effect model, meta-analysis showed a clear positive pooled overall effect for VPscompared to other educational methods (Odds Ratio: 2.39; 95% C.I. 1.48 O 3.84). A positive effect hasbeen documented both when VPs have been used as an additive resource (O.R.: 2.55; C.I. 1.36 O 4.79)and when they have been compared as an alternative to a more traditional method (O.R.: 2.19;1.06 O 4.52). When grouped for type of outcome, the pooled ES for studies addressing communicationskills and ethical reasoning was lower than for clinical reasoning outcome.There is evidence that VPs are effective but further research is needed to clarify which is their bestpossible integration in curricula and their value and cost/benefit ratio with respect to other activelearning methods.

� 2012 Elsevier Ltd. All rights reserved.

1. Introduction

Virtual patients (VPs) are computer-based simulations of clinical cases allowing the users to interact with the system and train theirclinical reasoning skills. The term “virtual patient” has been referred to various educational tools and methods like simulated patients in theform of actors (Cleland, Abe, & Rethans, 2009), software-based physiological simulators (Holzinger, Kickmeier, Wassertheurer, & Hessinger,2009), physical manikins and advanced technological simulators (Khan, Pattison, & Sherwood, 2011). A further new learningmethod near tothe concept of Virtual patients is based on platforms to use real patient records to teach undergraduate students decisionmaking and clinicalreasoning (Bloice, Simonic, Kreuzthaler, & Holzinger, 2011).

There is however a general consensus in considering a VP an “interactive computer simulation of real-life clinical scenarios for the purpose ofhealthcare and medical training, education or assessment” (Ellaway, Poulton, Fors, McGee, & Albright, 2008).

VPs can provide away to overcome the reduced student access to real patients, as well as a structured and safe environment for students topractice; they are therefore more and more extensively used in medical education but their efficacy is still a matter of debate (Poulton &Balasubramaniam, 2011). A recent review provided a critical evaluation of relevant literature but it could find mainly descriptive articles,many articlesmeasuring students’ satisfactionwith VPs but few studies providing evidence about the efficacy of VPs (Cook & Triola, 2009). Oneof the issues raised by Cook in his review was the way in which VPs are integrated in the curriculum. VPs have been used in many differentinstructional designs and educational applications (Huwendiek et al., 2009) but not always the terminology used to describe the learningenvironment and VPs usage was uniform. At this proposal, in her recent paper (Ellaway & Davies, 2011), Rachel Ellaway proposes to applyactivity theory to redefine Virtual patients “from a software artifact to an intrinsic part of an activity that mediates the ways that learners and their

iocircolatoria, Anestesiologia e Chirugia Generale, viale del Policlinico, 00161 Rome, Italy. Tel.: þ39 0649970634; fax:

F. Consorti), rosaria_mancuso1987@libero.it (R. Mancuso), martinanocioni@yahoo.it (M. Nocioni), annalisa.piccolo@

ll rights reserved.

F. Consorti et al. / Computers & Education 59 (2012) 1001–10081002

objectives interact. The same virtual patient artifact can be used in different learning activities and activity theory identifies each application asdiscrete and specific to the activity in hand.” This was an important theoretical contribution as a model to describe interventions that use VPs ina meaningful and consistent way. Ellaway and Davies (2011) and Huwendiek et al. (2009) work provided categories to identify in a clearer waycontext of use, objectives and type of interventions for VPs, allowing an easier comparison among different experiments.

This meta-analysis summarizes evidence from randomized studies comparing the effect of educational interventions in which VPs wereused either as an alternativemethod or additive to usual curriculumversus interventions based onmore traditional methods.We performedalso an evaluation of the efficacy of VPs when used to achieve outcomes oriented to clinical reasoning or outcomes addressed tocommunication skills and ethical reasoning.

2. Methods

This meta-analysis was designed, conducted and reported according to QUORUM statement on quality of reporting of meta-analyses(Moher et al., 1999).

2.1. Searching

We carried out a systematic search of MEDLINE, ERIC, PsycInfo, Cochrane from January 2000 through July 2010. This limited threshold intime was chosen to have a more homogenous sampling of studies as to their technological aspects. Search terms included MeSH or ERICdescriptors like distance education, computer simulation, virtual system. Since the MeSH term “virtual patient” does not exist, text strings likevirtual patient or virtual training or Internet-based case, limited to medical education were also used. Additional references were identifiedfrom reference lists of the retrieved articles.

2.2. Selection criteria

We searched for studies in any language, limited to graduate or undergraduate medical education, reporting the use of VPs as a teachingmethod and objectively assessing their efficacy with respect to an educational outcome in interventions in which VPs were used as analternative or additive method to other formats of teaching. We excluded studies without a control group or with a non-randomized design,as well as studies which considered a learner self-reported assessment both of satisfaction and of perceived achievement. Fig. 1 describesthe flow of the process of selection through the different stages.

2.3. Validity assessment and data abstraction

All the retrieved articles were independently screened by three of the Authors (R.M., M.N. and A.P.) by their title and abstract andpotentially interesting papers were retrieved in full text. F.C. checked the selection process and the doubtful abstracts. Disagreement wassolved by consensus.

Relevant information about each study was abstracted according to a standardized form, managed in an electronic format, whichsummarized type of study design, number and type of subjects, type of intervention, assessed outcome and instrument of assessment. Themethodological quality of the selected studies was assessed by the CONSORT statement (see CONsolidated Standards of Reporting Trials athttp://www.consort-statement.org), in its 2001 version. Item #19 about reporting of adverse events was not considered. Quality assessmentwas done in duplicate by two couples of Authors and then the results were merged during a consensus meeting. After a first phase of

1943 articles identified and screened for retrieval

1752 articles excluded because not dealing with VPs

191 articles retrieved for more detailed evaluation

157 articles excluded because only descriptive or qualitative

34 quantitative studies potentially appropriate to be included in meta-analysis 22 studies excluded because

without a control group or with a non-randomized design or with a self-assessed outcome or poor quality according to CONSORT12 randomized studies

included in meta-analysis, assessing 25 outcomes with objective instruments

Fig. 1. Progress through the stages of the meta-analysis.

F. Consorti et al. / Computers & Education 59 (2012) 1001–1008 1003

assessment, we decided to relax criteria of inclusion, due to the mean poor quality of presentation of studies. Actually the CONSORTstatement was designed to assess quality of clinical studies and its criteria are too stringent for the present quality of educational studies, inwhich it is more difficult to control the experimental setting. In a recent systematic review about quality of studies, Cook, Levinson, andGarside (2011) stated that the main positive factor for quality was the condition of two-group design. We then selected comparativestudies if they fulfilled at least 11 of the 21 considered items of CONSORT (2001) and – mandatorily – if they clearly indicated pre and postintervention figures needed to compute the Effect Size in meta-analysis and if they described outcome and related instrument of assess-ment. These last three requirements were considered sufficient to warrant validity of results.

2.4. Quantitative data synthesis

Studies were primarily considered separately as to the use of VPs as an alternative or an additional resource with respect to othermethods. We secondarily grouped studies in two classes, according to the type of assessed outcome: clinical reasoning or communicationand ethics.

The Effect Size (ES) was expressed as odds ratio (O.R.) and 95% confidence intervals (C.I.). More precisely, the ES in this meta-analysisexpresses the ratio between the odds of a positive effect for students exposed to VPs and for students not exposed to VPs. An ES > 1indicates an effect of VPs on the achievement of the learning outcome higher than the control group.

In its simpler formulation, the evaluation of ES does not take into account the extent of positive effect but only if a positive pre-postintervention effect was observed. Nevertheless, we expected a large variance across the effect size of the studies, also called heteroge-neity, due to the large difference in detailed outcomes, setting and educational methods compared. Heterogeneity was quantified by the I2

statistic (Higgins, Thompson, Deeks, & Altman, 2003), which describes the percentage of variation across studies that is due to heterogeneityrather than chance. A value greater than 50% indicates a large inconsistency and in our sample I2 was 85.8%. In the analysis we used then the‘random effects’ model, which allows the study outcomes to vary in a normal distribution among studies. This model uses the weightedaverage of the effect sizes to reduce the bias possibly introduced by a large variance of the effect size of the included studies (Senn, 2007).

Publication bias was controlled by a funnel plot of standard errors with random effects, which showed a symmetric distribution ofstudies about the effect size. Using “trim and fill”method (Sutton, Duval, Tweedie, Abrams, & Jones, 2000) under a random effects model andconsidering missing studies only to the left side of the mean effect, the values of ES and C.I. remained unchanged. Calculations and plottingwere done by Comprehensive Meta-Analysis� software (http://www.meta-analysis.com/).

3. Results

Table 1 summarizes the characteristics of the 12 randomized and controlled studies selected for meta-analysis. The 22 articles excludedfrommeta-analysis (Table 2) were primarily survey on students’ perception and satisfactionwith VPs or pre-post, non-randomized studies,in some instances addressed to validate VPs as an instrument of assessment.

Table 1Characteristics of the 12 studies included in the meta-analysis.

Study Participants Study design No. of VP cases Outcome No. ofoutcomes

Intervention Instrument

Botezatu et al., 2010 49 students Post 3 for each topic Clinical reasoning inhematology and cardiology

2 Alternative Writtenclinical test

Deladisma et al., 2007 84 students Post 1 Communication skills 1 Alternative Assessment gridFleetwood et al., 2000 173 students Post 2 Ethical reasoning 1 Alternative MCQFleming et al., 2009 102 mixed Pre-post 1 scenario

practised 10 timesCounseling in threedifferent scenarios

3 Alternative Score fromtrained SP

Kandasamy & Fung, 2009 55 students Pre-post 1 Clinical reasoning inlaryngeal stridor

1 Alternative MCQ

Kerfoot et al., 2006 210 students Pre-post 2 for each topic Clinical reasoning in urology 1 Additional 28-item validatedinstrument

Kumta, Tsang, Hung,& Cheng, 2003

163 students Pre-post n.a. Clinical reasoningand clinical skills

1 Alternative Written clinicaltest OSCE

Schittek et al., 2004 39 students Post 1 History taking 3 Additional TimeCritical questionsScore forprofessionalbehavior

Triola et al., 2006 55 hc professionals Pre-post 2 Diagnostic ability insub-diagnosticand acute stress,post-traumaticstress disorder (PSD),bereavement

4 Additional Clinical vignettes

Vash, Yunesian,Shariati,Keshvari,& Harirchi, 2007

48 students Post 14 History takingImaging and Labdata interpretationDiff diagnosisManagement planFactual Kn

6 Additional OSCEExtendedmatchingand MCQOpen questionsCloze test

Wahlgren, Edelbring, Fors,Hindbeck, & Ståhle, 2006

116 students Post n.a. Diff diagnosis,management, factual kn

1 Additional MCQ

Youngblood et al., 2008 30 mixed Pre-post 6 Team leadership skill 1 Alternative EMCRM scale

Table 2Characteristics of the 22 studies not included in the meta-analysis.

Study Participants Study design No. of VP cases Outcome Instrument

Bearman et al., 2001 255 students Not controlled, comparingtwo different VP format

1 for each group Communication skills Simulated patient

Bearman, 2003 12 students Phenomenological study 1 in twodifferent formats

Emotional effects on students Interview

Berman et al., 2009 545 students Survey 31 Students’ perception andsatisfaction with VPs

Validated instruments

Boyd et al., 2008 101 students Pre-post w/o control 1 User-perceived difficultyHealth-related knowledge

Validated instrumentsMCQs

Courteille, Bergin,Stockeld, Ponzer,& Fors, 2008

110 students Observational post 1 Clinical skills in an OSCE Grading

Critchley, Wong,& Leung, 2008

n.a. Survey 1 Students’ perception andsatisfaction with VPs

Rating of educational value

Deladisma et al., 2009 21 students RCT 1 Students’ confidence andanxiety with clinicalbreast examination

Validated instrument

Deladisma et al., 2008 23 students Survey 1 Students’ perception andsatisfaction with VPs

Rating of educational value

Jerant & Azari, 2004 89 students Validity study 4 Validity of automated scoresof diagnostic reasoning ability

–

Kamin, Deterding,& Lowry, 2002

n.a. Qualitative 1 Attitudes about threeteaching formats

Focus groups

Kamin, O’Sullivan,Deterding,& Younger, 2003

128 students Non-randomized comparison 1 Critical thinking inproblem-based learning

Indicators ofcritical-thinking stages

Kotranza, Lok, Deladisma,Pugh, & Lind, 2009

122 students Survey of three groups 1 for each group Acceptability Rating of acceptability

LeRoy Heinrichs,Youngblood, Harter,& Dev, 2008

43 students Pre-post w/o controlof three groups

6 þ 10 þ 10 Feeling of reality Validated instrument

Nathoo, Goldhoff,& Quattrocchi, 2005

16 students Survey 1 Students’ perception andsatisfaction with VPs

Interview

Poulton, Conradi, Kavia,Round, & Hilton, 2009

41 students Survey 5 Students’ perception andsatisfaction with VPs

Questionnaire

Sanders et al., 2007 94 students Pre-post w/o control 2 Knowledge and comfortlevel regarding the careof children

Validated instrument

Stevens et al., 2005 48 Pre-post w/o control 1 Knowledge structure Semantic organization of25 case-related concepts

Stevens et al., 2006 20 Survey 1 Students’ perception andsatisfaction with VPs

Rating of educational value

Vincent, Sherstyuk,Burgess, & Connolly, 2008

20 students Observational post 1 Triage and intervention ability Scoring system

Vukanovic-Criley, Boker,Criley, Rajagopalan,& Criley, 2008

82 students Non-randomized comparison 8 Cardiac examination skills Validated multimedia test

Waldmann, Gulich,& Zeitler, 2008

147 Validity study 3 GP competence Correlation with written exam

Zary, Johnson, Boberg,& Fors, 2006

70 medicalstudents123 dentistrystudents

Survey 6 medical20 dentistry

Students’ perception andsatisfaction with VPs

Rating of educational value

F. Consorti et al. / Computers & Education 59 (2012) 1001–10081004

Table 3 reports the number of studies providing information considered to be quality indicators. All studies gave details aboutthe educational intervention and assessment of outcome but less than half clearly expressed an experimental hypothesis andonly one study computed the needed sample size. The mean score according to CONSORT checklist was 13.7 out of 21 (range11 O 16).

Five studies used VPs as an additional intervention, assessing 15 different outcomes, while the other 7 studies compared VPs as analternative method, for a total of 10 outcomes. The overall pooled Effect Size (ES) for the 25 outcomes, expressed as O.R. was 2.39 in favor ofVPs (C.I. 1.48 O 3.84; p < 0.001), while the ES in favor of VPs as an additional resource was 2.55 (C.I. 1.36 O 4.79 p ¼ 0.003) and 2.19 (C.I.1.06 O 4.52; p ¼ 0.034) as an alternative method (Fig. 2).

Six studies performed a pre-post intervention assessment, while the other 6 assessed learners only at the end of the educationalintervention.

Eight out of the 12 studies assessed 17 outcomes which could be classified as pertinent to clinical reasoning, while the remaining4 assessed 8 outcomes in the domain of communication skills or ethical reasoning (Fig. 3). In this sub-analysis, while the pooled ESfor clinical reasoning remained significantly in favor of VPs (O.R. 2.59, C.I. 1.47 O 4.58; p ¼ 0.001), the ES for the outcomes in thedomain of communication skills and ethics was lower (O.R. 2.017, C.I. 0.93 O 4.369; p ¼ 0.075) and slightly below the level ofsignificance.

Table 3Quality indicators met in the 11 included studies.

Indicator N. of studies

Eligibility criteria and setting of data collection 12Precise details of the intervention 12Specific objectives and hypothesisa 5Clearly defined outcomes measures 12Sample size determination 1Details about randomizationb 4Blindingc 7Baseline data of sample (gender, age) 5Representation of effect sizeMean � S.D. 8Mean, sample size, p value 1Difference in means, sample size, p value 2Std difference in means, C.I. 1

a Expressed as a positive statement and not just as a generic statement like: to determine ifVPs have an effect.

b Method to determine the sequence, responsible, concealment.c 4 blind assessments and 3 declared blindness of students to the goals of the study.

F. Consorti et al. / Computers & Education 59 (2012) 1001–1008 1005

4. Discussion

Our analysis showed a clear positive pooled overall effect for VPs compared to other educational methods. A positive effect has beendocumented both when VPs have been used as an additive resource and when they have been compared as an alternative to a moretraditional method.

Nevertheless, heterogeneity among studies was high as shown by the rather wide dispersion in the forest plots (Figs. 2and 3). Thisinconsistency was expected, due to the great difference in the selected studies as to educational design of the interventions and in theassessed outcome.

4.1. Educational design and type of outcome

The value of VPs in medical education has been attributed to their potential use for many different educational outcomes and mainly topromote and/or assess clinical reasoning (Ellaway, Poulton, Smothers, & Greene, 2009). In being challenging for the learner, VPs are believedto foster an active learning and hence they should be consistently compared to other active methods of teaching/learning, particularly if theintervention is addressed to the acquisition of clinical competence and not to factual knowledge. Many of the selected studies report

Group by Study name Statistics

Additional

Odds

ratio Lower Upper p-Value

Vash 2007 History taking 21,624 6,537 71,531 0,000 Vash 2007 Imaging 1,553 0,554 4,349 0,403 Vash 2007 Lab data 0,626 0,223 1,753 0,372 Vash 2007 Diff Diagnosis 1,322 0,473 3,694 0,595 Vash 2007 Mng plan 1,179 0,422 3,293 0,753 Vash 2007 Factual Kn 1,274 0,456 3,558 0,645 Kerfoot 2006 15,752 9,422 26,335 0,000 Schittek 2002 Time 4,845 1,472 15,948 0,009 Schittek 2002 Critical quest. 12,591 3,536 44,836 0,000 Schittek 2002 Profess. behav. 4,845 1,472 15,948 0,009 Triola 2006 PSD 1,195 0,437 3,265 0,729 Triola 2006 Subdiagn. stress 2,750 0,988 7,652 0,053 Triola 2006 Ac. stress 1,301 0,476 3,558 0,608 Triola 2006 Bereavement 0,913 0,334 2,494 0,859 Wahlgren 2006 1,898 0,867 4,157 0,109

2,556 1,364 4,791 0,003

Alternative

Botezatu 2010 Haematology 12,809 4,119 39,829 0,000 Botezatu 2010 Cardiology 9,148 3,026 27,658 0,000 Deladisma 2007 0,196 0,085 0,450 0,000 Fleetwood 2000 1,000 0,582 1,717 1,000 Fleming 2009 Screening 3,427 1,661 7,068 0,001 Fleming 2009 Intervention 2,042 1,003 4,156 0,049 Fleming 2009 Referral 1,244 0,615 2,516 0,544 Kandasami 2009 4,801 1,761 13,085 0,002

000,0986,9789,2083,53002atmuK Youngblood 2008 0,570 0,154 2,111 0,400

2,190 1,059 4,527 0,034

Overall 2,393 1,488 3,848 0,000

Odds ratio and 95% C.I.

VPsno VPs 0.01 1001

Fig. 2. Random-effects meta-analysis of interventions based on VPs vs other interventions, overall and grouped for kind of intervention: VPs additional or VPs alternative to othermethods.

Group by Study name Statistics

Clin. reasoning Odds ratio Lower Upper p-Value

Vash 2007 History taking 21,624 6,537 71,531 0,000Vash 2007 Imaging 1,553 0,554 4,349 0,403Vash 2007 Lab data 0,626 0,223 1,753 0,372Vash 2007 Diff Diagnosis 1,322 0,473 3,694 0,595Vash 2007 Mng plan 1,179 0,422 3,293 0,753Vash 2007 Factual Kn 1,274 0,456 3,558 0,645Botezatu 2010 Haematol. 12,809 4,119 39,829 0,000Botezatu 2010 Cardiol. 9,148 3,026 27,658 0,000Kumta 2003 5,380 2,987 9,689 0,000Youngblood 2008 0,570 0,154 2,111 0,400Kandasamy 2009 4,801 1,761 13,085 0,002Kerfoot 2006 15,752 9,422 26,335 0,000Triola 2006 PSD 1,195 0,437 3,265 0,729Triola 2006 Subd.stress 2,750 0,988 7,652 0,053Triola 2006 Ac. stress 1,301 0,476 3,558 0,608Triola 2006 Bereavement 0,913 0,334 2,494 0,859Wahlgren 2006 1,898 0,867 4,157 0,109

2,599 1,473 4,586 0,001

comm and

ethics

Deladisma 2007 0,196 0,085 0,450 0,000Fleetwood 2000 1,000 0,582 1,717 1,000Fleming 2009 Screening 3,427 1,661 7,068 0,001Fleming 2009 Intervention 2,042 1,003 4,156 0,049Fleming 2009 Referral 1,244 0,615 2,516 0,544Schittek 2002 Time 4,845 1,472 15,948 0,009Schittek 2002 Critic. quest. 12,591 3,536 44,836 0,000Schittek 2002 Prof. behav. 4,845 1,472 15,948 0,009

2,017 0,932 4,369 0,075

Odds ratio and 95% C.I.

VPsno VPs

Fig. 3. Random-effects meta-analysis of interventions based on VPs vs other interventions, grouped for type of outcome: clinical reasoning or communication skills and ethic.

F. Consorti et al. / Computers & Education 59 (2012) 1001–10081006

a traditional curriculum for the control group, based on a combination of lectures and tutorial group activities. In these descriptions it is notalways clear the amount of active teaching/learning methods which students of the control group were exposed to and this may raise thequestion if the observed positive effect was due to VPs in themselves, as a specific consequence of their educational features or moregenerically to the introduction in the curriculum of a more active method of learning.

In our sample only by 4 studies addressed communication and ethical skills (Fig. 3). The ability of communicating in an effective andempathic way with patients, as well as the ability to identify and address the ethical aspects of dilemmatic situations are recognized as corecomponents of the construct of professionalism (Arnold, 2002). When grouped for type of outcome, the pooled ES for studies addressingcommunication skills and ethical reasoning was lower than for clinical reasoning outcome. This may reflect an actual difference in efficacy ofVPs but also a difference in reliability of scoring systems for these types of competencies. A systematic review on the reliability of objectivestructured clinical examination scores (Brannick, Erol-Korkmaz, & Prewett, 2011) showed that interpersonal skills were evaluated lessreliably across stations compared with clinical skills. The difficulty in obtaining an objective assessment in this domain could have thenintroduced a further element of heterogeneity.

4.2. Time

The amount of exposure is another issue to be considered. Only one of the studies in which VPs were alternative to another methodcompared the time spent on learning activities by the VPs group vs control group (Kandasamy & Fung, 2009), but the study compared twovery short interventions based on one VP case (9.06� 4.7min) vs reading of some articles (25.5�18.7min). In a recent meta-analysis (Cook,Levinson, & Garside, 2010) the Authors compared the learning time and efficiency of Internet-based vs non-computer instruction. Theirconclusions were that on average the two methods require the same amount of time but they found a substantial positive correlationbetween knowledge and time. So it could be argued that at least part of the positive effect of VPs could rely on a longer period of work spentby the learners. This assumption is surely true for studies inwhich VPs were additional to the normal curriculum, but it could be also true forVPs as an alternative method. A possible explanation to be tested for this hypothesis is that learners’ satisfaction with VPs is usually veryhigh (Gesundheit et al., 2009) and then that students could be more likely to engage themselves in a more prolonged learning activity.

4.3. Number of worked VPs cases

In his extensive review about research on clinical reasoning (Norman, 2005) Norman states that the best way to the acquisition of clinicalexpertise is “deliberate practice with multiple examples and feedback”. Most of the included studies used few VP cases for their experiment(Table 1). In the study of Botezatu, Hult, Tessma, and Fors (2010) just three VPs cases in the domains of cardiology and hematology wereenough to produce not only a positive early learning effect but also a persistent one, with better retention at a late assessment 4.5 monthslater. The Authors stress that the content of cases used for assessment was matched for topic and difficulty level with the training cases. Asimilar result was reached by Kerfoot et al. (2006) by the use of 2 VPs cases for each of four topics in urology. These findings are consistentwith the “case specificity” effect (Dory, Gagnon, & Charlin, 2010), which is based on the finding of variability of learners’ performance acrossdifferent cases and which is believed to reflect the fact that clinical reasoning is mainly based on non-analytic processes, hence stronglydependent on the mental database of cases owned by healthcare professionals (Eva, 2005). If the assessment of clinical skills after the

F. Consorti et al. / Computers & Education 59 (2012) 1001–1008 1007

exposure to an even limited set of cases on a specific topic is highly consistent with the training cases themselves, we may expect a goodperformance. Nevertheless, probably more cases are better than fewer and the issue of determining the right balance between efficacy andefficiency is still open.

4.4. Limitation of this study

This study has twomain limitations. Despite the original idea to consider only high quality evidence from randomized controlled studies,even if this would have reduced the number of selected articles, after a first phase of evaluation we decided to relax the inclusion criteria.Therefore information about the full educational design or details like the baseline data of participants was sometimes lacking. Poordescriptionmay have led tomisinterpretation andwrong classification of some studies as addressed to clinical or communication skills (e.g.:training history taking is a matter of clinical competence or a communication skill depending on the focus one gives to the assessment). Alsothe distinction between additional or alternative intervention may have been confused by the great heterogeneity of studies. In someinstances for example the two alternative interventions were actually comparable as to their load for learners (e.g. Deladisma et al., 2007),while in other cases (e.g. Botezatu et al., 2010) the VPs intervention substituted part of the traditional curriculum, but it was not possible todetermine if the load for students was the same and then to what extent VPs intervention was a real alternative or just a partial addition.

The second limitation is the time threshold we imposed to our search. We are aware that this choice could have caused to miss somestudies, but a review of methods used in research in medical education showed that randomized controlled studies were substantiallypresent in literature only after 2000 (Baernstein, Liss, Carney, & Elmore, 2007).

Relevant issues like comparison among different possible formats for VPs (Bearman, Cesnik, & Liddell, 2001) or long-term retention ofknowledge and skills after the use of VPs were outside the scope of our work.

5. Conclusions

Although limited to a rather small number of cases, this meta-analysis summarizes the best available evidence at present. The mainindication is that VPs are effective toward many different specific educational outcomes like clinical reasoning in selected topics, clinicaldata gathering and interpretation and – to amore limited extent – they are also useful in preparing students for further educational activitiesin the domain of communication skills and ethical reasoning. These last topics raise the issue of the inter-cultural differences among schoolsof medicine from different Countries and cultural areas, because both communication habits and ethical reasoning are strongly contextdependent. This issue too has been tackled by research (Zary et al., 2009) and it is a worth recalling it while evaluating the impact of the useof VPs in medical education.

Further research is needed to highlight the specific role, value and cost/benefit ratio of VPs with respect to other active learning methodslike team-based learning or game-based learning and other more sophisticated simulation techniques. Research should also address theeducational design with VPs, comparing different solutions as to the format and number of cases. Attention should be paid to the way inwhich VPs can be embedded in a whole curriculum. Possible topics are the scoring system (without any score but just feedback froma clinical tutor, with automatically generated score for formative or summative assessment) and the links of VPs with other educationalactivities (integrated in small groups tutorials, during clerkship, associated with lecturing). As other researches in contiguous fieldsdemonstrated, “successful application of simulations require careful additional guidance from medical professionals” (Holzinger et al., 2009;Khan et al., 2011). Therefore the way VPs are integrated with the whole curriculum is probably critical for their full efficacy.

Future research should clearly state the intended outcome and the size of the expected educational effect, in order to allow a properdefinition of the experimental setting, with special reference to the sample size of the study and control groups needed to obtain possiblysignificant findings and to a clear identification and control of confounding effects as differences among students in pre-test knowledge andattitudes to Internet-based learning.

References

Arnold, L. (2002). Assessing professional behavior: yesterday, today, tomorrow. Academic Medicine, 77(6), 502–515.Baernstein, A., Liss, H. K., Carney, P. A., & Elmore, J. G. (2007). Trends in study methods used in undergraduate medical education research, 1969–2007. Journal of American

Medical Association, 298(9), 1038–1045.Bearman, M. (2003). Is virtual the same as real? Medical students’ experiences of a virtual patient. Academic Medicine, 78(5), 538–545.Bearman, M., Cesnik, B., & Liddell, M. (2001). Random comparison of ‘virtual patient’models in the context of teaching clinical communication skills. Medical Education, 35(9),

824–832.Berman, N., Fall, L. H., Smith, S., Levine, D., Maloney, C. G., Potts, M., et al. (2009). Integration strategies for using virtual patient in clinical clerkships. Academic Medicine, 84(7),

942–949.Bloice, M. D., Simonic, K.-M., Kreuzthaler, M., & Holzinger, A. (2011). Development of an interactive application for learning medical procedures and clinical decision making.

Information Quality in e-Health. Springer Lecture Notes in Computer Science LNCS, 7058, 211–224.Botezatu, M., Hult, H., Tessma, M. K., & Fors, U. (2010). Virtual patient simulation: knowledge gain or knowledge loss? Medical Teacher, 32(7), 562–568.Boyd, S. E., Sanders, C. L., Kleinert, H. L., Huff, M. B., Lock, S., Johnson, S., et al. (2008). Virtual patient training to improve reproductive health care for womenwith intellectual

disabilities. Journal of Midwifery and Women’s Health, 53(5), 453–460.Brannick, M. T., Erol-Korkmaz, H. T., & Prewett, M. (2011). A systematic review of the reliability of objective structured clinical examination scores. Medical Education, 45(12),

1181–1189.Cleland, J. A., Abe, K., & Rethans, J. J. (2009). The use of simulated patients in medical education: AMEE guide no 42. Medical Teacher, 31(6), 477–486.CONSORT (2001). CONSORT statement available at http://www.consort-statement.org/.Cook, D. A., Levinson, A. J., & Garside, S. (2010). Time and learning efficiency in internet-based learning: a systematic review and meta-analysis. Advances in Health Sciences

Education Theory and Practice, 15(5), 755–770.Cook, D. A., Levinson, A. J., & Garside, S. (2011). Method and reporting quality in health professions education research: a systematic review. Medical Education, 45(3), 227–

238.Cook, D. A., & Triola, M. M. (2009). Virtual patients: a critical literature review and proposed next steps. Medical Education, 43(4), 303–311.Courteille, O., Bergin, R., Stockeld, D., Ponzer, S., & Fors, U. (2008). The use of a virtual patient case in an OSCE-based examda pilot study. Medical Teacher, 30(3), e66–e76.Critchley, L. A., Wong, J. W., & Leung, J. Y. (2008). Virtual patient and undergraduate anesthesia teaching. Medical Education, 42, 1120–1121.Deladisma, A. M., Gupta, M., Kotranza, A., Bittner, J. G., 4th, Imam, T., Swinson, D., et al. (2009). A pilot study to integrate an immersive virtual patient with a breast complaint

and breast examination simulator into a surgery clerkship. American Journal of Surgery, 197(1), 102–106.

F. Consorti et al. / Computers & Education 59 (2012) 1001–10081008

Deladisma, A. M., Johnsen, K., Rajj, A., Rossen, B., Kotranza, A., Kalapurakal, M., et al. (2008). Medical student satisfaction using a virtual patient system to learn history-takingcommunication skills. Studies in Health Technology and Informatics, 132, 101–105.

Deladisma, M., Cohen, M., Stevens, A., Wagner, P., Lok, B., Oxendine, C., et al. (2007). Do medical students respond empathetically to a virtual patient? American Journal ofSurgery, 193(6), 756–760.

Dory, V., Gagnon, R., & Charlin, B. (2010). Is case-specificity content-specificity? An analysis of data from extended-matching questions. Advances in Health Sciences EducationTheory and Practice, 15(1), 55–63.

Ellaway, R. H., & Davies, D. (2011). Design for learning: deconstructing virtual patient activities. Medical Teacher, 33(4), 303–310.Ellaway, R. H., Poulton, T., Fors, U., McGee, J. B., & Albright, S. (2008). Building a virtual patient commons. Medical Teacher, 30(2), 170–174.Ellaway, R. H., Poulton, T., Smothers, V., & Greene, P. (2009). Virtual patients come of age. Medical Teacher, 31(8), 683–684.Eva, K. W. (2005). What every teacher needs to know about clinical reasoning. Medical Education, 39(1), 98–106.Fleetwood, J., Vaught, W., Feldman, D., Gracely, E., Kassutto, Z., & Novack, D. (2000). MedEthEx online: a computer-based learning program in medical ethics and

communication skills. Teaching and Learning Medicine, 12(2), 96–104.Fleming, M., Olsen, D., Stathes, H., Boteler, L., Grossberg, P., Pfeifer, J., et al. (2009). Virtual reality skills training for health care professionals in alcohol screening and brief

intervention. Journal of American Board of Family Medicine, 22(4), 387–398.Gesundheit, N., Brutlag, P., Youngblood, P., Gunning, W. T., Zary, N., & Fors, U. (2009). The use of virtual patients to assess the clinical skills and reasoning of medical students:

initial insights on student acceptance. Medical Teacher, 31(8), 739–742.Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. British Medical Journal, 327(7414), 557–560.Holzinger, A., Kickmeier, M., Wassertheurer, S., & Hessinger, M. (2009). Learning performance with interactive simulations in medical education: lessons learned from results

of learning complex physiological models with the HAEMOdynamics SIMulator. Computers & Education, 52(1), 292–301.Huwendiek, S., De leng, B. A., Zary, N., Fischer, M. R., Ruiz, J. G., & Ellaway, R. (2009). Towards a typology of virtual patients. Medical Teacher, 31(8), 743–748.Jerant, A. F., & Azari, R. (2004). Validity of scores generated by a web-based multimedia simulation patient case software: a pilot study. Academic Medicine, 79(8), 805–811.Kamin, C., Deterding, R., & Lowry, M. (2002). Student’s perceptions of a virtual PBL experience. Academic Medicine, 77(11), 1161–1162.Kamin, C., O’Sullivan, P., Deterding, R., & Younger, M. (2003). A comparison of critical thinking in groups of third-year medical students in text, video, and virtual PBL case

modalities. Academic Medicine, 78(2), 204–211.Kandasamy, T., & Fung, K. (2009). Interactive internet-based cases for undergraduate otolaryngology education. Otolaryngology and Head Neck Surgery, 140(3), 398–402.Kerfoot, B. P., Baker, H., Jackson, T. L., Hulbert, W. C., Federman, D. D., Oates, R. D., et al. (2006). A multi-institutional randomized controlled trial of adjuvant web-based

teaching to medical students. Academic Medicine, 81(3), 224–230.Khan, K., Pattison, T., & Sherwood, M. (2011). Simulation in medical education. Medical Teacher, 33(1), 1–3.Kotranza, A., Lok, B., Deladisma, A., Pugh, C. M., & Lind, D. S. (2009). Mixed reality humans: evaluating behavior, usability and acceptability. IEEE Transaction of Visualisation and

Computer Graphics, 15(3), 369–382.Kumta, S. M., Tsang, P. L., Hung, L. K., & Cheng, J. C. Y. (2003). Fostering critical thinking skills through a web-based tutorial programme for final year medical studentsda

randomized controlled study. Journal of Educational Multimedia and Hypermedia, 12(3), 267–273.LeRoy Heinrichs, W., Youngblood, P., Harter, P. M., & Dev, P. (2008). Simulation for team training and assessment: case studies of online training with virtual worlds. World

Journal of Surgery, 32, 161–170.Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999). Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM

statement. Quality of reporting of meta-analyses. Lancet, 354(9193), 1896–1900.Nathoo, A. N., Goldhoff, P., & Quattrocchi, J. J. (2005). Evaluation of an interactive case-based online network (ICON) in a problem based learning. Advances in Health Sciences

Education Theory and Practice, 10(3), 215–230.Norman, G. (2005). Research in clinical reasoning: past history and current trends. Medical Education, 39(4), 418–427.Poulton, T., & Balasubramaniam, C. (2011). Virtual patients: a year of change. Medical Teacher, 33(11), 933–937.Poulton, T., Conradi, E., Kavia, S., Round, J., & Hilton, S. (2009). The replacement of “paper” cases by interactive online virtual patients in problem-based learning. Medical

Teacher, 31(8), 752–758.Sanders, C. L., Kleinert, H. L., Free, T., Slusher, I., Clevenger, K., Johnson, S., et al. (2007). Caring for children with intellectual and developmental disabilities: virtual patient

instruction improves students’ knowledge and comfort level. Journal of Pediatric Nursing, 22(6), 457–466.Schittek Janda, M., Mattheos, N., Nattestad, A., Wagner, A., Nebel, D., Färbom, C., et al. (2004). Simulation of patient encounters using a virtual patient in periodontology

instruction of dental students: design, usability, and learning, effect in history-taking skills. European Journal Dental Education, 8(3), 111–119.Senn, S. (2007). Trying to be precise about vagueness. Statistics in Medicine, 26, 1417–1430.Stevens, S. M., Goldsmith, T. E., Summers, K. L., Sherstyuk, A., Kihmm, K., Holten, J. R., et al. (2005). Virtual reality training improves students’ knowledge structures of medical

concepts. Studies in Health Technology and Informatics, 111, 519–525.Stevens, A., Hernandez, J., Johnsen, K., Dickerson, R., Raij, A., Harrison, C., et al. (2006). The use of virtual patients to teach medical students history taking and communication

skills. American Journal of Surgery, 191(6), 806–811.Sutton, A. J., Duval, S. J., Tweedie, R. L., Abrams, K. R., & Jones, D. R. (2000). Empirical assessment of effect of publication bias on meta-analyses. British Medical Journal, 320,

1574–1577.Triola, M., Feldman, H., Kalet, A. L., Zabar, S., Kachur, E. K., Gillespie, C., et al. (2006). A randomized trial of teaching clinical skills using virtual and live standardized patients.

Journal of General and Internal Medicine, 21, 424–429.Vash, J. H., Yunesian, M., Shariati, M., Keshvari, A., & Harirchi, I. (2007). Virtual patients in undergraduate surgery education: a randomized controlled study. Australia and New

Zealand Journal of Surgery, 77(1–2), 54–59.Vincent, D. S., Sherstyuk, A., Burgess, L., & Connolly, K. K. (2008). Teaching mass casualty triage skills using immersive three-dimensional virtual reality. Academic Emergency

Medicine, 15(11), 1160–1165.Vukanovic-Criley, J. M., Boker, J. R., Criley, S. R., Rajagopalan, S., & Criley, J. M. (2008). Using virtual patients to improve cardiac examination competency in medical students.

Clinical Cardiology, 31(7), 334–339.Wahlgren, C. F., Edelbring, S., Fors, U., Hindbeck, H., & Ståhle, M. (2006). Evaluation of an interactive case simulation system in dermatology and venereology for medical

students. BMC Medical Education, 6, 40.Waldmann, U. M., Gulich, M. S., & Zeitler, H. P. (2008). Virtual patients for assessing medical students – important aspects when considering the introduction of a new

assessment format. Medical Teacher, 30(1), 17–24.Youngblood, P., Harter, P. M., Srivastava, S., Moffett, S., Heinrichs, W. L., & Dev, P. (2008). Design, development, and evaluation of an online virtual emergency department for

training trauma teams. Simulation in Healthcare, 3(3), 146–153.Zary, N., Hege, I., Heid, J., Woodham, L., Donkers, J., & Kononowicz, A. A. (2009). Enabling interoperability, accessibility and reusability of virtual patients across

Europeddesign and implementation. Studies in Health Technology and Information, 150, 826–830.Zary, N., Johnson, G., Boberg, J., & Fors, U. G. (2006). Development, implementation and pilot evaluation of a web-based virtual patient case simulation environmentdWeb-SP.

BMC Medical Education, 6, 10.