Accepte

ReprintLaboratSchool oE-mail:

0039-60

� 2011

doi:10.1

750 S

Deconstructing laparoscopiccompetence in a virtual realitysimulation environmentConstantinos Loukas, PhD, Nikolaos Nikiteas, MD, Meletios Kanakis, MD, andEvangelos Georgiou, MD, Athens, Greece

Background. Virtual reality (VR) simulators play a substantial role in modern medical education andhave generated several performance parameters that are not always standardized and open to clear andeasy interpretation. Consequently, our study objective was to investigate how these parameters contributeto the enhancement of key competencies in laparoscopic surgical skills.Methods. We recruited 20 residents and 8 experienced surgeons to participate in this study. Theresidents were trained on 5 basic tasks (4 of them at two difficulty levels) using a commercially availableVR simulator. Study participants also performed an additional 3 complex tasks before and after trainingfor assessment purposes. The experienced surgeons served as controls and so only performed theassessment tasks. Performance parameters were grouped to reflect errors in dexterity, safety, and technicalskill. These errors, as well as the parameters of time and instrument velocity, were analyzed duringtraining and assessment.Results. Performance for training tasks demonstrated notable learning curves for most of the parametersthat were measured (ie, plateaus varied between the second and seventh VR training session). Velocitywas influenced least by the training (3 of the 5 tasks), while time and dexterity were influenced most (all5 tasks and for both difficulty levels). In the assessment tasks, technical skill was improved (P < .05) forsome study participants, but this improvement was not demonstrated in all of the complex procedurestested (eg, bowel suturing). There was a significant improvement in safety (all 3 tasks; P < .05), andtime to completion and dexterity (both of them in 2 tasks; P < .05). Experienced surgeons scored at agreater level than VR-trained residents in terms of time (all tasks; P < .05), safety and technical skill(bowel suturing; P < .05), as well as dexterity (adhesiolysis and bowel suturing; P < .05).Conclusion. VR simulation training contributed markedly to the enhancement of key surgicalcompetencies of residents. The proposed mapping of the simulator parameters may help program directorsand trainees evaluate important competency domains during VR-based surgical training. (Surgery2011;149:750-60.)

From the Medical Physics Laboratory Simulation Center (MPLSC), University of Athens School of Medicine,Athens, Greece

ACQUIRING FUNDAMENTAL SURGICAL SKILLS constitutes akey goal in every surgical training program. Estab-lished methods of assessing surgical competencesuch as the Objective Structured Clinical Examina-tion (OSCE) and the Objective Structured Assess-ment of Technical Skill (OSATS) have gainedwide acceptance as reliable tools for evaluatingoperative and cognitive skills in surgery, includinglaparoscopic surgery.1-3 The continued evolution

d for publication November 16, 2010.

requests: Constantinos Loukas, PhD, Medical Physicsory Simulation Center (MPLSC), University of Athensf Medicine, 75, Mikras Asias Str., 11527 Athens, Greece.cloukas@med.uoa.gr.

60/$ - see front matter

Mosby, Inc. All rights reserved.

016/j.surg.2010.11.012

URGERY

of minimally invasive surgery also has highlightedthe essential role of psychomotor skills.4 Modernsurgical practice requires the acquisition of severalcompetencies over a short period of time,imposing the need for development of efficient,high-quality educational programs.

Virtual reality (VR) simulators are frequentlyused for technical skill training in the early part ofthe learning curve.5 The main goal of a simulationtraining program should be to acquire skills in asimulated environment and then transfer thoseskills from VR to reality, thus providing a saferand more effective educational method.6-9 Toassess the effectiveness of that skills transfer, thereis a great need to create validated measures ofperformance using the output parameters of VRsimulators.

SurgeryVolume 149, Number 6

Loukas et al 751

Various programshave beenproposed for essentialand procedural skill training and assessment,6,10,11

and a whole-procedure VR training curriculum wasrecently proposed using a scientific methodology.9

Development of objective measures of laparoscopicskills is important to confirm the role of simulationin minimally invasive surgery. Hence, simulatorsshould be able to providemetrics that aremeaningfuland informative to trainees at low- and high-skilllevels.12

Most VR simulators output several performanceparameters such as the instruments’ path length,and completion time. Perhaps most importantly,VR simulators enumerate the different types oferrors that are specific to the task. Although thisenumeration is useful, the errors are usually ana-lyzed independently, which imposes difficultieswhen trying to establish a relationship betweenthe errors and potentially more meaningfulcompetency domains. In this study, we focusedon the contribution of VR simulation training tothe enhancement of competency skills that havedirect implications for applied surgical practice.

To move our investigation in this direction, wefirst tried to classify and then quantify the outputparameters of a VR simulator into 3 main cate-gories that reflect fundamental types of skill:Safety, Dexterity, and Technical Skill. This decon-struction model arose from relevant studies pro-viding a measurable definition of surgical competenceand an analysis of the technical, cognitive, andmanual skills necessary to perform a surgicalprocedure.13,14

Our main goal was to develop a unified quan-titative measure for each category by combiningseveral performance parameters of a VR simulator.Time to task completion and Velocity (which wedefined as the mean rate of change of the instru-ments’ position) also were included, leading to atotal of 5 endpoints per task. Progression in eachcomposite measure during training establishedlearning curves that provided validation of theseendpoints for skill assessment.

METHODS

Participants. The study was conducted within aperiod of 4 months in the Medical Physics Labo-ratory Simulation Center (MPLSC) of the AthensUniversity Medical School, Greece. Institutionalreview board approval was obtained for the studyprotocol. Group A (hereafter referred to as resi-dents) included 20 junior level surgery residents(7 were women; 5 were left-handed) who wererecruited from 5 different hospitals via personalcommunication.

All trainees were at the beginning of theirresidency; they had performed a range of 10 to20 (median, 17) laparoscopic cholecystectomies(LCs). Group B, referred to as experienced surgeonsthroughout the remainder of this study, included8 surgeons who had extensive experience in lapa-roscopic surgery ($300 LCs). This latter groupwas included for the following reasons: (1) estab-lish construct validity by examining the ability ofthe simulator to discriminate between experiencedand intermediate surgeons in the performance ofprocedural tasks; and (2) establish performancestandards by comparing experienced surgeonswith residents before and after training.

Surgical simulator. Training and assessmentwere performed using a commercially availableVR simulator (LapVR, software build 965; Immer-sion Medical, San Jose, CA), which was equippedwith technologic advances including interactive,3-dimensional graphic models, haptic forcefeedback control, and performance tracking. Theembedded software consisted of 3 modules: (1) ba-sic skills, which included 6 tasks primarily lever-aged from the Fundamentals in Laparoscopic Surgeryprogram; (http://www.flsprogram.org, Society ofAmerican Gastrointestinal and Endoscopic Sur-geons, Los Angeles, CA), (2) procedural skills(eg, running the bowel, bowel suturing, and adhes-iolysis); and (3) laparoscopic procedures (eg, sev-eral cholecystectomy and obstetrics/gynecologyprocedures).

The basic skills module allowed trainees to de-velop essential abilities to carry out laparoscopicsurgery. The procedural skills module advanced theuser into more complex tasks that were performedunder more surgically realistic conditions, such asbowel manipulation, stitching, and suturing. Eachof the tasks had 3 levels of difficulty: easy (L1),medium (L2), and hard (L3). Figure 1 shows sam-ple screenshots of 2 basic tasks (Fig 1, A and B),1 procedural task (Fig 1, C), and a cholecystectomy(Fig 1, D). The simulator registered a range ofparameters for each exercise that could assessperformance objectively. Table I shows a detaileddescription of the subset of simulator tasks used inthis study.

Training curriculum. Before training, all partic-ipants received an identical instructional tutorialto familiarize themselves with the equipment. Par-ticipants initially performed a trial session for eachtask at the easy level to enhance their familiariza-tion. Figure 2 shows the curriculum, which wasdivided into 3 sections: pre-assessment, trainingmodule, and post-assessment. Residents wereasked to follow the entire training curriculum,

Fig 1. Snapshots of 4 virtual reality simulation tasks during training: (A) needle driving; (B) knot tying; (C) bowelsuturing; and (D) laparoscopic cholecystectomy.

SurgeryJune 2011

752 Loukas et al

whereas experienced surgeons performed only thefirst section (the pre-assessment module).

The first module (pre-assessment) included 2repetitions (ie, sessions) of the following proce-dures: adhesiolysis, bowel suturing, and a LC. The2 scores for each endpoint of the 3 tasks wereaveraged to generate the final score (eg, G1i, G2i,G3i, in which the numerals 1, 2, and 3 denotethe task number, and the subscript i is the indexof the endpoint measured, as in i = 1, 2, 3, 4,and 5). A detailed description of the endpoints isprovided in Assessment methods and endpoints.

The tasks in this section were selected becauseof their variety in complexity (eg, adhesiolysis vscholecystectomy), competency (eg, suturing vscutting), difficulty (adhesiolysis vs suturing), andcognitive skill (eg, adhesion vs cholecystectomy).Experienced surgeons served as controls for thesepre-assessment tasks.

The second section (ie, the training module)contained 6 basic tasks with graduated difficulty andcomplexity (Table I). The number of sessions anddifficulty levels increased as residents advancedthrough the curriculum. For example, simple tasksincluded camera navigation. Although these taskswere important for the trainee to become fully en-gagedwith the simulator’s functionality and graphicenvironment, they were rather tedious even at themedium level, depending on each participant’sexperience.

In contrast, tasks such as clipping, knot tying,and needle driving were considered more chal-lenging due to their surgical and technical con-tent. These types of tasks required additionalrepetitions/sessions for the participants to achievethe task’s full educational value.

In the final section (ie, post-assessment), resi-dents performed the same tasks contained in thepre-assessment. As in the pre-assessment, each ofthe tasks was performed twice, allowing trainees torevise potentially unfortunate manipulations. Thefinal endpoint score for each of the 3 tasks was H1i,H2i, H3i (subscript i = 1, 2, 3, 4, and 5). The inter-session interval was set for a minimum of 1 hour toavoid the confounding influence of repetitionpatterns in the data collected.

Assessment method and endpoints. VR simula-tors generate a great number of outcome param-eters such as errors, completion time, and pathlength. As mentioned earlier, a great deal ofresearch9,12,13,15-17 has focused on the trainee’seconomy of movements (equivalent to the pathlength). Despite the crucial importance of errorparameters, the underlying surgical competencerepresented by a low error rate is rarely includedin VR output data.6,9-11,18

In this study, we tried to group the performanceparameters into 3 main sets that reflect the funda-mental categories of surgical competence, alsoknown as competence skills, namely: Dexterity,

Table I. Description of tasks performed using a virtual reality surgical simulation system*

Tasks Description

Essential tasksCamera navigation Navigate and maneuver the camera to locate a series of 4

objects (targets), and hold focus on the object for apredetermined length of time.

Peg transfer Reach for and grasp a 3D peg and then place it into apredetermined hole in the simulated pegboard (4 pegs/session).

Cutting Apply traction with one hand while using the other to cuttextural pattern along a circular boundary using scissors.

Clipping Place a number of clips on 4 predefined points in an artery tostop blood flow and then cut right in the middle.

Needle driving Orient and drive a curved needle through 2 circles (entry andexit) drawn on a block with tissue-like properties (3 blocks/session).

Knot tying Manipulate a length of suture that has been pre-driventhrough a tissue-like material into a knot consisting of adouble and then 2 single throws (3 knots/session).

Procedural tasksAdhesiolysis Manipulate the bowel safely with 1 hand while using the other

to cut adhesions using scissors.Bowel suturing Close an opening in a length of a bowel using 2 interrupted

stitches (tied knots), each 1 having 3 throws (a double andthen a single).

Surgical procedureLC Dissect the gallbladder; identify, dissect, clip, and divide the

cystic duct and cystic artery; and dissect the gallbladderfrom the liver bed.

*The virtual reality simulator used in this study was the LapVR (software build 965; Immersion Medical, San Jose, CA).3D, 3-Dimensional; LC, laparoscopic cholecystectomy.

SurgeryVolume 149, Number 6

Loukas et al 753

Safety, and Technical skill. Our purpose was to gener-ate a unified error measure for each of thesecategories.

Starting with the simplest of the skills, wedefined Dexterity as a motion-based skill represent-ing the instrument trajectory and motion pattern.Dosis et al,15 as well as other researchers cited intheir study, demonstrated that this parameterholds construct validity. Path length is consideredthe cornerstone of this type of skill. Kinematicanalysis of surgical instruments also has led to thedevelopment of dexterity assessment devices(such as the Imperial College Surgical AssessmentDevice19), which have been validated successfullyin both VR-simulated and real operatingconditions.20,21

The second composite measure, Safety, was re-lated primarily to the occurrence of an injury to atissue or material due to awkward manipulations(eg, tearing off a suture) while performing VRsimulator tasks. For this study, Safety was defined asthe sum of all the complications measured duringthe performance of a task. In the knot-tying task,for example, the safety error equaled the sum of

the number of times the following acts occurred:(1) the unprotected needle tip came in contactwith the haptic wall, (2) the target material was in-jured with the needle, and (3) the suture was over-stretched. Figure 1, B, provides a snapshot of thetask, andTable I describes the task. A correspondingsafety error was generated for each of the other tasksin a similar manner; Table II provides a detaileddescription of these errors for each task.

Technical skill, the third composite measure, wasassociated with cognitive ability and proficiency.Thus, trainees were assessed primarily on the tech-nical aspects associatedwith a task by demonstratingtheir knowledge of ‘‘what to do’’ rather than ‘‘how todo it’’ (Dexterity) or ‘‘what not to do’’ (Safety). Forexample, a technical skill error while performing acholecystectomy equaled the total number of tech-nical errors that occurred while performing thatprocedure. Such errors would include whether theproximal/distal duct (or the artery) was clippedincorrectly or the gallbladder was not dissectedthoroughly (ie, a minimum amount of peritoneumand fat tissue in Calot’s triangle had to be dissectedto perform that task correctly).

Fig 2. The components and grading system of the train-ing curriculum followed by residents (Group A). Experi-enced surgeons (Group B) only performed the tasksoutlined in first module (pre-assessment).

SurgeryJune 2011

754 Loukas et al

In addition to the metrics listed above, ouranalysis included the following variables: time (ie,the amount of time needed to complete a task)and velocity (ie, the mean rate of change in theinstruments position), resulting in a total of 5endpoints measured per task.

Statistical analysis. The learning curves gener-ated by residents were analyzed using the Friedmantest, a nonparametric repeated measures analysis ofvariance test (MATLAB Statistics Toolbox, versionR2009a; The MathWorks, Natick, MA). Statisticalsignificance for the study was set at P < .05. If therewas a statistically significant difference (P < .05),multiple comparisons procedures (ie, sequentialpairwise comparisons) were carried out to identifya session at which a point failed to show additionalsignificant decreases (ie, plateau).

To retain a prescribed familywise error ratea = 5%,a Bonferroni adjustment was used to compensate formultiple comparisons.Comparisonsof performances

before and after (pre-/post-assessment) training andbetween residents and experienced surgeons werecarried out with theMann-Whitney U test and, whereappropriate, the McNemar test (when the outcomewas binary).

RESULTS

Learning curves. As shown in Table III, all tasksdemonstrated notable learning curves for mostendpoints. Plateaus varied from the second andthird sessions (eg, simple tasks, such as camera nav-igation and peg transfer) to the seventh session(eg, complex tasks, such as needle driving andknot tying). Velocity seemed to improve with repe-tition throughout the duration of the study andhad the least number of plateaus (only for cameranavigation, peg transfer, and cutting) comparedwith the other parameters.

In contrast, Time and Dexterity showed a pla-teau for all tasks performed and at both difficultylevels in 4 tasks; some examples are shown inFigure 3, A and B. Safety and Technical skill showedplateaus in 2 of the 3 tasks measured (ie, needledriving at both difficulty levels and clipping at themedium level), whereas Safety showed an addi-tional plateau at the medium level in the knot-tying task. Figure 3, C and D, show some examplesof the Safety and Technical skill plateaus. Clippingdemonstrated a flat curve for Safety and Technicalskill at the easy level, although there was a plateauat the medium level. Technical skill was the only pa-rameter that did not demonstrate a plateau forknot tying in both the medium and difficulty levels.

Intergroup competency comparison. A summaryof all statistical comparisons between each grouppair (pre-assessment and post-assessment residentsand experienced surgeons) is shown in Table IV.Sample box plots for 3 different endpoints areshown in Figures 4 and 5. In particular, Figure 4shows examples in which residents showed no sig-nificant improvement after training (P < .05),whereas Figure 5 shows examples in which the im-provement was significant (P < .05).

The comparison between experienced surgeonsand residents pre-assessment revealed a significantstatistical difference in almost all tasks and end-points (Table IV). Only the Velocity parameter inthe adhesiolysis task did not demonstrate a statisti-cally significant difference. Figure 4, A and B,show examples of the cholecystectomy taskcomparing the study parameters of Velocity andTime, respectively. Figure 4, C, shows how the 2groups (residents pre-assessment and experiencedsurgeons) compare in suturing, a finding that hasbeen confirmed by others.22

Table II. Description of the 3 types of surgical skills measured for each of the simulation tasks performedthroughout the curriculum

Tasks Safety (error) Technical skill (error) Dexterity

Essential tasks(training module)

Camera navigation N/A N/A Camera path lengthPeg transfer N/A N/A Total path length (L + R)Cutting Total % cutting outside

boundary areaTotal no. unsuccessfulattempts for cutting

Total path length (L + R)

Clipping Total no. clips misplaced % maximum vessel stretch Total path length (L + R)Needle driving Total no. times that needle

came in contact with wall/ceiling or tag broken

Total no. times needle hitoutside entry/exit circle,dropped, or drive failure

Total path length (L + R)

Knot tying Total no. times needle camein contact with wall, tissuewas injured, suture wasoverstretched

Total count if knots notlocked, throws notalternating, taillength >5 cm

Total path length (L + R)

Procedural tasks(pre-/post-assessment)

Adhesiolysis Bowel injury (yes/no) % adhesions remaining Total path length (L+ R)Bowel suturing Total count if bowel

nicked or ripped orif suture damaged

Total no. (knots 3 throws)remaining

Total path length (L + R)

Surgical procedure(pre-/post-assessment)

LC Total count if injuryin duct, cystic artery,gallbladder, or liver

Total count if duct/arteryproximal/distal not clippedcorrectly or gallbladder notdissected thoroughly

Total path length (L+ R)

N/A, Not applicable; L, left; R, right; LC, laparoscopic cholecystectomy.

Table III. Learning curve data for the residents group*

Essential tasks(training module) Safety, error Technical skill, error

Dexterity,cm

Time,s Velocity, cm/s

Camera navigation NA NA 2nd (105) 3rd (46) 3rd (2.2)Peg transfer NA NA 3rd (530) 3rd (112) 4th (5.1)Cutting (L1) Xy X 3rd (440) 4th (117) 4th (3.8)Cutting (L2) X X 6th (372) 5th (158) XClipping (L1) X X 2nd (219) 2nd (69) XClipping (L2) 2nd (0) 2nd (18) 3rd (248) 3rd (58) XNeedle driving (L1) 4th (0) 5th (15) 5th (590) 5th (233) XNeedle driving (L2) 3rd (1) 6th (25) 7th (680) 6th (295) XKnot tying (L1) X X 5th (510) 5th (112) XKnot tying (L2) 7th (2) X 6th (1440) 6th (262) X

*Values outside parentheses denote plateau session; values within parentheses are the corresponding median values.yX denotes no significant plateau (ie, a flat learning curve).L1, Easy level; L2, medium level.

SurgeryVolume 149, Number 6

Loukas et al 755

Comparing residents post-assessment with expe-rienced surgeons, Time was the only parameter thatcontinued to discriminate the 2 groups in allassessment tasks (Table IV). Compared to thetrained residents, experienced surgeons couldcomplete the LC faster and with greater instrumentvelocity (see residents post-assessment vs experi-enced surgeons in Fig 4, A and B). Experienced

surgeons also performed bowel suturing withgreater Dexterity (Fig 4, C). The latter may be dueto the fine manipulations required in laparoscopicsuturing, a task in which skill is a product of exten-sive surgical experience.

Competency enhancement. Comparing residentspre-assessment with residents post-assessment, themost significant improvement was demonstrated in

Fig 3. Examples of learning curves (box plots) for 4 different endpoints and for 4 tasks performed by residents (GroupA) during the training phase: (A) Dexterity (path length) for camera navigation; (B) Time for peg transfer; (C) Tech-nical Skill error for needle driving (easy level); and (D) Safety for knot tying (medium level). The horizontal line withineach box represents median scores; the crosses correspond to outliers.

Table IV. Statistical comparison results (residents-pre/post-training and experienced surgeons) for the 5endpoints measured for each of the 3 tasks included in the assessment module of the curriculum

Assessmenttasks

Residents-pre vsexperienced surgeons

Residents-post vs.experienced surgeons

Residents-pre vsresidents-post

S TS D T V S TS D T V S TS D T V

Adhesiolysis O O* O O — — — O O — O O O O —Bowel suturing O O* O O O O O O O O O — — O OLC O O O O O — — — O O* O O O — —

Residents-pre, Residents before training; residents-post, residents after training; S, safety; TS, technical skill; D, dexterity; T, time; V, velocity; O, statistical sig-nificance (P < .05); ---, no significance; O*, great statistical significance (P < .01).

SurgeryJune 2011

756 Loukas et al

adhesiolysis, where only Velocity remained un-changed (P < .05; Table IV). As described earlier,Velocity showed the least improvement during train-ing. Figure 5, A, depicts an example of an improve-ment in Dexterity (residents pre-assessment vsresidents post-assessment). Experienced surgeonscould perform adhesiolysis with an even shorterpath length (ie, greater Velocity).

For cholecystectomy, significant improvementwas shown for all 3 categories: Dexterity, Safety,and Technical skill (P < .05). Figure 5, B, shows anexample of the Technical skill in the group of res-idents pre-assessment and post-assessment. Neither

Time nor Velocity was influenced after training(P > .05). Basic training enhanced the perfor-mance of residents by allowing them to completethe full procedure within the same amount oftime, with less complications (eg, injuries to theliver), greater proficiency (eg, precise dissectionof the gallbladder), and greater dexterity (shorterpath length).

For bowel suturing, residents demonstratedsignificant improvement in Safety (Fig 5, C) bymaking fewer mistakes that caused injuries, suchas a bowel nick or rip. Time and Velocity alsowere improved (P < .05), but neither Technical

Fig 4. Sample box plots for 3 different endpoints(Velocity and Time for laparoscopic cholecystectomyand Dexterity for bowel suturing) in which residents(Group A) did not show statistically significant improve-ment after training. Box plots for experienced surgeons(Group B) are shown for comparison. The graphs show2 tasks: laparoscopic cholecystectomy and bowel sutur-ing. Lap Chole, laparoscopic cholecystectomy; ResidentsPre, residents at the pre-assessment level of training;Residents Post, residents at the post-assessment level oftraining.

Fig 5. Sample box plots for 3 different endpoints(Dexterity, Technical Skill, and Safety) in which resi-dents (Group A) demonstrated marked improvementafter training. Box plots for experienced surgeons(Group B) are shown for comparison. The endpointscorrespond to the 3 tasks of the assessment modules:adhesiolysis, laparoscopic cholecystectomy, and bowelsuturing. Lap Chole, laparoscopic cholecystectomy; Resi-dents Pre, residents at the pre-assessment level of training;Residents Post, residents at the post-assessment level oftraining.

SurgeryVolume 149, Number 6

Loukas et al 757

SurgeryJune 2011

758 Loukas et al

skill nor Dexterity were affected, probably due tothe advanced nature of this task.

DISCUSSION

Laparoscopic surgery requires the fine coordi-nation of cognitive, dexterity, and technical skills.Analyzing this type of surgery in terms of thesecompetencies is a key step toward developingefficient criteria for training surgeons in a struc-tured and systematic manner. In most surgicalsimulation studies, great emphasis is placed onthe learning curves of basic and more advancedtasks, whereas performance comparison for morecomplex procedures is usually omitted.

In contrast, this study uses assessment tasks with agreater procedural and surgical context comparedwith tasks used for training alone. We selected thesetasks precisely because of their complexity andbecause they require trainees to demonstrate di-verse skills. Moreover, our assessment was based ontraditional parameters, such as time and pathlength but also on competence domains that arecrucial to clinical surgical practice.

VR simulators output several performance pa-rameters that could be broadly classified into 3categories: (1) time measurements, (2) motionmetrics (eg, path length and the number andsmoothness of hand movements), and (3) task-specific errors.6,9-13,15-18 When our study was pub-lished, the current literature suggested that motionmetrics may be the most sensitive parameter to dis-tinguish subtle performance differences, whereastime may be superior for basic task assessment dur-ing training.12 To our knowledge, the role of errormetrics in relation to the other parameters, how-ever, has yet to be elucidated. Rather, these errorsare typically presented in the form of an abstractscore without any connection to the underlyingtype of surgical competency being measured.

The error categories reported in this study arebased on the popular OSATS scoring system.2

Thus, established parameters, such as respect oftissue, time, motion, knowledge of specific proce-dures, and instrument handling, are closely relatedwith Safety, Time, Velocity, Proficiency, and Dexter-ity, respectively. Our intention was to classify theperformance parameters into skill categories thatare quantifiable and comparable. This approachdiffers from the methods reported in other studieswhere, apart from Dexterity, little distinction ismade among the various types of errors definedwithin the software used with the VR trainingsystem.9,22

Safety is a crucial parameter that, in this study,was tied to the occurrence of an injury. Technical

skill was associated primarily with technical knowl-edge, judgment, and the resulting psychomotoractions.13 Dexterity was related to the spatialmotionpattern of the instrument, and has been reported tohave construct validity.13 In addition, the pathlength and the numbers of movements are consid-ered to be the cornerstones of dexterity.19

Kinematic analysis of the instruments also hasled to the development of integrated surgicalassessment devices (such as the Imperial CollegeSurgical Assessment Device15), which have beenvalidated in VR, simulated, and real operating con-ditions.20,21 Our results showed that repetition ofthe task did not have much of an influence on Ve-locity during training, probably due to the carefuland slow manipulations required by the tasks. Theparadoxical increase in instrument velocity duringbowel suturing was mostly due to awkward, ratherthan technically proper, fast manipulation of theinstruments. This observation could be confirmedindirectly by the unimproved Dexterity and Tech-nical skill parameters associated with this task, be-cause very few trainees managed to perform therequired number of stitches or throws.

Measures of Time and Dexterity were influ-enced by the VR training more than any of theother parameters, except for the task involvingadhesiolysis. This finding is probably due to theresidents’ limited experience in performing dissec-tions that are both fast and elegant. For cholecys-tectomy, basic training led to improved safety,proficiency, and dexterity. These results suggestthat the VR training enabled the residents tomanage their operative time more effectively byminimizing idle or redundant movements and bybeing more efficient in certain parts of the proce-dure (eg, during dissection). Safety and Technicalskill also were improved notably by VR training,with the exception of complex tasks such as bowelsuturing.

Although values for Technical skill and Dexter-ity remained unchanged after training, there was asignificant difference in all 5 parameters betweenthe 2 groups, particularly for suturing tasks. Thisfinding may be explained by the fact that laparo-scopic suturing is considered a complex proce-dure, and its elegant performance requires regularand continuous practice. Alternatively, one mightargue that the absence of improvement is due tothe VR simulator’s inability to accurately representa suturing context.

Our experience has demonstrated that the VRsimulator that we used in this study (LapVR;Immersion Medical) is somewhat unique by incor-porating haptic and force feedback feeling

SurgeryVolume 149, Number 6

Loukas et al 759

properties, thus providing a sense of ‘‘touch’’during task performance. This feature does notusually come with VR simulators, making virtualsuturing often seem unrealistic to the user. Unfor-tunately, haptic feedback (ie, tactile feedback tech-nology) is only 1 aspect of simulator performancethat contributes to the user’s sense of ‘‘reality.’’

Other important measures of an effective simu-lation include the realistic deformation propertiesof the virtual object (eg, the bowel in this study) andthe object’s interaction with the appropriate tool(eg, needle or suture). Despite improvement incomputational matters such as navigation, imagerendering, texture mapping, and collision ofdeformable objects, other aspects of tool/tissueinteraction are needed to achieve realistic soft tissuerepresentation (eg, bowel indentation during sutur-ing, bowel puncturing, and suture manipulationduring knot tying). Unless these issues are resolved,any parameters relevant to the user’s commission oferrors, such as a bowel injury or tear, will retain adegree of uncertainty when analyzing trainingresults.

Traditional surgical training with low-fidelity sim-ulators (eg, box trainers) is the first choice in manycenters, primarily due to these models’ low cost,high availability, and realistic sensation during in-strument manipulation. The disadvantages of usingthis type of simulator include difficulty in obtainingperformance parameters (compared to VR), semi-automated assessment, reviewer-dependent evalua-tion, and limited flexibility to regulate difficultylevels. To date, an optimal solution has not beendeveloped. An answer might be found in the crea-tion of augmented reality simulation tasks thatcombine the haptic feedback sensed in reality withthe multifaceted flexibility of VR (ie, a combinationof various virtual models).

Regarding the differentiation among the groups’learning curves, all the tasks in this study demon-strated statistical significance for most endpoints.Only Velocity generated the least number of plateaus,implying that using it as a performance parameter inVR training isnot aneffectiveway tomonitorprogress.Technical skill was the only parameter that did notdemonstrate a plateau at both levels of knot tying, afinding that may be explained by the complex profi-ciency required to perform this task laparoscopically.

Several studies4,18,23 have assessed the validityand learning curves of different VR simulators. Inthis study, construct validity was defined as the abil-ity to differentiate between residents (inexperi-enced) and experienced surgeons. Our resultsshowed that the VR simulator used in this study(LapVR; Immersion Medical) has the appropriate

construct validity for 3 procedural tasks when stud-ied using additional subjects with a greater rangeof expertise.

In particular, our results demonstrated a nota-ble difference in Dexterity between the residentand experienced surgeon groups in the bowel-suturing task. The Technical skill performanceparameter had a marked sensitivity (P < .01) in dis-criminating the 2 groups in the adhesiolysis andsuturing procedures. This finding may be ex-plained by the great level of proficiency requiredto undertake these tasks, especially bowel suturing.Interestingly, experienced surgeons differed signif-icantly from the pre-assessment residents in Safetyfor all 3 tasks (P < .05), including adhesiolysis, thetask with the least risk of committing an injury.This finding suggests that Safety may have a sensi-tive role in discriminating groups with differentlevels of surgical experience.

We also studied the added value of basic train-ing in the performance of bowel suturing, adhes-iolysis, and cholecystectomy. These tasks werechosen because they had features in commonwith tasks that are practiced repeatedly duringtraining (eg, knot tying with bowel suturing andcutting with adhesiolysis) and because they re-quired the combined performance of several skills.Our results showed that deliberate training of basictasks improved various aspects of surgical perfor-mance in more complex tasks (Table IV; residentsbefore training [at the pre-assessment curriculumlevel] vs residents after training [at the post-assessment level). Our conclusion in terms of train-ing is echoed in a recent study by Kahol et al.24

Their study demonstrated that warm-up practicewith simple exercises can lead to a significant in-crease in proficiency in follow-up tasks with moreprocedural content, such as electrocautery.24

A research limitation of this study may be thedecision to exclude the hard level (ie, the greatestlevel of difficulty) during basic training. Althoughour results demonstrated improved performancewithout participants training at the hard level,future studies are needed to determine whether in-dividuals who practice simple tasks at the hard leveldemonstrate greater proficiency when performingmore complex tasks. Results of such a study mayshow that acquiring a standard psychomotor skilllevel may be enough for individuals to carry outmore complex procedures that also require precisesynchronization of individual skills. To resolvethese issues, we plan to carry out an expanded studyto investigate whether individuals who receiveadditional training at a more difficult level im-proves their performance of more complex tasks.

SurgeryJune 2011

760 Loukas et al

In conclusion, we propose an assessment meth-odology that maps the LapVR (ImmersionMedical)parameters to OSATS–based performance criteria.After basic training, safer performancewas achievedinmore complex tasks. Technical skill andDexterityimproved, but not for bowel suturing. Comparedwith expert surgeons, residents had an equivalentperformance regarding their Safety and Technicalskill in adhesiolysis, in addition to Dexterity incholecystectomy.

This study has left a significant issue unresolved:the effectiveness of skill transfer in the operatingroom. Yet, automatically obtaining measurementssuch as those depicted in Table II would be diffi-cult to gather in a ‘‘real’’ setting versus a virtualone. The LapVR (Immersion Medical) generatesseveral parameters that help untangle surgicalcompetence in a structured manner. Based onthis study, we believe that surgical training pro-grams should incorporate validated VR simulationsystems to enhance competencies. The operatingtheater could then become a place where traineesapply and refine the skills acquired in the VRenvironment with greater safety, dexterity, andproficiency.

REFERENCES

1. Tang B, Hanna GB, Carter F, et al. Competence assessmentof laparoscopic operative and cognitive skills: ObjectiveStructured Clinical Examination (OSCE) or ObservationalClinical Human Reliability Assessment (OCHRA). WorldJ Surg 2006;30:527-34.

2. Martin JA, Regehr G, Reznick R, et al. Objective structuredassessment of technical skill (OSATS) for surgical residents.Br J Surg 1997;84:273-8.

3. van der Vleuten CPM, Swanson DB. Assessment of clinicalskills with standardized patients: state of the art. TeachLearn Med 1990;2:58-76.

4. Gallagher AG, Satava RM. Virtual reality as a metric for theassessment of laparoscopic psychomotor skills. Learningcurves and reliability measures. Surg Endosc 2002;16:1746-52.

5. Aggarwal R, Darzi A. Technical-skills training in the 21stcentury. N Engl J Med 2006;355:2695-6.

6. Munz Y, Almoudaris AM, Moorty K, et al. Curriculum-basedsolo virtual reality training for laparoscopic intracorporealknot tying: objective assessment of the transfer of skillfrom virtual reality to reality. Am J Surg 2007;193:774-83.

7. Van Sickle KR, Ritter EM, Smith CD. The pretrained novice:using simulation-based training to improve learning in theoperating room. Surg Innov 2006;13:198-204.

8. Sturm LP, Windsor JA, Cosman PH, et al. A systematic re-view of skills transfer after surgical simulation training.Ann Surg 2008;248:166-79.

9. Aggarwal A, Rochet P, Dias A, et al. Development of a virtualreality training curriculum for laparoscopic cholecystec-tomy. Br J Surg 2009;96:1086-93.

10. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparo-scopic surgeons. Ann Surg 2006;244:310-4.

11. Aggarwal R, Grantcharov T, Moorthy K, et al. A competency-based virtual reality training curriculum for the acquisition oflaparoscopic psychomotor skill. Am J Surg 2006;191:128-33.

12. Stefanidis D, Scott DJ, Korndorffer JR Jr. Do metrics matter?Time versus motion tracking for performance assessment ofproficiency-based laparoscopic skills training. SimulHealthc 2009;4:104-8.

13. Bann S, Khan MS, Datta VK, et al. Technical performance:relation between surgical dexterity and technical knowl-edge. World J Surg 2004;28:142-7.

14. Satava RM, Gallagher AG, Pellegrini CA. Surgical compe-tence and surgical proficiency: definitions, taxonomy, andmetrics. J Am Coll Surg 2003;196:933-7.

15. Dosis A, Aggarwal R, Bello F, Moorthy K, et al. Synchronizedvideo and motion analysis for the assessment of proceduresin the operating theater. Arch Surg 2005;140:293-9.

16. Hogle NJ, Briggs WM, Fowler DM. Documenting a learningcurve and test-retest reliability of two tasks on a virtual real-ity training simulator in laparoscopic surgery. J Surg Ed2007;64:424-30.

17. Pellen M, Horgan L, Barton JR, et al. Laparoscopic surgicalskills assessment: can simulators replace experts? WorldJ Surg 2009;33:440-7.

18. Grantcharov TP, Funch-Jensen P. Can everyone achieve pro-ficiency with the laparoscopic technique? Learning curvepatterns in technical skills acquisition. Am J Surg 2009;197:447-9.

19. Mackay S, Datta V, Mandalia M, et al. Electromagnetic mo-tion analysis in the assessment of surgical technical skill: re-lationship between time and movement. ANZ J Surg 2002;72:632-4.

20. Datta V, Gillies DF, Mackay S, et al. Motion analysis in theassessment of surgical skill. Comp Meth Biomech BiomedEng 2001;4:515-23.

21. Datta V, Mackay S, Mandalia M, et al. The use of electromag-netic motion tracking analysis to objectively measure opensurgical skill in the laboratory-based model. J Am CollSurg 2001;193:479-85.

22. BalasundaramI,AggarwalR,DarziA. Short-phase trainingona virtual reality simulator improves technical performance intele-robotic surgery. Int J Med Robot 2008;4:139-45.

23. McDougall EM, Corica FA, Boker JR, et al. Construct validitytesting of a laparoscopic surgical simulator. J Am Coll Surg2006;202:779-87.

24. Kahol K, Satava RM, Ferrara J, Smith ML. Effect of short-term pretrial practice on surgical proficiency in simulatedenvironments: a randomized trial of the ‘‘preoperativewarm-up’’ effect. J Am Coll Surg 2009;208:255-68.