351

TELEMEDICINE AND e-HEALTHVolume 12, Number 3, 2006© Mary Ann Liebert, Inc.

Multimedia Data Capture and Management for SurgicalEvents: Evaluation of a System

STEPHEN W. CONE, M.D., ANNA LEUNG, M.D., FRANCISCO MORA, M.D., AZHAR RAFIQ, M.D., and RONALD C. MERRELL, M.D.

ABSTRACT

The objective of this study was to design an electronic form of documentation of surgical pro-cedures, which would include audio and video recording of the entire surgical procedure.Video clips have shown promise for teaching surgical procedures. To date, no systems havebeen described to fully record video and audio of all events during a surgical procedure. Muchas such systems have aided the airline industry, surgical safety, documentation, and educa-tion could benefit from comprehensive, multimedia documentation systems. Four camcordersprovided views of: (1) anesthetic monitors, (2) laparoscopic images, (3) room view, and (4) sur-gical field view. All video and audio were combined with real-time written documentationof events within a simple, inexpensive database for archiving, review, and evaluation. Elec-tronic records provided answers to more than 90% of the structured review questions, leav-ing only 6% unanswered, versus 92% unanswerable based on the traditional paper records.This electronic documentation system provides a much more comprehensive and easily minedmeans of surgical documentation than traditional paper records.

Medical Informatics and Technology Applications Consortium, Department of Surgery, Virginia CommonwealthUniversity, Richmond, Virginia.

INTRODUCTION

DOCUMENTATION IS IMPORTANT in medicine.This is true for purposes of scientific in-

tegrity, legality, and education. Many areas ofmedicine are currently investigating and navi-gating to electronic means of documenta-tion.1–4 Much of the previous work in this areahas focused on specific tasks or segments of thesurgery, especially for skill evaluation and ed-ucational purposes.5–11

Following capture, editing of video docu-mentation becomes an issue. There should besome determination of the correct amount ofediting acceptable. Previous studies have found

the amount of storage necessary for completesurgical documentation to be excessive and be-yond the means of most users.12 However, asstorage and computing capabilities have be-come more within reach, these obstacles havebeen overcome. As Yamakawa et al.13 discov-ered, some forms of editing and compression,such as Moving Pictures Expert Group (MPEG)-4 formatting, degrade the surgical image toomuch for adequate documentation and mayonly be used for presentation purposes.

Determination of the ideal placement of cam-eras to capture all events of the surgery becomesan important issue. Some studies have deter-mined that an all-encompassing view of the en-

6238_08_p351-358 6/12/06 1:33 PM Page 351

tire room becomes important for documentationand evaluation of team interaction.14–16 Thisstudy concentrated on full video capture of sur-gery (from multiple views) integrated with datarepresentations of the surgical events for easilynavigable, real-time documentation of surgery.Archer et al.,17 found that residency traininghelps diminish the incidence of bile duct injuriesduring laparoscopic cholecystectomy. It is ourhope that full documentation may further aid inthe education of emerging laparoscopic surgeons.

We hypothesize that full multimedia surgi-cal documentation is possible with mainte-nance of data quality to ensure retrospectiveanalysis for any purpose.

MATERIALS AND METHODS

Operative documentation

In collaboration with the Cinterandes Foun-dation of Cuenca, Ecuador, researchers from theMedical Informatics and Telemedicine Appli-cations Consortium (MITAC) at Virginia Com-monwealth University (VCU) in Richmond,Virginia, installed four Sony Handycam®

DCR/TRV-27 MiniDV (Sony, New York, NY)camcorders in the Cinterandes Foundation Mo-bile Surgery Unit. Figure 1 shows the approxi-mate locations of the four cameras, designed toallow select views during surgery. Camera 1was directed to record video of the display of

anesthetic patient monitoring data, includingelectrocardiogram (ECG) and pulse oximetry.Camera 2 was directly connected to the laparo-scopic camera for video recording of the intra-abdominal view. Cameras 3 and 4 were directedto provide opposite views of the operatingtable, attempting to capture all activities of thesurgical team. Figure 2 shows video from eachof the four cameras, while Figure 3 shows ac-tual positioning of cameras 2 and 3.

Audio streams were recorded using three ofthe four cameras, as well. Acting as a videorecorder for the laparoscopic camera, camera 2did not have the ability to record audio. Cam-eras 1 and 3 recorded room audio throughbuilt-in microphones. Camera 4 recorded au-dio directly from the investigating surgeon, us-ing a wireless microphone to dictate surgicalevents as they occurred.

Additionally, a handheld Canon PowerShot®

S300 digital still camera (Canon, Lake Success,NY) was used in the operating room to captureadditional images, including patient airwayevaluation and images of the paper records.

A Microsoft Access® (Microsoft Corporation,Redmond, WA) -based database was devel-oped to record all pertinent events. The data-base was designed to attach date and time in-formation to each event description manuallyentered. All entries were to correspond withverbal cues from the attending surgeon, indi-cating important surgical events.

CONE ET AL.352

3

1

2

282

Operating Table 4

FIG. 1. Diagram of mobile operating room floor plan, showing locations of the four camcorders utilized for videocapture. Camera 1—for capture of anesthetic data; camera 2—for direct laparoscopy capture; cameras 3 and 4—forcapture of operative table video.

6238_08_p351-358 6/12/06 1:33 PM Page 352

This system was utilized for all laparoscopiccholecystectomies performed in the mobile surgical facility during a 2-day mission toManglaralto, Ecuador. During this time period,seven cholecystectomies were performed la-paroscopically, with digital recording as indi-cated. One of the authors (R.C.M.) was eitherthe attending surgeon, or present to orchestrateand dictate all of the procedures. The tapes foreach case were labeled and stored separatelyfor return to the United States and further eval-uation. The standard paper records of the Cin-terandes Foundation were photographed fortransport to the laboratory and comparison tothe video documentation. These photographicrecords were printed on paper for review pur-poses.

Video editing

Upon return to the United States, the videorecordings were transferred to .wmv format us-ing Microsoft MovieMaker® software with cap-ture parameters of 768 kilobits per second (Kbps)on a personal computer. Video and audio feedfrom the four cameras were combined within the

aforementioned database for simultaneous dis-play on a split screen for review, as in Figure 2.

Record review

Using questions designed by the investigat-ing surgeon (R.C.M.) (Table 1), one attendingsurgeon and three surgery residents (fluent inSpanish) from VCU reviewed the paper andelectronic records of six cases for comparison.The questions related to parts of the operationswell-recognized for possible error or associa-tion with complications. The first case was ex-cluded from the study because of electrical fluc-tuations resulting in loss of video recording.Review of the records, based on the study-spe-cific questionnaire, required an average of15–20 minutes per surgery. All responses wererecorded, with “unknown” accepted as a viableresponse when the reviewer felt the questioncould not be answered with the record pre-sented.

The query of electronic medical records ac-quired from patients undergoing surgery wasbased on 17 questions, 4 of which had a follow-up question for elaboration, totaling 21 ques-

MULTIMEDIA DATA CAPTURE AND MANAGEMENT 353

TABLE 1. QUESTIONS THAT COULD NOT BE ANSWERED BASED ON REVIEW OF PAPER OR ELECTRONIC RECORDS

Paper Electronic

1. What was the SaO2 at intubation? 16 12. Were the breath sounds bilateral and equal after intubation? 24 243. How much time transpired from the inclusion to cannulation for the 24 0

umbilical port?a. Was this excessive? 24 0

4. How much time transpired from the initial incision to the insertion of the 24 0last port?a. Was this excessive? 24 0

5. How much time transpired from the beginning of dissection to scissor 24 0division of the clipped duct?a. Was this excessive? 24 0

6. Was the CBD identified? 24 07. Was the origin of the cystic duct identified? 18 08. Was the gallbladder retracted laterally to expose the triangle of Callot? 24 09. What was the EBL for the gallbladder neck dissection? 24 0

10. Was the gallbladder perforated? 24 0a. More than once? 24 0

11. What was the EBL for the dissection of the gallbladder bed? 24 012. How much time was required for the gallbladder bed dissection? 24 013. Was the liver bed entered? 24 014. Was a specimen bag used? 24 015. How much time was required to remove the gallbladder after introduction 24 0

of the umbilical clamp?16. Was the umbilical fascia closed? 4 017. What was the SaO2 at extubation? 17 5

CBD, common bile duct; EBL, estimated blood loss.

6238_08_p351-358 6/12/06 1:33 PM Page 353

tions. Classification of the questions yieldedfour categories. The first category was con-cerned with anesthetic events. The second cat-egory asked reviewers to evaluate the capacityto identify anatomic landmarks (summarizedin Table 2) relevant to the cholecystectomy pro-cedure. A third category of questions dealt withthe quantifiable values of time required forcompletion of specific tasks essential in chole-cystectomies. The fourth category dealt withthe surgeons’ technique and the reviewers’abilities to estimate blood loss as a result. Be-cause this study was not designed to evaluatethe skills of the surgeons, these categories ofquestions were analyzed as the reviewers’ abil-ities to determine answers (rather than to spec-ify the quantified answers).

The reviewers were asked to complete thequestionnaire in two parts. The paper recordswere reviewed while completing one num-bered questionnaire per surgical case. Reviewof paper records required less than 2 minutesper case on average, once it was determinedthat the paper records could not possibly pro-vide the information sought. However, rootcause analysis for an adverse surgical outcomerequires weeks of paper review and interviews.After completion of the paper records, the elec-tronic records were reviewed, with identicalquestionnaires, by each reviewer.

RESULTS

Four reviewers evaluated both the digitaland the paper records for six cholecystectomiesperformed in Ecuador during April 2004. Eval-uations consisted of 21 questions, resulting ina total of 504 possible responses each for thedigital and the paper records. The paper recordcould not be interpreted or inform a response

in 463 (92%) instances. The electronic records,or data sets, could be reviewed in approxi-mately 20 minutes to answer virtually all ques-tions; only 30 questions could not be answeredof the 504 possibilities (approximately 6%).

Table 2 reveals some disparity among eval-uators with regard to identification of anatom-ical structures. Some evaluators thought theconfirmation of anatomy was adequate whileothers disagreed. Some disparity based on ad-equate anatomical exposure is realistic. Dis-agreements occurred only in particular in-stances, and were usually associated with asingle reviewer. All evaluators were able to es-timate the timed events related to technique butthey differed somewhat in the times recorded.In all instances, the evaluators were able to in-terpret the data (Table 3) but differed some-what in the interpretation in the beginning andending times.

DISCUSSION

Public expectation of airline safety has beenunderstandably high, but the tragedy of an air-line failure rarely left witnesses to the eventwho could reliably reconstruct the failure andinform decisions about safer operation in thefuture. Therefore, the passive acquisition of information by cockpit voice recorder and instrumentation data have made it possible to accurately reconstruct the interpersonal, me-chanical and even meteorological events pre-ceding an airline disaster. In surgery there aremany people who are in the operating roombut they may not be able to identify the failurethat leads to a bad result. For so long we havereasonably relied upon the judgment of the sur-geon confronted with the frailty of the patientto make the best of a situation. That reliance

CONE ET AL.354

TABLE 2. CONFIRMED OR NOT CONFIRMED ANATOMIC FEATURES IN 72 (3 � 24) QUERIES FOR EACH QUESTION

Anatomic Not Datastructure Confirm (%) confirm (%) transparency (%)

CBD identified 83.0 17.0 100Origin of cystic duct 87.8 13.0 100Triangle of Callot exposed 75.0 25.0 100

The electronic data always permitted a response but evaluators did not necessarily agree.CBD, common bile duct.

6238_08_p351-358 6/12/06 1:33 PM Page 354

should in no way be diminished. The dictatednote of the experienced surgeon is still the mostinsightful document we can have with regardto events in an operation. However, Way etal.,18 have noted that in laparoscopic error thesurgeon may see what is expected and not whatis actually happening on the path to an error.This certainly proved to be the case in airlinedisaster in which concerns about weather, me-chanical failure, etc., were predicated on the be-lief that the pilot and crew were performing totask and saw no problem they might have cor-

rected prior to failure. However, it is quite clearthat many of the aircraft disasters are causedby pilot error among experienced and highlyqualified individuals.19

The ability to capture events in the operatingroom (OR) is increasing just as the expectationof the public is rising with regard to OR safetyand accountability. The OR can be digitallycaptured in records and data can be stored andreviewed.1,2 However, the very amount of in-formation we have in analog, digital, and pa-per records can be a disincentive to accurate

MULTIMEDIA DATA CAPTURE AND MANAGEMENT 355

TABLE 3. EVALUATION OF ADEQUACY IN 72 (3 � 24) QUERIES

Skill parameters identified Concur (%) Not concur (%) Uncertain (%)

Cannulation of umbilical port 100.0 0.0 0.0Insertion of the last trocar port 100.0 0.0 0.0Scissor division of clipped duct 100.0 0.0 0.0

All electronic data sets permitted evaluation, though time quantities (not displayed) differed.

FIG. 2. Screenshot of display from all four cameras during capture of laparoscopic cholecystectomy.

6238_08_p351-358 6/12/06 1:33 PM Page 355

and in-depth review of events. Root cause anal-ysis of an untoward outcome can consumedays of time for a reviewer poring throughreams of information trying to find the reasonfor a failure of the system, an individual or in-deed the physiology of the patient. Anesthesi-ology has done an enviable job of capturing theevents of an anesthetic. The sensors in anes-thesia such as end tidal CO2, pulse oximetry,and ECG have allowed much safer anestheticmanagement in the last 20 years3–5 and com-puter management of those data promise fur-ther improvements in the future when softwaremay recognize problems in real time and callfor a response.10

The remarkable quality of laparoscopic

video has invited surgeons to examine theirown work and that of others for the purposeof education and communication. This videorecord can be used to evaluate trainees6,9,11

and has enjoyed wide application in the teach-ing of techniques at conferences, webcasts andother live interactions with the OR.15,20 Thestrong video signal of anesthesia monitors canbe supplemented with video for distant inter-action in teleanesthesia.21 Management ofvideo has gotten easier with desktop editingand has brought videoconferencing into thereach of almost all teachers of medicine andsurgery.12,13,20

Putting together coherent video records ofoperations is an area actively being reported;

CONE ET AL.356

FIG. 3. View from foot of operating table demonstrating positioning of cameras 2 and 3 at the head of the operat-ing table.

6238_08_p351-358 6/12/06 1:33 PM Page 356

as in the case of the University of Virginia sys-tem reported this year22 and earlier efforts.8Our report combines the lessons learned invideo capture by other and our efforts with ex-cellent software tools for data management andretrieval.5,7,8,12,23,24 The program we used istermed Surgical data Collection, Integration,Storage and Retrieval System (SdCISRS). Thetotal data set for a typical laparoscopic chole-cystectomy is several megabytes. However, thedata are easily managed in the relational data-base using a master clock to marshal the data.The data are easily retrieved to answer cogentquestions relative to surgical outcome in thisprocedure that continues to be fraught with er-ror.17 In our study the reviewers did not nec-essarily agree with one another in small detailsbut strongly agreed that the questions posedwere entirely answerable using the softwarebut not with paper records. The rapidity of re-view (20 minutes per record) contrasts dra-matically with the laborious review process ofstandard records.

Future applications of this approach will in-clude analysis of open surgical procedureswhere video capture is steadily improv-ing.20,23–28 There are features of the system thatinvite improvement in voice control and auto-matic data capture. One can readily imaginethe critics who will be concerned about col-lecting so very much data about the operationand confounding the tort exposure of surgeons.However, we are gravely challenged nowwhen experts contradict one another and thesurgeon in a near void of hard data, while ajury of our peers struggles to make sense ofevents. With proper data collection, tort mat-ters may move from matters that are moot andpossibly actionable to res ipsa loquitor. Whetherthat exonerates the surgeon or makes the errorclear, legal costs should be less. Furthermore,if the experience in airline disaster is repeated,recognition of error will improve practices byidentifying errors and near misses as well as fa-cilitate training and credentialing.

The practice of surgery is under severescrutiny for errors. The cost of malpractice isconsidered a regional, if not a national, crisisfor access to care. The cost of a simple videocapture system and data storage is obviouslyminiscule (less than $4,000 with operational

costs of approximately $20 per case) in com-parison to the costs incurred by medical error.The average cost of a single extra day in thehospital after a surgical complication is $4,000.Would this system reduce medical error? At theleast, it could move a contested adverse med-ical event from moot to res ipsa loquitur. Withsystematic documentation of error and bestpractices, convergence toward a high and re-producible national standard of surgical per-formance may be anticipated.

ACKNOWLEDGMENTS

We would like to thank the staff of the Cin-terandes Foundation for their help with coor-dinating the surgical cases. We would also liketo thank Ms. Chasity Roberts for her editorialwork with this article. This work was fundedin part by a grant from NASA.

REFERENCES

1. Hippisley-Cox J, Pringle M, Cater R, Wynn A, Ham-mersley V, Coupland C, Hapgood R, Horsfield P,Teasdale S, Johnson C. The electronic patient recordin primary care—regression or progression? A crosssectional study. BMJ 2003;326:1439–1443.

2. Beuscart-Zephir MC, Anceaux F, Crinquette V, Re-nard JM. Integrating users’ activity modeling in thedesign and assessment of hospital electronic patientrecords: the example of anesthesia. Int J Med Inform2001;64:157–171.

3. Weiss YG, Maliar A, Eidelman LA, Berlatzky Y, Han-son CW, III, Deutschman CS, Zajicek G. Computer as-sisted physiologic monitoring and stability assess-ment in vascular surgical patients undergoing generalanesthesia—Preliminary data. J Clin Monit Comput2000;16:107–113.

4. Byrne AJ, Jones JG. Inaccurate reporting of simulatedcritical anaesthetic incidents. Br J Anaesth 1997;78:637–641.

5. Beckmann CR, Lipscomb GH, Ling FW, BeckmannCA, Johnson H, Barton L. Computer-assisted videoevaluation of surgical skills. Obstet Gynecol 1995;85:1039–1041.

6. Eubanks TR, Clements RH, Pohl D, Williams N,Schaad DC, Horgan S, Pellegrini C. An objective scor-ing system for laparoscopic cholecystectomy. J AmColl Surg 1999;189:566–574.

7. Miron H, Blumenthal EZ. Bridging analog and digi-tal video in the surgical setting. J Cataract Refract Surg2003;29:1874–1877.

MULTIMEDIA DATA CAPTURE AND MANAGEMENT 357

6238_08_p351-358 6/12/06 1:33 PM Page 357

8. Wurnig PN, Hollaus PH, Wurnig CH, Wolf RK, Oht-suka T, Pridun NS. A new method for digital videodocumentation in surgical procedures and minimallyinvasive surgery. Surg Endosc 2003;17:232–235.

9. Tang B, Hanna GB, Joice P, Cuschieri A. Identifica-tion and categorization of technical errors by Obser-vational Clinical Human Reliability Assessment(OCHRA) during laparoscopic cholecystectomy. ArchSurg 2004;139:1215–1220.

10. Rohrig R, Junger A, Hartmann B, Klasen J, QuinzioL, Jost A, Benson M, Hempelmann G. The incidenceand prediction of automatically detected intraopera-tive cardiovascular events in noncardiac surgery.Anesth Analg 2004;98:569–77.

11. Derossis AM, Fried GM, Abrahamowicz M, SigmanHH, Barkun JS, Meakins JL. Development of a modelfor training and evaluation of laparoscopic skills. AmJ Surg 1998;175:482–487.

12. Vincent-Hamelin E, Sarmiento JM, de la Puente JM,Vicente M. Computer-based desktop system for sur-gical videotape editing. Surg Endosc 1997;11:464–467.

13. Yamakawa T, Toyabe S, Cao P, Akazawa K. Web-based delivery of medical multimedia contents usingan MPEG-4 system. Comput Methods Programs Biomed2004;75:259–264.

14. Amiel GE, Cohen R, Amar R, Sarafov I, Safadi T, SauteM, Porat E, Sharony R, Bolotin G, Tann M, UretzkyG. Using a video conference to teach clinical clerks ina one-week cardiothoracic clerkship. Am J Surg1997;173:330–332.

15. Graschew G, Rakowsky S, Balanou P, Schlag PM. In-teractive telemedicine in the operating theatre of thefuture. J Telemed Telecare 2000;6(Suppl 2):S20–S24.

16. Mackenzie CF, Xiao Y. Video techniques and datacompared with observation in emergency traumacare. Qual Saf Health Care 2003;12(Suppl 2):ii51–ii57.

17. Archer SB, Brown DW, Smith CD, Branum GD,Hunter JG. Bile duct injury during laparoscopiccholecystectomy: Results of a national survey. AnnSurg 2001;234:549–558.

18. Way LW, Stewart L, Gantert W, Liu K, Lee CM,Whang K, Hunter JG. Causes and prevention of lap-aroscopic bile duct injuries: analysis of 252 cases froma human factors and cognitive psychology perspec-tive. Ann Surg 2003;237:460–469.

19. Pilot Error Declining as Proportion of Crash Causes,Study Suggests. Air Safety Week, March 26, 2001.Phillips Publishing International, Inc. www.findarti-cles.com/p/articles/mi_m0UBT/is_13_15/ai_72329–443 Last accessed June 3, 2005.

20. Rosser JC, Jr., Bell RL, Harnett B, Rodas E, MurayamaM, Merrell R. Use of mobile low-bandwidth telemed-ical techniques for extreme telemedicine applications.J Am Coll Surg 1999;189:397–404.

21. Cone SW, Gehr L, Hummel R, Rafiq A, Doarn CR,Merrell RC. Case report of remote anesthetic moni-toring using telemedicine. Anesth Analg 2004;98:386–388.

22. Guerlain S, Adams RB, Turrentine FB, Shin T, Guo H,Collins SR, Calland JF. Assessing team performancein the operating room: development and use of a“black-box” recorder and other tools for the intraop-erative environment. J Am Coll Surg 2005;200:29–37.

23. Rafiq A, Moore JA, Doarn CR, Merrell RC. Asyn-chronous confirmation of anatomical landmarks byoptical capture in open surgery. Arch Surg 2003;138:792–795.

24. Rafiq A, Moore JA, Zhao X, Doarn CR, Merrell RC.Digital video capture and synchronous consultationin open surgery. Ann Surg 2004;239:567–573.

25. Broderick TJ, Harnett BM, Doarn CR, Rodas EB, Mer-rell RC. Real-time Internet connections: Implicationsfor surgical decision making in laparoscopy. Ann Surg2001;234:165–171.

26. Broderick TJ, Russell KM, Doarn CR, Merrell RC. Anovel telemedicine method for viewing the open sur-gical field. J Laparoendosc Adv Surg Tech A 2002;12:293–298.

27. Rodas EB, Latifi R, Cone S, Broderick TJ, Doarn CR,Merrell RC. Telesurgical presence and consultationfor open surgery. Arch Surg 2002;137:1360–1363.

28. Russell KM, Broderick TJ, Demaria EJ, Kothari SN,Merrell RC. Laparoscopic telescope with alpha portand aesop to view open surgical procedures. J Lap-aroendosc Adv Surg Tech A 2001;11:213–218.

Address reprint requests to:Ronald C. Merrell, M.D.

Medical Informatics and Technology Applications Consortium

Department of SurgeryVirginia Commonwealth University

P.O. Box 9804801101 East Marshall Street

Richmond, VA 23298

E-mail: ronald.merrell@vcu.edu

CONE ET AL.358

6238_08_p351-358 6/12/06 1:33 PM Page 358