ww.sciencedirect.com

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8

Available online at w

ScienceDirect

journal homepage: www.JournalofSurgicalResearch.com

Association for Academic Surgery

Practicality of intraoperative teamworkassessments

Roy Phitayakorn, MD, MHPE,a,b,* Rebecca Minehart, MD,b,c

May C.M. Pian-Smith, MD,b,c Maureen W. Hemingway, RN, MSN,d,c

Tanya Milosh-Zinkus, BA,b Danika Oriol-Morway, BA,b

and Emil Petrusa, PhDa,b

aDepartment of Surgery, Massachusetts General Hospital, Boston, MassachusettsbMGH Learning Laboratory, Massachusetts General Hospital, Boston, MassachusettscDepartment of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston,

MassachusettsdDepartment of Perioperative Services, Massachusetts General Hospital, Boston, Massachusetts

a r t i c l e i n f o

Article history:

Received 7 January 2014

Received in revised form

4 April 2014

Accepted 9 April 2014

Available online 18 April 2014

Keywords:

OR teamwork assessment

Transprofessional/transdisciplinary

OR simulations

Nontechnical OR performance

OR teamwork assessment tools

Feasibility of OR team assessment

* Corresponding author. Department of Surgeþ1 617 643 7935; fax: þ1 617 643 4802.

E-mail address: rphitayakorn@mgh.harva0022-4804/$ e see front matter ª 2014 Elsevhttp://dx.doi.org/10.1016/j.jss.2014.04.024

a b s t r a c t

Background: High-quality teamwork among operating room (OR) professionals is a key to

efficient and safe practice. Quantification of teamwork facilitates feedback, assessment,

and improvement. Several valid and reliable instruments are available for assessing

separate OR disciplines and teams. We sought to determine the most feasible approach for

routine documentation of teamwork in in-situ OR simulations. We compared rater agree-

ment, hypothetical training costs, and feasibility ratings from five clinicians and two

nonclinicians with instruments for assessment of separate OR groups and teams.

Materials and methods: Five teams of anesthesia or surgery residents and OR nurses (RN) or

surgical technicians were videotaped in simulations of an epigastric hernia repair where the

patient develops malignant hyperthermia. Two anesthesiologists, one OR clinical RN

specialist, one educational psychologist, one simulation specialist, and one general surgeon

discussed and then independently completed Anesthesiologists’ Non-Technical Skills, Non-

Technical Skills for Surgeons, Scrub Practitioners’ List of Intraoperative Non-Technical

Skills, and Observational Teamwork Assessment for Surgery forms to rate nontechnical

performance of anesthesiologists, surgeons, nurses, technicians, and the whole team.

Results: Intraclass correlations of agreement ranged from 0.17e0.85. Clinicians’ agreements

were not different from nonclinicians’. Published rater training was 4 h for Anesthesiolo-

gists’ Non-Technical Skills and Scrub Practitioners’ List of Intraoperative Non-Technical

Skills, 2.5 h for Non-Technical Skills for Surgeons, and 15.5 h for Observational Team-

work Assessment for Surgery. Estimated costs to train one rater to use all instruments

ranged from $442 for a simulation specialist to $6006 for a general surgeon.

Conclusions: Additional training is needed to achieve higher levels of agreement; however,

costs may be prohibitive. The most cost-effective model for real-time OR teamwork

ry, Massachusetts General Hospital, 460 Wang ACC, 15 Parkman St., Boston, MA 02114. Tel.:

rd.edu (R. Phitayakorn).ier Inc. All rights reserved.

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 23

assessment may be to use a simulation technician combined with one clinical rater to

allow complete documentation of all participants.

ª 2014 Elsevier Inc. All rights reserved.

1. Introduction

professionals’ teamwork in routine high-fidelity OR simula-

Effective teamwork presumably results in improved medical

decision making, patient safety and therefore improved pa-

tient outcomes [1e3]. This benefit is especially pertinent to

operating room (OR) teams during unexpected urgent or

emergent situations [4e6]. However, these clinical situations

are relatively rare. Therefore, high-fidelity simulation is

increasingly used to allow OR teams to practice and improve

their teamwork skills in emergent situations. Documentation

and assessment of teamwork performance is essential for

effective feedback, benchmarking, and quantifying overall

improvement. Several tools to assess OR professionals’ per-

formance, both individually and collectively, have been

developed.

For anesthesiologists, a form for rating nontechnical skills

of anesthesiologists called Anesthesiologists’ Non-Technical

Skills (ANTS) was published in 2003 [7]. Based on human

factors research methods, ANTS has four categories: task

management, situation awareness, teamwork, and decision

making. A total of 15 elements serve as items for the cate-

gories and are marked on a four-point scale of good, accept-

able, marginal, and poor. For surgeons, the Non-Technical

Skills for Surgeons (NOTSS) tool has five categories of behavior

to be rated: situational awareness, decision making, leader-

ship, task management, and communication and teamwork.

Each category has three items to be marked on a four-point

scale of good, acceptable, marginal, and poor [8]. The other

key professional groups in ORs are nurses and surgical tech-

nicianswhose nontechnical skills greatly contribute to patient

safety and quality of care. A form similar to both the NOTSS

and ANTS has been developed for this group called Scrub

Practitioners’ List of Intraoperative Non-Technical Skills

(SPLINTS). This form consists of three categories: situation

awareness, communication and teamwork, and task man-

agement. Each category has three elements/items that are

marked on a four-point scale of good, acceptable, marginal,

and poor [9].

There are a number of tools for assessing overall commu-

nication and teamwork for OR professionals including Team-

STEPPS, Communication and Teamwork Skills (CATS), and

OTAS [10e12]. The OTAS was also used for this study as it did

not require specialized training for use and integratedwell with

ANTS, NOTSS, and SPLINTS. First, there are five behavioral

categories: (1) communication, (2) coordination, (3) cooperation

and back up, (4) leadership, and (5) team monitoring and situ-

ation awareness, each applied to the three professional groups

(nurses/technicians, surgeons, and anesthesiologists) in the

OR. Behaviors are rated on a seven-point scale indicating the

degree to which each behavior hampers (zero points) or en-

hances (six points) team functioning during the three major

phases (preoperative, intraoperative, and postoperative) of an

operation.

The research question for this study was practical and

sought to determine the most feasible approach to assess OR

tions. Criteria for feasibility include reliability, validity, ease of

use, low cost, convenience, andmost informative for feedback

and documentation of improvement. Ease of use and low costs

include requiring minimal training with equivalent use by the

least expensive observers, in addition to real-time recording

with immediate use for feedback during the debriefing.

2. Methods

2.1. OR simulation room and scenario

Our hospital has designated three ORs for in-situ OR

simulation-based training. With a few exceptions, all equip-

ments, drugs, and supplies are real and exactly the same as

would be found in an actual clinical OR. The “patient” in these

simulations is a SimMan Essential high-fidelity patient

simulation mannequin whose changing physical character-

istics and vital signs are controlled from an observational deck

above the OR separated by one-way glass.

With the approval of the Institutional Review Board (Part-

ners Healthcare, Boston, MA), this studywas conductedwith a

mixed group of participants for each scenario: typically two

anesthesiology residents, one general surgery resident, two

practicing OR nurses or one OR nurse and one surgical tech-

nician. Each OR simulation began with all participants in a 20-

min orientation that reviewed the goals for crisis resource

management training and special considerations for in-situOR

simulation. This structure created a “psychologically safe”

learning environment for the participants. Personal in-

troductions were exchanged, an overview of the surgical case

was given, and consent was obtained from participants to

allow the videotape to be used for research purposes.

All participants were then moved to the in-situ OR where a

confederate anesthesiology team had already induced and

intubated the patient and then handed over the case to the

participants. The nursing staff was also afforded a patient

handoff by the confederate nurse. Next, the OR team con-

ducted the Joint Commission of Accredited Hospital’s preop-

erative “time out,” and the confederate attending surgeon

started an open epigastric hernia repair with the participating

general surgery resident. Soon after, the surgeon was called

away and left the participants alone with the mannequin

patient who quickly developed signs of malignant hyper-

thermia. Participants managed this unexpected situation and

needed to obtain all the equipment normally used (malignant

hyperthermia kit, ice, chilled intravenous fluids, urinary

catheter, arterial line, laboratory test results, and so forth).

Two cameras and two microphones recorded the partici-

pants’ actions. B-line (Washington, DC) video technology was

used to synchronize the patient’s vital signs with the video-

tape for debriefing and research purposes. At the conclusion

of the scenario, participants were debriefed using an

objectives-oriented advocacy-inquiry approach [13]. The

Table 1 e Absolute agreement between pairs of observers for each OR team.

Anesthesiologist 1 Anesthesiologist 2 OR clinical RN specialist Psychologist Simulation specialist

ANTS (four items)

Surgeon 0.52 0.61 0.33 0.17 0.51

Anesthesiologist 1 d 0.57 0.55 0.57 0.05

Anesthesiologist 2 d 0.66 0.62 0.62

OR clinical nurse specialist e 0.35 0.56

Psychologist e 0.57

NOTSS (12 items)

Surgeon 0.48 0.69 0.15 0.19 0.72

Anesthesiologist 1 d 0.83 0.40 0.52 0.82

Anesthesiologist 2 d 0.70 0.72 0.81

OR clinical nurse specialist d 0.12 0.71

Psychologist d 0.83

SPLINTS (nine items)

Surgeon 0.23 0.54 0.45 0.67 0.36

Anesthesiologist 1 d 0.72 0.62 0.80 0.28

Anesthesiologist 2 d 0.75 0.85 0.82

OR clinical nurse specialist d 0.79 0.71

Psychologist d 0.85

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 824

debrief process ensured that all the participants were able to

discuss the crisis resource management training goals. Five

OR teams participated in the pilot MH simulation fromApril to

June of 2013.

2.2. Raters

Raters were two anesthesiology attendings (R.M. and M.P.-S),

one OR clinical nurse specialist (M.W.H.), one psychologist

(E.P.), one simulation specialist (D.O.-M), and one general

surgeon (R.P.). Raters were members of the Massachusetts

General Hospital Inter-professional OR Simulation Team.

2.3. Rating forms

ANTS, NOTSS, and SPLINTSwere used as published. For OTAS,

each OR professional group was rated on performance during

the case as recommended. In addition, raters were asked to

make an overall rating of each team’s collective teamwork. To

avoid numerical problems when attempting between instru-

ment comparisons, OTAS scale values were shifted from 0e6

to 1e7. Rater data of videotaped performances were collected

beginning 1 mo after the last simulation.

2.4. Rating procedure

All raters met for 1 h to review the assessment tools and study

procedure as well as clarify definitions and terms. Each rater

received secured, encrypted links to the five videos, and used

all four assessment tools in the same order for the five

recorded simulations. Specific simulation sessions were rated

in the same order by all raters over a time period of 3 wk.

Raters did not discuss ratings with one another. Raters could

stop, rewind, and review each recording as often as needed

while completing forms. After completing ratings of all five

simulations, each rater answered six questions (see Appendix

1) regarding the feasibility of each tool. None of the raters had

used any of the assessment tools before this study.

2.5. Calculation of costs

The suggested amount of training published and recommended

to prepare raters to use each of the instruments were substan-

tial. The original authors’ recommendation to train users of

ANTS,NOTSS,andSPLINTSrequired2e3dofpreparation,which

included education on human factors of team performance,

implications of poor nontechnical performance, and dedicated

scoring using each of the three instruments with a standard set

of videos followedbydiscussionwith repeat scoring.Developers

acknowledged that this amount of training is impractical and

additionally reported psychometric results after 4 h of training

for ANTS and 2.5 h for NOTSS and SPLINTS [7,9,14]. Recom-

mended preparation to properly use OTAS included over 15 h of

discussion, practice, and coachingduring 10 real operations [15].

All the raters in this study felt comfortable interpreting the

items and scales on all instruments, so the cost of training one

rater from each professional group was considered equivalent

to the others in any professional group and was therefore

estimated for each of the four instruments.

2.6. Statistical analysis

Overall rater agreement was assessed with intraclass corre-

lations for absolute agreement. Average rater agreement was

computed by transforming correlation coefficients from rez,

averaging the z scores and then transforming the average z

back to an r. Responses to feasibility items were analyzed

descriptively. Performance differences between teams were

analyzed with a one-way analysis of variance. Minimal sta-

tistical significance was P ¼ 0.05. All data were entered into

and analyzed with SPSS 22 (IBM Analytics) .

3. Results

Intraclass correlation coefficients for NOTSS ranged from

0.12e0.83, for ANTS 0.17e0.57, for SPLINTS 0.23e0.85, and

Table 2 e Absolute agreement between pairs of observers for each OR team using OTASeentire OR team.

Anesthesiologist 1 Anesthesiologist 2 OR clinical RN specialist Psychologist Simulation specialist

Surgeon 0.72 0.74 0.42 0.64 0.71

Anesthesiologist 1 d 0.90 0.57 0.72 0.75

Anesthesiologist 2 d 0.70 0.81 0.87

OR clinical nurse specialist d 0.50 0.62

Psychologist d 0.74

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 25

OTAS 0.42e0.90. Table 1 illustrates agreement coefficients for

each rater pair. Agreements for OTAS ratings are shown in

Tables 2 and 3.

The average rater agreements on each instrument for the

videos are illustrated in Table 4. There was considerable

variability in agreement for both instruments and videos. This

suggests that for some videos, raters were attending to similar

aspects of performance, while for other videos raters tended

to focus on different aspects.

Raters were consistent in their evaluation of feasibility for

all instruments. There were no significant differences in rat-

ings for instruments on most of the items, with the exception

of “time and mental effort” for OTAS compared with ANTS,

with OTAS requiring significantlymore time andmental effort

than ANTS (5.50 versus 3.33, P ¼ 0.04). Half of the raters

thought that OTAS was better suited for video replay whereas

the other half felt that OTAS would be feasible during real-

time observation. The mean time to complete each team-

work instrument was 12.9 � 8.7 min for ANTS, 12.5 � 8.3 min

for SPLINTS, 10.1� 6.7min for NOTSS, and 22.54� 22.1min for

OTAS.

Based on local rates of pay (Boston, MA), the hypothetical

costs for rater training to use all the instruments were $442 for

the simulation specialist, $1248 for the psychologist, $1820 for

the OR clinical RN specialist, $2470 for an anesthesiologist,

and $6006 for the general surgeon.

Table 3 e Absolute agreement between pairs of observers for e

Anesthesiologist1

Anesth

OTASeanesthesiologists (five items)

Surgeon 0.72 0

Anesthesiologist 1 d 0

Anesthesiologist 2

OR clinical nurse specialist

Psychologist

OTASenurses (five items)

Surgeon 0.72 0

Anesthesiologist 1 d 0

Anesthesiologist 2

OR clinical nurse specialist

Psychologist

OTASesurgeons (five items)

Surgeon 0.72 0

Anesthesiologist 1 d 0

Anesthesiologist 2

OR clinical nurse specialist

Psychologist

4. Discussion

The goal of this study was to assess the practicality of using

published instruments to assess teamwork and communica-

tion during OR simulations. Optimal feasibility includes inter-

rater consistency, lowcosts, andease of use includingminimal

training and completion in real time and rapid analysis for use

during debriefings.

In this study, inter-rater consistency was variable, and

moderate to moderately high even with minimal rater

training. Interestingly, clinician and nonclinician raters had

similar ranges of agreement. These results suggest that ele-

ments of effective OR teamwork and communication may be

recognizable even to individuals who are not members of an

OR team. This has significant implications to costs of OR

teamwork assessment since our calculations demonstrated

that nonclinician raters are substantially more affordable for

large-scale OR-based team training. One application of this

result would be to use a nonclinician to performOR teamwork

assessments during an actual simulation, which can then be

used by a clinician to guide the debriefing of the participants.

Ultimately, nonclinicians who have sufficient training and

experience with the clinical scenario may be used to debrief

participants as well. This process would dramatically reduce

costs to a simulation center, which is a major deterrent to

ach OR team using OTASespecialty specific.

esiologist2

OR clinical RNspecialist

Psychologist Simulationspecialist

.74 0.42 0.64 0.71

.90 0.57 0.72 0.75

d 0.70 0.81 0.87

d 0.50 0.62

d 0.74

.74 0.42 0.64 0.71

.90 0.57 0.72 0.75

d 0.70 0.81 0.87

d 0.50 0.62

d 0.74

.74 0.42 0.64 0.71

.90 0.57 0.72 0.75

d 0.70 0.81 0.87

d 0.50 0.62

d 0.74

Table 4eAverage rater agreement for each instrument byOR team.

ORteam 1

ORteam 2

ORteam 3

ORteam 4

ORteam 5

ANTS 0.64 0.40 0.75 0.34 0.53

NOTSS 0.47 0.48 0.58 0.79 0.82

SPLINTS 0.38 0.63 0.71 0.38 0.31

OTAS-A 0.29 0.22 0.43 0.71 0.77

OTAS-N 0.14 0.07 0.59 0.71 0.63

OTAS-S 0.14 0.13 0.48 0.68 0.83

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 826

routine team-based simulation. In general, a simulation pro-

gram would likely want more than one trained rater for

scheduling purposes. These costs would increase rapidly for

training additional raters.

Limitations of this study are similar to any using simula-

tion, whereby participants may not take the simulation seri-

ously or behave as they normally would. However, there was

no indication that this was the case in exit surveys taken from

the groups involved. Additionally, although the camera angles

captured much of the environment, it is possible that some

teamwork behaviors were not assessed if they occurred off-

camera, which may have affected the scoring performed

using only video review. Finally, although our raters made

estimates of whether they would be able to use each tool in

real time, none of the assessment tools were used during an

actual simulation scenario in real time and applied to the

debriefing. However, some of the raters did comment on the

amount of time spent rewinding and replaying certain video

clips, particularly in the midst of the crisis where activity was

high. This may have contributed to the higher mean time to

complete the OTAS instrument and the predominant rater

opinion that using the more complex OTAS scoring sheet

might be more challenging during these highly active team-

work situations than the more simplistic ANTS, NOTSS, or

SPLINTS instruments. Finally, the scenario of malignant hy-

perthermia may have influenced the pattern of performance,

because it is typically considered a challenge for the anes-

thesiologists. Although all members of each OR team

appeared to be fully engaged in the simulation, more research

using additional scenarios is necessary to allowdirect analysis

of the scenarios’ influences.

This study demonstrates that it may be feasible to assess

the teamwork and communication ability of one role group

within an OR team in real time using ANTS, SPLINTS, or

NOTSS. However, assessment of overall OR teamwork using

OTAS or completion of multiple instruments by one rater

without the benefit of video review or instrument modifi-

cation is complicated and appears impractical. This study

did not find high correlations between data from the indi-

vidual instruments and profession-specific ratings with

OTAS. This result has implications in the broader theoretical

frameworks of OR teamwork and implies that effective

teamwork within each discipline (anesthesia, nursing, and

surgery) may function differently than when all disciplines

are assessed working together [16]. Future research into real-

time OR-based teamwork assessment would likely require

two representatives from each professional group who

complete an overall teamwork instrument such as OTAS in

real-time and then compare data with ratings from video

replay.

It will be important to understand the causes of the vari-

ability in agreement for pairs of our raters. It is likely that each

rater has a different mental model of ideal teamwork perfor-

mance by each discipline/professional group. Future qualita-

tive research is required to further explore these mental

models and modify existing teamwork instruments accord-

ingly. Finally, the significance of technical skills integration

into overall OR teamwork is unclear. In this study, clinician

raters noted instances where participants demonstrated ideal

teamwork elements, but made ineffective or incorrect clinical

decisions. This dichotomy resulted in decreased scores by

clinician raters, but was not noticed by nonclinician raters. It

is possible that these errors result from perceptual “anchors”

that are difficult for clinical raters to overlook, but are possibly

out of proportion to their contributions to team effectiveness.

5. Conclusions

Our approach that approximated real-world implementation

of assessment tools did not produce a satisfactory model for

assessing individual professional groups and overall team

performance. More training for any observer is needed, but

nonclinicians hold promise as less expensive raters of

nontechnical OR team skills. Research about the influence of a

simulation team’s incorrect clinical decisions on clinician

rating of nontechnical skills will contribute to better training

and more accurate ratings by clinicians. Whether non-

clinicians can be trained to accurately complete checklists and

ratings of clinical actions remains an open question, as does

the question of the relationship between nontechnical and

technical behaviors. Authentic, in-situ OR simulations with

transdiscipline participants provide a laboratory to help

determine the ideal combination of teamwork skills with ac-

curate patient care to improve clinical outcomes during un-

expected OR emergencies.

Acknowledgment

Authors’ contributions: R.P., R.M., M.C.M.P.-S., M.W.H., and

E.P. contributed to conception and design. R.P., D.O.-M., and

E.P. analyzed and interpreted the data. R.P., R.M., M.C.M.P.-S.,

M.W.H., D.O.-M., and E.P. collected the data. R.P., R.M.,

M.C.M.P.-S., M.W.H., T.M.-Z., and E.P. made the critical revi-

sion of the article.

Disclosure

The authors reported no proprietary or commercial interest in

any product mentioned or concept discussed in this article.

r e f e r e n c e s

[1] Mills P, Neily J, Dunn E. Teamwork and communication insurgical teams: implications for patient safety. J Am Coll Surg2008 Jan;206:107.

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 27

[2] Buljac-Samardzic M, Dekker-van Doorn CM, vanWijngaarden JD, van Wijk KP. Interventions to improve teameffectiveness: a systematic review. Health Policy 2010 Mar;94:183.

[3] Lingard L, Espin S, Whyte S, et al. Communication failures inthe operating room: an observational classification ofrecurrent types and effects. Qual Saf Health Care 2004 Oct;13:330.

[4] Catchpole K, Mishra A, Handa A, McCulloch P. Teamworkand error in the operating room: analysis of skills and roles.Ann Surg 2008 Apr;247:699.

[5] Russ S, Hull L, Rout S, Vincent C, Darzi A, Sevdalis N.Observational teamwork assessment for surgery: feasibilityof clinical and nonclinical assessor calibration with short-term training. Ann Surg 2012 Apr;255:804.

[6] Greenberg CC, Regenbogen SE, Studdert DM, et al. Patterns ofcommunication breakdowns resulting in injury to surgicalpatients. J Am Coll Surg 2007 Apr;204:533.

[7] Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R.Anaesthetists’ Non-Technical Skills (ANTS): evaluationof a behaviouralmarker system. Br J Anaesth 2003May;90:580.

[8] Yule S, Flin R, Paterson-Brown S, Maran N. Non-technicalskills for surgeons in the operating room: a review of theliterature. Surgery 2006 Feb;139:140.

[9] Mitchell L, Flin R, Yule S, Mitchell J, Coutts K, Youngson G.Development of a behavioural marker system for scrub

practitioners’ non-technical skills (SPLINTS system). J EvalClin Pract 2013 Apr;19:317.

[10] Clancy CM, Tornberg DN. TeamSTEPPS: assuring optimalteamwork in clinical settings. Am J Med Qual 2007 May-Jun;22:214.

[11] Frankel A, Gardner R, Maynard L, Kelly A. Using theCommunication and Teamwork Skills (CATS) Assessment tomeasure health care team performance. Jt Comm J QualPatient Saf 2007 Sep;33:549.

[12] Healey AN, Undre S, Vincent CA. Developing observationalmeasures of performance in surgical teams. Qual Saf HealthCare 2004 Oct;13(Suppl 1):i33.

[13] Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB.Debriefing with good judgment: combining rigorous feedbackwith genuine inquiry. Anesthesiol Clin 2007 Jun;25:361.

[14] Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons’ non-technical skills in the operatingroom: reliability testing of the NOTSS behavior ratingsystem. World J Surg 2008 Apr;32:548.

[15] Sevdalis N, Lyons M, Healey AN, Undre S, Darzi A,Vincent CA. Observational teamwork assessment forsurgery: construct validation with expert versus noviceraters. Ann Surg 2009 Jun;249:1047.

[16] Healey AN, Undre S, Sevdalis N, Koutantji M, Vincent CA.The complexity of measuring interprofessional teamwork inthe operating theatre. J Interprof Care 2006 Oct;20:485.

Appendix 1 e Items regarding feasibility of assessmenttools.*

1. Rate your comfort interpreting the categories and elements of

each observation tool.

2. Rate your level of confidence that your marks on each of these

tools will align with marks form other observers for these

videos.

3. Indicate your level of confidence that you would have similar

marks on the same tool if you rated the same videos 1 wk

later.

4. Rate the amount of effort (time and mental energy) you

expended to use each instrument.

5. What is the feasibility for you to use each tool for marking

behavior in real time?

6. What is the feasibility for you to use each tool with video

replay?

* Scale for each item: very low to very high (1e7).

j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 828