Post on 02-May-2023
ww.sciencedirect.com
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8
Available online at w
ScienceDirect
journal homepage: www.JournalofSurgicalResearch.com
Association for Academic Surgery
Practicality of intraoperative teamworkassessments
Roy Phitayakorn, MD, MHPE,a,b,* Rebecca Minehart, MD,b,c
May C.M. Pian-Smith, MD,b,c Maureen W. Hemingway, RN, MSN,d,c
Tanya Milosh-Zinkus, BA,b Danika Oriol-Morway, BA,b
and Emil Petrusa, PhDa,b
aDepartment of Surgery, Massachusetts General Hospital, Boston, MassachusettsbMGH Learning Laboratory, Massachusetts General Hospital, Boston, MassachusettscDepartment of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston,
MassachusettsdDepartment of Perioperative Services, Massachusetts General Hospital, Boston, Massachusetts
a r t i c l e i n f o
Article history:
Received 7 January 2014
Received in revised form
4 April 2014
Accepted 9 April 2014
Available online 18 April 2014
Keywords:
OR teamwork assessment
Transprofessional/transdisciplinary
OR simulations
Nontechnical OR performance
OR teamwork assessment tools
Feasibility of OR team assessment
* Corresponding author. Department of Surgeþ1 617 643 7935; fax: þ1 617 643 4802.
E-mail address: rphitayakorn@mgh.harva0022-4804/$ e see front matter ª 2014 Elsevhttp://dx.doi.org/10.1016/j.jss.2014.04.024
a b s t r a c t
Background: High-quality teamwork among operating room (OR) professionals is a key to
efficient and safe practice. Quantification of teamwork facilitates feedback, assessment,
and improvement. Several valid and reliable instruments are available for assessing
separate OR disciplines and teams. We sought to determine the most feasible approach for
routine documentation of teamwork in in-situ OR simulations. We compared rater agree-
ment, hypothetical training costs, and feasibility ratings from five clinicians and two
nonclinicians with instruments for assessment of separate OR groups and teams.
Materials and methods: Five teams of anesthesia or surgery residents and OR nurses (RN) or
surgical technicians were videotaped in simulations of an epigastric hernia repair where the
patient develops malignant hyperthermia. Two anesthesiologists, one OR clinical RN
specialist, one educational psychologist, one simulation specialist, and one general surgeon
discussed and then independently completed Anesthesiologists’ Non-Technical Skills, Non-
Technical Skills for Surgeons, Scrub Practitioners’ List of Intraoperative Non-Technical
Skills, and Observational Teamwork Assessment for Surgery forms to rate nontechnical
performance of anesthesiologists, surgeons, nurses, technicians, and the whole team.
Results: Intraclass correlations of agreement ranged from 0.17e0.85. Clinicians’ agreements
were not different from nonclinicians’. Published rater training was 4 h for Anesthesiolo-
gists’ Non-Technical Skills and Scrub Practitioners’ List of Intraoperative Non-Technical
Skills, 2.5 h for Non-Technical Skills for Surgeons, and 15.5 h for Observational Team-
work Assessment for Surgery. Estimated costs to train one rater to use all instruments
ranged from $442 for a simulation specialist to $6006 for a general surgeon.
Conclusions: Additional training is needed to achieve higher levels of agreement; however,
costs may be prohibitive. The most cost-effective model for real-time OR teamwork
ry, Massachusetts General Hospital, 460 Wang ACC, 15 Parkman St., Boston, MA 02114. Tel.:
rd.edu (R. Phitayakorn).ier Inc. All rights reserved.
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 23
assessment may be to use a simulation technician combined with one clinical rater to
allow complete documentation of all participants.
ª 2014 Elsevier Inc. All rights reserved.
1. Introduction
professionals’ teamwork in routine high-fidelity OR simula-Effective teamwork presumably results in improved medical
decision making, patient safety and therefore improved pa-
tient outcomes [1e3]. This benefit is especially pertinent to
operating room (OR) teams during unexpected urgent or
emergent situations [4e6]. However, these clinical situations
are relatively rare. Therefore, high-fidelity simulation is
increasingly used to allow OR teams to practice and improve
their teamwork skills in emergent situations. Documentation
and assessment of teamwork performance is essential for
effective feedback, benchmarking, and quantifying overall
improvement. Several tools to assess OR professionals’ per-
formance, both individually and collectively, have been
developed.
For anesthesiologists, a form for rating nontechnical skills
of anesthesiologists called Anesthesiologists’ Non-Technical
Skills (ANTS) was published in 2003 [7]. Based on human
factors research methods, ANTS has four categories: task
management, situation awareness, teamwork, and decision
making. A total of 15 elements serve as items for the cate-
gories and are marked on a four-point scale of good, accept-
able, marginal, and poor. For surgeons, the Non-Technical
Skills for Surgeons (NOTSS) tool has five categories of behavior
to be rated: situational awareness, decision making, leader-
ship, task management, and communication and teamwork.
Each category has three items to be marked on a four-point
scale of good, acceptable, marginal, and poor [8]. The other
key professional groups in ORs are nurses and surgical tech-
nicianswhose nontechnical skills greatly contribute to patient
safety and quality of care. A form similar to both the NOTSS
and ANTS has been developed for this group called Scrub
Practitioners’ List of Intraoperative Non-Technical Skills
(SPLINTS). This form consists of three categories: situation
awareness, communication and teamwork, and task man-
agement. Each category has three elements/items that are
marked on a four-point scale of good, acceptable, marginal,
and poor [9].
There are a number of tools for assessing overall commu-
nication and teamwork for OR professionals including Team-
STEPPS, Communication and Teamwork Skills (CATS), and
OTAS [10e12]. The OTAS was also used for this study as it did
not require specialized training for use and integratedwell with
ANTS, NOTSS, and SPLINTS. First, there are five behavioral
categories: (1) communication, (2) coordination, (3) cooperation
and back up, (4) leadership, and (5) team monitoring and situ-
ation awareness, each applied to the three professional groups
(nurses/technicians, surgeons, and anesthesiologists) in the
OR. Behaviors are rated on a seven-point scale indicating the
degree to which each behavior hampers (zero points) or en-
hances (six points) team functioning during the three major
phases (preoperative, intraoperative, and postoperative) of an
operation.
The research question for this study was practical and
sought to determine the most feasible approach to assess OR
tions. Criteria for feasibility include reliability, validity, ease of
use, low cost, convenience, andmost informative for feedback
and documentation of improvement. Ease of use and low costs
include requiring minimal training with equivalent use by the
least expensive observers, in addition to real-time recording
with immediate use for feedback during the debriefing.
2. Methods
2.1. OR simulation room and scenario
Our hospital has designated three ORs for in-situ OR
simulation-based training. With a few exceptions, all equip-
ments, drugs, and supplies are real and exactly the same as
would be found in an actual clinical OR. The “patient” in these
simulations is a SimMan Essential high-fidelity patient
simulation mannequin whose changing physical character-
istics and vital signs are controlled from an observational deck
above the OR separated by one-way glass.
With the approval of the Institutional Review Board (Part-
ners Healthcare, Boston, MA), this studywas conductedwith a
mixed group of participants for each scenario: typically two
anesthesiology residents, one general surgery resident, two
practicing OR nurses or one OR nurse and one surgical tech-
nician. Each OR simulation began with all participants in a 20-
min orientation that reviewed the goals for crisis resource
management training and special considerations for in-situOR
simulation. This structure created a “psychologically safe”
learning environment for the participants. Personal in-
troductions were exchanged, an overview of the surgical case
was given, and consent was obtained from participants to
allow the videotape to be used for research purposes.
All participants were then moved to the in-situ OR where a
confederate anesthesiology team had already induced and
intubated the patient and then handed over the case to the
participants. The nursing staff was also afforded a patient
handoff by the confederate nurse. Next, the OR team con-
ducted the Joint Commission of Accredited Hospital’s preop-
erative “time out,” and the confederate attending surgeon
started an open epigastric hernia repair with the participating
general surgery resident. Soon after, the surgeon was called
away and left the participants alone with the mannequin
patient who quickly developed signs of malignant hyper-
thermia. Participants managed this unexpected situation and
needed to obtain all the equipment normally used (malignant
hyperthermia kit, ice, chilled intravenous fluids, urinary
catheter, arterial line, laboratory test results, and so forth).
Two cameras and two microphones recorded the partici-
pants’ actions. B-line (Washington, DC) video technology was
used to synchronize the patient’s vital signs with the video-
tape for debriefing and research purposes. At the conclusion
of the scenario, participants were debriefed using an
objectives-oriented advocacy-inquiry approach [13]. The
Table 1 e Absolute agreement between pairs of observers for each OR team.
Anesthesiologist 1 Anesthesiologist 2 OR clinical RN specialist Psychologist Simulation specialist
ANTS (four items)
Surgeon 0.52 0.61 0.33 0.17 0.51
Anesthesiologist 1 d 0.57 0.55 0.57 0.05
Anesthesiologist 2 d 0.66 0.62 0.62
OR clinical nurse specialist e 0.35 0.56
Psychologist e 0.57
NOTSS (12 items)
Surgeon 0.48 0.69 0.15 0.19 0.72
Anesthesiologist 1 d 0.83 0.40 0.52 0.82
Anesthesiologist 2 d 0.70 0.72 0.81
OR clinical nurse specialist d 0.12 0.71
Psychologist d 0.83
SPLINTS (nine items)
Surgeon 0.23 0.54 0.45 0.67 0.36
Anesthesiologist 1 d 0.72 0.62 0.80 0.28
Anesthesiologist 2 d 0.75 0.85 0.82
OR clinical nurse specialist d 0.79 0.71
Psychologist d 0.85
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 824
debrief process ensured that all the participants were able to
discuss the crisis resource management training goals. Five
OR teams participated in the pilot MH simulation fromApril to
June of 2013.
2.2. Raters
Raters were two anesthesiology attendings (R.M. and M.P.-S),
one OR clinical nurse specialist (M.W.H.), one psychologist
(E.P.), one simulation specialist (D.O.-M), and one general
surgeon (R.P.). Raters were members of the Massachusetts
General Hospital Inter-professional OR Simulation Team.
2.3. Rating forms
ANTS, NOTSS, and SPLINTSwere used as published. For OTAS,
each OR professional group was rated on performance during
the case as recommended. In addition, raters were asked to
make an overall rating of each team’s collective teamwork. To
avoid numerical problems when attempting between instru-
ment comparisons, OTAS scale values were shifted from 0e6
to 1e7. Rater data of videotaped performances were collected
beginning 1 mo after the last simulation.
2.4. Rating procedure
All raters met for 1 h to review the assessment tools and study
procedure as well as clarify definitions and terms. Each rater
received secured, encrypted links to the five videos, and used
all four assessment tools in the same order for the five
recorded simulations. Specific simulation sessions were rated
in the same order by all raters over a time period of 3 wk.
Raters did not discuss ratings with one another. Raters could
stop, rewind, and review each recording as often as needed
while completing forms. After completing ratings of all five
simulations, each rater answered six questions (see Appendix
1) regarding the feasibility of each tool. None of the raters had
used any of the assessment tools before this study.
2.5. Calculation of costs
The suggested amount of training published and recommended
to prepare raters to use each of the instruments were substan-
tial. The original authors’ recommendation to train users of
ANTS,NOTSS,andSPLINTSrequired2e3dofpreparation,which
included education on human factors of team performance,
implications of poor nontechnical performance, and dedicated
scoring using each of the three instruments with a standard set
of videos followedbydiscussionwith repeat scoring.Developers
acknowledged that this amount of training is impractical and
additionally reported psychometric results after 4 h of training
for ANTS and 2.5 h for NOTSS and SPLINTS [7,9,14]. Recom-
mended preparation to properly use OTAS included over 15 h of
discussion, practice, and coachingduring 10 real operations [15].
All the raters in this study felt comfortable interpreting the
items and scales on all instruments, so the cost of training one
rater from each professional group was considered equivalent
to the others in any professional group and was therefore
estimated for each of the four instruments.
2.6. Statistical analysis
Overall rater agreement was assessed with intraclass corre-
lations for absolute agreement. Average rater agreement was
computed by transforming correlation coefficients from rez,
averaging the z scores and then transforming the average z
back to an r. Responses to feasibility items were analyzed
descriptively. Performance differences between teams were
analyzed with a one-way analysis of variance. Minimal sta-
tistical significance was P ¼ 0.05. All data were entered into
and analyzed with SPSS 22 (IBM Analytics) .
3. Results
Intraclass correlation coefficients for NOTSS ranged from
0.12e0.83, for ANTS 0.17e0.57, for SPLINTS 0.23e0.85, and
Table 2 e Absolute agreement between pairs of observers for each OR team using OTASeentire OR team.
Anesthesiologist 1 Anesthesiologist 2 OR clinical RN specialist Psychologist Simulation specialist
Surgeon 0.72 0.74 0.42 0.64 0.71
Anesthesiologist 1 d 0.90 0.57 0.72 0.75
Anesthesiologist 2 d 0.70 0.81 0.87
OR clinical nurse specialist d 0.50 0.62
Psychologist d 0.74
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 25
OTAS 0.42e0.90. Table 1 illustrates agreement coefficients for
each rater pair. Agreements for OTAS ratings are shown in
Tables 2 and 3.
The average rater agreements on each instrument for the
videos are illustrated in Table 4. There was considerable
variability in agreement for both instruments and videos. This
suggests that for some videos, raters were attending to similar
aspects of performance, while for other videos raters tended
to focus on different aspects.
Raters were consistent in their evaluation of feasibility for
all instruments. There were no significant differences in rat-
ings for instruments on most of the items, with the exception
of “time and mental effort” for OTAS compared with ANTS,
with OTAS requiring significantlymore time andmental effort
than ANTS (5.50 versus 3.33, P ¼ 0.04). Half of the raters
thought that OTAS was better suited for video replay whereas
the other half felt that OTAS would be feasible during real-
time observation. The mean time to complete each team-
work instrument was 12.9 � 8.7 min for ANTS, 12.5 � 8.3 min
for SPLINTS, 10.1� 6.7min for NOTSS, and 22.54� 22.1min for
OTAS.
Based on local rates of pay (Boston, MA), the hypothetical
costs for rater training to use all the instruments were $442 for
the simulation specialist, $1248 for the psychologist, $1820 for
the OR clinical RN specialist, $2470 for an anesthesiologist,
and $6006 for the general surgeon.
Table 3 e Absolute agreement between pairs of observers for e
Anesthesiologist1
Anesth
OTASeanesthesiologists (five items)
Surgeon 0.72 0
Anesthesiologist 1 d 0
Anesthesiologist 2
OR clinical nurse specialist
Psychologist
OTASenurses (five items)
Surgeon 0.72 0
Anesthesiologist 1 d 0
Anesthesiologist 2
OR clinical nurse specialist
Psychologist
OTASesurgeons (five items)
Surgeon 0.72 0
Anesthesiologist 1 d 0
Anesthesiologist 2
OR clinical nurse specialist
Psychologist
4. Discussion
The goal of this study was to assess the practicality of using
published instruments to assess teamwork and communica-
tion during OR simulations. Optimal feasibility includes inter-
rater consistency, lowcosts, andease of use includingminimal
training and completion in real time and rapid analysis for use
during debriefings.
In this study, inter-rater consistency was variable, and
moderate to moderately high even with minimal rater
training. Interestingly, clinician and nonclinician raters had
similar ranges of agreement. These results suggest that ele-
ments of effective OR teamwork and communication may be
recognizable even to individuals who are not members of an
OR team. This has significant implications to costs of OR
teamwork assessment since our calculations demonstrated
that nonclinician raters are substantially more affordable for
large-scale OR-based team training. One application of this
result would be to use a nonclinician to performOR teamwork
assessments during an actual simulation, which can then be
used by a clinician to guide the debriefing of the participants.
Ultimately, nonclinicians who have sufficient training and
experience with the clinical scenario may be used to debrief
participants as well. This process would dramatically reduce
costs to a simulation center, which is a major deterrent to
ach OR team using OTASespecialty specific.
esiologist2
OR clinical RNspecialist
Psychologist Simulationspecialist
.74 0.42 0.64 0.71
.90 0.57 0.72 0.75
d 0.70 0.81 0.87
d 0.50 0.62
d 0.74
.74 0.42 0.64 0.71
.90 0.57 0.72 0.75
d 0.70 0.81 0.87
d 0.50 0.62
d 0.74
.74 0.42 0.64 0.71
.90 0.57 0.72 0.75
d 0.70 0.81 0.87
d 0.50 0.62
d 0.74
Table 4eAverage rater agreement for each instrument byOR team.
ORteam 1
ORteam 2
ORteam 3
ORteam 4
ORteam 5
ANTS 0.64 0.40 0.75 0.34 0.53
NOTSS 0.47 0.48 0.58 0.79 0.82
SPLINTS 0.38 0.63 0.71 0.38 0.31
OTAS-A 0.29 0.22 0.43 0.71 0.77
OTAS-N 0.14 0.07 0.59 0.71 0.63
OTAS-S 0.14 0.13 0.48 0.68 0.83
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 826
routine team-based simulation. In general, a simulation pro-
gram would likely want more than one trained rater for
scheduling purposes. These costs would increase rapidly for
training additional raters.
Limitations of this study are similar to any using simula-
tion, whereby participants may not take the simulation seri-
ously or behave as they normally would. However, there was
no indication that this was the case in exit surveys taken from
the groups involved. Additionally, although the camera angles
captured much of the environment, it is possible that some
teamwork behaviors were not assessed if they occurred off-
camera, which may have affected the scoring performed
using only video review. Finally, although our raters made
estimates of whether they would be able to use each tool in
real time, none of the assessment tools were used during an
actual simulation scenario in real time and applied to the
debriefing. However, some of the raters did comment on the
amount of time spent rewinding and replaying certain video
clips, particularly in the midst of the crisis where activity was
high. This may have contributed to the higher mean time to
complete the OTAS instrument and the predominant rater
opinion that using the more complex OTAS scoring sheet
might be more challenging during these highly active team-
work situations than the more simplistic ANTS, NOTSS, or
SPLINTS instruments. Finally, the scenario of malignant hy-
perthermia may have influenced the pattern of performance,
because it is typically considered a challenge for the anes-
thesiologists. Although all members of each OR team
appeared to be fully engaged in the simulation, more research
using additional scenarios is necessary to allowdirect analysis
of the scenarios’ influences.
This study demonstrates that it may be feasible to assess
the teamwork and communication ability of one role group
within an OR team in real time using ANTS, SPLINTS, or
NOTSS. However, assessment of overall OR teamwork using
OTAS or completion of multiple instruments by one rater
without the benefit of video review or instrument modifi-
cation is complicated and appears impractical. This study
did not find high correlations between data from the indi-
vidual instruments and profession-specific ratings with
OTAS. This result has implications in the broader theoretical
frameworks of OR teamwork and implies that effective
teamwork within each discipline (anesthesia, nursing, and
surgery) may function differently than when all disciplines
are assessed working together [16]. Future research into real-
time OR-based teamwork assessment would likely require
two representatives from each professional group who
complete an overall teamwork instrument such as OTAS in
real-time and then compare data with ratings from video
replay.
It will be important to understand the causes of the vari-
ability in agreement for pairs of our raters. It is likely that each
rater has a different mental model of ideal teamwork perfor-
mance by each discipline/professional group. Future qualita-
tive research is required to further explore these mental
models and modify existing teamwork instruments accord-
ingly. Finally, the significance of technical skills integration
into overall OR teamwork is unclear. In this study, clinician
raters noted instances where participants demonstrated ideal
teamwork elements, but made ineffective or incorrect clinical
decisions. This dichotomy resulted in decreased scores by
clinician raters, but was not noticed by nonclinician raters. It
is possible that these errors result from perceptual “anchors”
that are difficult for clinical raters to overlook, but are possibly
out of proportion to their contributions to team effectiveness.
5. Conclusions
Our approach that approximated real-world implementation
of assessment tools did not produce a satisfactory model for
assessing individual professional groups and overall team
performance. More training for any observer is needed, but
nonclinicians hold promise as less expensive raters of
nontechnical OR team skills. Research about the influence of a
simulation team’s incorrect clinical decisions on clinician
rating of nontechnical skills will contribute to better training
and more accurate ratings by clinicians. Whether non-
clinicians can be trained to accurately complete checklists and
ratings of clinical actions remains an open question, as does
the question of the relationship between nontechnical and
technical behaviors. Authentic, in-situ OR simulations with
transdiscipline participants provide a laboratory to help
determine the ideal combination of teamwork skills with ac-
curate patient care to improve clinical outcomes during un-
expected OR emergencies.
Acknowledgment
Authors’ contributions: R.P., R.M., M.C.M.P.-S., M.W.H., and
E.P. contributed to conception and design. R.P., D.O.-M., and
E.P. analyzed and interpreted the data. R.P., R.M., M.C.M.P.-S.,
M.W.H., D.O.-M., and E.P. collected the data. R.P., R.M.,
M.C.M.P.-S., M.W.H., T.M.-Z., and E.P. made the critical revi-
sion of the article.
Disclosure
The authors reported no proprietary or commercial interest in
any product mentioned or concept discussed in this article.
r e f e r e n c e s
[1] Mills P, Neily J, Dunn E. Teamwork and communication insurgical teams: implications for patient safety. J Am Coll Surg2008 Jan;206:107.
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 8 27
[2] Buljac-Samardzic M, Dekker-van Doorn CM, vanWijngaarden JD, van Wijk KP. Interventions to improve teameffectiveness: a systematic review. Health Policy 2010 Mar;94:183.
[3] Lingard L, Espin S, Whyte S, et al. Communication failures inthe operating room: an observational classification ofrecurrent types and effects. Qual Saf Health Care 2004 Oct;13:330.
[4] Catchpole K, Mishra A, Handa A, McCulloch P. Teamworkand error in the operating room: analysis of skills and roles.Ann Surg 2008 Apr;247:699.
[5] Russ S, Hull L, Rout S, Vincent C, Darzi A, Sevdalis N.Observational teamwork assessment for surgery: feasibilityof clinical and nonclinical assessor calibration with short-term training. Ann Surg 2012 Apr;255:804.
[6] Greenberg CC, Regenbogen SE, Studdert DM, et al. Patterns ofcommunication breakdowns resulting in injury to surgicalpatients. J Am Coll Surg 2007 Apr;204:533.
[7] Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R.Anaesthetists’ Non-Technical Skills (ANTS): evaluationof a behaviouralmarker system. Br J Anaesth 2003May;90:580.
[8] Yule S, Flin R, Paterson-Brown S, Maran N. Non-technicalskills for surgeons in the operating room: a review of theliterature. Surgery 2006 Feb;139:140.
[9] Mitchell L, Flin R, Yule S, Mitchell J, Coutts K, Youngson G.Development of a behavioural marker system for scrub
practitioners’ non-technical skills (SPLINTS system). J EvalClin Pract 2013 Apr;19:317.
[10] Clancy CM, Tornberg DN. TeamSTEPPS: assuring optimalteamwork in clinical settings. Am J Med Qual 2007 May-Jun;22:214.
[11] Frankel A, Gardner R, Maynard L, Kelly A. Using theCommunication and Teamwork Skills (CATS) Assessment tomeasure health care team performance. Jt Comm J QualPatient Saf 2007 Sep;33:549.
[12] Healey AN, Undre S, Vincent CA. Developing observationalmeasures of performance in surgical teams. Qual Saf HealthCare 2004 Oct;13(Suppl 1):i33.
[13] Rudolph JW, Simon R, Rivard P, Dufresne RL, Raemer DB.Debriefing with good judgment: combining rigorous feedbackwith genuine inquiry. Anesthesiol Clin 2007 Jun;25:361.
[14] Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons’ non-technical skills in the operatingroom: reliability testing of the NOTSS behavior ratingsystem. World J Surg 2008 Apr;32:548.
[15] Sevdalis N, Lyons M, Healey AN, Undre S, Darzi A,Vincent CA. Observational teamwork assessment forsurgery: construct validation with expert versus noviceraters. Ann Surg 2009 Jun;249:1047.
[16] Healey AN, Undre S, Sevdalis N, Koutantji M, Vincent CA.The complexity of measuring interprofessional teamwork inthe operating theatre. J Interprof Care 2006 Oct;20:485.
Appendix 1 e Items regarding feasibility of assessmenttools.*
1. Rate your comfort interpreting the categories and elements of
each observation tool.
2. Rate your level of confidence that your marks on each of these
tools will align with marks form other observers for these
videos.
3. Indicate your level of confidence that you would have similar
marks on the same tool if you rated the same videos 1 wk
later.
4. Rate the amount of effort (time and mental energy) you
expended to use each instrument.
5. What is the feasibility for you to use each tool for marking
behavior in real time?
6. What is the feasibility for you to use each tool with video
replay?
* Scale for each item: very low to very high (1e7).
j o u r n a l o f s u r g i c a l r e s e a r c h 1 9 0 ( 2 0 1 4 ) 2 2e2 828