Using Scripted Video to Assess Interdisciplinary Team Effectiveness Training Outcomes

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [University of South Florida]On: 4 February 2010Access details: Access Details: [subscription number 917282056]Publisher RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Gerontology & Geriatrics EducationPublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t792304009

Using Scripted Video to Assess Interdisciplinary Team EffectivenessTraining OutcomesKathryn Hyer a; John H. Skinner b; Robert L. Kane c; Judith L. Howe d; Nancy Whitelaw e; NancyWilson f; Ellen Flaherty g; Lois Halstead h; Terry Fulmer i

a Training Academy on Aging, Florida Policy Exchange Center on Aging, University of South Florida,Tampa, FL, USA b Department of Gerontology, University of South Florida, Tampa, FL, USA c Long-Term Care and Aging, University of Minnesota School of Public Health, Minneapolis, MN, USA d

Brookdale Department of Geriatrics and Adult Development, Mount Sinai School of Medicine, NewYork, NY, USA e Vital Aging Center-Health Studies, National Council on the Aging, Washington, DC,USA f Medicine-Geriatrics and Assistant Director, Huffington Center on Aging, Department ofMedicine-Geriatrics, Baylor College of Medicine, Houston, TX, USA g Geriatric NursePractitioner/Adult Nurse Practitioner Program and Co-Principal Investigator, GITT Resource Center,New York University, Division of Nursing, New York, NY, USA h Rush University College of Nursing,Chicago, IL, USA i Erline Perkins McGriff, New York University, School of Education, New York, NY,USA

To cite this Article Hyer, Kathryn, Skinner, John H., Kane, Robert L., Howe, Judith L., Whitelaw, Nancy, Wilson, Nancy,Flaherty, Ellen, Halstead, Lois and Fulmer, Terry(2004) 'Using Scripted Video to Assess Interdisciplinary TeamEffectiveness Training Outcomes', Gerontology & Geriatrics Education, 24: 2, 75 — 91To link to this Article: DOI: 10.1300/J021v24n02_07URL: http://dx.doi.org/10.1300/J021v24n02_07

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

http://www.informaworld.com/smpp/title~content=t792304009

http://dx.doi.org/10.1300/J021v24n02_07

http://www.informaworld.com/terms-and-conditions-of-access.pdf

Using Scripted Videoto Assess Interdisciplinary TeamEffectiveness Training Outcomes

Kathryn Hyer, PhDJohn H. Skinner, PhDRobert L. Kane, MDJudith L. Howe, PhD

Nancy Whitelaw, PhDNancy Wilson, MA, LMSW

Ellen Flaherty, MSN, RNLois Halstead, PhD, RN

Terry Fulmer, PhD, RN, FAAN

Kathryn Hyer is Director, Training Academy on Aging, Florida Policy ExchangeCenter on Aging, University of South Florida, 13301 Bruce B. Downs Blvd.MHC1715, Tampa, FL 33612. John H. Skinner is Associate Professor, Department ofGerontology, University of South Florida, USF #30437, 4202 E. Fowler Avenue,Tampa, FL 33620. Robert Kane is Minnesota Chair in Long-Term Care and Aging,University of Minnesota School of Public Health, Mayo Mail Code 197, Minneapolis,MN 55455. Judith L. Howe is Assistant Professor, Brookdale Department of Geriatricsand Adult Development, Mount Sinai School of Medicine, One Gustave L. Levy Place,Box 1070, New York, NY 10029. Nancy Whitelaw is Director,Vital Aging Cen-ter-Health Studies, National Council on the Aging, 300 D St. SW, Suite 801, Washing-ton, DC 20024. Nancy L. Wilson is Assistant Professor of Medicine-Geriatrics andAssistant Director, Huffington Center on Aging, Department of Medicine-Geriatrics,Baylor College of Medicine, One Baylor Plaza M-320, Houston, TX 77030-3498. El-len Flaherty is Coordinator of the Geriatric Nurse Practitioner/Adult Nurse PractitionerProgram and Co-Principal Investigator, GITT Resource Center, New York University,Division of Nursing, 246 Greene Street, 5th Floor, New York, NY 10003. LoisHalstead is Associate Dean, Rush University College of Nursing. 600 S. Paulina, Suite1080, Chicago, IL 60612-3873. Terry Fulmer is Erline Perkins McGriff Professor andHead of the Division of Nursing, New York University, School of Education, 246Greene Street, 8th Floor, New York, NY 10003-6677.

The authors wish to thank the John A. Hartford Foundation of New York City for itsgenerous support of the GITT program and of this research. All findings are the responsi-bility of the authors and do not necessarily reflect the policies of the Foundation.

Gerontology & Geriatrics Education, Vol. 24(2) 2003http://www.haworthpress.com/store/product.asp?sku=J021

2003 by The Haworth Press, Inc. All rights reserved.Digital Object Identifier: 10.1300/J021v24n02_07 75

Downloaded By: [University of South Florida] At: 14:39 4 February 2010

ABSTRACT. As part of the Geriatric Interdisciplinary Team Training(GITT) Program funded by the John A. Hartford Foundation, the authorsof this article worked to create an instrument, the Trainee Test of TeamDynamics, to assess health care trainees’ understanding of team dynam-ics. The Trainee Test of Team Dynamics is a five-question written testdesigned to capture GITT trainees’ knowledge of team process and skillsin addressing conflict that is administered after watching a five-minutevideotape of a simulated interdisciplinary health care team meeting. Thetest was created to measure health professions students’ abilities to rec-ognize effective geriatric health care teams, to respond to effective andineffective team behaviors, and to determine whether or not the teammeeting achieved its purpose: to meet the patient’s needs for an interdis-ciplinary care plan. Scripts and test items developed and tested by practi-tioners in social work, medicine, public health, nursing and othersassured a product that compensated for differences in educational leveland occupation, yet captured accurate and appropriate responses. The re-sults reported here include an analysis of 740 trainees’ baseline re-sponses from the multi-site educational programs to determine theconstruct validity of the new measure. [Article copies available for a feefrom The Haworth Document Delivery Service: 1-800-HAWORTH. E-mail ad-dress: <[email protected]> Website: <http://www.HaworthPress.com> © 2003 by The Haworth Press, Inc. All rights reserved.]

KEYWORDS. Interdisciplinary education, geriatrics, team training,videotape, evaluation

In 1997, nationally recognized geriatric leaders, as part of the GeriatricInterdisciplinary Team Training (GITT) Program funded by the John A.Hartford Foundation of New York City (Hyer, 1998), met to developtools to assess health care trainees’ understanding of teams and team dy-namics for a new interdisciplinary team training program. Spearheadedby the GITT Resource Center,1 this paper describes the steps taken tocreate and validate a new test, the Trainee Test of Team Dynamics. De-signed to capture GITT trainees’ knowledge of team process and skillsin addressing conflict, the Trainee Test of Team Dynamics is afive-question written test that is administered after trainees watch afive-minute videotape of a simulated interdisciplinary health care teammeeting. This endeavor was undertaken since no adequate measures tocapture trainees’ knowledge of team process were available for use asevaluative tools (Fulmer & Hyer, 1998).

76 GERONTOLOGY & GERIATRICS EDUCATION


Interdisciplinary training in geriatrics has long been championed inthe geriatric and gerontology literature (Drinka & Clark, 2000; Gallo,Fulmer, Paveza, & Reichel, 2000; Heinemann, 1995; Drinka, 1996; Toner,Miller, & Gurland, 1994; Qualls & Czirr, 1988). Despite the calls for in-creased training in interdisciplinary education (O’Neil, 1993; Holmes &Osterwies, 1999), little substantive work has been done to establishmethods to measure either students’ or clinicians’ team skills andknowledge (Heinemann & Zeiss, 2002; Fulmer & Hyer, 1998; Schmitt,Farrell, & Heinemann, 1988).

The difficulty in establishing measures for teams originates from theconceptual complexity of teams. It is complicated by the differences inevaluating performance of ongoing teams versus the educational impactof team training for health care trainees. Schmitt et al. (1988, p. 763)noted the absence of information “about the processes and outcomes as-sociated with systematically organized and implemented interdisciplin-ary team care.” They have criticized many previous studies assessingthe impact of teams on patient care on the grounds of poor construct va-lidity and inability to identify all the elements of the complex multidi-mensional concept of team participation. Other concerns in evaluatingthe outcome of health care teams have been the variability of knowl-edge possessed by team members, the manner in which patients havebeen grouped for team interventions, the role of families in team out-comes, the possibility of other uncontrolled covariates, and the pecu-liarities of specific team characteristics.

Interdisciplinary geriatric and gerontological education has focusedon the development of clinical teams (Drinka & Clark, 2000; Qualls &Czirr, 1988; Toner et al., 1994; Drinka, 1996; Heinemann, 1995) and theoutcomes of the team process (Heinemann & Zeiss, 2002; Schmitt et al.,1988; Saltz, McVey, Becker, Feussner, & Cohen, 1988).

Educational programs emphasize measuring changes in learners’skills and knowledge. Videotapes and simulations are increasingly usedfor educational purposes as well as for evaluation of performance. Vid-eotapes have become more common in the education of health profes-sionals to provide feedback on a trainee’s performance (Frankel &Beckman, 1982; Epstein et al., 1998). Videotapes have been used forcase studies in teaching medical sociology (Pescosolido 1990) in pa-tient education to aid communication between patients and doctors andto help trainees learn how to draw out patient concerns (Mechanic,1998) and to examine the interaction between doctors and their patients(Epstein et al., 1998).

Hyer et al. 77


https://www.researchgate.net/publication/51298023_Awkward_moments_in_patient-physician_communication_about_HIV_risk?el=1_x_8&enrichId=rgreq-555951f3-1602-4369-9580-ee86ee619d0e&enrichSource=Y292ZXJQYWdlOzc4NjY1MDU7QVM6MTkyNzkxMzQwNDc4NDY1QDE0MjI5NzY0NDY5MDI=

During the 1990s, medical education began to shift evaluation of stu-dents from individual clinical faculty evaluations to more uniform evalu-ations, using observations of standardized patients in the objectivestructured clinical examination (OSCE) process, a process during whichmedical students were evaluated on their interactions with non-physi-cians trained to play the role of patients (Van der Vleuten & Swanson,1990; Ferrell, 1995). Kalet, Earp, and Kowlowitz (1992) emphasized theimportance of experienced clinical faculty directly observing student in-teractions with standardized patients. They discussed the use of videotap-ing student-patient interviews for subsequent coding. They found thatwhile clinicians were able to identify effective interviewing skills of stu-dents, they were unable to provide specificity about their assessments.

Our review of the literature uncovered few measures to evaluatechanges in skills of students learning team skills (Fulmer & Hyer,1998). There were also no uses of videotapes to test knowledge of teamdynamics. While observational techniques have been a part of the em-pirical method of research, there is no evidence in the literature of geri-atric team training of using scripted videotapes designed to representtheoretical team behaviors as an assessment tool for student learning ina training program. Further, there is no literature on the use of observa-tional measurement processes to assess team effectiveness based onscripted videotape scenarios.

Yet GITT faculty, familiar with the increased importance of evaluat-ing clinicians in a systematic way, believed it was important to developa standard evaluation of team skills that would be familiar to medical,nursing, and other health professions educators. Without the ability tomeasure a trainee’s knowledge of team dynamics, faculty recognized itwould be impossible to document the impact of team training on stu-dents’ ability to distinguish ineffective from effective teams. The GITTfaculty also understood that the very process of describing team dynam-ics in a written format provided inadvertent cues to the trainees abouteffective and ineffective behaviors. Consequently, it was decided to testthe trainees’ knowledge of team dynamics through videotapes of simu-lated geriatric health care team meetings.

CONSTRUCTION OF THE TRAINEE TESTOF TEAM DYNAMICS

The Trainee Test of Team Dynamics, the focus of this paper, was cre-ated to capture whether GITT trainees learned effective team dynamics.



The GITT faculty agreed to a specific set of learning objectives fortrainees at a national meeting prior to the full launch of GITT. Table 1includes the domains, or learning objectives for trainees, the specificquestions used in the test to measure that concept and the scoring forcorrect answers. Specifically, the faculty agreed that scripted video-tapes needed to be written and filmed to test the capacity of students to:(Domain 1) distinguish team dynamics and meeting processes fromteam outcomes (teams should produce an interdisciplinary care planthat establishes or improves the care plan for the patient); (Domain 2)identify effective and ineffective team behaviors; and (Domain 3) de-lineate approaches for coping with ineffective team behaviors. GITTdistinguishes team outcome, an integrated care plan, Domain 1, fromteam process and team members’ abilities to effectively handle inter-personal relationships, covered in Domains 2 and 3. Simply listing nu-merous conditions or problems does not improve a patient’s care.Unless a team sets priorities, creates an action care plan, and agrees onhow they will implement and evaluate their plan, the meeting has not ad-vanced patient care (Hyer, Fairchild, Abraham, Mezey, & Fulmer, 2000).

Creating the Video Scripts and Videotapes

The simulated health care team meeting scripts were carefully writ-ten and professionally produced to ensure that the scripts tested thelearning objectives and that the videos were of consistent difficulty. Anationally recognized expert in team dynamics was hired to create cate-gories to characterize common team behavior. These behaviors werelinked to agreed-upon educational objectives, and positive and negativeteam behaviors were organized into seven categories: conflict creation,conflict management, teaching/learning behaviors, leadership styles,meeting behavior, meeting skills, and willingness to recognize otherprofessional roles (Drinka, 1997). Upon review by GITT faculty, twonew categories were added: (a) respect for the patient; and (b) respectfor the family’s role on the team.

Teams of clinicians and gerontologists wrote video scripts. The RCdistributed the scripts to Principal Investigators for comments and revi-sions. Each video script presented the following health care team dy-namics: competent professionals who knew the patient and family;identification of important new medical and psychosocial issues thatwere addressed by the team; insights that reflected each discipline’sunique contribution; a team leader who started meetings with an orga-nized structure that later fell apart; teams that did not prioritize pa-

Hyer et al. 79



TABLE 1. Trainee Test of Team Dynamics

Domain 1:

Ability to distinguish team dynamics and meeting processes from team outcomes (aninterdisciplinary care plan that establishes or improves the care plan for the patient).

Criterion Scoring per item

(Max score possible on anyitem = 10 points)

Item 1: On the scale below(1-10) please circle thenumber representing yourperception of the teamsfunctioning (10 highlyeffective and 1 highlyineffective).

Congruence with goldstandard established from

expert responses.

Within 1 SD of theexpert mean = 5 points

Within .5 SD of theexpert mean = 10 points

Item 4: How valuable wasthe team meeting inestablishing or improvingthe care plan for thepatient. Please circle thenumber that representsyour judgment (10 veryvaluable and 1 not at allvaluable).


expert responses.

Within 1 SD of the expertmean = 5 points

Within .5 SD of the expertmean = 10 points

Item 5: Please give threedifferent reasons why youcircled the number you didin question 4.


expert responses.

3 points credit given for eachidentified behavior.

One positive behaviorallowed.

Domain 2:

Recognize effective and ineffective team behaviors.

Item 2: Please list thespecific behaviors youobserved in the meeting.Be as concrete as possible:

a) list discipline andeffective behaviors

b) list discipline andineffective behaviors.

Identify at least ONE effec-tive behavior congruent withgold standard established by

expert responses AND atleast ONE ineffective behav-ior congruent with gold stan-dard established by expert

responses.

5 points possible for EACHbehavior type.

(Max for effectivebehaviors = 5 pointsMax for ineffective

behaviors = 5 points)

Domain 3:

Delineate ways to respond to ineffective team behaviors.

Item 3: List three differentways you would haveresponded to theseineffective behaviors.

Three ineffective responsesto behaviors in congruence

with “gold standard.”

3 points credit given foreach of three identified

responses.


tient/family needs; teams that did not agree on a care plan or thedevelopment of an interdisciplinary care plan with priorities and jointaccountability; teams that had interruptions from beepers or memberswho disrupted the process with side conversations; teams that managedconflict poorly (sarcasm, defensiveness, withdrawal, inappropriate com-ments); and teams that failed to resolve conflict among team members.

Professional actors and directors were hired to produce the video-tapes. Each of four scripted videotapes simulated a team meeting andincluded four members of the team (a physician, a nurse practitioner, asocial worker, and a pharmacist), with each actor representing the sameprofession in each of the tapes. However, the actors in each of the video-tapes exhibited different effective/ineffective behaviors. Each video in-cluded a case presentation of a patient with discussion among the repre-sentatives of the four disciplines at the team meeting. Scripted negativebehaviors included interruptions such as beepers, late arrivals, and dis-agreements during the meeting. The team’s interactions were scriptedto present a preponderance of ineffective team behaviors, but there werealso examples of positive behaviors to keep the simulation real. All cli-nicians were familiar with the patient and appropriately raised new is-sues or provided information to the team. Scripted positive actionsincluded such behaviors as soliciting information from quiet teammembers or redirecting a discussion after an interruption.

METHODS

Developing the “Gold Standard” Test Answers

Even though there was agreement about the domains and questions,the actual responses that would be considered correct and incorrect an-swers to our test needed to be developed. Thirty-one GITT faculty andnational experts rated the videotapes to create a “gold standard” towhich trainee ratings were compared. Expert answers were solicitedfrom 24 GITT faculty representing medicine, nursing, social work,physical and occupational therapy, public health and health administra-tion and from seven national team experts representing social work,medicine, nursing, occupational therapy, psychology, and gerontology.Faculty were requested to be as comprehensive as possible in answeringthe open-ended responses. The goal was to create a compendium of allpossible correct answers that could be used as a benchmark to grade theanswers of trainees.

Hyer et al. 81


All open-ended answers (Items 2, 3, and 5) were reviewed andgrouped into qualitative codes. For the question on effective and inef-fective behaviors (Item 2), the answers were coded into the nine catego-ries of effective and ineffective behaviors that originally drove thescripted behaviors of the actors. Qualitative answers about how to re-spond to ineffective behaviors (Item 3) were coded into categories de-pending upon the videotape specifics, but included codes such as“establish meeting structure and ground rules,” “appoint a timekeeper,”“ask team member to contribute to discussion,” or “facilitate meetingby refocusing discussion after interruption.” Item 5, justifying reasonswhy teams were effective or ineffective, included positive responses,such as “patient problems identified and discussed” and “team mem-bers were engaged in discussion,” and negative responses, such as “pa-tient problems identified but not resolved,” “team members were rudeor disrespectful,” and “no interdisciplinary care plan was established.”

To ensure that behaviors identified less frequently by faculty werereviewed and categorized as correct or incorrect responses, a specialwork group was established. The Videotape Work Group (VTWG) con-sisted of eight faculty from multiple disciplines (medicine, nursing, so-cial work, gerontology, and public health) and represented the GITTnational program and the Resource Center. They performed a second-ary validation of the qualitative code categories. The group met via con-ference calls to discuss responses for each video. Background datadisplayed each expert answer, the discipline of the expert, and the quali-tative codes created for the response. All qualitative codes created to re-flect expert answers were reviewed. Responses cited by fewer than 20%of the experts were voted on by the workgroup to determine whether theanswer provided by the expert was correct or incorrect according to theintent of the script. After the videotapes were distributed and trainees’answers were assessed, the VTWG was consulted again because actualtrainee responses did not always match expert answers. The VTWG re-viewed these new responses. As expected, the new categories createdwere generally new codes for incorrect answers.

Creating a Grade for Trainees

Two questions on the test involve rating the simulated team meetingon a ten-point scale. The scoring of the two ten-point numeric scales ofeffectiveness and value of teams (Items 1 and 4) were determined by thedegree to which the trainees’ ratings were congruent with those of theseven team experts. Trainees with ratings within plus or minus one-half



standard deviation (SD = ± .5) of the expert mean received full credit(10 points), and those falling within plus or minus one standard devia-tion ( ± .5 < SD < ± 1.0) of the mean were given half credit (5 points).All trainee ratings greater than 1.0 SD of the mean were assigned scoresof 0. By adjusting the scoring of trainee items to the mean and standarddeviations of expert criterion referenced responses, it was possible tomake comparisons across videos. This process compensated for differ-ent means and standard deviations of individual videos since all scoreswere assigned based on their relative distance from the respective ex-pert mean. Hence, a trainee with an adjusted score of 5 on one video and5 on another video performed comparably, with the same relative posi-tion from the mean of all trainees rating the two videos. This process re-sulted in a relative ranking of trainee performance.

Open-ended responses to the other three test items were assignedscores based on whether the response matched those of faculty and teamexperts. If a response met the “gold standard,” it was considered correctand assigned a positive score. Incorrect responses were assigned a nega-tive score. This approach was adopted to correct for guessing. Missing,neutral, and redundant responses were assigned a score of 0. The maxi-mum score on any item was 10 points. In the case of the item elicitingresponses of effective and ineffective behaviors, Item 2, trainees had tolist at least one effective and one ineffective behavior to receive anycredit. This two-part item provided for a maximum of 5 points for iden-tifying effective behaviors and 5 points for identifying ineffective be-haviors.

In order for trainees to receive full credit for Item 3, trainees had tomatch the ineffective behaviors identified by the experts. Less thanthree listed behaviors were scored proportionately for Item 3. The lastopen-ended item, Item 5, asked trainees to justify the score they as-signed to the team’s ability to improve the patient’s care plan. Traineesreceived no credit for Item 5 if their answers were more than 1 standarddeviation away from the mean of the experts’ rating on Item 4, since thetrainee did not recognize that the team was not effective. Full credit onthe justification was given for listing three different reasons that con-curred with those of the experts. Although the scripted tapes were de-signed to present ineffective teams that did not meet the needs ofpatients, some positive aspects of the team process were identified suchas “the team identified new patient issues and addressed these in themeeting.” Up to one positive reason was accepted as a correct answer ifthe expert panel had also identified it.

Hyer et al. 83


Every effort was made to ensure that response categories were codedto be compatible across videos yet represented the specific scenariospresented in each video. Even though great care was given to designingvideos that followed similar scripts representing similar behaviors, itwas impossible to assure that they were statistically equivalent. To ad-dress the concern of the statistical equivalency, the scales of the evalua-tion instrument were established to make the scores of different videoscomparable by developing standard scores that permitted comparisonsacross videos despite different means and standard deviations. The totalscores in the evaluation instrument were summed and standardized sothat each video had a mean of 20 and a standard deviation of five.

The objectives of the analysis are to determine if (1) the evalua-tion items measure a single domain or more than one domain; (2) theitems support the construct validity of the domains of team behav-iors; (3) evaluation videos are equivalent at the beginning of the train-ing; and (4) the evaluation items are sensitive to and differentiateamong trainees from various disciplines at Time One.

A protocol for the administration of all GITT measures, including thevideotape, was developed by the Resource Center and approved by theGITT investigators. The GITT core measures, including a demographicquestionnaire, tests attitudes towards health care teams, measures ofknowledge of care planning and the 5-minute video, were generally ad-ministered to all GITT trainees as a group at the first the team trainingmeeting or class. The video was shown only once, so the entire set ofcore measures could be administered within approximately 60 minutes.Every year, videotapes were randomly assigned to sites, and the videoused for the pre-test was different from the video used for the post-test.

RESULTS

Characteristics of Program Participants

This study examines the responses of 740 geriatric interdisciplinaryteam trainees at baseline. Table 2 presents the characteristic of the pro-gram trainees. Seventy-two percent of trainees were female, and sev-enty percent were white. Asian/Pacific Islanders made up 16 percent ofthe trainees, with approximately equal percents (approximately 5%) ofAfrican American, Native American/Alaskan Islander, and Latinotrainees. The average age of trainees was 32, with ages ranging from 22to 70 years of age. Most of the trainees were physicians (41%). Nurses



accounted for 17 percent of trainees, social work trainees for 14 percent,and pharmacy trainees for 9 percent. Other disciplines not specificallyidentified made up 19 percent of the trainees. The majority of traineeswere full-time students. Their average paid work experience in geriat-rics was about two-and-a-half years. The length of GITT training variedby students’ discipline, with the average length of training at approxi-mately one semester.

Table 3 shows the inter-item correlations of standardized scores.This table shows correlations ranging from �.01 to .48. The data revealthat items 1, 4, 5 remain highly intercorrelated, while items 2a, 2b, and 3seem to cluster together. The data appear to create two clusters that

Hyer et al. 85

TABLE 2. Characteristics of the Program Participants

Items Frequency Percent Mean

Gender

Female 530 72.5

Male 201 27.5

Race/Ethnicity

White 519 70.4

African American 36 5.0

Black non-African American 6 .8

Asian/Pacific Islander 121 16.6

Native American/Alaskan Islander 1 .1

Latino 36 5.0

Other 44 5.9

Age 32.37

Discipline

Medical 303 41.4

Nursing 122 16.7

Social Work 100 13.7

Pharmacy 65 8.9

Other 142 19.4

Student Status

Full Time 536 79.9

Part Time 82 12.2

Other 53 7.9

Days of Training 101.09

Years Worked in Geriatrics 2.48


were confirmed in principal component factor analysis with Varimaxrotation. The two factors are similar to a prior design of the three do-mains that were tested in the Team Dynamics Test. The exception to thea priori model is that Item 3 does not seem to operate as a single factor,but is more logically part of a domain addressing effective and ineffec-tive behaviors.

The planning and design of the scripted videos were intended a priorito result in four videos that were essentially equal. To determine if re-sponses to videos were perceived as equal, the pre-test data were ana-lyzed using one-way ANOVA of standardized item mean scores foreach video. As shown in Table 4, there were no significant differencesin the one-way ANOVAs of the standardized mean scores between vid-eos, suggesting that the standardization process was successful in estab-lishing the equivalency of videotapes.

To determine whether trainees with different formal training respondsimilarly, we performed a one-way ANOVA on the standardized itemscores by trainee discipline. It was expected that pre-test results wouldyield differences among trainees by disciplines. Table 5 presents theone-way ANOVA of standardized item means for five groups of healthcare trainees: medicine, nursing, social work, pharmacy, and otherhealth care trainees. Two items were found to differentiate trainees bytheir discipline. Significant differences in responses of trainees forItems 1 and 4 were observed. The lowest mean score for Item 1, the abil-ity to rate the team’s effectiveness, was by pharmacy trainees, and thehighest mean score was registered by “other” trainees. A post hoc multi-ple comparison using the Scheffe test of differences between groups in-dicated that there were no significant pairwise differences found for


TABLE 3. Inter-Item Correlation of Standardized Scores

Item 1 Item 2a Item 2b Item 3 Item 4 Item 5

Item 1 1.00

Item 2a �.01 1.00

Item 2b �.02 .20*** 1.00

Item 3 .02 .09* .13*** 1.00

Item 4 .22*** .01 �.03 .05 1.00

Item 5 .10** .02 .09* .15*** .48*** 1.00

N = 739*p < .05; ** p < .01; ***p < .001


Item 1. However, for Item 4, a pairwise comparison revealed that themean score of medicine trainees was statistically different from themean score of social work trainees (p = .01). Social work trainees atbaseline were more likely to recognize that the simulated health careteams were not advancing or improving the patient’s care plan. Thesedata verify that trainees from various disciplines did respond differentlyto test items at baseline, suggesting that the test item is sensitive to theeducational background of trainees.

DISCUSSION

We set out to create an evaluation instrument to assess team dynam-ics using scripted videos. The response included five assessment ques-tions with one item consisting of two parts. These five items wereintended to assess the process and outcomes of team activity using threedomains: (1) distinguishing team process from team outcomes, (2) rec-ognizing effective and ineffective team behaviors, and (3) identifyingways to address ineffective team behaviors.

The review of pre-test inter-item correlations was also supported byprinciple component factor analysis (not reported here) and failed tosupport the three a priori concepts that were tested. Instead, the analysesrevealed that only two concepts evolved. The first domain, ability todistinguish team dynamics and meeting processes from team outcomes,was fully supported by correlation analysis. However, the second do-main, recognition of effective and ineffective team behaviors, includedthe two items that had been hypothesized and also included a third item,having the knowledge of how to respond to ineffective behaviors, that

Hyer et al. 87

TABLE 4. One-Way Anova of Means of Standardized Score Items on EachVideo

StandardizedScores

Video 1 Video 2 Video 3 Video 4 F Ratio P

Item 1 20.00 19.95 20.00 20.04 .005 1.0n.s.

Item 2a 20.00 20.03 20.00 19.98 .002 1.0n.s.

Item 2b 20.02 20.00 20.03 20.03 .001 1.0

Item 3 20.00 20.03 20.00 19.99 .002 1.0

Item 4 20.00 19.94 20.00 20.03 .007 .999

Item 5 20.00 19.96 20.00 20.02 .003 1.0

N = 250 N = 173 N = 174 N = 142


had been thought to be an independent concept. It appears that the abil-ity to recognize effective and ineffective behaviors is indistinguishablefrom the knowledge of how to respond to those behaviors. If one has theability to recognize ineffective and effective behaviors, one is also ca-pable of suggesting ways to respond to ineffective behaviors. There-fore, the two-domain solution reflects an improvement over the originalconceptual framework.

Another concern at the beginning of the study was to ensure that thefour final videos were statistically equivalent at pre-testing. Our datashowed that standardized item scores showed no statistical differencesbetween videos and provided evidence that the videos are essentiallyequivalent for establishing baseline data on trainees.

We also wanted the items to be sensitive to trainees from differenteducational backgrounds. Our analysis demonstrates that, at baseline,the videotape test appears to be sensitive to differences in formal train-ing. Furthermore, Item 4, the ability to recognize whether or not theteam is advancing the patient’s needs, seems to capture prior disciplin-ary sensitivity to team dynamics. Social workers generally receivetraining on group process; they scored significantly higher on thepre-test than physicians who do not typically receive such training. Thisis consistent with literature on professional communication, roles, andconflict between social workers and physicians (Netting & Williams,1996).

The assessment instrument for the Trainee Test of Team Dynamicsappears to be most useful when the raw scores are standardized atpre-test. This standardization process adjusts for differences betweenvideos and provides assurance that we have equivalency at pre-test.


TABLE 5. One-Way ANOVA of Standardized Item Scores by Degreeof the Trainees

StandardizedScores

Med Nurse Soc. Wrk Pharm Other F Ratio P.

Item 1 19.53 20.55 20.25 19.07 20.73 2.459 .044

Item 2a 19.98 20.28 19.60 19.76 20.25 .385 .819

Item 2b 20.09 20.36 19.01 20.87 19.82 1.711 .146

Item 3 20.21 20.06 20.03 19.73 19.61 .392 .814

Item 4 19.48 19.57 21.52 20.09 20.31 3.545 .007

Item 5 20.05 20.18 20.29 19.53 19.77 .345 .848

N = 302 N = 123 N = 100 N = 65 N = 142


Pre-test and post-test comparative analyses of item differencesshould be made using the two domains identified in this analyses. Whilethe two domains identified in these results may be combined intosummative scales, the use of individual test items may yield more quali-tative information about the training experience. Therefore, it is recom-mended only single items of this test be used to assess team training.

We employed an innovative approach to quantifying and correlatingresponses from students working in multidisciplinary health fields.Scripts and test items developed and tested by practitioners in socialwork, medicine, public health, nursing, and other disciplines assured aproduct that compensated for differences in educational level and occu-pation, yet captured accurate and appropriate responses.

Several limitations of this study should be noted. Because scriptedvideo is intended to be a contrived social situation, factors might skewthe psychometric validity of an item’s difficulty, and therefore addi-tional work needs to be done to examine the pre-test and post-test oftrainees exposed to the videos. Re-testing the differential effects of vid-eos should include testing the order in which videos are presented andthe possibility of using videos as their own controls. In future scripts, theemphasis on subtle behaviors could be greater, potentially capturing astronger validity of legitimate change. With these refinements, this tech-nique offers a good baseline on which others can build.

NOTE

1. The GITT Resource Center was housed within the Division of Nursing at theLeinhart School of Education, New York University.

REFERENCES

Drinka, T. J. K. (1996). Applying learning from self-directed work teams in business tocurriculum development for interdisciplinary geriatric teams. Educational Geron-tology, 22, 433-50.

Drinka, T. J. K. (1997). Effective and ineffective “Team Behaviors” consulting report.New York University for GITT Resource Center.

Drinka, T. J. K, & Clark, P. G. (2000). Health care teamwork: Interdisciplinary prac-tice and teaching. Westport, CT: Auburn House.

Epstein, R. M., Morse, D. S., Frankel, R. M., Frarey, L., Anderson, K., & Beckman,H. B. (1998). Awkward moments in patient-physician communication about HIVrisk. Annals of Internal Medicine, 128, 435-442.

Hyer et al. 89


Ferrell, B. G. (1995). Clinical performance assessment using standardized patients: Aprimer. Family Medicine, 27, 14-19.

Frankel, R. M., & Beckman, H. B. (1982). Impact: An interaction-based method forpreserving and analyzing clinical transaction. In L. S. Pettegrew (Ed.) Straight talk:Explorations in provider and patient interaction (pp. 71-84). Louisville, KY:Humana, Inc.

Fulmer, T., & Hyer, K. (1998). Evaluating the effects of geriatric interdisciplinaryteam training. In E. L. Siegler, K. Hyer, T. Fulmer, & M. Mezer (Eds.) Geriatric in-terdisciplinary team training (pp. 115-149). New York, NY: Springer.

Gallo, J. J., Fulmer, T., Paveza, G. J., & Reichel, W. (2000). Handbook of geriatric as-sessment (3rd ed.). Gaithersburg, MD: Aspen.

Heinemann, G. D. (1995). Best education and training: In geriatrics, the team ap-proach. Generations, 19 (2), 20-22.

Heinemann, G. D., & Zeiss, A.M. (2002). Team performance in health care: Assess-ment and development. New York, NY: Kluwer Academic/Plenum Publishers.

Holmes, D. E., & Osterwies, M. (1999). Catalysts in interdisciplinary education: Inno-vation by Academic Health Centers. Washington, D.C.: Association of AcademicHealth Centers.

Hyer, K. (1998). The John A Hartford Foundation Geriatric Interdisciplinary TeamTraining Program. In E. L. Seigler, K. Hyer, T. Fulmer, & M. Mezey (Eds.) Geriat-ric interdisciplinary team training (pp. 3-12) New York: Springer.

Hyer, K., Fairchild, S., Abraham, I., Mezey, M., & Fulmer, T. (2000). Measuring atti-tudes related to interdisciplinary training: Revisiting the Heinemann, Schmitt andFarrell “Attitudes Toward Health Care Teams” Scale. Journal of InterprofessionalCare, 14, 249-258.

Kalet, A., Earp, J. A., & Kowlowitz, V. (1992). How well do faculty evaluate the inter-viewing skills of medical students? Journal of General Internal Medicine, 7,499-505.

Mechanic, D. (1998). Public Trust and initiatives for new health care partnerships.Milbank Quarterly, 76, 281-302.

Netting, F. E., & Williams, F. G. (1996). Case manager-physician collaboration: Impli-cations for professional identity, roles, and relationships. Health and Social Work,21, 216-224.

O’Neil, E. H. (1993). Health professions education for the future: Schools in service tothe nation. San Francisco, CA: Pew Health Professions Commission.

Pescosolido, B. A. (1990). Teaching medical sociology through film: Theoretical per-spectives and practical tools. Teaching Sociology, 18, 337-346.

Qualls, S. H., & Czirr, R. (1988). Geriatric health teams: Classifying models of profes-sional and team functioning. Gerontologist, 28, 372-76.

Saltz, C. C., McVey, L. J., Becker, P. M., Feussner, J. R., & Cohen, H. J. (1988). Im-pact of geriatric consultation team on discharge placement and repeat hospitaliza-tion. Gerontologist, 28(3), 344-50.

Schmitt, M. H., Farrell, M. P., & Heinemann, G. D. (1988). Conceptual and method-ological problems in studying the effects of interdisciplinary geriatric teams. TheGerontologist, 28, 753-764.



Toner, J. A., Miller, P., & Gurland, B. J. (1994). Conceptual, theoretical, and practicalapproaches to the development of interdisciplinary teams: A transactional model.Educational Gerontology, 20(1) 53-69.

Van der Vleuten, C. P. M., & Swanson, D. B. (1990). Assessment of clinical skills withstandardized patients: State of the art. Teaching and Learning in Medicine, 2(2),58-76.

Hyer et al. 91


Using Scripted Video to Assess Interdisciplinary Team Effectiveness Training Outcomes

Documents

Transcript of Using Scripted Video to Assess Interdisciplinary Team Effectiveness Training Outcomes