Affect Recognition in Traumatic Brain Injury

J Head Trauma RehabilCopyright c© 2013 Wolters Kluwer Health | Lippincott Williams & Wilkins

Affect Recognition in Traumatic BrainInjury: Responses to Unimodal andMultimodal Media

Barbra Zupan, PhD; Dawn Neumann, PhD

Objectives: To compare affect recognition by people with and without traumatic brain injury (TBI) for (1) unimodaland context-enriched multimodal media; (2) positive (happy) and negative emotions; and (3) neutral multimodalstimuli. Participants: A total of 60 people with moderate to severe TBI and 60 matched controls. Measures: (1)facial affect, (2) vocal affect, and (3) multimodal affect. Results: Compared with controls, people with TBI scoredsignificantly lower on both unimodal measures but not on the multimodal measure. Within- group comparisonsfor people with TBI revealed that they were better at recognizing affect from multimodal than unimodal stimuli.As a group, participants with TBI who were categorized as having impaired facial/vocal affect recognition wereless accurate at recognizing all emotions, including happy, than unimpaired participants. Neutral stimuli were morepoorly identified by participants with TBI than by those with controls. Conclusion: Context-enriched multimodalstimuli may enhance affect recognition for people with TBI. People with TBI who have impaired affect recognitionmay have problems identifying both positive (happy) and negative expressions. Furthermore, people with TBI mayperceive affect when there is none. Key words: affect recognition, media, multimodal, traumatic brain injury, unimodal

IN TRAUMATIC BRAIN INJURY (TBI), areas of thebrain commonly associated with emotion are particu-

larly susceptible to damage1; as a result, people with TBIoften exhibit impaired recognition of others’ emotions,despite the heterogeneity of their injuries.2–12 Quality ofinterpersonal interactions and relationships partially de-pends on accurate interpretation of others’ emotions vianonverbal, semantic, and contextual cues.10,13–15 Thus,it is important to consider how these cues are processedunder various conditions so that interventions that tar-get specific issues can be designed.

A recent meta-analysis has shown that 13% to39% of people with moderate to severe TBI aresignificantly impaired at recognizing emotion fromstatic facial cues.16 Recognition of vocal affect is also

Author Affiliations: Department of Applied Linguistics, BrockUniversity, St. Catharines, Ontario, Canada (Dr Zupan); andDepartment of Physical Medicine and Rehabilitation, Indiana UniversitySchool of Medicine and Rehabilitation Hospital of Indiana, Indiana(Dr Neumann).

This work was funded through the Humanities Research Institute at BrockUniversity in St. Catharines, Ontario, and by the Cannon Research Center atCarolinas Rehabilitation in Charlotte, North Carolina.

The authors declare no conflicts of interest.

Corresponding Author: Barbra Zupan, PhD, Department of Applied Lin-guistics, Brock University, 500 Glenridge Avenue, St. Catharines, Ontario,Canada, L2S 3A1 ([email protected]).

DOI: 10.1097/HTR.0b013e31829dded6

problematic.3,5,9,10,17,18 For instance, Dimoska et al3

found that people with moderate to severe TBI hadmore difficulty than controls at discriminating andlabeling vocal expressions. Recently, studies haveinvestigated whether adding situational context helps orhinders affect recognition after TBI. Situational contextprovides important information about circumstances,such as whether or not the situation is consistent withthat person’s wants and expectations, which wouldaffect how that person feels.19 As such, situationalcontext might be expected to foster accurate inferencesabout how a person is feeling. However, the findingsfrom studies thus far have been mixed. Croker andMcDonald2 reported that people with TBI were betterable to match and label photographs depicting variousfacial emotion expressions when social context wasadded. In contrast, Milders and colleagues9,20 reportedthat people with TBI showed significant difficulty intheir ability to identify and explain the intentions andfeelings of others using contextual information. Thus,it remains unclear what effect contextual informationhas on emotion recognition in people with TBI.

Prior work has helped inform the development oftreatment programs for people with TBI. However, be-cause previous studies focused primarily on emotionrecognition from isolated cues (rather than the multi-modal cues more characteristic of everyday interactions),further work is needed using ecologically valid stimuli.To our knowledge, only 2 studies have attempted to

Copyright © 2013 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

1

2 JOURNAL OF HEAD TRAUMA AND REHABILITATION

compare how people with TBI process affective cuesfrom unimodal (ie, face only and voice only) versusmultimodal media. McDonald and Saunders8 presented34 participants with severe TBI and 28 controls withunimodal and multimodal media extracted from theEmotion Evaluation Test,21 a video-based test that dis-plays audiovisual emotion expressions within the con-text of an emotionally ambiguous dialogue. Participantswith TBI had significantly more difficulty recognizingemotions from multimodal media and from unimodalvocal affect cues. Inconsistent with prior findings,16 theirparticipants with TBI performed as well as controls atrecognizing emotions from unimodal facial affect stim-uli (static and dynamic).8 In a similar study by Williamsand Wood,12 64 participants with TBI and 64 matchedcontrols were presented with multimodal stimuli fromthe Emotion Evaluation Test and unimodal static fa-cial stimuli (Ekman 60 Faces Test22). Participants withTBI were significantly less accurate at identifying emo-tions from both the unimodal and multimodal stimuli.However, within-group comparisons revealed that par-ticipants with TBI were more accurate at recognizingemotions from the multimodal media than from uni-modal facial stimuli. This result contradicts McDonaldand Saunders’8 findings that participants with TBI betterrecognized unimodal facial affect than affect from multi-modal media. Finally, Williams and Wood12 found thatparticipants with TBI had significantly more difficultythan controls at recognizing expressions of positive andnegative affect as well as neutral displays.

CURRENT STUDY

Although both McDonald and Saunders8 andWilliams and Wood12 investigated affect recognition inresponse to unimodal and multimodal media, neitherstudy considered the influence of semantic or contextualinformation on interpretation of emotion expressions.Thus, it remains unclear whether the presence of such in-formation might improve affect recognition through in-creased intersensory redundancy or hinder recognitionbecause of increased cognitive demands (eg, attentionand inferencing). The aim of this study was to expandupon the work of these investigators by assessing affectrecognition for multimodal media stimuli that also in-cludes contextual information. Combining contextualinformation with multimodal stimuli offers greater eco-logical validity of affect recognition assessments. Thisstudy also differs from former research in that our mul-timodal stimuli are derived from commercial televisionand film excerpts. Although still not genuine expressionsof emotion, these commercial-grade stimuli using pro-fessional actors to depict real-world social experiencesare likely to be more realistic and believable than emo-tion stimuli designed for research purposes.

To experience and express appropriate responses toeveryday emotional situations requires the ability simul-taneously to interpret a combination of nonverbal emo-tion cues, including contextual ones. Determining howwell people with TBI can do this—and what limits theirperformance—may influence the design of interventionsfor improving affect recognition by people with TBI.If contextual information enhances affect recognition,people with TBI may benefit from interventions thatmatch various affective cues (eg, vocal emotion expres-sions) to contextual situations. Ecologically valid stim-uli, such as the film and television clips proposed here,could potentially be used as training tools, allowing clin-icians to demonstrate subtle and obvious emotion ex-pressions and point out contextual cues that might helppatients to infer emotions of others.

Objectives

1. To compare affect recognition by people withand without TBI for unimodal (face only andvoice only) and context-enriched multimodal (filmclips) media. Given the results of McDonald andSaunders8 and Williams and Wood,12 we predictedthat participants with TBI would perform morepoorly than controls in all media conditions.

2. To compare affect recognition for positive (happy)and negative emotions by people with and withoutTBI. Although it has been well documented thatpeople with TBI have more difficulty recognizingnegatively valenced than positively valenced facialstimuli,2,6,11,12,23 we felt it was important to ex-plore whether valence would continue to influenceaffect recognition in context-enriched multimodalstimuli. Although we only use 1 positive exem-plar (happy), this exploratory objective will provideuseful preliminary data regarding valence effectswith this novel stimulus set for future studies. Wehypothesized that people with TBI would havemore difficulty identifying emotions with a neg-ative than positive valence.

3. To compare participants’ recognition of neutralstimuli in the context-enriched multimodal condi-tion. Although recognition of neutral was investi-gated by Williams and Wood,12 we were interestedin the effect of contextual information, which mayfacilitate participants’ interpretation of the neutralmultimodal stimuli.

METHODS

Participants

Sixty adults (37 males and 23 females) with moderateto severe TBI and 60 age-matched controls (38 malesand 22 females) were recruited from local rehabilitation


Affect Recognition in TBI Injury 3

centers in Charlotte, North Carolina, andSt. Catharines, Ontario, Canada (30 from eachgroup, from each site). Controls were either staff mem-bers or volunteers at these centers or friends and familymembers of the participants with TBI. People with TBIhad to be 6 months postinjury and have a Glasgowcoma scale (GCS) score, posttraumatic amnesia (PTA),or loss of consciousness (LOC) indicative of a moderateto severe TBI (GCS ≤ 12; PTA ≥ 24 hours; LOC ≥24 hours). Participants were excluded for the followingreasons: presence of a developmental affective disorder;acquired neurological disorder; major psychiatricdisorder; and/or impaired vision or hearing that wouldprohibit full participation in the experimental tasks. Inaddition, control participants were excluded if they hada history of TBI of any severity, including concussionresulting in postconcussive syndrome.

Participants with TBI were between the ages of 21.6and 63 years (mean = 40.98; SD = 12.45), and theage range of the control participants was 18 to 63.2years (mean = 40.64; SD = 13.04). There was no signif-icant age difference between groups [F(1, 118) = 0.021;P = 0.884]. Table 1 lists demographics and injury-relatedvariables for the participants with TBI. Given that thisstudy aimed to expand on the work of McDonald andSaunders8 and Williams and Wood,12 Table 1 also pro-vides demographic comparisons from these studies.

For all participants English was their primary lan-guage, and they were able to comprehend and respondto basic written and oral language, as indicated via self-report and interaction with the investigator during theconsent process. On average, participants with TBI hadcompleted 14.43 years of education (SD = 2.29)—-17had earned a college or university degree, 23 had at-tained some postsecondary education or training (eg,skills certificate), 16 had completed high school, and4 had completed no more than grade school. Controlparticipants had completed an average of 15.72 yearsof education (SD = 1.96)—28 had obtained a collegeor university degree, 22 reported having some postsec-ondary education or training, and 10 had completedhigh school.

Measures and Procedures

This study includes a subset of measures adminis-tered to participants as part of a larger investigation.24

Only measures relevant to this study are discussed here.Characteristic features of these measures are provided inTable 2, as are features of the affect measures used byMcDonald and Saunders8 and Williams and Wood.12

Participants in this study were seen either individuallyor in small groups (maximum = 3), and the order oftasks was randomized across testing sessions.

Unimodal Measure: Facial Affect Recognition

The Diagnostic Analysis of Nonverbal Affect 2-AdultFaces (DANVA-Faces)25 was selected for the facial affecttask because of its prior use with people with TBI.10,14

The DANVA-Faces includes 24 colored photographs ofyoung adults, portraying happy, sad, angry, and fearfulfacial expressions. Each photograph was displayed fora total of 5 seconds, and participants were instructedto select the emotion portrayed using from a set of 5choices (happy, sad, angry, fearful, and I don’t know). Thisstandardized tool is reported to have good internal con-sistency and high test–retest reliability. In addition, ithas been shown to correlate well with measures thatassess related constructs such as personality and socialcompetence (see Nowicki25 for a summary of validityevidence).

Unimodal Measure: Vocal Affect Recognition

The Diagnostic Analysis of Nonverbal Affect 2-AdultParalanguage (DANVA-Voices)25 has also been used pre-viously with people with TBI.10,14 The DANVA-Voicesconsists of 24 repetitions of a single, emotionally neutralsentence (“I’m going out of the room now, and I’ll beback later”). Participants heard each sentence only onceand were instructed to indicate the emotion portrayedthrough the tone of voice by selecting 1 of 5 possibleresponses: happy, sad, angry, fearful, and I don’t know. Asindicated by Nowicki,25 the DANVA-Voices has goodinternal consistency and test–retest reliability. Good cri-teria validity has also been indicated in studies that havecompared the performance with the performance onsimilar constructs (ie, social competence).25

Context-Enriched Multimodal Measure: Film Clips

Fifteen film clips were extracted from commercialmovies and television because of their presumed eco-logical validity (see Appendix). Unlike the video vi-gnettes used in previous studies, which were generatedfor research,8,12 cinematic film clips provide a naturalway to include context and appear very real because oftheir approximation of everyday situations.26 Clips werechosen according to the selection criteria suggested byGross and Levenson.27 In a previous study (n = 70 typ-ical adults), the film clips included in this study suc-cessfully elicited the target emotion 84.73% (range: 64-96) of the time. The clips ranged in length from 45to 103 seconds (mean = 71.87). Three restricted ran-domized orders of the film clips were created so thatthe series of clips always started with a neutral portrayaland 2 clips that targeted the same emotion never oc-curred consecutively. Participants were assigned to 1of the 3 randomized orders before recruitment using acomputerized random number generator.


www.headtraumarehab.com


TABLE 1 Demographics and injury characteristics for people with TBI in this study andpeople with TBI in similar studies conducted by McDonald and Saunders8 and Williamsand Wood12

Zupan andNeumann

McDonaldand Saunders

Williams andWood

Age, yrMean 40.98 41 35.84SD 12.45 Not reported 13.33Range 21.6-63 Not reported 20-62

SexMale 37 25 53Female 23 9 11Severity Moderate to severe Severe Moderate to severe

GCS (n = 17)Mean 4.47 Not reported 9.3SD 2.48 4.46Range 3-12 3-15

PTA (n = 31), dMean 19.9 76 16.85SD 38.16 59 27.84Range 0.5-180 Not reported 1-210

LOC (n = 40), dMean 43.05 Not reported Not reportedSD 50.66Range 0.5-180

Time since injury, yrMean 13.68 9.5 3.19SD 10.53 8 2.58Range 0.5-37 Not reported 0.63-16.27

Age at injury, yrMean 27.31 Not reported 32.77SD 13.87 13.32Range 4.5-56.3 15-62

Cause of injury, %MVA 72 65 Not reportedFall 13 15Assault 15Other 13 5

Education (n = 58)Mean years completed 14.43 12.14 13SD 2.29 2.18 3.1Range 9-18 8-20

Abbreviations: GCS, Glasgow coma score; LOC, loss of consciousness; PTA, posttraumatic amnesia; SD, standard deviation; TBI,traumatic brain injury.

RESULTS

Statistical Analysis

Descriptive statistics were calculated for all demo-graphics, injury-related variables, and outcome mea-sures. Average years of education significantly dif-fered between groups [F(1, 103) = 9.39; P = 0.003;Np

2 = 0.05]. Education was negatively correlated withDANVA-Faces (r = 0.30; P = 0.002) and DANVA-Voices (r = 0.34; P = 0.001). Therefore, group compar-isons involving the DANVA-Faces or DANVA-Voicesincluded education as a covariate.

Analyses were conducted using mean percentage ac-curacy scores for the DANVA-Faces, DANVA-Voices,and film clips. Positive affect scores consisted of happyitems; scores for negative affect were determined by col-lapsing responses to sad, angry, and fearful stimuli into asingle category. Mixed model analyses of variance (AN-COVA) were conducted to examine affect recognitionperformance by TBI and control participants within andacross the 3 types of media using education as a covari-ate. When examining differences across media, we ex-cluded neutral from the film clip accuracy scores so thatonly emotions that were represented in all 3 tasks were



TABLE 2 Characteristic features of affect measures administered in this study and byMcDonald and Saunders8 and Williams and Wood12

Zupan and NeumannMcDonald and Saunders/Williams

and Wood

DANVA-FacesDANVA-

Voices Film clipsEET: modified

formatsEET: original

formatEkman 60Faces Test

Number ofitems

24 24 15 14 28 60

Emotionsassessed

H, S, A, F H, S, A, F H, S, A, F, N H, S, A, F, N,surprised,disgusted

H, S, A, F, Nsurprised,disgusted

H, S, A, F,surprised,disgusted

Exemplars peremotion

6 6 3 2 4 10

Type of media Unimodal Unimodal Multimodal Unimodal Multimodal UnimodalNature of

mediaStatic (colored

photographs)Dynamic Dynamic Static (colored

photographs)Dynamic (facial

only, vocalonly)

Dynamic Static (black andwhitephotographs)

Available cues Facial only Vocal only Face Facial only Face Facial onlyEmotionally

neutralsentenceb

VoiceContext

Emotionallyneutraldialogue

(static anddynamicvisualformats)Vocal only

VoiceEmotionally

ambiguousdialogue

Backgroundmusic in 9of 15 clips

Responseoptions

H, S, A, F, Idon’t know

H, S, A, F, Idon’t know

H, S, A, F, N,I don’t know

H, S, A, F, N,surprised,disgusted



Procedure Displayed oncomputerscreen for 5 seach

Sentenceplayed onetime only

Participantstold whichcharacter tofocus on

Details notprovided

Details notprovided

Displayed oncomputerscreen for 5 seach

No feedbackprovided

No feedbackprovided

Played onetime only

No feedbackprovided

No feedbackprovided

Abbreviations: DANVA-Faces, Diagnostic Analysis of Nonverbal Affect 2-Adult Faces; DANVA-Voices, Diagnostic Analysis of NonverbalAffect 2-Adult Paralanguage; EET, Emotion Evaluation Test.

analyzed (happy, sad, angry, fearful). Neutral clips wereincluded when analyses focused solely on film clips.Main effects found in the ANCOVA were explored fur-ther to compare performance across and within tasks.Finally, a 1-way analysis of variance was conducted tocompare group performance on responses to neutral filmclip stimuli.

Relations Between Affect Recognition andInjury-Related Variables (TBI Only)

Spearman correlations indicated no significant rela-tions between age at injury, years postinjury, and in-jury severity (GCS, PTA, LOC) with affect recogni-tion scores on the DANVA-Faces, DANVA-Voices, and

film clips for people with TBI after applying an α

less than 0.003 to account for multiple comparisons(see Table 3).

Relation of Affect Recognition WithTesting Condition

Spearman rho analyses were conducted to investigatewhether there was a relation between testing condition(ie, individual versus group and task order) and taskperformance. Thirty-five participants with TBI and 24control participants were tested individually. Twenty-five participants with TBI and 35 control participantswere tested in small groups (maximum = 3). After ap-plying an α less than 0.008 to account for multiple




TABLE 3 Spearman correlations for injury variables with affect recognition measures(DANVA-Faces, DANVA-Voices, and film clips)

Age atinjury

(n = 60)

Yearssinceinjury

(n = 60)GCS

(n = 17)PTA

(n = 31)LOC

(n = 40)

DANVA-Faces

(n = 59)

DANVA-Voices

(n = 58)

Filmclips

(n = 60)

Age at injury 1.000 − 0.523a 0.487 0.033 − 0.278 − 0.036 − 0.191 − 0.075Years since injury 1.000 − 0.206 − 0.209 0.273 − 0.168 − 0.084 − 0.010GCS 1.000 − 0.647 0.000 − 0.038 0.095 − 0.044PTA 1.000 0.169 0.093 − 0.124 − 0.228LOC 1.000 − 0.172a − 0.039 − 0.250DANVA-Faces 1.000 0.570a 0.227DANVA-Voices 1.000 0.354Film clips 1.000

Abbreviations: DANVA-Faces, Diagnostic Analysis of Nonverbal Affect 2-Adult Faces; DANVA-Voices, Diagnostic Analysis of NonverbalAffect 2-Adult Paralanguage; GCS, Glasgow coma score; LOC, loss of consciousness; PTA, posttraumatic amnesia;aCorrelations that are significant after correcting for (15) multiple comparisons (α = 0.003).

comparisons, testing style (individual vs group) wasnot significantly correlated with performance on theDANVA-Faces, DANVA-Voices, or film clips tasks forpeople with TBI (r = −0.214, P = 0.10; r = −0.123,P = 0.36; r = −0.264, P = 0.04, respectively) or con-trols (r = −0.047, P = 0.72; r = −0.027, P = 0.84; r= −0.176, P = 0.18, respectively). Similarly, Spearmancorrelations indicated no significant relation betweenthe order in which the tasks were administered and per-formance on the DANVA-Faces, DANVA-Voices, andfilm clips for either group (r = −0.022, P = 0.87; r =−0.074, P = 0.58; r = 0.115, P = 0.38 and r = −0.248,P = 0.06; r = −0.008, P = 0.95; r = −0.018, P = 0.89,TBI and controls, respectively).

Objective 1: To Compare Affect Recognition byPeople With and Without TBI in Response toUnimodal and Multimodal Emotion Stimuli

Figure 1 shows the mean performance accuracy foreach task by participants with TBI and controls. An AN-COVA examining group differences across the 3 affectrecognition tasks found significant main effects for thegroup [F(1, 100) = 9.31; P = 0.003; Np

2 = 0.08] and task[F(2, 200) = 11.05; P < 0.001; Np

2 = 0.10], indicatingthat affect recognition differed between people with andwithout TBI (group) and media type (ie, unimodal facialexpressions vs film clips). Using a Bonferroni-adjustedα of 0.017 (0.05/3), follow-up univariate comparisonsacross groups indicated that people with TBI had

Figure 1. Mean responses to positive and negative unimodal and multimodal media, by group.



significantly more errors than controls on DANVA-Faces [F(1, 101) = 6.77; P = 0.005; Np

2 = .06] andDANVA-Voices [F(1, 100) = 9.64; P = 0.002; Np

2

= 0.09]. No significant difference was found betweengroups on the context-enriched film clips [F(1, 118) =2.12; P = 0.15; Np

2 = 0.02]. Within-group comparisonsacross media showed that both groups were better atrecognizing emotions from multimodal than unimodalmedia (see Table 4 for paired sample t test results).

Because people with TBI had significantly more errorsthan controls for unimodal stimuli, we further exploredwhether people with TBI had more difficulty with facesversus voices or they were equally impaired for bothtypes of stimuli. Because group average can be skewedby the heterogeneity of the group, we compared thenumber of people with TBI who were classified as im-paired (defined as 2 SD below the standardized norms)on each task with those who were not impaired. Thenumber of participants with TBI who were impaired atidentifying facial emotion expressions (n = 29) was sig-nificantly greater than the number of participants whowere significantly impaired at identifying vocal affect(n = 8) [X2(1) = 17.81; P < 0.001].

Objective 2: To Compare Affect Recognition forPositive Versus Negative Emotions AcrossUnimodal Media

An ANCOVA was conducted to explore the impactof valence on response accuracy by participants with

TABLE 4 Paired sample t test results bygroup comparing affect recognitionaccuracy for unimodal versus multimodalmedia

Group Comparison Statistic

TBI DANVA-Faces vsDANVA-Voices

t(57) = 0.244;P = 0.81

DANVA-Faces vsfilm clips

t(58) = −8.99;P < 0.001a

DANVA-Voices vsfilm clips

t(57) = −12.06,P < 0.001a

Controls DANVA-Faces vsDANVA-Voices

t(57) = 0.774;P = 0.44

DANVA-Faces vsfilm clips

t(57) = −7.38;P < 0.001a

DANVA-Voices vsfilm clips

t(57) −8.40;P < 0.001a

Abbreviations: DANVA-Faces, Diagnostic Analysis of NonverbalAffect 2-Adult Faces; DANVA-Voices, Diagnostic Analysis of Non-verbal Affect 2-Adult Paralanguage; TBI, traumatic brain injury.aSignificant at α = 0.017, Bonferroni corrected for multiplecomparisons.

and without TBI across the unimodal media. Althoughour original objective was to compare across all media,the context-enriched multimodal media data were notincluded because previous analyses showed no groupdifferences in overall responses for this task. Results re-vealed a significant group × valence interaction [F(1,100) = 4.00; P = 0.04; Np

2 = 0.04], indicating that par-ticipants with TBI responded differently than controlsto positive and negative stimuli. Follow-up ANCOVAs,using a Bonferroni-adjusted α level of 0.012 (.05/4), re-vealed no significant group difference in recognizingpositive affect on the DANVA-Faces [F(1, 101) = 0.322;P = 0.57; Np

2 = 0.003] and the DANVA-Voices [F(1,100) = 0.026; P = 0.87; Np

2 = 0.00] tasks. However, par-ticipants with TBI had significantly more difficulty thancontrols at identifying negative emotion expressions inthe DANVA-Faces task [F(1, 101) = 6.8; P = 0.01; Np

2 =0.06] and the DANVA-Voices task [F(1, 100) = 10.85;P = 0.001; Np

2 = 0.10].Responses to positive versus negative stimuli also dif-

fered by tasks (ie, face vs voice) as indicated by thesignificant valence × task interaction [F(1, 100) = 6.64;P = 0.01; Np

2 = 0.06]. Both groups identified positiveexpressions portrayed in the DANVA-Faces with signifi-cantly greater accuracy than negative ones [TBI: t(58) =11.64, P < 0.001; controls: t(59) = 9.71, P < 0.001]. Nosignificant difference was found for either group for re-sponses to positive versus negative expressions portrayedin the DANVA-Voices task. However, people with TBIdemonstrated a trend toward greater accuracy in iden-tifying positive vocal affect expressions versus negativeones. This trend led to the observed interaction shownin Figure 2.

We further explored whether participants who wereclassified as having significant affect recognition impair-ments only had problems recognizing negative emo-tional expressions, or whether this included positiveaffect as well. Table 5 details the mean accuracy byTBI participants with and without affect recognitionimpairment for positive versus negative stimuli forthe DANVA-Faces and DANVA-Voices. Participantswith TBI and impaired facial affect recognition had agreater number of errors for both positive and nega-tive facial emotion expressions than participants withTBI who did not have impaired facial affect recog-nition [positive: F(1, 57) = 4.43, P = 0.04, Np

2 =0.07; negative: F(1, 57) = 87.19, P < 0.001, Np

2 =0.60]. Similarly, participants with TBI and impaired vo-cal affect recognition had significantly more difficultyidentifying both happy vocal expressions [F(1, 56) =7.00; P = 0.01; Np

2 = 0.11] and negative vocal emo-tion expressions [F(1, 56) = 33.93; P < 0.001, Np

2 =0.38] than those who were unimpaired at vocal affectrecognition.




Figure 2. Mean performance accuracy for emotions categorized by valence across unimodal media for participants with TBI andcontrols.

Objective 3: To Compare Groups in TheirRecognition of Neutral in Context-EnrichedMultimodal Stimuli

The maximum number of errors that could be at-tained for neutral stimuli was 3. On average, people withTBI had 2.05 errors (SD = 1.02) and controls had 1.6errors (SD = .89), a significant difference [F(1, 118) =6.68; P = 0.01].

DISCUSSION

Responses to Unimodal Versus Context-EnrichedMultimodal Stimuli

Participants with TBI were less accurate than controlsat identifying unimodal (static) facial and vocal affect,

but not at recognizing affect from our context-enrichedmultimodal media. In fact, people with TBI identifiedemotion expressions in the multimodal media (whenneutral was excluded) almost as accurately as controls.We were surprised by this latter finding, which con-flicts with other8,12 reports that people with TBI hadmore difficulty identifying emotion in multimodal me-dia than controls. Our finding suggests that the peoplewith TBI in this study may have used the contextual in-formation contained in the film clips to facilitate theiremotion recognition. However, because we did not alsotest multimodal stimuli that were not context enriched,we cannot definitively conclude that it was the addi-tion of context that led to enhanced performance andnot the use of other cues (eg, gestures and body pos-ture). Presenting these film clips with the sound off may

TABLE 5 Means (and standard deviations) of performance accuracy for recognition ofunimodal positive and negative stimuli by participants with TBI identified as havingimpaired affect recognition and those without impaired affect recognition

DANVA-Faces DANVA-Voices

Valence Positive Negative Valence Positive Negative

TBI group Impaired FAR(n = 28)

0.82 (0.22) 0.34 (0.11) Impaired VAR(n = 8)

0.40 (0.27) 0.31 (0.15)

UnimpairedFAR (n = 29)

0.92 (0.10) 0.66 (0.14) Unimpaired VAR(n = 48)

0.65 (0.25) 0.63 (0.14)

Abbreviations: DANVA-Faces, Diagnostic Analysis of Nonverbal Affect 2-Adult Faces; DANVA-Voices, Diagnostic Analysis of NonverbalAffect 2-Adult Paralanguage; FAR, facial affect recognition; TBI, traumatic brain injury; VAR, vocal affect recognition.



have provided further insight into what cues producedthe improvement in emotion recognition in responseto the multimodal versus unimodal media. Althoughwe cannot be certain that the contextual informationimproved our participants’ performance, the fact thatit did not worsen performance suggests it would notbe harmful to include contextual information in affectrecognition interventions for people with TBI to achievegreater ecological validity.

It is important to note that these film clips includedextraneous cues that were not controlled for, cues thatcould have contributed to recognition of emotion byour participants. For instance, 9 of the 15 film clips hadaccompanying affective background music that is oftenused in film to induce specific emotions.28,29 Further,it has been shown to activate many of the same neu-ral structures important to emotion perception.30 Ourset of film clips was not large enough to permit directcomparison of emotion recognition for clips with andwithout background music. Without this comparison,we cannot say for certain that participants were betterat recognizing emotions from the film clip stimuli be-cause of the added contextual information, and not thepresence of affective music. If further study using filmclips that do not have accompanying affective musicindicates that contextual cues facilitate affect recogni-tion for people with TBI, therapeutic interventions canbe designed to initially maximize these cues and thenslowly diminish them while drawing attention to otherrelevant nonverbal cues. Inclusion of background mu-sic in the film clips, a limitation in this study, couldpotentially be a focal point for future investigations. Acomparison could be made between clips with and with-out accompanying music to investigate whether musicpositively influences affect recognition for people withTBI. Similar to context, music may also have importantimplications for treatment approaches.

Responses to our context-enriched multimodal stim-uli seem to suggest that affect recognition in everydaysocial interactions may be less challenging than iso-lated testing indicates. However, people with TBI andtheir loved ones continue to report affect recognitionto be a considerable problem.14,24 It seems, then, thatalthough the film clips may be more ecologically validthan unimodal stimuli, the information available stilldiffers from what actually occurs during interpersonalinteractions. People with TBI may have more difficultywhen they are actually involved in an emotional inter-action in a normal environment with all of its distrac-tions, rather than observing an emotional interactionwithin a controlled environment where they have beeninstructed to focus on the available emotion cues.

In addition to comparing unimodal with multimodalrecognition, this study compared affect recognition bypeople with TBI across facial and vocal modalities to

determine whether one is more problematic than theother. A significantly greater number of people withTBI were impaired for facial (29 participants) versus vo-cal (8 participants) affect recognition. It seems that fa-cial affect recognition poses a greater challenge for peo-ple with TBI, even though recognition of isolated vocalaffect is reported to be more difficult for the generalpopulation.31–33

Affect Recognition for Positive and NegativeEmotions

The second objective of this study was to investigatethe impact of valence in the identification of emotionexpressions across the different forms of media. Becauseno significant difference was found between groups foraffect recognition on the film clips tasks, multimodalmedia was not considered in this analysis. At first glance,it may seem that participants with TBI were able tocompetently recognize positive affect because they werebetter at recognizing happy facial and vocal expressionsthan negative affect. However, once we categorized ourTBI participants as either impaired or unimpaired, wefound that participants with TBI and impaired affectrecognition actually had significantly more trouble thanthe unimpaired TBI group at recognizing both positiveand negative affect for faces and voices.

These results contrast with the literature on valencethat suggests recognition of positive expressions is notimpaired for people with TBI.2,6,11,12,23 Although thisstudy included only 1 positive exemplar (happy), thisfinding has important implications for the developmentof treatment programs, as it suggests that positively va-lenced emotions should be given as much credence inrehabilitation programs as negatively valenced ones. Fu-ture studies should include a wider range of positiveand negative emotions to more thoroughly investigatethe influence of valence on affect recognition for bothunimodal and multimodal media.

Affect Recognition to Neutral Stimuli

Our final objective was to compare responses by peo-ple with and without TBI to context-enriched neutralmultimodal stimuli. Williams and Wood12 found neu-tral to be particularly difficult for people with TBI toidentify, even when given combined facial and vocalaffect cues. Our results showed that although the in-tersensory redundancy offered by the context-enrichedcues in the multimodal film clips led to high recog-nition accuracy for happy, sad, angry, and fearful, bothgroups of participants had difficulty identifying neu-tral. It seems that the recognition of neutral is moreelusive than the recognition of positive and negativeemotions, even in multimodal representations. This maybe because neutral stimuli are more likely to convey




affective information when presented alongside contex-tual information.34 Furthermore, participants with TBIwere found to have significantly more difficulty withthe neutral film clips than controls. This has importantimplications for affect recognition during social interac-tions, because erroneously perceiving an emotion whenthere is not one, and vice versa, may lead to communi-cation breakdowns.34 As a result, these findings implythat treatment programs should also include trainingrecognition to neutral stimuli.

Limitations and Future Directions

The use of cinematic film clips from commercialfilms and television to assess affect recognition in peo-ple with TBI was a novel approach that revealed in-teresting results. However, the lack of comparison tosimilar unimodal cues (eg, dynamic facial expressions)and various combinations of these cues (eg, contextu-ally enriched film clips played with no sound) limitsour ability to extend the findings to everyday situa-tions. We compared performance on our multimodalmedia with performance on unimodal subtests froman existing standardized tool, the Diagnostic Analysisof Nonverbal Accuracy-2 (DANVA-2).25 Although thesesubtests have been shown to have good reliability andvalidity25 and have been previously used with peoplewith TBI,14,24 they are not without limitations. Onelimitation of comparing performance on the DANVA-Faces and DANVA-Voices tasks with performance onthe film clips was the lack of neutral exemplars in theDANVA-2. Because we have no direct comparison tohow people with TBI responded to unimodal neutral ex-pressions, we cannot discern whether neutral is simplyan elusive state to identify or the addition of contextualcues led people with TBI to infer emotions that were notpresent. Given the implications this may have for socialinteraction, future studies should explore the perceptionof neutral in various forms of media.

The use of the DANVA-2 further limits interpreta-tion of our results because we were unable to com-pare performance for unimodal visual stimuli that wasequivalent, or at least similar, to the multimodal filmclips (ie, dynamic facial expression). Not only have dy-namic facial stimuli been associated with enhanced neu-ral activity,35–39 they contain important temporal cueswithin the facial movements themselves that facilitateoverall performance.8,40 Because we did not test visual-only affect recognition of dynamic facial expressions,we do not know whether the dynamic nature of thefacial stimuli contributed to the increased recognitionof emotion for the film clips. Simply playing the filmclips without sound would not have fully answered thisquestion because their multimodal nature inherently in-cludes visual contextual cues (eg, social environment) in

addition to the nonverbal facial expressions. The con-tribution of each of these cues to multimodal affectrecognition is an important query that awaits furtherinvestigation with larger sample sizes and carefully se-lected stimuli that limit the extraneous visual cues indynamic visual-only displays.

Factors associated with the film clips themselves (eg,the previously discussed potential influence of affectivemusic in some film clips) may also limit the conclusionswe can make. Furthermore, familiarity with the film it-self or with the actors may have led to improved perfor-mance. Familiarity has been shown to contribute to thediscrimination of facial expressions.41,42 Although pre-vious work with these same film clips did not show a re-lation between prior viewing and participant responses,(B. Zupan and D. R. Babbage, unpublished data, 2010)we cannot be certain it was not a factor for our partici-pants with TBI.

The use of only 1 positive exemplar among a limitedset of emotion categories constrains our interpretationof valence effects because recognition of happy may havebeen influenced by the use of exclusion rules in responseselection. Including additional positive emotions (eg,amusement and pleasant surprise) in future studies mightprovide a more accurate representation of the ability ofpeople with TBI to recognize emotions that differ byvalence. The use of a forced-choice format may alsohave led to increased recognition rates. However, weattempted to overcome the tendency for participantsto guess or select an emotion they did not perceiveby including an alternate response category (ie, I don’tknow), a practice shown to reduce artificial agreement.43

We did not collect data on depression or anxiety, bothof which may have influenced affect recognition perfor-mance. Depression in people with TBI has been asso-ciated with changes in personality and mood;44 thus,the lack of these measures in this study is a limitation.Similarly, the generalization of results from this study toreal-world functioning is restricted because we did notcollect observer data from people who interact regularlywith our participants with TBI. However, we are unawareof any standardized questionnaires for observers that fo-cus on the ability of people with TBI to use affective cueswithin social contexts to interpret the feelings of others.

CONCLUSIONS

This study supports earlier reports of deficits in peoplewith TBI for facial and vocal affect recognition. Of notewas the finding that people with TBI identified as hav-ing impaired facial or vocal affect recognition were im-paired for both positive (happy) and negative emotions.Recognition of neutral in context-enriched stimuli wasalso found to be problematic, a finding that has impor-tant implications for real-world interactions. Finally, our



findings with context-enriched multimodal media stim-uli suggest that people with TBI are better at recognizingemotions when they are given an array of affective cuesthat include contextual information. These results addto our knowledge in this area, but leave us with furtherquestions about how people with TBI combine multipleaffective cues in their interpretation of affect, and whatcues most influence their interpretation of a complexsocial event. Some questions that have emerged includethe following: (a) Do people with TBI use all availablecues equally or do they compensate for deficits in 1

modality (eg, facial affect recognition) by relying moreheavily on other available cues (eg, context)? (b) Is therea minimum number of affective cues necessary for suc-cess with affect recognition (eg, face and voice only)?(c) Does successful affect recognition for people withTBI depend on a specific combination of affective cues(eg, face, posture, and context)? (d) How are other emo-tions affected? (e) Does the presence of affective musicimpact recognition? Answering these questions will as-sist us in developing more suitable and rigorous affectrecognition interventions for people with TBI.

REFERENCES

1. Radice-Neumann D, Zupan B, Babbage DR, Willer B. Overview ofimpaired facial affect recognition in persons with traumatic braininjury. Brain Inj. 2007;21(8):807–816.

2. Croker V, McDonald S. Recognition of emotion from fa-cial expression following traumatic brain injury. Brain Inj.2005;19(10):787–799.

3. Dimoska A, McDonald S, Pell MC, Tate RL, James CM. Recog-nizing vocal expressions of emotion in patients with social skillsdeficits following traumatic brain injury. J Int Neuropsychol Soc.2010;16:369–382.

4. Green REA, Turner GR, Thompson WF. Deficits in facial emotionperception in adults with recent traumatic brain injury. Neuropsy-chologia. 2004;42:133–141.

5. Ietswaart M, Milders M, Crawford JR, Currie D, Scott CL. Longi-tudinal aspects of emotion recognition in patients with traumaticbrain injury. Neuropsychologia. 2008;46:148–159.

6. Jackson HF, Moffat NJ. Impaired emotional recognition followingsevere head injury. Cortex. 1987;23(2):293–300.

7. McDonald S, Flanagan S. Social perception deficits after trau-matic brain injury: interaction between emotion recognition,mentalizing ability, and social communication. Neuropsychology.2004;18(3):572–579.

8. McDonald S, Saunders JC. Differential impairment in recognitionof emotion across different media in people with severe traumaticbrain injury. J Int Neuropsychol Soc. 2005;11:392–399.

9. Milders M, Fuchs S, Crawford JR. Neuropsychological im-pairments and changes in emotional and social behavior fol-lowing severe traumatic brain injury. J Clin Exp Neuropscyhol.2003;25(2):157–172.

10. Spell LA, Frank E. Recognition of nonverbal communicationof affect following traumatic brain injury. J Nonverbal Behav.2000;24(4):285–300.

11. Hopkins MJ, Dywan J, Segalowitz SJ. Altered electrodermal re-sponse to facial expression after closed head injury. Brain Inj.2002;16(3):245–257.

12. Williams C, Wood RLI. Impairment in the recognition of emotionacross different media following traumatic brain injury. J Clin ExpNeuropscyhol. 2010;32(2):113–122.

13. Marquardt TP, Rios-Brown M, Richburg T. Comprehension andexpression of affective sentences in traumatic brain injury. Apha-siology. 2001;15(10/11):1091–1101.

14. Radice-Neumann D, Zupan B, Tomita MR, Willer B. Trainingemotion processing in persons with brain injury. J Head TraumaRehabil. 2009;24(5):313–323.

15. Zupan B, Neumann D, Babbage DR, Willer B. The importanceof vocal affect to bimodal processing of emotion: implications forindividuals with traumatic brain injury. J Commun Disord. 2009;42:1–17.

16. Babbage DR, Yim J, Zupan B, Neumann D, Tomita MR, Willer B.Meta-analysis of facial affect recognition difficulties after traumaticbrain injury. Neuropsychology. 2011;25(3):277–285.

17. Hornak J, Rolls ET, Wade D. Face and voice expression identi-fication in patients with emotional and behavioural changes fol-lowing ventral frontal lobe damage. Neuropsychologia. 1996;34(4):247–261.

18. Pell MD. Cerebral mechanisims for understanding emotionalprosody in speech. Brain Lang. 2006;96:221–234.

19. Wierzbicka A. Defining emotional concepts. Cogn Sci.1992;16:539–581.

20. Milders M, Ietswaart M, Crawford JR, Currie D. Impairments intheory of mind shortly after traumatic brain injury and at 1-yearfollow-up. Neuropsychology. 2006;20(4):400–408.

21. McDonald S, Flanagan S, Rollins J. The Awareness of Social InferenceTest. San Antonio, TX: Harcourt Assessment; 2002.

22. Ekman P, Friesen WV. Pictures of Facial Affect. Palo Alto, CA:Consulting Psychological Press; 1976.

23. Kucharska-Pietura K, Phillips ML, Gernand W, David AS. Percep-tion of emotions from faces and voices following unilateral braindamage. Neuropsychologia. 2003;41(8):1082–1090.

24. Neumann D, Zupan B, Hammond F, Malec J. Relationships be-tween affect recognition, empathy, and alexythymia after TBI.J Head Trauma Rehabil. 2013 [Epub ahead of print].

25. Nowicki S. The Manual for the Receptive Tests of the Diagnostic Analysisof Nonverbal Accuracy 2 (DANVA 2). Atlanta, GA: Department ofPsychology, Emory University; 2008.

26. Rottenberg J, Ray RD, Gross JJ. Emotion elicitation using films.In:Coan JA, Allen JJB, eds. Handbook of Emotion Elicitation andAssessment. New York: Oxford University Press; 2007.

27. Gross JJ, Levenson RW. Emotion elicitation using films. CognEmot. 1995;9(1):87–108.

28. Busselle R, Bilandzic H. Fictionality and perceived realism in ex-periencing stories: a model of narrative comprehension and en-gagement. Commun Theory. 2008;18:255–280.

29. Zagalo N, Barker A, Branco V. Story Reaction Structures to EmotionDetection. Paper presented at: Proceedings of ACM Workshop onStory Representation, Mechanisms and Context, New York; 2004.

30. Koelsch S, Fritz T, V Cramon DY, Muller K, Friederici AD. Inves-tigating emotion with music: an fMRI study. Hum Brain Mapping.2006;27:239–250.

31. Russell JA. Is there universal recognition of emotion from fa-cial expression? a review of the cross-cultural studies. Psychol Bull.1994;115(1):102–141.

32. Scherer KR. Vocal communication of emotion: a review of re-search paradigms. Speech Commun. 2003;40(1–2):227–256.

33. Walbott HG, Scherer KR. Cues and channels in emotion recogni-tion. J Pers Soc Psychol. 1986;51(4):690–699.




34. Carrera-Levillain P, Fernandez-Dols J-M. Neutral faces in context:their emotional meaning and their function. J Nonverbal Behav.1994;18(4):281–299.

35. Biele C, Grabowska A. Sex differences in perception of emotionintensity in dynamic and static facial expressions. Exp Brain Res.2006;171(1):1–6.

36. Collignon O, Girard S, Gosselin F, et al. Audio-visual intergrationof emotion expression. Brain Res. 2008;1242:126–135.

37. LaBar KS, Crupain MJ, Voyvodic JT, McCarthy G. Dynamic per-ception of facial affect and identity in the human brain. CerebralCortex. 2003;13(10):1023–1047.

38. Mayes AK, Pipingas A, Silberstein RB, Johnston P. Steady state vi-sually evoked potential correlates of static and dynamic emotionalface processing. Brain Topogr. 2009;22(3):145–157.

39. Schulz J, Pilz KS. Natural facial motion enhances cortical re-sponses to faces. Exp Brain Res. 2009;194(3):465–475.

40. Cunningham DW, Wallraven C. Dynamic information for therecognition of conversational expressions. J Vision. 2009;9(13):1–17.

41. Dobel C, Geiger L, Bruchmann M, Putsche C, SchweinbergerSR, Junghofer M. On the interplay between familiarity andemotional expression in face perception. Psychol Res. 2008;72:580–586.

42. Wild-Wall N, Dimigen O, Sommer W. Interaction of facial ex-pressions and familiarity: ERP evidence. Biol Psychol. 2008;77:138–149.

43. Frank MG, Stennett J. The forced-choice paradigm and theperception of facial expressions of emotion. J Pers Soc Psychol.2001;80(1):75–85.

44. Ownsworth TL, Oei TPS. Depression after traumatic brain in-jury: conceptualization and treatment considerations. Brain Inj.1998;12(9):735–751.

APPENDIX Description of Film Clips

Emotion Movie Length (ms) Scene

Happy Sweet Home Alabama, D & D Films,2002, director: A. Tennant

76 A man surprises his girlfriend with amarriage proposal at a jewelry store

Happy Wedding Crashers, New Line Cinema,2005, director: D. Dobkin

62 A young couple is sitting on the beachplaying flirtatious games

Happy Remember the Titans, JerryBruckheimer Films, 2000, director:B. Yakin

65 A black family is accepted into a whitecommunity after the high schoolfootball team wins the championshipgame

Sad Grey’s Anatomy, ABC, 2006, director:P. Horton

98 A father is going into surgery and isonly able to communicate with hisfamily by blinking his eyes

Sad Armageddon, Touchstone Pictures,1998, director: M. Bay

75 A daughter must say goodbye to herfather while he is on a dangerousmission

Sad Grey’s Anatomy, ABC, 2006, director:M. Tinker

87 A woman is heart broken after herfinance has passed away

Angry Anne of Green Gables, CanadianBroadcast Corporation, 1985,director: K. Sullivan, Canada: Anne ofGreen Gables Productions

62 An older woman speaks openly abouta child’s physical flaws in front of her

Angry Enough, Columbia Pictures, 2002,director: M. Apted, United States:Columbia Pictures

103 A wife confronts her husband aboutinfidelity and he responds byexerting physical and emotionaldominance

Angry Pretty Woman, Touchstone Pictures,1990, director: G. Marshall

65 A call girl attempts to buy clothingfrom an expensive store and thestore clerks turn her away

Fearful Blood Diamond, Warner Bros Pictures,2006, director: E. Zwick

50 A father and son are walking togetherwhen numerous vehicles carryingmilitia approach

Fearful The Life Before Her Eyes, 2929Productions, 2007, director: V.Perelman

79 Two teenage girls are talking in a highschool bathroom when they heargunshots that continue to get closer

Fearful The Sixth Sense, Barry MendelProductions, 1999

70 A young boy wakes from his sleep andenters the kitchen to find his mothermaking dinner

Neutral The Other Sister, Mandeville Films,1999

45 A mother and daughter are sitting anddiscussing an art book

Neutral The Game, Polygram FilmedEntertainment, 1999

85 One gentleman is explaining the rulesof a game to another gentleman

Neutral The Notebook, New Line Cinema,2004, director: N. Cassavetes

56 An older gentlemen and his doctor areconversing during a routine check-up


Affect Recognition in Traumatic Brain Injury

Documents

Transcript of Affect Recognition in Traumatic Brain Injury