Available online at www.sciencedirect.com

www.elsevier.com/locate/brainres

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6

http://dx.doi.org/10.0006-8993/& 2014 El

nCorresponding autE-mail addresses:

Research Report

Semantic relation vs. surprise: The differential effectsof related and unrelated co-verbal gestures on neuralencoding and subsequent recognition

Benjamin Strauben, Lea Meyer, Antonia Green, Tilo Kircher

Department of Psychiatry und Psychotherapy, Philipps-University Marburg, Marburg, Germany

a r t i c l e i n f o

Article history:

Accepted 9 April 2014

Speech-associated gesturing leads to memory advantages for spoken sentences. However,

unexpected or surprising events are also likely to be remembered. With this study we test

Available online 16 April 2014

Keywords:

Iconic gestures

Multimodal processing

Congruency

Speech associated gestures

Memory

Attention

1016/j.brainres.2014.04.01sevier B.V. All rights rese

hor. Fax: þ49 3212 758660straubeb@med.uni-marb

a b s t r a c t

the hypothesis that different neural mechanisms (semantic elaboration and surprise) lead

to memory advantages for iconic and unrelated gestures. During fMRI-data acquisition

participants were presented with video clips of an actor verbalising concrete sentences

accompanied by iconic gestures (IG; e.g., circular gesture; sentence: “The man is sitting at the

round table”), unrelated free gestures (FG; e.g., unrelated up down movements; same sentence)

and no gestures (NG; same sentence). After scanning, recognition performance for the

three conditions was tested. Videos were evaluated regarding semantic relation and

surprise by a different group of participants. The semantic relationship between speech

and gesture was rated higher for IG (IG4FG), whereas surprise was rated higher for FG

(FG4IG). Activation of the hippocampus correlated with subsequent memory performance

of both gesture conditions (IGþFG4NG). For the IG condition we found activation in the left

temporal pole and middle cingulate cortex (MCC; IG4FG). In contrast, for the FG condition

posterior thalamic structures (FG4IG) as well as anterior and posterior cingulate cortices

were activated (FG4NG). Our behavioral and fMRI-data suggest different mechanisms for

processing related and unrelated co-verbal gestures, both of them leading to enhanced

memory performance. Whereas activation in MCC and left temporal pole for iconic co-

verbal gestures may reflect semantic memory processes, memory enhancement for

unrelated gestures relies on the surprise response, mediated by anterior/posterior cingu-

late cortex and thalamico-hippocampal structures.

& 2014 Elsevier B.V. All rights reserved.

2rved.

5.urg.de, benjamin.straube@web.de (B. Straube).

1. Introduction

Human speech is typically accompanied by gestures. Thesegestures confer advantages in comprehension, learning andmemory (e.g., Cook et al., 2010, 2013; Kelly and Church, 1998;

Kelly et al., 1999; Straube et al., 2009, 2011a; Valenzeno et al.,2003). For example Kelly et al. (1999) demonstrated that certainnonverbal behaviors, such as deictic and iconic gestures, helppeople comprehend and remember pragmatic communication.Consistent with this finding Valenzeno et al. (2003) showed that

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 43

children learn more when they watch a video containing theteacher's gestures as compared to videos without gestures.However, whether the semantic content of gestures and itsrelation to language are important for gesture related advan-tages in comprehension and memory are unknown. Only forgestures in an abstract sentence context (metaphoric gestures)it has been shown that the semantic relation between speechand gesture information is important for neural encoding andsubsequent recognition (Straube et al., 2009). However, memoryadvantages for semantically incongruous sentences (e.g., Mosset al., 2008; Worthen et al., 2000; Worthen and Roark, 2002) andfor incongruous picture information (e.g., Michelon et al., 2003;Nicolas and Worthen, 2009; Worthen and Eller, 2002), has alsobeen reported. Thus, different mechanisms might contribute togesture related advantages for subsequent memory. To get abetter understanding of these mechanisms, we investigated thedifferent neural processes underlying the encoding of iconicand unrelated speech–gesture information in relation to sub-sequent memory performance.

Iconic gestures are an important accompaniment tospeech, sharing a formal relationship with the co-occurringconcrete speech content (McNeill, 1992). Iconic gesturesillustrate forms, shapes, events or actions of the speechcontent. Therefore iconic gestures directly refer to concretefeatures of the world, as in the sentence “The ball is round”(round hand gesture), “The cottage is located on a very highmountain” (lift of the right hand) or “The fisherman caught ahuge fish” (where the gesture indicates the size of the fish).An important characteristic of iconic gestures is the referenceto concrete sentence contents (By contrast, metaphoric ges-tures, which are also illustrative co-verbal gestures, refer toabstract speech contents, e.g., Kircher et al., 2009; Straubeet al., 2011a).

The neural basis of co-verbal gesture processing is beinginvestigated by an increasing number of functional magneticresonance imaging (fMRI) studies (Dick et al., 2009; Greenet al., 2009; Holle et al., 2008, 2010; Kircher et al., 2009; Nagelset al., 2013; Skipper et al., 2007, 2009; Straube et al., 2011a,2012, 2013c; Willems et al., 2007, 2009). For the processing oficonic co-verbal gestures the left posterior temporal cortexseems to be especially relevant (Green et al., 2009; Holle et al.,2008, 2010; Straube et al., 2011a, 2013b), whereas left inferiorfrontal and parietal brain activations were reported for mis-matches between unrelated concrete speech and iconicgesture information (Green et al., 2009; Willems et al., 2007).For the processing of metaphoric co-verbal gestures bothinferior frontal as well as posterior temporal activations seemto be relevant (Kircher et al., 2009; Straube et al., 2009, 2011a,2013a, 2013b). Whereas these studies always compared dif-ferent gesture conditions, studies that investigate the neuralcorrelates of memory processes for speech and gestureutterances are rare. For iconic and emblematic gestures andcorresponding sentences it was found that body orientationof the actor affects memory processes in hippocampal andmedial frontal brain regions (Straube et al., 2011b). Formetaphoric co-verbal gestures left lateral posterior temporaland inferior frontal as well as bilateral hippocampal brainregions are involved in successful encoding processes forspeech and gesture (Straube et al., 2009). Other than thesefindings the effect of related vs. unrelated speech–gesture

information on neural memory encoding processes for iconicgestures is unknown. Straube et al. (2009) showed activationof the left occipito-temporo-cerebellar region, the right infer-ior frontal cortex and the bilateral medial temporal lobe forthe successful encoding of unrelated gesture information inthe context of abstract speech. However, with reference tothe differences of iconic and metaphoric co-verbal gesturesreported above (Straube et al., 2011a), manipulation of con-gruency in iconic gestures most likely led to different neuralencoding processes in relation to subsequent recognition.

In general we would expect gesture related advantages incomprehension and memory encoding only in semantic relatedspeech and gesture pairs (as basically supported for metapho-rical gestures, Straube et al., 2009). However, in our recent fMRI-study about iconic gesture integration we found even a slightlyhigher hit-rate for unrelated in contrast to related iconicgestures (Green et al., 2009). Such an in-congruency effect onmemory has repeatedly been found, for example, in semanti-cally incongruous sentences (e.g., Moss et al., 2008; Worthenet al., 2000; Worthen and Roark, 2002) and for incongruouspicture information (e.g., Michelon et al., 2003; Nicolas andWorthen, 2009; Worthen and Eller, 2002), which were betterrecalled subsequently than ordinary ones. This paradoxicaleffect, the so-called “bizarreness effect” (Hirshman et al., 1989;McDaniel et al., 1995; Moss et al., 2008; Nicolas and Marchal,1998; Nicolas and Worthen, 2009; Riefer and Rouder, 1992;Worthen and Wood, 2001; Worthen and Roark, 2002; Worthenand Deschamps, 2008) may result from a surprise response dueto violated expectations, which enhances the contextual cuesthat help in subsequent recognition (surprise hypothesis,Hirshman et al., 1989; Michelon et al., 2003). The surprise maybe part of an orienting response that includes arousal, capturedattention and memory formation (see Shulman et al., 2009;Sokolov, 1963). Thus, the surprise hypothesis predicts an emo-tional response for incongruous items (Michelon et al., 2003;Strange et al., 2000), which should not occur in congruousitems. For utterances with congruent speech and gestureinformation we expect that deeper elaboration of the presentedmaterial is related to enhanced memory performance(Staresina et al., 2009; Straube et al., 2009). As alternative tothe semantic elaboration theory recent studies have relatedschema activation to enhancing effects of congruently learnedassociations (e.g., McKenzie and Eichenbaum, 2011; McKenzieet al., 2013; van Kesteren et al., 2012, 2013; Wang et al., 2012). Inaccordance with this theory an activated schema leads to fasterincorporation of congruent information into neocortical net-works, including the medial prefrontal cortex. Oppositely, theprocessing of unexpected or surprising events are suggested torather relate to processing in the medial temporal lobe, parti-cularly the hippocampus (van Kesteren et al., 2012, 2013).Nevertheless, we assumed that the memory effect of unrelatedgestures relays on different mechanisms than the encoding oficonic gestures in context of related concrete speech, which istherefore the focus of the current study.

To get a better understanding of these mechanisms, weinvestigated the different neural processes underlying theencoding of iconic and unrelated speech–gesture informationin relation to subsequent memory performance. We performeda rating of the speech–gesture video-clips with regard to thesemantic relationship as well as surprise (rating study) to

Fig. 1 – Experimental conditions. (A) Illustrates example videos for each condition presented during encoding: related iconicgesture (IG), unrelated free gesture (FG) and no gesture condition (NG). During this phase fMRI data were acquired, whileparticipants performed an implicit encoding task (judging the color on the sweater). (B) Illustrates the eight video conditions inthe recognition phase. Different new distractor stimuli (New1 and New2) were presented in randomized order with theoriginal stimuli (Old) during recognition. Half of the new videos (n¼16) consisted of completely new items (New1), and theother half of the items (n¼16) were videos including the same gesture but accompanying new sentences (New2, bottom).Therefore in New1 gestures may support subsequent recognition whereas in New2 gestures should interfere and thereforenot lead to a better discrimination of old and new items (gesture interference; here memory of gestures might leaderroneously to an old response). The recognition phase took place after fMRI data acquisition. The spoken German sentences,translated into English, are written beneath each picture. The gesture movements were presented with arrows.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 644

Fig. 2 – Results of the rating experiment. The figure showsthe participants ratings of the related (IG) and unrelatedgesture conditions (FG) with regard to the semanticrelatedness (dark bar) and surprise (light bar). We found a

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 45

support our hypothesis that semantic elaboration (or schemaactivation) and surprise at encoding might mediate subsequentrecognition. During fMRI-data acquisition a different set ofparticipants (fMRI study) were presented with video clips of anactor performing concrete sentences accompanied by iconicgestures (IG), unrelated free gestures (FG) or no gestures (NG,Green et al., 2009). After scanning, recognition performance wasacquired. In general we assumed cortical and hippocampalstructures would be involved in memory encoding for speechand gesture information and their activation would thereforepredict subsequent memory performance (Straube et al., 2009,2011b; for a review see Straube, 2012). However, we expectdifferent encoding mechanisms for related iconic and unrelatedfree co-verbal gesture conditions. Whereas the encoding of iconicgestures is probably related to frontal, temporal and hippocam-pal structures related to semantic elaboration, memory encodingand schema activation (Staresina et al., 2009; van Kesterenet al., 2012, 2013), we expect for the FG condition activity inareas underlying attention (e.g., fronto-parietal brain regions;Blankenburg et al., 2010; Chica et al., 2011; Greenberg et al., 2010;Heinen et al., 2011; Li et al., 2010; Nardo et al., 2011) andemotional responses (e.g., thalamus, the amygdala, and/or thefrontal operculum; Michelon et al., 2003; Strange et al., 2000).

significant interaction with regard to the rating question andgesture condition (see Section 2).

2. Results

Results are reported for the experimental conditions illu-strated in Fig. 1. Videos of the related iconic gesture condition(IG), unrelated free gesture condition (FG) and the isolatedspeech condition (NG) were presented during encoding (fMRIdata acquisition; Fig. 1A).

All the videos of the gesture conditions and half of the videosof the NG condition were also presented in the recognition phase(Old; Fig. 1B, top). The new items consisted of two types ofdistractors: (a) 16 completely new items including new speechand new gesture information (New1; Fig. 1B, middle) and (b) 16items including new sentences but old gesture information(New2; Fig. 1B, bottom). In the recognition phase participantshad to indicate via key presses whether they had seen that clip(including both speech and gesture information) before or not.During encoding the participants only judged the color of thespot of the actors' sweater (independent control task).

2.1. Rating results

All videos presented in the encoding phase of the fMRI experi-ment had been evaluated by a separate group of participantsregarding semantic relatedness and surprise on a Likert-likescale between 1 (low) and 7 (high). For the results of the ratingexperiment we obtained in an analyses of variance significantmain effects (semantic-relation4surprise; F(1,19)¼52.183, po.001,partial-eta-squared¼ .733; IG4FG; F(1,19)¼15.641, po.001, partial-eta-squared¼ .451) and an interaction of rating question andgesture-congruency, as videos of the IG condition were ratedmore semantic related (IG: mean¼5.97, SD¼ .53; FG: mean¼2.83;SD¼ .60; t19¼16.677, po.001) and less surprising (IG: mean¼2.01,SD¼ .50; FG: mean¼4.46; SD¼ .89; direct comparison: t19¼10.681,po.001) than clips of the FG condition (Interaction: F(1,19)¼155.883, po.001, partial-eta-squared¼ .919; see Fig. 2). Further

post-hoc comparisons indicate, that videos in the IG conditionswere rated more semantically related than surprising (IG:semantic related4surprise; t19¼18.804, po.001) and videos inthe FG conditions were rated more surprising than semanticallyrelated (FG: surprise4semantic related; t19¼5.726, po.001).

2.2. Behavioral results (fMRI)

2.2.1. Implicit encoding taskThe average reaction time for the control task (“indicate the colorof the spot on the actor's sweater”) did not differ between condi-tions (F(2,12)¼ .519, p¼ .608, partial-eta-squared¼ .080; within-subjects ANOVA; IG mean¼ .964 s, SD¼ .20; FG mean¼ .948 s,SD¼ .20; mean¼ .950 s, SD¼ .20). The participants showed anaverage accuracy rate of 99% which did not differ across condi-tions (F(2,12)¼ .160, p¼ .854, partial-eta-squared¼ .026; within-subjects ANOVA). Thus, the attention control task indicates thatparticipants did pay attention to the video clips in the fMRI.

2.2.2. Memory performanceThe proportion of study items correctly identified as old (hits)in the gesture support conditions (New1; see Fig. 1) was 56.70%(SD¼19.68%) for IG and 47.77% (SD¼16.92%) for FG. Thecorresponding false alarm rate (FA) was 18.30% (SD¼14.29%)for the IG and 14.73% (SD¼14.00%) for the FG condition. For theNG condition we obtained 33.48% (SD¼15.23%) hits and 18.75%(13.20%) FA. For the comparison of the hit rates a withinsubject ANOVA showed a significant effect of gesture type onmemory (F(2,26)¼9.419, po.001, partial-eta-squared¼ .420). Apriori comparisons between gesture types and the NG condi-tion indicate significantly enhanced hit-rates for IG (F(1,13)¼13.731, po.005, partial-eta-squared¼ .514) and FG (F(1,13)¼7.157,po.05, partial-eta-squared¼ .355). Post-hoc comparisons between

Fig. 3 – Memory performance for both recognition and the no-gesture control condition. The figure shows the participantsdiscrimination performance (d0, left) and response behavior (c, right) for each gesture (IG, FG, NG ) and recognition condition(gesture support, New1: light gray; gesture interference, New2: dark gray). The performance of the no gesture controlcondition (NG) is presented in white, because no gesture information was present during encoding and recognition.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 646

the gesture conditions failed to reach significance (p¼ .065).In contrast to the hit-rate, we found no differences in theFA-rate between conditions (F(2,12)¼1.000, p¼ .397, partial-eta-squared¼ .143).

To control for the effect of differences in response behaviorbetween the participants the discrimination index d-prime (d0)and the corresponding response criteria (c) were furtherdefined (see Fig. 3; light gray and white bars). We obtained anaverage discrimination performance of d0 ¼1.507 (SD¼1.39) forthe IG-condition and d0 ¼1.346 (SD¼1.225) for the FG-condition.The corresponding response criteria (c) were .576 (SD¼ .518) forthe IG and .723 (SD¼ .423) for the FG-condition. For the NGcondition we obtained a d0 of .688 (SD¼ .829) and a c of .804(SD¼ .546). A comparison of the d0 rates between the conditionsshowed a significant differences (F(2,26)¼4.031, po.05, partial-eta-squared¼ .237). A priori comparisons between gesturetypes and the NG condition indicate significant enhancedd0-rates for IG (F(1,13)¼6.626, po.05, partial-eta-squared¼ .338)and a marginal effect for FG (F(1,13)¼4.285, p¼ .056, partial-eta-squared¼ .248). Post-hoc comparisons between the gestureconditions indicate no significant difference (p¼ .575; seeFig. 3, left: light gray and white bars). In contrast to the d0-rate,we found no differences in the response criterion (c) betweenconditions (F(2,26)¼ .939, p¼ .404, partial-eta-squared¼ .067; seeFig. 3, right: light gray and white bars).

In sum, these results indicate better memory for sentencesaccompanied by gestures in contrast to the no gesture condi-tion. This effect is most pronounced in the IG condition, and didnot differ significantly between the gesture conditions.

2.2.3. Recognition performance in the gesture interferenceconditionsTo investigate the effect of gestures on recognition we useda second gesture-interference control recognition conditionwhere the same gestures were presented in context of

different sentences (New2; Gesture interference; see Fig. 1).In this condition the participants cannot make their decisionbased on the recognition of gesture information becausegesture information is present in both old (hits) and newvideos (FA). This analyses include for both conditions (IG, FG)the other half of the stimuli of the encoding phase (n¼16: old)and corresponding distractor stimuli including the samegesture information than the old items (see Fig. 1B, New2).To investigate the gesture effect for this recognition condition(New2) these data were also compared to the no gesturecondition (NG, see Section 5.4)

The proportion of study items correctly identified as ‘Old'in the New2 condition (Gesture interference condition; seeFig. 1) was 56.70% (SD¼15.79%) for IG and 52.68% (SD¼10.32%)for FG. The corresponding false alarm rate (FA) was 25.89%(SD¼12.47%) for the IG and 20.54% (SD¼13.30%) for the FGcondition. For the comparison of the hit rates between IG, FGand NG a within subject ANOVA showed an significant effectof gesture condition (F(2,26)¼12.401, po.001, partial-eta-squared¼ .488). A priori comparisons between gesture typesand the NG condition indicate significantly enhanced hit-rates for IG (F(1,13)¼18.347, po.001, partial-eta-squared¼ .585)and FG (F(1,13)¼12.358, po.005, partial-eta-squared¼ .487). ForFA-rates we also found a significant effect (F(2,26)¼3.852,po.05, partial-eta-squared¼ .229) which resulted from a sig-nificantly higher FA rate in the IG condition in contrast to theNG condition (F(1,13)¼7.051, po.05, partial-eta-squared¼ .352).

To control for the effect of differences in response beha-vior between the participants the discrimination indexd-prime (d0) and response criterion (c) were further defined(see Fig. 3; dark gray bars). We obtained an average discrimi-nation performance of d0 ¼ .892 (SD¼ .728) for the IG andd0 ¼ .972 (SD¼ .628) for the FG condition. The correspondingresponse criteria (c) were .258 (SD¼ .246) for the IG and .415(SD¼ .226) for the FG condition. A comparison of the d0 rates

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 47

between the conditions (IG, FG and NG) showed no significantdifferences (F(2,26)¼1.557, p¼ .230, partial-eta-squared¼ .107). Incontrast to the d0-rate, we found significant differences in theresponse criterion (c) between conditions (F(2,26)¼ .939, p¼ .404,partial-eta-squared¼ .067). A priori comparisons between ges-ture types and the NG condition indicate significantly reducedc-rates for IG (F(1,13)¼18.098, po.001, partial-eta-squared¼ .582)and FG (F(1,13)¼7.537, po.017, partial-eta-squared¼ .367). Post-hoc comparisons between the gesture conditions indicated nosignificant differences (p¼ .076).

Together, for the control recognition condition were justsentences but not the gestures were new (New2) the differ-ences between d0 observed in the New1 condition no longerexisted. Thus, the use of gesture information in the recogni-tion task led to better memory performances in contrast to nogesture condition. In contrast the response criteria for IG andFG were reduced (as a sign of more liberal response behavior)in contrast to the NG condition. These results indicate thatfor both, IG and FG, gesture information support the old-newdiscrimination in the subsequent recognition task.

Direct comparisons of the discrimination performance inboth recognition conditions (New1, New2) indicate a significantdifference for the IG (New14New2: t13¼2.185, po.05), butnot in the FG conditions (New14New2: t13¼1.516, po.15). Bycontrast, the response behavior was influenced by the differentrecognition conditions in both gesture conditions (IG, New14New2: t13¼2.226, po.05; FG, New14New2: t13¼2.845, po.05)indicating a more liberal response behavior when old gestureswere presented with new sentences (more false alarms in theNew2 condition).

Despite the relatively low memory performance (between48 and 57% hits in the old gesture condition) the discrimina-tion performance is above zero for all conditions (R1-IG: t(13)¼4.043, po.001; R2-IG: t(13)¼4.582, po.001; R1-FG: t(13)¼4.146,po.001; R2-FG: t(13)¼5.791, po.001) and comparable to pre-vious investigations into implicit encoding of speech andgesture videos (Straube et al., 2009, 2011a; 2011b).

Fig. 4 – Activation pattern of the three conditions in contrast to bapattern for the IG (left), FG (middle) and NG (right) condition in ctemporal, inferior frontal and parietal activation are present in aadditionally related to occipito-temporal activations.

2.3. fMRI imaging results

2.3.1. Activation for each condition in contrast to baseline(fixiation cross)To investigate the general activation pattern related to IG, FGand NG, we contrasted each condition with the low levelbaseline (fixiation cross). We found distributed bilateral tem-poral, inferior frontal and parietal activations for all conditions(see Fig. 4). Despite the similar activations of all three conditionsin contrast to baseline, memory related activations are expectedto differ between conditions and were investigated in thefollowing analyses. Other contrasts between conditions, focus-ing on integration processes or supramodal semantic proces-sing, were published elsewhere (Green et al., 2009; Straube et al.,2012) and are not of interest here.

2.3.2. Activation in relation to memory performanceTo identify memory related activation patterns, which arebased on both, speech and gesture information, the analysesdescribed below always refer to the “gesture support condi-tion” (New1). Thus, in the applied covariate approach (seeSection 5) subsequent memory performance (d-prime) of theNew1 condition had been correlated with condition specificactivation during encoding.

2.3.2.1. Memory related activations of the gesture vs. nogesture conditions. To identify activation related to gesturedependent memory enhancements, memory related activa-tion patterns for both gesture conditions were contrastedwith the no gesture condition (IG4NG and FG4NG). ForIG4NG we found activation of the right temporal pole andinsular cortex as well as the left middle frontal cortex (MFG;see Table 1; Fig. 5A). In contrast anterior and posterior partsof the cingulate cortex (ACC, PCC) and the bilateral posteriorthalamus were related to subsequent memory strength forthe FG condition (see Table 1; Fig. 5B).

seline (fixiation cross). The figure shows the brain activationontrast to baseline (fixation contrast). Distributed bilateralll conditions. Both gesture conditions (IG and FG) are

Table 1 – Memory related activation pattern.

Contrast Anatomical region Lat. Coordinates t-Value Volume (mm3)

x y z

IG4NG MFG/IFG L �32 40 36 5.13 1408Insula, Putamen, Temporal pole R 40 8 �12 4.32 3136Putamen R 32 8 �8 4.11Temporal pole R 52 12 �20 3.54

FG4NG ACC/medial frontal cortex L �12 40 0 5.00 1600PCC/Precuneus L �12 �40 28 4.21 1536IPL L �36 �52 28 3.75Thalamus (extensions to Hippo.) L �4 �36 12 4.14 1984Thalamus L �12 �32 12 4.07Hippocampus L �20 �40 8 3.27ACC R 16 40 8 3.99 1152MFG/IFG L �28 28 20 3.91 1408IFG L �28 32 12 3.43MFG L �20 36 16 3.22

IG4FG MCC L �8 8 32 4.71 2240MCC L �8 �8 40 3.92STG, Insula, Temporal Pole L �48 0 �12 3.77 2176Insula L �36 8 12 3.69Temporal Pole L �48 12 �8 3.58

FG4IG Thalamus (extensions to Hippo.) L �20 �36 12 4.47 1216

Significance level (t-value), size of the respective activation cluster (No. voxels; number of voxels 417) at po.05 corrected were presented.Coordinates are listed in MNI space. Anatomical regions refer to the area nearest to the coordinate and should be considered approximate.Contrasts were named as in Fig. 5 (IG¼Iconic gesture condition; FG¼Free gesture condition; NG¼No gesture condition). ACC: Anteriorcingulate cortex; IFG: Inferior frontal gyrus; MCC: Middle cingulate cortex; MFG: Middle frontal gyrus; PCC: Posterior cingualte cortex; STG:Superior temporal gyrus.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 648

Direct contrasts of memory related activations of bothgesture conditions showed enhanced activation for the FGcondition (FG4IG) in thalamic and hippocampal structures(see Table 1, Fig. 6). The reversed contrast (IG4FG) showedenhanced activation in the middle cingulate cortex (MCC) andthe left temporal pole (with extensions to the left insular andsuperior temporal gyrus; see Table 1, Fig. 6).

2.3.2.2. Region of interest analyses. Region of interest (ROI)analyses were applied to examine in more detail the contribu-tion of the hippocampus (HC; including all anatomically definedsubregions; defined by the SPM Antomy-Toolbox, see Amuntset al., 2005; Eickhoff et al., 2005). The ROI analyses indicatean involvement of the bilateral hippocampus for the processingof both, IG and FG, in contrast to the NG condition (right hippo-campus: MNIxyz¼28, �36, �8, t¼4.94, volume¼256mm3,p¼0.012 FWE corrected; left hippocampus: MNIxyz¼�24, �36,�4, t¼3.95, volume¼256mm3, p¼0.129 FWE corrected; seeFig. 5C). However, no significant differences between gestureconditions (IG, FG) were observed in hippocampal structuresusing ROI analyses.Taken together, the analyses indicate the existence of differentneural processes and thus different encoding processes for freegestures that are unrelated with speech content, as opposed toiconic gestures that are related with the speech content.Whereas the hippocampus is related to subsequent memoryperformance for both gesture conditions, differences in fronto-temporal and thalamic structures suggest different neuralprocesses related to the successful encoding of related versusunrelated speech and gesture videos.

3. Discussion

This study aimed to identify the neural correlates of memoryencoding of related in contrast to unrelated speech–gestureinformation. The present study supports the hypothesis thatdifferent mechanisms for related and unrelated gestures incontext of concrete speech lead to memory advantages.In line with our expectations gesture conditions differed notonly in the semantic relationship, but also with regard to thesurprise response elicited by the stimuli. Unrelated gestureswere rated high on surprise and low on the semantic-relationof speech and gesture. The opposite effect was true forrelated iconic co-verbal gestures.

The fMRI-data support the assumption that different brainmechanisms are related to memory advantages for relatedand unrelated speech–gesture information. Our data suggestthat the left temporal pole with cluster extensions to theinsula and the MCC were specifically involved in memoryprocesses for iconic gestures and related sentences. In con-trast, thalamic–hippocampal structures play a role in mem-ory processes for unrelated speech and gesture information.

Thus, the memory effect for unrelated gestures is mostlikely driven by a surprise response, as indicated by the dataof our rating. In line with this assumption, not only thalamicactivation found for FG4IG, but also PCC and ACC activationsfound for FG4NG might reflect a surprise related attentionreaction (Clark et al., 2001; Mathiak et al., 2002; Stevens et al.,2005; Strange et al., 2000). This explanation is consistent withsuggestions about how attention and surprise affect the

Fig. 5 – Differences in memory related activation pattern. The figure shows activations related to gesture dependent memoryenhancements as memory related activations (correlations with the d0 parameter) for both gesture conditions contrasted with the nogesture condition (5A: IG4NG; 5B: FG4NG; 5C: IGþFG4NG; see Table 1, for coordinates and statistics). Contrast estimates illustrate thememory related activation differences between conditions for the peak voxel of the cluster in the left middle frontal gyrus (MFG forIC4NG; (A)), the anterior cingulated cortex (ACC for FG4NG; (B)) and the right hippocampus (HC for IGþFG4NG; (C)). All other clustershow similar memory related activation pattern across conditions. (C) Shows the results of the region of interest (ROI) analyses. IG:Sentences with related iconic gestures; FG: Sentences with unrelated free gestures, NG: German sentences without gestures.

Fig. 6 – Differences between related iconic und unrelated free gesture conditions. The figure illustrates the differences inmemory related activity between related iconic (IG) und unrelated free gesture conditions (FG). For related speech and gesture(IG) videos we found activation of the middle cingulate cortex and left insula/temporal pole to be stronger related tosubsequent memory performance than for the unrelated condition (FG; see left). For unrelated speech and gesture (FG) videoswe found a stronger subsequent memory related activation in the thalamus compared to the related condition (IG; see right).IG: Sentences with related iconic gestures; FG: Sentences with unrelated free gestures, NG: German sentences withoutgestures.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 650

processing of unrelated information (e.g., Hirshman et al.,1989; Michelon et al., 2003).

For related speech and gesture conditions we foundactivation in the right temporal pole and left middle frontalgyrus in contrast to the no gesture condition. The importanceof the temporal poles for memory processes is known frompatients with dominant anterior temporal lobectomy for therelief of intractable epilepsy which resulted in postsurgicalanomia (e.g., Damasio et al., 2004; Fukatsu et al., 1999;Grabowski et al., 2001; Tranel, 2006). The related memorydysfunctions are relatively specific to proper names, butspare the ability to provide semantic information regardingthe individuals that could not be named. Grabowski et al.(2001) observed that positron emission tomography (PET)activation of the left temporal pole was associated withproper retrieval of names of people and famous landmarks

among healthy normal subjects (Grabowski et al., 2001;Tranel, 2006). These findings suggest that the temporal poleis not exclusively tied to producing names of known facesper se, but is more accurately associated with the naming ofunique entities. The present study did not require explicitname retrieval but related speech–gesture information maycreate a unique neural entity which did require increasedprocessing in the bilateral temporal poles during encoding.A further activation cluster was located in the left middlefrontal cortex, which is basically in line with the theory thatthe frontal cortex and the related activation of schemataplays a specific role in congruency related memory enhance-ments (van Kesteren et al., 2012, 2013). Activation of the leftfrontal cortex had previously been found to be related tosuccessful memory encoding of related gestures in the con-text of abstract speech (Straube et al., 2009). However, a more

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 51

inferior part of the frontal cortex as well as bilateral and moreposterior parts of the temporal cortex has been found in thisprevious study. Therefore this region seemed to reflectbimodal integrative or elaborative encoding processes, whichmight be specific for concrete speech–gesture utterances incontrast to the no-gesture condition.

For incongruous speech and gesture information weobserved activation in anterior and posterior cingulate as wellas thalamo-hippocampal structures. Activity of the thalamuswith extensions to the left hippocampus was increased incontrast to both the no gesture and iconic gesture condition.Therefore this activation most likely reflected specific proces-sing necessary for the processing and subsequent recognition ofunrelated speech and gesture information. This finding is inline with evidence from Michelon et al. (2003) who showedgreater responses to incongruous visual stimuli compared toordinary pictures (the physiological bizarreness effect) in anetwork of regions including thalamic, frontal, and parietalareas. In a second experiment they also showed that partici-pants had better memory for incongruous vs. ordinary stimuli.In keeping with Michelon et al. we also obtained enhancedactivation in parietal and posterior thalamic areas for unrelatedspeech–gesture items. Here we found direct encoding relatedactivation in the ACC, the PCC and the thalamus in relation tosubsequent memory performance. There is evidence for thala-mic responses to oddball stimuli (Kiehl and Liddle, 2001;Klostermann et al., 2006; Stevens et al., 2005), which couldindicate orienting or arousal functions associated with emotion,triggered by the infrequent or unexpected stimuli. Thus, ourresults might be in line with certain aspects of the surprisehypothesis (Hirshman et al., 1989), which proposes that incon-gruous materials are better recalled because of enhancedavailability of episodic cues triggered by attention and surprise.However, thalamic activation can also indicate enhancedelaboration or memory encoding as in the notion of Aggletonand Brown (1999) where the thalamus-hippocampal axis repre-sented an extended but unitary memory network (Aggleton andBrown, 1999; Aggleton et al., 2010). We found bilateral thalamicactivation accompanied by bilateral posterior hippocampalactivation suggesting a common functional relevance for thesubsequent memory performance. The widespread anatomicalconnections between the temporal lobe and the thalamus onthe one hand and the diencephalon connecting the MTL andthe frontal lobe on the other suggest an essential role of thediencephalon in the network underlying memory formation(Vann and Aggleton, 2004). This could indicate that in our studythalamic and hippocampal activation co-occurs and is thereforetogether responsible for successful encoding. These processesmay be further mediated by the frontal cortex. Attention andmemory processes are probably working hand in hand here.However, here we used an implicit encoding task where neitherincongruence detection nor memorization is the primary taskof our participants. Nonetheless, we found activation of thefronto-thalamic–hippocampal network in relation to subse-quent memory performance. This indicates automatic stimulidriven processes related to speech and gesture comprehension,detection of in-congruency and memory.

With regard to memory encoding we focused on hippo-campal activation using ROI analyses and found bilateralhippocampal activation for iconic and unrelated free gestures

in contrast to the isolated speech condition. This finding sup-ports the role of the hippocampal formation for the successfulencoding of relational multimodal episodes such as speech–gesture utterances (Straube et al., 2009, 2011b). In the presentstudy the hippocampus was related to subsequent memorystrength for both incongruous and congruous speech gestureutterances. This indicates that at least some memory relatedhippocampal mechanisms are similar for both gesture pairs andmay characterize the construction of a visual–spatial memoryrepresentation. We explicitly investigated recognition memoryunder two conditions to ensure that both modalities supportsubsequent recognition. In the gesture support condition (New1;see Fig. 1) both speech and gesture information supportedrecognition. In the gesture interference condition (New2; seeFig. 1) sentences have to be discriminated from items thatinclude the same gestures, but in the context of new sentences.Increased discrimination performance in the gesture supportcondition (New1) in contrast to the no-gesture condition, andmore liberal response behavior when the same gestures werepresented with new sentences (New2) both indicated thatgestures play a significant role in the recognition process. Thusmemory related activation specifically in the hippocampusduring encoding of the bimodal conditions probably indicatedmore binding effort for the verbal and gestural information.

The behavioral data suggest that a semantic relation orcongruency between concrete speech and gesture is generallynot the critical feature for gesture related advantages in memory.However, despite the comparable memory performancesbetween related and unrelated speech and gesture items wefound different memory related activation patterns for eachcondition. Whereas temporal and frontal regions used for mem-ory about related speech–gesture information probably reflectsemantic memory processes (Noppeney et al., 2007; Pobric et al.,2007, 2010a, 2010b; Straube et al., 2009, 2011b), memory enhance-ments through unrelated gestures seem to rely on attentionprocesses, related to medial frontal and parietal structures (e.g.,Chica et al., 2011; Loayza et al., 2011; Nardo et al., 2011; Nataleet al., 2009; Sato et al., 2009; Zenon et al., 2010). These attentionprocesses are probably further associated with specific enhancedthalamico–hippocampal storage processes (Aggleton and Brown,1999; Aggleton et al., 2010; Caulo et al., 2005).

The reported medial frontal, parietal and thalamus activa-tions are somewhat inconsistent with findings from thesuccessful encoding of unrelated (in contrast to related)gesture information in the context of abstract speech(Straube et al., 2009). However, the main difference betweeniconic and metaphoric co-verbal gestures is the sentencecontent. The surprise or incongruence effect could be reducedif unrelated gestures were presented in the context ofabstract sentences and therefore not elicit the same attentionresponse in medial frontal and thalamic structures.

In general other explanations for differences in memoryrelated activity such as increased integration load (Willemset al., 2007) for additional activation in the FG condition arepossible and cannot be fully disentangled here. The relativelylow memory performance in our memory task can be con-sidered as further limitation of our study. This low perfor-mance is probably an effect of the easy and implicit encodingtask (evaluating the colour of the spot of the sweater of theactor). Similar low performances have been previously observed

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 652

in conditions were the participants did not know that asubsequent memory task will follow (Straube et al., 2009,2011a, 2011b). However, despite the low performance we stillobtained significant differences between conditions on beha-vioural and neural level supporting the validity of this task.In future studies more elaborative encoding tasks might help toincrease the general memory performance across conditions.Here a covariate approach had been applied to reveal memoryrelated activation pattern for each condition. This has theadvantage of a robust basis fMRI analyses (e.g., the same trialsper condition and subject), but basically reveals brain regionscorrelating with memory strength. A more conventionalapproach were remembered versus forgotten trials are sepa-rately modelled for each subjects and contrasted directly couldbe considered to better reveal brain regions important forremembering. However, due to the limited trial number percondition such an analysis was not possible here.

4. Conclusion

The present study suggests that different neural memorymechanisms underlie memory advantages generated byrelated and unrelated speech and gesture information. Ourresults support suggestions about attention and the effect ofsurprise in the processing of unrelated information, whereassemantic elaboration or schema processing seem to be morerelevant for related speech and gesture memory encoding.Finally, our results imply an involvement of the thalamus–hippocampus axis in memory mechanisms affected by incon-gruous events. Further studies are necessary to investigatequalitative discrepancy between these different ways ofencoding, for example with regard to long time effects orthe practical use of related and unrelated gestures duringteaching or presentations. However, this study alreadydemonstrates the often neglected potential of gestures duringnatural language encoding.

5. Experimental procedures

5.1. Participants

5.1.1. RatingTwenty healthy subjects (10 female, 10 males), which did notparticipate in the fMRI-study, were included in the rating study(mean age: 27 years, range: 24–32 years). All participants wereright handed, native German speakers and had normal orcorrected-to-normal vision, none reported hearing deficits.

5.1.2. fMRI-studyFourteen healthy male subjects were included in the study(mean age: 29 years, range 23–55 years). All participants wereright handed (Oldfield, 1971) native German speakers and hadnormal or corrected-to-normal vision, none reported hearingdeficits. All subjects gave written informed consent prior toparticipation in the study. The study was approved by the localethics committee. Part of the data has been used previously for

comparisons between conditions (Green et al., 2009; Straubeet al., 2012).

5.2. Stimulus material

The details of the stimulus production are described in moredetail in Green et al. (2009) for the same iconic gesture condi-tions. For a more information about the stimulus material seealso (Straube et al., 2010, 2011b, for a comparison of iconic andemblematic coverbal gestures; in this study, parts of the stimulusmaterial were used in event-related design and a differentsubject sample (Straube et al., 2010, 2011b)) and (Straube et al.,2011a, 2012, 2013a,b, for a comparison of iconic and metaphoriccoverbal gestures; in this study, parts of the stimulus materialwere used in an blocked design and a different subject sample(Straube et al., 2011a, 2013a,b)). Therefore only a short descriptionof stimulus production is given here.

The stimulus materials consisted of short video clips of anactor who performed combinations of speech and relatediconic gestures, unrelated free gestures and a control condi-tion including no gestures (Fig. 1). Additionally videos werecreated including the same related iconic or unrelated freegestures but different German sentences, which were relatedwith the iconic and unrelated with the free gestures. There-fore for each related or unrelated gesture there exist twovideos that differ only in the accompanying sentences.

This study focused on three stimulus types (cf. Fig. 1):(1) sentences with related iconic gestures [IG], (2) sentenceswith unrelated free gestures [FG], (3) sentences withoutgestures [NG]. As a distractor in the memory recognition task(New2) videos with the same gestures and different sen-tences were used as a control for iconic (Control-IG, C-IG) andfree gesture conditions (Control-FC, C-FG; see Fig. 1). Higherfalse alarm rates were expected in the recognition task forthese interfering distractor trials. Such an interference effectwould support the importance of gesture information in ourexperiment.

Each clip had a duration of 5000 ms, including at least500 ms before and after the scene, where the actor neitherspoke nor moved. This was done to get standardized videoclips and to account for variable utterance lengths.

The recorded sentences presented here had an averagespeech duration of 2330 ms (SD¼390 ms) and an averagegesture duration of 2700 ms (SD¼450 ms) and did not differbetween conditions.

Videos were previously rated on understandability, image-ability and naturalness (Green et al., 2009) as well as parameterslike movement characteristics, pantomimic content, transitivityand handedness. Based on these ratings stimuli were dividedinto four sets in order to present each participant with 32different items per condition during the scanning procedure,counterbalanced across subjects. Across subjects, each itemwas presented in all eight conditions but a single participantonly saw complementary derivatives of one item, i.e., the samesentence or gesture information was only seen once perparticipant. This was done to prevent speech or gesture repeti-tion effects. Again, all parameters listed above were used for anequal assignment of video clips to the four experimental sets, toavoid set-related between-subject differences.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 53

Previous research has shown that coverbal gestures aretemporally aligned with the corresponding utterance (McNeill,1992). The alignment of stroke and speech occurred on average2150 ms (SD¼535 ms) after the video start (1650 ms afterspeech onset) and was used for the modulation of events inthe event-elated fMRI analysis (Green et al., 2009).

5.3. Experimental design and procedure

5.3.1. RatingDuring rating each participant were presented with 128videos-clips including 64 items of the IG condition and 64items of the FG condition, counterbalanced across subjects.Thus across subjects all German speech–gesture videos usedin the fMRI-study were rated. The participants evaluated eachvideo clip with regard to the semantic relatedness betweenspeech and gesture (question 1) and the surprise elicit by thecombination of speech and gesture information (question 2)on a Likert-like scale between 1 (low) and 7 (high).

5.4. fMRI-experiment

An experimental session was comprised of 256 trials (32 foreach condition) and consisted of four 11-min blocks. Eachblock contained 64 trials with a matched number of itemsfrom each condition. The stimuli were presented in an event-related design in pseudo-randomized order and counterba-lanced across subjects. Each clip was followed by a fixationcross on a grey background with a variable duration of3750 ms to 6750 ms (average: 5000 ms; see Green et al., 2009).

During scanning, participants were instructed to watchthe videos and to indicate via left hand key presses at thebeginning of each video whether the spot displayed on theactor's sweater was light- or dark-colored. This task waschosen to focus participants' attention on the middle of thescreen and enabled us to investigate implicit speech andgesture processing without possible instruction-relatedattention biases. Performance rates and reaction times wererecorded. Before scanning, each participant received at least10 practice trials outside the scanner, which were differentfrom those used in the main experiment. Before the experi-ment started, the volume of the videos was individuallyadjusted so that the clips were clearly audible.

Fifteen minutes after scanning an unheralded recognitionparadigm was used to examine the influence of related andunrelated gestures on memory performance. All videos of therelated iconic gesture condition (IG) and unrelated free gesturecondition (FG; 32 each) and half of the isolated speech condi-tion (16) were presented in random order, together with anequal number of new items for each of the three conditions(altogether 160 videos). The new items consisted of two typesof distractors: (a) 16 completely new items including newspeech and new gesture information (New1) and (b) 16 itemsincluding new sentences but old gesture information (New2).Conditions including new gestures but old sentences were notinvestigated, because (1) the focus of the study was on theinfluence of gesture congruency on the encoding of verbalsentences and (2) it is practically very difficult to find newnatural gestures to the same sentence. Therefore for each ofthe three conditions (IG, FG, NG) 16 completely new videos

were presented and for the gesture conditions (IG, FG), anadditional 16 videos with the same gesture but differentsentence information were presented. Participants had toindicate via key presses whether they had seen that clip(including both speech and gesture information) before or not.

5.5. fMRI data acquisition

The video clips were presented via MR-compatible videogoggles (stereoscopic display with up to 1024�768 pixel reso-lution; VisuaStim XGA©, Resonance Technology, Inc.) andnonmagnetic headphones (audio presenting systems forstereophonic stimuli: Commander XG; Resonance Technology,Inc.). Furthermore, participants wore ear plugs, which act as anadditional low-pass filter.

All MRI data were acquired on a Philips Achieva 3 T scanner.Functional images were acquired using a T2* weighted echoplanar image sequence (TR¼2 s, TE¼30 ms, flip angle 901, slicethickness 3.5 mm with a 0.3 mm interslice gap, 64�64 matrix,FoV 240 mm, in-plane resolution 3.5�3.5 mm, 31 axial slicesorientated parallel to the AC-PC line covering the whole brain).Four runs of 330 volumes were acquired during the experi-ment. The onset of each trial was synchronized to a scannerpulse. Additionally, an anatomical scan was acquired for eachparticipant using a high resolution T1-weighted 3D-sequenceconsisting of 180 sagittal slices (TR¼9863 ms, TE¼4.59 ms,FoV¼256mm, slice thickness 1 mm, interslice gap¼1 mm).

5.6. fMRI data analysis

Statistical parametric mapping (SPM, www.fil.ion.ucl.ac.uk)standard routines and templates were used for analysis offMRI data. The first 5 volumes of every functional run werediscarded from the analysis to minimize T1-saturationeffects. To correct for different acquisition times, the signalmeasured in each slice was shifted relative to the acquisitiontime of the middle slice using a slice interpolation in time. Allimages of one session were realigned to the first image of arun to correct for head movement and normalized intostandard stereotaxic anatomical MNI-space by using thetransformation matrix calculated from the first EPI-scan ofeach subject and the EPI-template. Afterwards, the normal-ized data with a resliced voxel size of 4 mm�4 mm�4 mmwere smoothed with a 6-mm FWHM isotropic Gaussiankernel to accommodate intersubject variation in brain anat-omy. Proportional scaling with high pass filtering (cut-offperiod 128 s) was used to eliminate confounding effects ofdifferences in global activity within and between subjects.

Statistical analysis was performed in a two-level, mixed-effects procedure. At the first level, single-subject BOLDresponses were modeled by a design matrix comprising theonsets of each event (i.e., time of alignment, see stimulusconstruction) of all eight experimental conditions. The hemo-dynamic response was modeled by the canonical hemody-namic response function (HRF) and its temporal derivative.Parameter estimate (ß-) images for the HRF were calculatedfor each condition and each subject. A group analysis wasperformed by entering contrast images into a flexible factorialanalysis implemented in SPM5. This analysis, in whichsubjects are treated as random variables, enables the

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 654

correlation of condition specific activation with the corre-sponding memory performances (d0, discrimination index) ascovariates. Thus, a single flexible factorial analysis wascreated including activation maps and d-prime parameters(covariates) for all conditions (IG, FG, NG). In this analysis theinteraction between condition and covariate was calculated.With this analysis, memory effects can not only be analyzedand compared, but also disentangled from confounding sub-ject or condition specific effects. Age was included as anadditional covariate of no interest to consider age relateddifferences between subjects.

To identify memory related activation patterns, which arebased on both speech and gesture information, the analysesdescribed above always refer to activation correlating withthe covariate “memory performance” (d0) of the “gesturesupport condition” (New1; see Fig. 1). We further includedthe recognition performances (d0) from the gesture interfer-ence control condition (New2) as a covariate of no interest tocontrol for differences in gesture interference effects. Forother comparisons between conditions independent frommemory performance see Green et al. (2009) and Straubeet al. (2012), for the same stimulus set and partly the samesubject sample.

We chose to employ Monte-Carlo simulation of the brainvolume to establish an appropriate voxel contiguity threshold(Slotnick et al., 2003). This correction has the advantage ofhigher sensitivity to smaller effect sizes, while still correctingfor multiple comparisons across the whole brain volume.Assuming an individual voxel type I error of po0.005, a clusterextent of 17 contiguous resampled voxels was necessary tocorrect for multiple voxel comparisons at po0.05. The reportedvoxel coordinates of activation peaks are located in MNI space.For the anatomical localization and ROI analyses the functionaldata were referenced to probabilistic cytoarchitectonic maps(Amunts et al., 2005; Eickhoff et al., 2005; for hippocampalregions). The ROI analyses have been performed by the use ofthe wfu-pickatlas toolbox (http://www.nitrc.org/projects/wfu_pickatlas/).

Statistical analyses of data other than fMRI were performedusing SPSS version 14.0 for Windows (SPSS Inc., Chicago, IL,USA). T-tests and ANOVAs were used to find significant effects.Greenhouse–Geisser correction was applied whenever neces-sary. Discrimination performance (d0) and response criterion (c)were calculated following the signal detection theory (d0 ¼z(Hits) �z(FA); c¼�1/2 [z(Hits)þz(FA)]; e.g., (Macmillan andCreelman, 1991). Statistical analyses were two-tailed withα-levels of significance of po.05.

Acknowledgments

This research project is supported by a grant from theInterdisciplinary Centre for Clinical Research “BIOMAT”within the Faculty of Medicine at the RWTH AachenUniversity (project no. VV N68-e), by the “Von Behring-Rontgen-Stiftung” (project no. 59-0002) and by the “DeutscheForschungsgemeinschaft” (project no. DFG: Ki 588/6-1). B.S. isfunded by the German Federal Ministry of Education andResearch (BMBF; project no. 01GV0615). The authors declareno competing financial interests.

r e f e r e n c e s

Aggleton, J.P., Brown, M.W., 1999. Episodic memory, amnesia, andthe hippocampal–anterior thalamic axis. Behav. Brain Sci. 22,425–444 (discussion 444-89).

Aggleton, J.P., O’Mara, S.M., Vann, S.D., Wright, N.F., Tsanov, M.,Erichsen, J.T., 2010. Hippocampal–anterior thalamic pathwaysfor memory: uncovering a network of direct and indirectactions. Eur. J. Neurosci. 31, 2292–2307.

Amunts, K., Kedo, O., Kindler, M., Pieperhoff, P., Mohlberg, H.,Shah, N.J., Habel, U., Schneider, F., Zilles, K., 2005.Cytoarchitectonic mapping of the human amygdala,hippocampal region and entorhinal cortex: intersubjectvariability and probability maps. Anat. Embryol. (Berlin) 210,343–352.

Blankenburg, F., Ruff, C.C., Bestmann, S., Bjoertomt, O., Josephs,O., Deichmann, R., Driver, J., 2010. Studying the role of humanparietal cortex in visuospatial attention with concurrent TMS–fMRI. Cerebral Cortex 20, 2702–2711.

Caulo, M., Van Hecke, J., Toma, L., Ferretti, A., Tartaro, A.,Colosimo, C., Romani, G.L., Uncini, A., 2005. Functional MRIstudy of diencephalic amnesia in Wernicke–Korsakoffsyndrome. Brain 128, 1584–1594.

Chica, A.B., Bartolomeo, P., Valero-Cabre, A., 2011. Dorsal andventral parietal contributions to spatial orienting in thehuman brain. J. Neurosci. 31, 8143–8149.

Clark, V.P., Fannon, S., Lai, S., Benson, R., 2001.Paradigm-dependent modulation of event-related fMRIactivity evoked by the oddball task. Hum. Brain Mapp. 14,116–127.

Cook, S.W., Yip, T.K., Goldin-Meadow, S., 2010. Gesturing makesmemories that last. J. Mem. Lang. 63, 465–475.

Cook, S.W., Duffy, R.G., Fenn, K.M., 2013. Consolidation andtransfer of learning after observing hand gesture. Child Dev.84, 1863–1871.

Damasio, H., Tranel, D., Grabowski, T., Adolphs, R., Damasio, A.,2004. Neural systems behind word and concept retrieval.Cognition 92, 179–229.

Dick, A.S., Goldin-Meadow, S., Hasson, U., Skipper, J.I., Small, S.L.,2009. Co-speech gestures influence neural activity in brainregions associated with processing semantic information.Hum. Brain Mapp. 30, 3509–3526.

Eickhoff, S.B., Stephan, K.E., Mohlberg, H., Grefkes, C., Fink, G.R.,Amunts, K., Zilles, K., 2005. A new SPM toolbox for combiningprobabilistic cytoarchitectonic maps and functional imagingdata. NeuroImage 25, 1325–1335.

Fukatsu, R., Fujii, T., Tsukiura, T., Yamadori, A., Otsuki, T., 1999.Proper name anomia after left temporal lobectomy: a patientstudy. Neurology 52, 1096–1099.

Grabowski, T.J., Damasio, H., Tranel, D., Ponto, L.L., Hichwa, R.D.,Damasio, A.R., 2001. A role for left temporal pole in theretrieval of words for unique entities. Hum. Brain Mapp. 13,199–212.

Green, A., Straube, B., Weis, S., Jansen, A., Willmes, K., Konrad, K.,Kircher, T., 2009. Neural integration of iconic and unrelatedcoverbal gestures: a functional MRI study. Hum. Brain Mapp.30, 3309–3324.

Greenberg, A.S., Esterman, M., Wilson, D., Serences, J.T., Yantis, S.,2010. Control of spatial and feature-based attention infrontoparietal cortex. J. Neurosci. 30, 14330–14339.

Heinen, K., Ruff, C.C., Bjoertomt, O., Schenkluhn, B., Bestmann,S., Blankenburg, F., Driver, J., Chambers, C.D., 2011. ConcurrentTMS–fMRI reveals dynamic interhemispheric influences of theright parietal cortex during exogenously cued visuospatialattention. Eur. J. Neurosci. 33, 991–1000.

Hirshman, E., Whelley, M.M., Palij, M., 1989. An investigation ofparadoxical memory effects. J. Mem. Lang. 28, 594–609.

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 6 55

Holle, H., Gunter, T.C., Ruschemeyer, S.A., Hennenlotter, A.,Iacoboni, M., 2008. Neural correlates of the processing ofco-speech gestures. NeuroImage 39, 2010–2024.

Holle, H., Obleser, J., Rueschemeyer, S.A., Gunter, T.C., 2010.Integration of iconic gestures and speech in left superiortemporal areas boosts speech comprehension under adverselistening conditions. NeuroImage 49, 875–884.

Kelly, S.D., Church, R.B., 1998. A comparison between childrenand adults’ ability to detect conceptual information conveyedthrough representational gestures. Child Dev. 69, 85–93.

Kelly, S.D., Barr, D.J., Church, R.B., Lynch, K., 1999. Offering a handto pragmatic understanding: the role of speech and gesture incomprehension and memory. J. Mem. Lang. 40, 577–592.

Kiehl, K.A., Liddle, P.F., 2001. An event-related functionalmagnetic resonance imaging study of an auditory oddball taskin schizophrenia. Schizophr. Res. 48, 159–171.

Kircher, T., Straube, B., Leube, D., Weis, S., Sachs, O., Willmes, K.,Konrad, K., Green, A., 2009. Neural interaction of speech andgesture: differential activations of metaphoric co-verbalgestures. Neuropsychologia 47, 169–179.

Klostermann, F., Wahl, M., Marzinzik, F., Schneider, G.H., Kupsch, A.,Curio, G., 2006. Mental chronometry of target detection: humanthalamus leads cortex. Brain 129, 923–931.

Li, L., Gratton, C., Yao, D., Knight, R.T., 2010. Role of frontal andparietal cortices in the control of bottom-up and top-downattention in humans. Brain Res. 1344, 173–184.

Loayza, F.R., Fernandez-Seara, M.A., Aznarez-Sanado, M.,Pastor, M.A., 2011. Right parietal dominance in spatialegocentric discrimination. NeuroImage 55, 635–643.

Macmillan, N.A., Creelman, C.D., 1991. Detection Theory: A User’sGuide. Cambridge University Press, New York.

Mathiak, K., Rapp, A., Kircher, T.T., Grodd, W., Hertrich, I.,Weiskopf, N., Lutzenberger, W., Ackermann, H., 2002.Mismatch responses to randomized gradient switching noiseas reflected by fMRI and whole-headmagnetoencephalography. Hum. Brain Mapp. 16, 190–195.

McDaniel, M.A., Einstein, G.O., DeLosh, E.L., May, C.P., Brady, P.,1995. The bizarreness effect: it not surprising, it complex.J. Exp. Psychol. Learn. Mem. Cogn. 21, 422–435.

McKenzie, S., Eichenbaum, H., 2011. Consolidation andreconsolidation: two lives of memories?. Neuron 71, 224–233.

McKenzie, S., Robinson, N.T., Herrera, L., Churchill, J.C.,Eichenbaum, H., 2013. Learning causes reorganization ofneuronal firing patterns to represent relatedexperiences within a hippocampal schema. J. Neurosci. 33,10243–10256.

McNeill, D., 1992. Hand and Mind: What Gestures Reveal AboutThought. University of Chicago Press, Chicago, IL.

Michelon, P., Snyder, A.Z., Buckner, R.L., McAvoy, M., Zacks, J.M.,2003. Neural correlates of incongruous visual information. Anevent-related fMRI study. NeuroImage 19, 1612–1626.

Moss, B.J., Worthen, J.B., Haydel, L.A., Mahon, B.D.,Savoy, S.C., 2008. Relationships between personalityvariables and bizarreness effects in free recall. Am. J. Psychol.121, 175–187.

Nagels, A., Chatterjee, A., Kircher, T., Straube, B., 2013. The role ofsemantic abstractness and perceptual category in processingspeech accompanied by gestures. Front. Behav. Neurosci., 7.

Nardo, D., Santangelo, V., Macaluso, E., 2011. Stimulus-drivenorienting of visuo-spatial attention in complex dynamicenvironments. Neuron 69, 1015–1028.

Natale, E., Marzi, C.A., Macaluso, E., 2009. FMRI correlates ofvisuo-spatial reorienting investigated with an attentionshifting double-cue paradigm. Hum. Brain Mapp. 30,2367–2381.

Nicolas, S., Marchal, A., 1998. Implicit memory, explicit memoryand the picture bizarreness effect. Acta Psychol. (Amsterdam)99, 43–58.

Nicolas, S., Worthen, J.B., 2009. Adult age differences in memory

for distinctive information: evidence from the bizarreness

effect. Q. J. Exp. Psychol. (Colchester) 62, 1983–1990.Noppeney, U., Patterson, K., Tyler, L.K., Moss, H., Stamatakis, E.A.,

Bright, P., Mummery, C., Price, C.J., 2007. Temporal lobe lesions

and semantic impairment: a comparison of herpes simplex

virus encephalitis and semantic dementia. Brain 130,

1138–1147.Oldfield, R.C., 1971. The assessment and analysis of handedness:

the Edinburgh inventory. Neuropsychologia, 9(1), 97–113.Retrieved from /http://www.ncbi.nlm.nih.gov/pubmed/

5146491S.Pobric, G., Jefferies, E., Ralph, M.A., 2007. Anterior temporal lobes

mediate semantic representation: mimicking semantic

dementia by using rTMS in normal participants. Proc. Nat.

Acad. Sci. U.S.A 104, 20137–20141.Pobric, G., Jefferies, E., Lambon Ralph, M.A., 2010a. Category-

specific versus category-general semantic impairment

induced by transcranial magnetic stimulation. Curr. Biol. 20,

964–968.Pobric, G., Jefferies, E., Ralph, M.A., 2010b. Amodal semantic

representations depend on both anterior temporal lobes:

evidence from repetitive transcranial magnetic stimulation.

Neuropsychologia 48, 1336–1342.Riefer, D.M., Rouder, J.N., 1992. A multinomial modeling analysis

of the mnemonic benefits of bizarre imagery. Mem. Cogn. 20,

601–611.Sato, W., Kochiyama, T., Uono, S., Yoshikawa, S., 2009.

Commonalities in the neural mechanisms underlying

automatic attentional shifts by gaze, gestures, and symbols.

NeuroImage 45, 984–992.Shulman, G.L., Astafiev, S.V., Franke, D., Pope, D.L., Snyder, A.Z.,

McAvoy, M.P., Corbetta, M., 2009. Interaction of stimulus-

driven reorienting and expectation in ventral and dorsal

frontoparietal and basal ganglia-cortical networks. J. Neurosci.

29, 4392–4407.Skipper, J.I., Goldin-Meadow, S., Nusbaum, H.C., Small, S.L., 2007.

Speech-associated gestures, Broca area, and the human

mirror system. Brain Lang. 101, 260–277.Skipper, J.I., Goldin-Meadow, S., Nusbaum, H.C., Small, S.L., 2009.

Gestures orchestrate brain networks for language

understanding. Curr. Biol. 19, 661–667.Slotnick, S.D., Moo, L.R., Segal, J.B., Hart Jr., J., 2003. Distinct

prefrontal cortex activity associated with item memory and

source memory for visual shapes. Brain Res. Cogn. Brain Res.

17, 75–82.Sokolov, E.N., 1963. Higher nervous functions; the orienting

reflex. Annu. Rev. Physiol. 25, 545–580.Staresina, B.P., Gray, J.C., Davachi, L., 2009. Event congruency

enhances episodic memory encoding through semantic

elaboration and relational binding. Cerebral Cortex 19,

1198–1207.Stevens, M.C., Calhoun, V.D., Kiehl, K.A., 2005. fMRI in an oddball

task: effects of target-to-target interval. Psychophysiology 42,

636–642.Strange, B.A., Henson, R.N., Friston, K.J., Dolan, R.J., 2000. Brain

mechanisms for detecting perceptual, semantic, and

emotional deviance. NeuroImage 12, 425–433.Straube, B., Green, A., Weis, S., Chatterjee, A., Kircher, T., 2009.

Memory effects of speech and gesture binding: cortical and

hippocampal activation in relation to subsequent memory

performance. J. Cogn. Neurosci. 21, 821–836.Straube, B., Green, A., Jansen, A., Chatterjee, A., Kircher, T., 2010.

Social cues, mentalizing and the neural processing of speech

accompanied by gestures. Neuropsychologia 48, 382–393.Straube, B., Green, A., Bromberger, B., Kircher, T., 2011a. The

differentiation of iconic and metaphoric gestures: common

b r a i n r e s e a r c h 1 5 6 7 ( 2 0 1 4 ) 4 2 – 5 656

and unique integration processes. Hum. Brain Mapp. 32,

520–533.Straube, B., Green, A., Chatterjee, A., Kircher, T., 2011b. Encoding

social interactions: the neural correlates of true and false

memories. J. Cogn. Neurosci. 23, 306–324.Straube, B., 2012. An overview of the neuro-cognitive processes

involved in the encoding, consolidation, and retrieval of true

and false memories. Behav. Brain Funct. 8, 35.Straube, B., Green, A., Weis, S., Kircher, T., 2012. A supramodal

neural network for speech and gesture semantics: an fMRI

study. PLoS One 7, e51207.Straube, B., Green, A., Sass, K., Kircher, T., 2013a.

Superior temporal sulcus disconnectivity during

processing of metaphoric gestures in schizophrenia.

Schizophr. Bull..Straube, B., Green, A., Sass, K., Kirner-Veselinovic, A., Kircher, T.,

2013b. Neural integration of speech and gesture in

schizophrenia: evidence for differential processing of

metaphoric gestures. Hum. Brain Mapp. 34, 1696–1712.Straube, B., He, Y., Steines, M., Gebhardt, H., Kircher, T., Sammer,

G., Nagels, A., 2013c. Supramodal neural processing of

abstract information conveyed by speech and gesture. Front.

Behav. Neurosci. 7, 120.Tranel, D., 2006. Impaired naming of unique landmarks is

associated with left temporal polar damage. Neuropsychology

20, 1–10.Valenzeno, L., Alibali, M.W., Klatzky, R., 2003. Teacher’s gestures

facilitate students learning: a lesson in symmetry. Contemp.

Educ. Psychol. 28, 187–204.van Kesteren, M.T., Ruiter, D.J., Fernandez, G., Henson, R.N., 2012.

How schema and novelty augment memory formation. Trends

Neurosci. 35, 211–219.

van Kesteren, M.T., Beul, S.F., Takashima, A., Henson, R.N.,Ruiter, D.J., Fernandez, G., 2013. Differential roles for medialprefrontal and medial temporal cortices in schema-dependentencoding: from congruent to incongruent. Neuropsychologia51, 2352–2359.

Vann, S.D., Aggleton, J.P., 2004. The mammillary bodies: twomemory systems in one? Nat. Rev. Neurosci. 5, 35–44.

Wang, S.H., Tse, D., Morris, R.G., 2012. Anterior cingulate cortex inschema assimilation and expression. Learn. Mem. 19, 315–318.

Willems, R.M., Ozyurek, A., Hagoort, P., 2007. When languagemeets action: the neural integration of gesture and speech.Cerebral Cortex 17, 2322–2333.

Willems, R.M., Ozyurek, A., Hagoort, P., 2009. Differential roles forleft inferior frontal and superior temporal cortex inmultimodal integration of action and language. NeuroImage47, 1992–2004.

Worthen, J.B., Garcia-Rivas, G., Green, C.R., Vidos, R.A., 2000. Testsof a cognitive-resource-allocation account of the bizarrenesseffect. J. Gen. Psychol. 127, 117–144.

Worthen, J.B., Wood, V.V., 2001. Memory discrimination for self-performed and imagined acts: bizarreness effects in falserecognition. Q. J. Exp. Psychol. A 54, 49–67.

Worthen, J.B., Eller, L.S., 2002. Test of competing explanations ofthe bizarre response bias in recognition memory. J. Gen.Psychol. 129, 36–48.

Worthen, J.B., Roark, B., 2002. Free recall accuracy for commonand bizarre verbal information. Am. J. Psychol. 115, 377–394.

Worthen, J.B., Deschamps, J.D., 2008. Humour mediates thefacilitative effect of bizarreness in delayed recall. Br. J. Psychol.99, 461–471.

Zenon, A., Filali, N., Duhamel, J.R., Olivier, E., 2010. Saliencerepresentation in the parietal and frontal cortex. J. Cogn.Neurosci. 22, 918–930.