Na

TKa

b

c

a

ARRAA

KCLMCfSG

1

tofrr

apf&

a

0d

Neuropsychologia 47 (2009) 169–179

Contents lists available at ScienceDirect

Neuropsychologia

journa l homepage: www.e lsev ier .com/ locate /neuropsychologia

eural interaction of speech and gesture: Differentialctivations of metaphoric co-verbal gestures

ilo Kirchera,∗,1, Benjamin Straubea,1, Dirk Leubea, Susanne Weisb, Olga Sachsa,laus Willmesb, Kerstin Konradc, Antonia Greena

Department of Psychiatry and Psychotherapy, RWTH Aachen University, Pauwelsstr. 30, D-52074 Aachen, GermanyDepartment of Neurology, RWTH Aachen University, Pauwelsstr. 30, D-52074 Aachen, GermanyDepartment of Child and Adolescent Psychiatry and Psychotherapy, RWTH Aachen University, Pauwelsstr. 30, D-52074 Aachen, Germany

r t i c l e i n f o

rticle history:eceived 21 January 2008eceived in revised form 23 July 2008ccepted 1 August 2008vailable online 14 August 2008

eywords:ross-modal processinganguageultisensory integration

ommunication

a b s t r a c t

Gestures are an important part of human communication. However, little is known about the neural cor-relates of gestures accompanying speech comprehension. The goal of this study is to investigate the neuralbasis of speech–gesture interaction as reflected in activation increase and decrease during observation ofnatural communication.

Fourteen German participants watched video clips of 5 s duration depicting an actor who performedmetaphoric gestures to illustrate the abstract content of spoken sentences. Furthermore, video clips ofisolated gestures (without speech), isolated spoken sentences (without gestures) and gestures in the con-text of an unknown language (Russian) were additionally presented while functional magnetic resonanceimaging (fMRI) data were acquired.

Bimodal speech and gesture processing led to left hemispheric activation increases of the posterior

MRIpeechestures

middle temporal gyrus, the premotor cortex, the inferior frontal gyrus, and the right superior temporalsulcus. Activation reductions during the bimodal condition were located in the left superior temporalgyrus and the left posterior insula. Gesture related activation increases and decreases were dependent onlanguage semantics and were not found in the unknown-language condition.

Our results suggest that semantic integration processes for bimodal speech plus gesture comprehen-sion are reflected in activation increases in the classical left hemispheric language areas. Speech related

langu

NHoc1Stcm

gestures seem to enhance

. Introduction

Humans are highly social beings, who communicate mainlyhrough language. Direct personal interaction furthermore reliesn non-verbal cues, such as facial movements or gestures. Whereasaces and facial affect have been studied extensively in the neu-osciences and functional imaging, co-verbal gestures have yeteceived little attention.

During language perception the multimodal integration of

uditory and visual input facilitates understanding. Multimodalrocessing of an event can lead to advantages, such as enabling

aster and more accurate discrimination (e.g. Frens, Van Opstal,Van der Willigen, 1995; Hershenson, 1962; Hughes, Reuter,

∗ Corresponding author. Tel.: +49 241 8089821; fax: +49 241 8082422.E-mail address: tkircher@ukaachen.de (T. Kircher).

1 These authors contributed equally to this work and should be considered co-firstuthors.

ri

sp1Ad2B

028-3932/$ – see front matter © 2008 Elsevier Ltd. All rights reserved.oi:10.1016/j.neuropsychologia.2008.08.009

age comprehension during the face-to-face communication.© 2008 Elsevier Ltd. All rights reserved.

ozawa, & Fendrich, 1994; Morrell, 1968; Stein, Meredith,uneycutt, & McDade, 1989). For example there is evidence thatbservation of mouth movements strongly influences auditory per-eption already at the phoneme level (e.g. Grant, Walden, & Seitz,998; McGurk & MacDonald, 1976; Sekiyama, Kanno, Miura, &ugita, 2003; Sumby & Pollack, 1954). Functional MRI studies onhe relationship between lip movements and speech found spe-ific activation for processing mismatches versus matches of mouthovements and accompanying phonemes in the superior tempo-

al sulcus, the inferior parietal sulcus, regions of frontal, and thensular cortex (e.g. Calvert, 2001).

With regard to gestures and language, a line of behavioraltudies has shown that listeners use gestural information in com-rehending the speaker’s overall message (e.g. Beattie & Shovelton,

999, 2002; Goldin Meadow & Momeni Sandhofer, 1999; Graham &rgyle, 1975) and that there are facilitation effects for speech pro-uction (e.g. Hanlon, Brown, & Gerstman, 1990; Rose & Douglas,001) as well as for speech comprehension (Gauger, 1952; Kelly,arr, Church, & Lynch, 1999; Valenzeno, Alibali, & Klatzkya, 2003).

1 sychol

RgCintmlMhb&gmEthTransr

roBLBC2aÖTit(pstatrpv(

r1tSeTpHastosSSia

er

todrpiloutgtaigcritEft2tfcBi(dre

2

2

mvnce

2

tmfRcgg

ati(

rp

70 T. Kircher et al. / Neurop

Gestures are an important part of “face to face” communication.ecent theories suggest that gestures may be the phylogenetic ori-in of human speech (e.g. Corballis, 2003; Gentilucci, Bernardis,risi, & Dalla, 2006). These theories are supported by neuroimag-

ng data which have shown an overlap between speech and gestureetworks (see Willems & Hagoort, 2007, for a review). Brain regionsypically associated with motor functions (e.g. motor cortex, pre-

otor cortex, and cerebellum) are also activated during receptiveanguage tasks (Fried et al., 1991; Pulvermüller, 2005; Stringaris,

edford, Giampietro, Brammer, & David, 2007). On the otherand, classical ‘language areas’ (e.g. Broca’s area) were found toe activated during motor tasks (Nishitani, Schurmann, Amunts,

Hari, 2005) and had a functional role in verbal responses toestures (Gentilucci et al., 2006). The observation of body move-ents (Allison, Puce, & McCarthy, 2000; Bonda, Petrides, Ostry, &

vans, 1996) and body-referred gestures (Lotze et al., 2006) leadso increased activity in the bilateral posterior STS – a region thatas been related to speech processing (e.g. Price, 2000; Demonet,hierry, & Cardebat, 2005). Conversely, language processing duringeading leads to an increase in the excitability in the hand motorrea of the dominant hemisphere, measured with transcranial mag-etic stimulation (TMS; e.g. Meister et al., 2003). These resultsuggest a functional connection between motor- and language-elated brain networks.

In functional imaging studies, a number of brain regions haveepeatedly been implicated in cross-modal processing, particularlyf simple speech and audiovisual stimuli (such as lip-movements;eauchamp, Argall, Bodurka, Duyn, & Martin, 2004; Beauchamp,ee, Argall, & Martin, 2004; Callan, Callan, Kroos, & Vatikiotis-ateson, 2001; Callan et al., 2003, 2004; Calvert et al., 1999; Calvert,ampbell, & Brammer, 2000; Calvert, 2001; Olson, Gatenby, & Gore,002; Pekkola et al., 2005). In contrast, the integration of speechnd gestures is believed to occur on a higher semantic level (e.g.zyürek, Willems, Kita, & Hagoort, 2007; Willems & Hagoort, 2007).he first fMRI study on speech and gesture perception focused onconic co-verbal gestures (Willems, Ozyurek, & Hagoort, 2007). Inhis study differential activation of the left inferior frontal cortexBA 45), the premotor cortex (BA 6) and the left superior tem-oral sulcus was demonstrated for the processing of gesture andpeech mismatches. Activation increases in response to seman-ic mismatches in two modalities were interpreted by the authorss correlates of additional processing demands in comparison tohe natural speech–gesture condition. The second fMRI study withegard to speech and gesture processing used a disambiguationaradigm and identified the left superior temporal sulcus to be acti-ated for iconic co-verbal gestures in contrast to grooming gesturesHolle, Gunter, Ruschemeyer, Hennenlotter, & Iacoboni, 2008).

Beside integration-related activation enhancements activationeductions usually occur in response to repeated (e.g. Buckner et al.,998; Schacter & Buckner, 1998; Wiggs & Martin, 1998) or seman-ically related stimuli (e.g. Rissman, Eliassen, & Blumstein, 2003;achs, Weis, Krings, Huber, & Kircher, 2008; Sachs, Weis, Zellagui,t al., 2008; Wheatley, Weisberg, Beauchamp, & Martin, 2005).here is also some evidence from EEG studies for gesture relatedriming effects (e.g. Wu & Coulson, 2007; Holle & Gunter, 2007).owever, in those priming experiments a subsequent trial is gener-lly influenced by one or more prior stimuli. In naturally producedentences gestures accompany speech without much delay. Never-heless, we assume reduced processing demand for comprehensionf speech accompanied by meaningful gestures in contrast to

peech alone. Using fMRI, Skipper, Goldin-Meadow, Nusbaum, andmall (2007) showed reduced connectivity between the anteriorTG and the Broca’s area when speech was accompanied by mean-ngful gestures. This reduced connectivity was further interpreteds facilitation of speech comprehension. However, to our knowl-

at5cat

ogia 47 (2009) 169–179

dge no fMRI investigations have demonstrated speech–gestureelated activation reductions so far.

Thus, little is known about the interaction of speech and ges-ures, and particularly about the influence of metaphoric gesturesn comprehension of abstract concepts. Metaphorical gestures areescriptive, and in contrast to iconic gestures they refer to abstractather than to concrete speech contents (McNeill, 1992). For exam-le, the sentence “The presentation is given on a very high level”

s accompanied by a lift of the right hand representing the “highevel” of the presentation. Metaphoric gestures provide illustrationsf concrete features in order to visualize abstract concepts in thetterance. In contrast, iconic gestures directly refer to concrete fea-ures of the world, as in the sentence “The ball is round” (round handesture) or “The cottage is located on a very high mountain” (lift ofhe right hand). In our study, we presented short videos showing anctor speaking and performing a corresponding metaphoric gesturen combination (SG) or speech and gesture in isolation (metaphoricestures: G; speech: S), as well as a control condition of gesture inontext of an unknown language (Russian; rSG). We used an event-elated design tailored to the co-occurrence of speech and gesturenformation in these utterances. We expected signal increases inhe left IFG due to higher processing demands (e.g. Kircher, Leube,rb, Grodd, & Rapp, 2007; Rapp, Leube, Erb, Grodd, & Kircher, 2004,or metaphor processing; Willems et al., 2007, for co-verbal ges-ure processing), as well as in the temporal lobes (e.g. Callan et al.,004; Holle et al., 2008) for the combined condition (SG) in con-rast to both the isolated (S, G) and the control condition (rSG). Weurther assumed that metaphoric gestures would require less pro-essing demand for the comprehension of the sentence message.ased on the existing evidence that reduced processing demand

s related to lower activation in the task-depended brain regionse.g. Wheatley et al., 2005), we predict gesture related activationecrease for speech comprehension, predominantly in the left ante-ior temporal lobe (e.g. Mummery, Shallice, & Price, 1999; Rissmant al., 2003).

. Methods

.1. Participants

Fourteen male, right handed (Oldfield, 1971) healthy volunteers, all native Ger-an speakers (mean age = 28.96 years, range: 23–38 years) without impairments of

ision or hearing, participated in the study. None of the participants had any medical,eurological or psychiatric illness, past or present. All participants gave informedonsent, and were paid 20D for participation. The study was approved by the localthics committee.

.2. Stimulus construction

A set of 1296 (162 × 8) short video clips containing eight conditions were ini-ially created: (1) German sentences with abstract contents and corresponding

etaphoric gestures; (2) German sentences with abstract contents and unrelated,ree gestures; (3) Russian sentences and corresponding metaphoric gestures; (4)ussian sentences and unrelated, free gestures; (5) German sentences with abstractontents in isolation (without gestures); (6) Russian sentences in isolation (withoutestures); (7) metaphoric gestures in isolation (without speech); (8) unrelated, freeestures in isolation (without speech).

The current study focuses on four stimulus types: (1) German sentences withbstract content and corresponding metaphoric gestures (SG), (2) German sen-ences with abstract content in isolation [S], (3) meaningful metaphoric gesturesn isolation (G) and (4) Russian sentences and corresponding metaphoric gesturesrSG).

Metaphoric gestures constitute one of McNeill’s (1992) basic gesture types andesemble something concrete in order to represent something abstract. For exam-le, the sentence “The twins had a spiritual bond between them” is accompanied by

short connection of the fingertips of both hands in the body middle representing

he “bond” between them (see Fig. 1). All 1296 sentences had the same length of–8 words, with an average duration of 2.47 s (S.D. = 0.4) and a similar grammati-al form (subject–predicate–object). The stimuli were produced by the same malector, performing them in a natural, spontaneous way, to illustrate the content ofhe sentences. The prepared standardized sentences (standardized in terms of word

T. Kircher et al. / Neuropsychologia 47 (2009) 169–179 171

F ts of vw th ans e. A foa

nwTseowdnohapv

mambwwwcmwhwartgrfoec

iSaulmcgaimfa

mSG

sGp

msscctttdpptspayiaitnss

2

vtpo

rsAetAg5

nmo

ig. 1. Examples for the three experimental conditions. The stimulus material consisith abstract contents (SG), isolated metaphoric gestures (G) and isolated speech wi

poken German sentences, translated into English, are written beneath each picturnd analysed additionally.

umber in any given sentence and its grammatical form) were given to the actorho was instructed to speak out the sentences and illustrate them with a gesture.

he synchrony of speech and gesture was determined by the actor. In nearly allituations the gesture automatically corresponded to the most illustrating word ofach sentences (so-called “keyword”), otherwise the sentences were excluded. Ifne sentence was recorded in a natural way subsequently the control conditionsere produced with the aim to hold all item characteristics constant (e.g. sentenceuration or movement complexity), regardless of the language (German, Russian,o speech) and gesture (metaphoric, free, no gesture). This procedure was continu-usly supervised by two of the authors (B.S., A.G.) and timed digitally. All video clipsad the same length of 5 s with at least 0.5 s before and after the sentence onsetnd offset, respectively, where the actor neither speaks nor moves. With the pur-ose to keep the stimuli as natural as possible the head was not removed from theideo.

For stimulus validation, 20 raters who did not take part in the fMRI experi-ent rated each video on a scale from 1 to 7 on understandability, imageability

nd naturalness (1 = very low to 7 = very high). Other parameters such as move-ent characteristics, pantomimic content, transitivity or handedness were coded

y two of the authors (B.S., A.G.). A set of 1024 video clips (128 abstract sentencesith metaphoric gestures with their counterparts in the other seven conditions)ere chosen from the total pool of stimuli for the fMRI experiment. The stimuliere chosen on the basis of high naturalness and movement characteristics (across

onditions), as well as high understandability for the German conditions. Move-ent characteristics were coded by two of the authors (B.S., A.G.) for each video clipith regard to gesture direction and gesture extent. Out of these 1024 videos fouromogeneous sets were created with the aim that each participant was presentedith 256 sentences during the scanning procedure. The videos were counterbal-

nced across the subjects so that one subject did not see either gesture or speechepetitions of any single item. In each of the four complementary sets only thosewo conditions of an item were repeated which contained different speech andesture information. For example German sentences with abstract contents and cor-esponding metaphoric gestures (condition 1) and Russian sentences and unrelated,ree gestures (condition 4). Further examples are condition (2) and (7), (3) and (5)r condition (6) and (8; see above). Across all participants each item occurred inach condition leading to an additional control for possible differences in stimulusharacteristics.

The stimuli for the three conditions analysed in this study scored higher than 3.5n the rating of naturalness (SG m = 4.74, S.D. = 0.57; G m = 4.27, S.D. = 0.33; S m = 3.93,.D. = 0.34; and rSG m = 4.27, S.D. = 0.33), indicating that our material was perceiveds natural. Furthermore, video clips with German speech scored higher than 6 onnderstandability (SG m = 6.77, S.D. = 0.18 and S m = 6.44, S.D. = 1.05), whereas iso-

ated gestures and gestures in context of Russian sentences scored lower than 3 (G= 2.04, S.D. = 0.54; and rSG m = 2.04, S.D. = 0.71). These results represented a high

omprehension of the German sentences with abstract contents independent of theesture condition and were in line with the assumption that metaphoric gesturesre meaningful only in the respective meaningful sentence context. Imageability rat-ngs indicated that items of the combined speech–gesture video condition induced

ental images in the participants (SG m = 4.74, S.D. = 0.57) but this was not the case

or the isolated forms of an item (G m = 3.25, S.D. = 0.56, and S m = 2.95, S.D. = 0.51)nd the Russian control condition (rSG m = 2.87, S.D. = 0.39).

Recorded sentences did not differ significantly in average duration (SG= 2532 ms, S.D. = 382 ms, S m = 2470 ms, S.D. = 366 ms, and rSG m = 2486 ms,

.D. = 414 ms, F(2,381) = 0.891, p = .411), measured from speech onset to speech offset.esture duration, measured from arm movement onset to arm movement off-

ateirt

ideo clips of an actor performing metaphoric gestures and corresponding sentencesabstract content (S). A screenshot of a typical video is shown for each condition. Theurth control condition including Russian speech and gesture (rSG) was presented

et, had the same length in the gesture conditions (SG m = 2394 ms, S.D. = 418 ms,m = 2381 ms, S.D. = 449 ms, and rSG m = 2381 ms, S.D. = 449 ms F(2,381) = 0.945,

= .389).Previous research has pointed out that speakers tend to produce the peak move-

ent of a gesture, the so-called stroke, simultaneously with the relevant speechegment (e.g. Levelt, Richardson, & la Heij, 1985; Nobe, 2000). We used this stroke-peech synchrony in our study for the modulation of events. For the combinedonditions “points of integration” were defined as the time point of the highestonnection between speech and gesture content (stroke). For example, for the sen-ence “The twins had a spiritual bond between them” the integration point was fixedo the end of the word (“bond”) corresponding to the metaphoric gesture (connec-ion of the fingertips) in the combined condition. The keyword of each sentence wasefined in relation to the gesture in the combined condition. This gesture is tem-orally and semantically aligned with the corresponding speech segment. The timeoint of integration was then transferred to the isolated conditions and the con-rol condition (rSG). This was possible because of the standardized timing, form andtructure of each of the eight conditions corresponding to an item. These integrationoints occurred on average 2557 ms (S.D. = 707 ms) after the video start (2057 msfter speech onset) and were used as event sequence in our event-related fMRI anal-sis (see below). With regard to metaphoric gestures the “time point of integration”s strongly dependent on the preceding (abstract) sentence context. Locally speechnd gesture refer to a concrete entity (e.g. “bond” in the example sentences), whereasn the context of abstract speech (e.g. “The twins had a spiritual bond between them”)he bond and the corresponding gesture refer to an abstract concept of a mental con-ection. So our study mainly targeted the effects of bimodal integration on a highemantic level, with a temporal focus on the simultaneously perceived gestural andpoken information.

.3. Experimental design and procedure

During the fMRI scanning procedure, videos were presented via MR-compatibleideo goggles (stereoscopic display with up to 1024 × 768 pixel resolution; VisuaS-im XGA©, Resonance Technology, Inc.) and non-magnetic headphones (audioresenting systems for stereophonic stimuli: Commander XG; Resonance Technol-gy, Inc.), which also dampened scanner noise.

Thirty-two items of each of the eight conditions were presented in an event-elated design, in a pseudo-randomized order and counterbalanced across subjects,o that one subject did not see gesture or speech repetitions of any single item.cross all participants every video was presented in each of its eight conditions, butach participant saw only complementary items of the 8 possible derivatives, thus,he same sentence or gesture information was seen only once by each participant.ll videos had a duration of 5 s and were followed by a baseline condition (gray back-round with a fixation cross) with a variable duration of 3750–6750 ms (average:000 ms).

Subjects were instructed to watch the videos and then to respond at the begin-ing of each video by pressing one of two buttons with the left hand (index andiddle finger) to indicate whether a spot displayed on the actor’s sweater was dark

r light (see Fig. 1, for a dark example). This task was chosen to focus participants’

ttention on the middle of the screen, to avoid instruction biases and to have aask that could be manipulated independent of the video conditions. Using this tasknabled us to investigate implicit speech and gesture processing without possiblenstruction-related attention biases. Before the scanning session, each participanteceived at least 10 practice trials outside the scanner, which were different fromhose used in the main experiment. To adjust the volume level of the video clips,

1 sychol

dat

tawh(dssr

2

wp[otact

2

wTmmstsfmwaes

errfiescgeg

atbo

ssacowuoenrcoac

3

3

(nbaGA(p

qetp

3

TA

C

S

G

S

SridM

72 T. Kircher et al. / Neurop

uring the overview scans additional clips were presented and the volume wasdjusted. During fMRI each participant performed four runs with 64 video clips andhe duration of approximately 11 min each.

Memory performances for the German speech conditions were examined, ∼10o 15 min after scanning, with a recognition paradigm. All videos of the metaphoricnd free gesture condition (32 each) and half of the isolated speech condition (16)ere presented with an equal number of new items for each condition. Participantsad to indicate via “yes”/“no” response, if they had seen the presented video before“old”: left button) or not (“new”: right button). Together 160 videos of the three con-itions were presented randomly distributed in the recognition phase. Presentationoftware (Version 9.2, 2005) was used for stimulus presentation and response mea-urement. Memory data for two of the fourteen participants are missing for technicaleasons.

.4. fMRI data acquisition

MRI acquisition was performed on a Philips scanner (Philips MRT Achieva series)ith 3-Tesla magnetic field strength. Functional data were acquired with echolanar images in 31 transversal slices (repetition time [TR] = 2000 ms; echo timeTE] = 30 ms; flip angle = 90; slice thickness = 3.5 mm; interslice gap = 0.35 mm; fieldf view [FoV] = 240 mm, voxel resolution = 3.5 mm × 3.5 mm). Slices were positionedo cover the participant’s whole brain. After the functional run for each participantn anatomical scan was acquired using a high resolution T1-weighted 3D-sequenceonsisting of 180 sagittal slices (TR = 9863 ms; TE = 4.59 ms; FoV = 256 mm; slicehickness = 1 mm; interslice gap = 1 mm).

.5. Data analysis

MR images were analysed using Statistical Parametric Mapping (SPM2;ww.fil.ion.ucl.ac.uk) implemented in MATLAB 6.5 (Mathworks Inc., Sherborn, MA).

he first 5 volumes of every functional run were discarded from the analysis toinimize T1-saturation effects. To correct for different acquisition times, the signaleasured in each slice was shifted relative to the acquisition time of the middle

lice using a slice interpolation in time. All images of one session were realigned tohe first image of a run to correct for head movement and normalized into standardtereotaxic anatomical MNI-space by using the transformation matrix calculatedrom the first EPI-scan of each subject and the EPI-template. Afterwards, the nor-

alized data with a resliced voxel size of 4 mm × 4 mm × 4 mm were smoothedith a 6-mm FWHM isotropic Gaussian kernel to accommodate intersubject vari-

tion in brain anatomy. Proportional scaling with high pass filtering was used toliminate confounding effects of differences in global activity within and betweenubjects.

The expected hemodynamic response at the defined “points of integration” forach event-type was modelled by two response functions, a canonical hemodynamicesponse function (HRF; Friston et al., 1998) and its temporal derivative. The tempo-al derivative was included in the model to account for the residual variance resultingrom small temporal differences in the onset of the hemodynamic response, whichs not explained by the canonical HRF alone. The functions were convolved with the

vent sequence, with the onsets corresponding to the integration points of gesturetroke and sentence keyword (see stimulus construction), to create the stimulusonditions in a general linear model. A fixed event duration of 1-s was chosen toet a broader range of data around the assumed time point of integration. There isvidence that the semantic information provided by both spoken words and visualestures is integrated within 350–550 ms after word and gesture onset (Özyürek et

3

arg

able 1ctivation peaks and cluster extensions of the S, G and SG conditions against baseline (fix

ond. Anatomical region Cluster extent BA Lat.

Middle temporal gyrus Bilat. TL, IFG,OL, BA6 21 RPrimary motor cortex PL, premotor cortex 6/4/7 RMedial frontal cortex 8/9 LPremotor cortex 6/8 LOccipital lobe 18Premotor cortex 6/8 R

Extrastriate cortex Bilat. OL, IFG, BA6 19 RInferior frontal gyrus 45 L

G Middle temporal gyrus Bilat. TL, IFG,OL, BA6 21 RPrecentral gyrus Bilat. Precuneus 4/6 RMedial frontal gyrus 8 L

ignificance level, t-values and the size of the respective activation cluster (number of voxate (FDR) were mentioned. Lateralization of the activation clusters is indicated by L (left) as the Brodmann area nearest to the coordinate and should be considered approximate. Clusifferent brain areas and should be considered approximate. (OL: occipital lobe; TL: tempoTL: medial temporal lobe; HC: hippocampus).

ogia 47 (2009) 169–179

l., 2007). We used a somewhat broader time frame of data acquisition because theime course of semantic integration processes on the global sentence level is possi-ly more variable and has a longer duration for metaphoric gestures in the contextf abstract sentences.

An SPM2 group analysis was performed by entering contrast images into one-ample t-tests, in which subjects are treated as random variables. Voxels with aignificance level of p < .05 uncorrected belonging to clusters with at least 23 voxelsre reported for all analyses. A Monte Carlo simulation of the brain volume of theurrent study was conducted to establish an appropriate voxel contiguity thresh-ld (Slotnick, Moo, Segal, & Hart, 2003). The result of the Monte Carlo simulationsas based on the 4 mm × 4 mm × 4 mm interpolated voxels. Further informationsed for the cluster-size calculations were the 6-mm smoothing kernel, the fieldf view and the number of slices (see above). Assuming an individual voxel type Irror of p < .05, a cluster extent of 23 contiguous resampled voxels was indicated asecessary to correct for multiple voxel comparisons at p < .05. Most of the reportedesults also hold a false discovery rate correction (FDR) with p < .05. This is indi-ated for each analysis separately (*FDR; Tables 1–3). The reported voxel coordinatesf activation peaks were transformed from MNI space to Talairach and Tournouxtlas space (Talairach & Tournoux, 1988) by non-linear transformations (www.mrc-bu.cam.ac.uk/Imaging/mnispace.html).

. Results

.1. Behavioral results

The average reaction time for the implemented control task“indicate the color of the spot on the actor’s sweater”) didot differ across colors (left button: m = 1.04 s; S.D. = 0.39; rightutton: m = 1.03 s; S.D. = 0.35; t(13) = 0.244, p = .811, paired t-test)nd conditions (SG: m = 1.04 s, S.D. = 0.37; S: m = 1.00 s, S.D. = 0.32;: m = 1.05 s, S.D. = 0.35; F(2,26) = 2.109, p = 0.142; within subjectNOVA). The participants showed an average accuracy rate of 99.2%

S.D. = 0.81), which did not differ across the conditions (F(2,26) = .582,= .566; within subject ANOVA).

In contrast to the performance in the control task, the subse-uent recognition performance of our participants was significantlynhanced for the bimodal condition (SG: 60.18% correct) in con-rast to the isolated speech condition (S: 42.22% correct; t(11) = 2.80,< .05, paired t-test).

.2. fMRI results

.2.1. Gesture processingFor the gesture condition (G) versus baseline (fixation cross)

n extended network of bilateral occipital–temporal, parietal andight lateral frontal activation was revealed (see Table 1 and Fig. 2A:reen).

ation cross)

Coordinates t-Value *FDR No. voxels

X Y Z

63 −24 −6 10.81 .001 39374 −39 68 5.73 .003 297

−8 60 26 5.11 .005 102−44 10 51 4.80 .007 50

0 −93 1 4.01 .017 3655 18 40 3.22 .044 43

48 −70 −7 8.37 .004 5717−59 24 6 3.97 .023 86

59 −20 −6 16.37 .001 47854 −39 68 6.20 .001 372

−8 60 26 3.35 .031 40

els) at p < .05, corrected (MC), and p-value for a correction with the false discoverynd R (right). Coordinates are listed in Talairach and Tournoux (1988) atlas space. BAter extensions denominate activated regions for larger voxel clusters encompassingral lobe; PL: parietal lobe; IFG: inferior frontal gyrus; STG: superior temporal gyrus;

T. Kircher et al. / Neuropsychologia 47 (2009) 169–179 173

Table 2Activation peaks and cluster extensions of the difference contrasts SG > S and SG > G

Contrast Anatomical region Cluster extent BA Lat. Coordinates t-Value *FDR No. voxels

X Y Z

SG > S Fusiform gyrus Bilat. ITG, Cerebellum, OL, PL 21 R 48 −66 −7 7.58 .014 2508Inferior frontal gyrus Precentral gyrus 44/45 L −44 17 21 4.70 .024 545Orbital gyrus IFG 11/47 R 28 42 −12 3.98 .046 121Inferior frontal gyrus Precentral gyrus 44/45 R 48 10 47 3.89 .049 140Medial temporal lobe HC 27 R 20 −27 −5 2.95 .112 45Medial temporal lobe HC, Amygdala 34 L −8 −4 −10 2.24 .223 24

SG > G Middle temporal gyrus STG, STS, ITG 21 R 59 −16 −6 16.89 .000 832Middle temporal gyrus STG, STS, ITG, IFG 21 L −59 −20 −2 15.24 .000 1736Medial frontal cortex 8/9 L −8 56 27 4.65 .006 87Premotor cortex 6 L −48 6 44 3.93 .018 101Medial temporal lobe HC 27/28 R 16 −12 −16 3.50 .035 110Parietal lobe HC 23/7 L −24 −41 32 2.84 .093 24

Significance level, t-values and the size of the respective activation cluster (number of voxels) at p < .05, corrected (MC), and p-value for a correction with the false discoveryr (left) ai e. Clusd empoM

tiSOwp

3

ci

t((ca

3

c

ftTe(tapF−(t6(mtt

TA

C

G

S

SridM

ate (FDR) were mentioned. Lateralization of the activation clusters is indicated by Ls the Brodmann area nearest to the coordinate and should be considered approximatifferent brain areas and should be considered approximate. (OL: occipital lobe; TL: tTL: medial temporal lobe; HC: hippocampus).

We further calculated the gesture effect with the difference con-rast of the combined speech–gesture condition (SG) minus thesolated speech condition (S; see Table 2). The distinct (G: green;G > S: red) and overlapping areas (yellow) are presented in Fig. 2A.verlapping areas, calculated as conjunction analysis (G ∩ [SG > S]),ere mainly located in the bilateral occipito-temporal and rightarietal regions (Table 3).

.2.2. Speech processingFor the isolated speech condition (S) versus baseline (fixation

ross) we found an extended network of bilateral temporal and leftnferior frontal activation (see Table 1 and Fig. 2B: blue).

The difference contrast of the combined condition (SG) minushe isolated gesture condition (G) led to similar activationsTable 2). The distinct (S: blue; SG > G: red) and overlapping areaspurple) are presented in Fig. 2B. In particular, the bilateral temporalortices and the left inferior frontal gyrus showed an overlapping

ctivation pattern (see Table 3).

.2.3. Activation increasesTo investigate enhanced activation in the bimodal condition, a

onjunction analysis (conjunction null) was performed as a test

a(tta

able 3ctivation peaks and cluster extensions of the difference contrasts SG > S and SG > G in co

ontrast Anatomical region Cluster extent BA La

esture effect: (SG > S) ∩ GInferior temporal gyrus OL, PL, Cerebellum 37 RInferior occipital gyrus OL, ITG, Cerebellum 37 LMiddle frontal gyrus IFG, premotorcortex 6/44 ROrbital frontal gyrus 8 RInferior frontal gyrus 45 RInferior frontal gyrus 45 L

peech effect: (SG > G) ∩ SMiddle temporal gyrus TL, MTL, HC 21 RMiddle temporal gyrus TL, IFG, MTL, HC 21 LSuperior frontal gyrus 8 LMiddle frontal gyrus IFG, Premotor cortex 6/44 LMedial temporal lobe HC 28 R

ignificance level, t-values and the size of the respective activation cluster (number of voxate (FDR) were mentioned. Lateralization of the activation clusters is indicated by L (left) as the Brodmann area nearest to the coordinate and should be considered approximate. Clusifferent brain areas and should be considered approximate. (OL: occipital lobe; TL: tempoTL: medial temporal lobe; HC: hippocampus).

nd R (right). Coordinates are listed in Talairach and Tournoux (1988) atlas space. BAter extensions denominate activated regions for larger voxel clusters encompassingral lobe; PL: parietal lobe; IFG: inferior frontal gyrus; STG: superior temporal gyrus;

or two independently significant effects compared to the samehreshold (see Nichols, Brett, Andersson, Wager, & Poline, 2005).his conjunction analysis showed activation in both the differ-nce contrast “speech and gesture (SG)” minus “isolated gestureG)” (red and purple in Fig. 2B) as well as in the difference con-rast “speech and gesture (SG)” minus “isolated speech (S)” (rednd yellow in Fig. 2A). Four areas were significantly activated (all< .05, corrected for multiple comparisons) in this conjunction (seeig. 3A): the left posterior middle temporal gyrus (BA21, TalX, Y, Z:63, −43, −1; t = 5.02; 312 voxels), the left inferior frontal gyrus

BA45, TalX, Y, Z: −51, 24, 10; t = 4.62 and BA47, TalX, Y, Z: −55, 27, −1;= 4.62; 233 voxels), the left premotor cortex (BA6, TalX, Y, Z: −48,, 44; t = 3.90; 80 voxels) and the right superior temporal sulcusBA21, TalX, Y, Z: 51, −31, 2; t = 4.74; 107 voxels). These areas show

ore activation in the bimodal speech and gesture condition (SG)han in the isolated counterparts as well as in the Russian con-rol condition (rSG, see parameter estimates Fig. 3B). Therefore

ll reported activations also hold a further restriction of SG > rSG[SG > S] ∩ [SG > G] ∩ [SG > rSG]), which only resulted in the reduc-ion of cluster size (BA21, TalX, Y, Z: −63, −43, −1; t = 5.02; 269 voxels,he left inferior frontal gyrus, BA45, TalX, Y, Z: −51, 24, 10; t = 4.62nd BA47, TalX, Y, Z: −55, 27, −1; t = 4.34; 202 voxels, the left pre-

njunction with the isolated conditions against baseline (fixation cross)

t. Coordinates t-Value *FDR No. voxels

X Y Z

51 −51 −4 7.38 .039 817−48 −78 −3 5.89 .042 690

51 10 47 3.61 .124 9124 38 −15 3.30 .170 2951 39 −2 3.06 .212 59

−51 39 −2 2.98 .229 58

63 −20 −6 1.22 .001 649−59 −16 −6 9.80 .001 1266−8 60 30 4.49 .015 60

−48 10 47 3.79 .039 3116 −12 −16 3.50 .059 85

els) at p < .05, corrected (MC), and p-value for a correction with the false discoverynd R (right). Coordinates are listed in Talairach and Tournoux (1988) atlas space. BAter extensions denominate activated regions for larger voxel clusters encompassingral lobe; PL: parietal lobe; IFG: inferior frontal gyrus; STG: superior temporal gyrus;

174 T. Kircher et al. / Neuropsychologia 47 (2009) 169–179

Fig. 2. Gesture and speech: distinct and overlapping areas in the processing ofisolated and bimodal video stimuli. (A) Brain areas related to gesture effects, sepa-rate for isolated gesture (G) versus baseline (green) and combined speech–gesture(ssa

mr7

ficsi(r

3

tccvasavraymt−pav−ttlribs

Fig. 3. Activation increases and reductions in response to bimodal speech and ges-ture stimuli. (A) The conjunction analysis of the two difference contrasts (SG > S) and(SG > G) reflects the activation increase (see Section 3) due to the bimodal presen-tation in the combined condition (see parameter estimates, B). (C) The conjunctionanalysis of the two difference contrasts (S > SG) and (SG > G), reflecting activationreductions (see Section 3) in speech related areas for the combined condition (SG)in contrast to the isolated speech condition (S; see parameter estimates, D). For eachcrrsc

SG) versus isolated speech (S) condition (SG > S: red). (B) Brain areas related topeech effects, separate for isolated speech (S) versus baseline (blue) and combinedpeech–gesture (SG) versus isolated gesture (G) condition (SG > G: red). Overlappingctivations are indicated in yellow (A) and purple (B).

otor cortex, BA6, TalX, Y, Z: −48, 6, 44; t = 3.47; 59 voxels, and theight superior temporal sulcus, BA21, TalX, Y, Z: 51, −31, 2; t = 4.33;4 voxels).

Next we specified regions of integration also receiving inputsrom each modality in isolation, by including the constraint thatntegration sites should also respond significantly to both unimodalonditions in isolation [(SG > S) ∩ (SG > G) ∩ S ∩ G]. Two areas wereignificantly activated (all p < .05, corrected for multiple compar-sons) in this conjunction: the left posterior middle temporal gyrusBA37, TalX, Y, Z: −63, −47 −4; t = 4.04; 120 voxels) and the left infe-ior frontal gyrus (BA47, TalX, Y, Z: −55, 27, −5; t = 2.73; 38 voxels).

.2.4. Activation decreasesTo test for the regions in speech specific areas that show ges-

ure related reduced activation in the combined speech and gestureondition (SG) compared to the isolated speech condition (S), aonjunction analysis was performed, testing for the regions acti-ated in both the S versus SG and the SG versus G contrasts. Thisnalysis was preferred to an analysis masked with the isolatedpeech condition (S) because the isolated condition contains allctivations stronger than baseline. By contrast, the regions acti-ated in the difference contrast (SG > G) are more constrained toegions related to speech processing (see Table 2) and do not includectivations due to general video observation. This conjunction anal-sis ([S > SG] ∩ [SG > G]) led to activations (all p < .05, corrected forultiple comparisons) in the left STG (BA22, TalX, Y, Z: −51, 4, −7;

= 3.11; 38 voxels; see Fig. 3C) and the left insula (TalX, Y, Z: −40, −16,6; t = 3.25; 43 voxels; see Fig. 3C). To control for general bimodalrocessing we included the bimodal control condition (rSG). Thisnalysis ([S > SG] ∩ [SG > G] ∩ [rSG > SG]) led also to significant acti-ations in the same brain regions (left STG, BA22, TalX, Y, Z: −51, 4,7; t = 3.11; 23 voxels, and the left insula, TalX, Y, Z: −40, −16, −6;= 3.25; 35 voxels). These results showed that the activation reduc-ion is specific to metaphoric gestures in the context of German

anguage and is not an effect of simple bimodal processing. Theeduced activation in the combined condition (SG) compared to thesolated speech condition (see parameter estimates Fig. 3D) possi-ly indicates reduced processing demand for the comprehension ofpeech. To confirm this assumption regression analyses were per-

ffail

ondition (SG, left; S, middle/left; G, middle/right, and the control condition: rSG,ight) the bars of (B) and (D) represent the parameter estimates (arbitrary units) ineference to the baseline (fixation cross). The black line indicates the correspondingtandard error. (SG: speech and gesture, S: isolated speech, G: isolated gesture, rSG:ontrol “Russian” speech and gesture).

ormed with the activation pattern of the SG condition (extracted

rom both activation peaks using the VOI function of SPM2 and

sphere of 4 mm) and the proportion of correctly rememberedtems in the recognition task. We obtained a significant negativeinear relationship between the activation in the left STG (BA22,

ycholo

T(impvcS

4

sl(tiscpilttmbgh

uCbi2tfestfbtagtWat“ttpicHsocviwtiifl

nsiar

sfapeigctaLaatstascfttto

msJ2cctacwptbttK

citTibidBci1

T. Kircher et al. / Neurops

alX, Y, Z: −51, 4, −7) with the subsequent recognition performanceˇ = −.539; t = −2.023; p < .035). In contrast, the activation of the leftnsula (TalX, Y, Z: −40, −16, −6) in relation to the recognition perfor-

ance failed to reach the significance level (ˇ = −.474; t = −1.701;< .061). These results support the interpretation that reduced acti-ation reflects a reduction in the processing demands for speechomprehension. Predominantly the activation reduction of the leftTG predicts subsequent memory performance of our participants.

. Discussion

In our study, we investigated the neural correlates of metaphoricpeech–gesture interaction. Previous studies on the neural corre-ates of speech and gestures focused on mismatch manipulationse.g. Kelly, Kravitz, & Hopkins, 2004; Willems et al., 2007). In con-rast, we investigated the processing of natural speech and gesturenteraction contrasted with either of them in isolation. We foundtronger activation in the left MTG, the right STS, the premotorortex and the left IFG during simultaneous speech and gesturerocessing (SG), most likely reflecting an increase in the process-

ng demands or a semantic integration effect. Further, we foundeft lateralized activation decreases in the STG and the insula, forhe SG compared to the isolated speech condition which is likelyo reflect lower processing demands for the comprehension of the

essage. Our results suggest that when speech is accompaniedy metaphoric co-verbal gestures the integration of speech andesture possibly leads to reduced processing demands for compre-ension.

A number of experimental paradigms and analyses have beensed to characterize neural correlates of audiovisual interactions.lassically, multisensory integration areas have been identifiedy superimposition of auditory and visual activations (e.g. usingmplicit masking or conjunction analyses, Friston, Penny, & Glaser,005), audiovisual interaction effects, and congruency manipula-ions (e.g. Calvert, 2001). Congruency manipulations were included,or instance, for the integration of speech and lip movements, tonsure that identified neurons respond specifically to the corre-ponding auditory and visual input. They showed less activation inhe incongruent conditions than in the congruent ones. However,or speech and gesture integration mismatching information foroth modalities seems to elicit higher activation in regions assumedo be involved in natural speech–gesture integration (Willems etl., 2007). Therefore, congruency manipulations of speech andesture result in activations different from other domains of mul-isensory integration (e.g. Calvert, 2001; Laurienti et al., 2002;

right, Pelphrey, Allison, McKeown, & McCarthy, 2003). Hein etl. (2007) also showed activation increase for incongruent pic-ure (for instance a “dog”) and sound information (for instancemeow”) in predominantly inferior frontal areas and the pos-erior superior temporal sulcus. Willems et al. (2007) assumedhat semantic integration processes for speech and gesture com-rehension are more strongly taxed when the integration load

ncreases, leading to higher activation levels in the “mismatch”onditions. Another explanation comes from a recent study ofocking and Price (2008), suggesting a conceptual matching, irre-

pective of whether the stimuli are audiovisual, auditory–auditoryr visual–visual. They interpreted activation for incongruent versusongruent concepts of objects as a result of two concepts acti-ated separately (Hocking & Price, 2008). The focus of our studys rather the natural integration of speech and gesture processing

ithout mismatch manipulations. To control for the fact that activa-ion increase of the combined in contrast to the isolated conditionss simply the result of a general sensitivity for bimodal stimuli, wencluded a bimodal control condition. To avoid problems of con-icting information, mismatch reactions or unnaturalness, we used

aoiga

gia 47 (2009) 169–179 175

atural spoken sentences of an unknown language with a corre-ponding gesture (rSG). All reported regions showing activationncrease in comparison to the isolated condition were also morectivated for SG in contrast to rSG. Therefore our results most likelyepresent integration or at least semantic bimodal processes.

Consistent with previous findings on the integration of multi-ensory speech–gesture information (e.g. Callan et al., 2004) weound activation in the posterior part of the left MTG for speechnd gesture information. Besides the fact that MTG activation wasresent in many language studies (e.g. Demonet et al., 2005) andven in relation to metaphor processing (e.g. Stringaris et al., 2007),n our study this activation is more likely related to speech andesture integration than to the comprehension of distantly relatedontents. Evidence for an involvement of the posterior MTG in mul-isensory integration processes comes from studies investigatingction representation (Beauchamp, Argall, et al., 2004; Beauchamp,ee, et al., 2004; Lewis, Brefczynski, Phinney, Janik, & DeYoe, 2005nd Lewis, 2006 for a review). With regard to co-verbal gesturesrecent study of Holle et al. (2008) indicates the crucial role of

he left posterior STS in audiovisual integration processes. In theirtudy the activation of the STS might reflect the interaction betweenhe meaning of gesture and the ambiguous sentence. Willems etl. (2007) also found activation of the left STS, specifically for thepeech mismatch and not for the gesture or the double mismatchondition in contrast to the non-mismatch condition. Despite theact that the activation peak in our study is located in the left pos-erior MTG, the corresponding activation cluster encompasses alsohe posterior STS. Therefore the left posterior temporal lobe seemso have a specific function with regard to the semantic processingr integration of speech and gesture.

In line with previous studies we found activation in the left IFGost likely reflecting language processing on a high semantic level,

uch as metaphor comprehension (Ahrens et al., 2007; Eviatar &ust, 2006; Kircher et al., 2007; Lee & Dapretto, 2006; Rapp et al.,004; Stringaris et al., 2007). Despite the fact that our sentences areonventional, listeners have to integrate the concrete features (e.g.ircle, high location, physical connection) represented by the ges-ures into the abstract sentence contents (e.g. the whole thematic,high level of a talk, a mental connection). Therefore only in the SGondition the concrete gesture information had to be interpretedith reference to the speech content. For instance, a depicted archerformed with the right hand may illustrate the concept of smoothransition of two different topics in the sentence: “The politicianuilds a bridge to the next topic”. Therefore our results of IFG activa-ion may reflect processes on a high semantic level, comparable tohose involved in the metaphor comprehension (Rapp et al., 2004;ircher et al., 2007).

However, activation of the inferior frontal cortex can also reflectomplex semantic integration processing of speech and gesturenformation. Willems et al. (2007) showed that this region is par-icularly sensitive to the processing of speech–gesture mismatches.he processing of mismatches is often assumed to reflect a higherntegrational load leading to enhanced activation in the relatedrain areas. For the processing of sentences with an abstract mean-

ng accompanied by metaphoric gestures the integration load is byefinition high thus leading to the increased activation in the IFG.esides these semantic functions (predominantly BA45/47) espe-ially the dorsal part of left inferior prefrontal cortex (BA44/45)s also specialized for phonological processing (e.g. Poldrack et al.,999) which is consistent with its relation to speech conditions (SG

nd S) in our study. Despite the heterogeneous functional potentialf the inferior frontal cortex, this region seems to be specificallynvolved in co-verbal gesture processing, at least for metaphoricestures, in incongruent speech–gesture information (Willems etl., 2007) and in successful encoding of speech–gesture stimuli

1 sychol

(Hcie

arrfHtp2satsiuc

ibvRNces2bopiFpiiituhmtSiiilFFmsahdat2h

st(

itcota

tiiii1aNniiHibwsdgTgsrwsps2t2spb22lmwOittsgebeevsii(o

76 T. Kircher et al. / Neurop

Straube, Green, Weis, Chatterjee, & Kircher, 2008). The fact thatolle et al. (2008) did not find any activation in the IFG for the pro-essing of iconic co-verbal gestures indicates that the IFG is engagedn higher order semantic or other demanding processes (e.g. Kirchert al., 2007).

In addition to the left hemispheric activations, we also foundsmall right hemispheric activation in the STS. Activation of the

ight hemisphere may be related to the processing of distantlyelated meanings (e.g. Beeman, 1998, 2005). However, it can also beound in response to imitation of finger movements (Goldenberg,ermsdörfer, & Laimgruber, 2001), observation of meaningless ges-

ures (Decety et al., 1997; Grezes, Costes, & Decety, 1999), androcessing of sign language (Levanen, Uutela, Salenius, & Hari,001). Still, because no right hemispheric activation has been foundo far for the processing of iconic gestures and speech (Willems etl., 2007), the activation in our study seems to be more relatedo the processing of illustrating gestures in the context of abstractpeech contents than to gesture processing per se. Further researchs necessary to disentangle the gesture-content interaction, e.g. bysing a direct comparison of iconic and metaphoric gestures withorresponding speech.

Besides the activation in speech related areas we also foundncreased activation of the left premotor cortex (BA6) for theimodal condition which is consistent with studies of action obser-ation (Buccino et al., 2001; Costantini et al., 2005; Grezes, Armony,owe, & Passingham, 2003; Hari et al., 1998; Jeannerod, 2001;ishitani & Hari, 2000; Rizzolatti, Fogassi, & Gallese, 2001), pro-essing of human language (e.g. Meister et al., 2003; Stringarist al., 2007; Wilson, Saygin, Sereno, & Iacoboni, 2004) and theemantic processing of speech gesture mismatches (Willems et al.,007). The meta-analysis of Grezes and Decety (2001) indicates ailateral involvement of the dorsolateral premotor cortex in actionbservation. However, we found activation specifically in the rightremotor cortex for isolated gestures. One explanation of this find-

ng is that metaphoric gestures in isolation are quite meaningless.or example Decety et al. (1997) showed a similar right hemisphericreference for the processing of meaningless in contrast to mean-

ngful gestures. In contrast, the combined speech–gesture conditionn our experiment led to a stronger activation in the left BA6 regionn comparison to the observation of isolated gestures or listeningo the isolated sentences. This suggests that this region is partic-larly sensitive to the presence of both modalities. On the otherand, this region has been implicated in the processing of abstractetaphoric sentences (e.g. Stringaris et al., 2007) and its activa-

ion is possibly enhanced by the additional metaphoric gestures.upportive of this explanation is the fact, that the left BA6 regions also activated in the isolated speech condition but not in thesolated gesture condition. Thus, it seems that the activity in BA6s more related to the processing of abstract speech content, or ateast to related phonological or syntactical speech components (e.g.riederici, Opitz, & von Cramon, 2000; Poldrack et al., 1999; Roskies,iez, Balota, Raichle, & Petersen, 2001), than to the isolated armovements. However, the combination of sentences with corre-

ponding metaphoric gestures leads to an additional activation inmore ventrally located part of this region. The absence of a leftemispheric activation of the BA6 region in our study is possiblyue to the fact that the presented isolated metaphoric gesturesre very complex, meaningless and not goal-directed in contrasto simple hand actions investigated by others (e.g. Buccino et al.,001; Hari et al., 1998). These differences might lead to the right

emispheric lateralization.

Taken together we found activation increases in a left hemi-pheric network of inferior frontal, premotor and bilateral posterioremporal regions. Consistent with these findings Straube et al.2008) showed that a similar integration network that included left

ctwcs

ogia 47 (2009) 169–179

nferior frontal, premotor and posterior temporal areas was relatedo subsequent memory performance for video clips of metaphorico-verbal gestures. These results are in line with our interpretationf the integration-related activation increase, showing the func-ional relevance of these regions in the memory binding of speechnd gesture information.

When speech is presented with co-verbal gestures in contrasto the isolated speech condition an activation reduction was foundn the left STG and the left insula. Activation reductions are oftennterpreted as inhibition or suppression in studies of multisensoryntegration (e.g. Calvert, 2001; Wright et al., 2003) and as facil-tation in studies of repetition priming (e.g. Schacter & Buckner,998; Wiggs & Martin, 1998), semantic priming (e.g. Rissman etl., 2003; Wheatley et al., 2005) and cross-modal priming (e.g.oppeney, Josephs, Hocking, Price, & Friston, 2008). One expla-ation of our finding is that the presentation of gestures had an

ndependently suppressive effect on the activation of the STS andnsula, leading to the reduced activation in the bimodal condition.owever the reduced activation of the isolated gesture condition

n contrast to both speech conditions (see Fig. 3D) can be explainedy the fact that we explicitly investigated speech related areashich are not likely to process isolated metaphoric gestures to the

ame extent. There was no reduction for the bimodal control con-ition (rSG), which is also inconsistent with the assumption thatestures had a general suppressive effect on related brain areas.herefore, the activation reductions in our study most likely reflectesture-related reduction of processing demand for comprehen-ion. This interpretation is in line with the increased gesture-relatedecognition performance of our participants which is consistentith the activation reduction of the left STG. Many behavioural

tudies showed gesture related facilitation effects in language com-rehension and advantages in memory performance for spokenentences (e.g. Gauger, 1952; Kelly et al., 1999; Valenzeno et al.,003). Furthermore, activation of the left STG has been attributedo various semantic functions (e.g. Price, 2000; Demonet et al.,005). Evidence from the semantic priming literature shows thatemantically related pairs generate less activity than unrelatedairs, mirroring the pattern of behavioral facilitation as measuredy word reading times (e.g. Rissman et al., 2003; Wheatley et al.,005; Sachs, Weis, Krings, et al., 2008; Sachs, Weis, Zellagui, et al.,008). Consistent with our finding Mummery et al. (1999) found

ess activity in the left anterior temporal cortex when participantsade lexical decisions during runs with a high proportion of relatedords compared to runs with a lower proportion of related words.ther evidence for semantic priming related activation reduction

n the STG comes from the fMRI study of Rissman et al. (2003). Here,he reduced activity in the temporal lobe suggests that the percep-ion of the prime word activates a lexical-semantic network thathares common elements with the target word, and, thus, the tar-et can be recognized with enhanced neural efficiency. There is alsovidence from EEG studies showing cross-modal priming processesetween speech and gesture items (e.g. Holle & Gunter, 2007; Kellyt al., 2004; Wu & Coulson, 2005). Therefore, such neural processingfficiency possibly also occurs when speech is accompanied by co-erbal gestures. In priming studies prime and target were generallyequentially presented and measured as two events. In our study, asn natural conversation, speech and gesture co-occur, which makest impossible to clarify the direction of possible priming effectswhether gesture primes speech or vice versa) or to clarify potentialther explanations. Nevertheless, we assume that semantic pro-

essing of the spoken sentences in our study is less demanding ifhey are accompanied by meaningful metaphoric gestures. In lineith this interpretation, Skipper et al. (2007) showed a reduced

onnectivity between the anterior STG and the Broca’s area whenpeech is accompanied by meaningful gestures in comparison to

ycholo

miffiasibistsoTwpGap

as2&I(1mcrsrttsc(ttt

alMrsosdg

A

dtA

C(taswa

R

A

A

B

B

B

B

B

B

B

B

B

C

C

C

C

C

C

C

C

C

C

D

D

E

F

F

T. Kircher et al. / Neurops

eaningless or no-gesture conditions. Their findings reinforce ournterpretation of gesture related reduction of processing demandor speech comprehension. Conversely, another explanation for ournding of reduced activation would be that in the case of speechlone, speech-related areas are fully involved in processing thepeech (since there is nothing else present). In contrast to that,n the bimodal speech–gesture condition neural resources woulde further distributed across the brain since more input is coming

nto the system. This could lead to fewer resources available to thepeech-related areas and hence to a ‘reduction’ when compared tohe isolated speech condition. However, this assumption is incon-istent with the behavioural performance in the recognition task inur study, which correlates with the activation reduction of the STG.he performance of our participants in the recognition task is in lineith other evidence for gesture related advantages for speech com-rehension (e.g. Beattie & Shovelton, 1999; Gauger, 1952; Holle &unter, 2007; Kelly et al., 1999; Valenzeno et al., 2003) and pointsgainst gesture related interference or reduced resources for speechrocessing.

Besides the left STG, the left insula exhibits gesture relatedctivation reduction. This area has been related to action repre-entation and body schema (e.g. Chaminade, Meltzoff, & Decety,005), subjective emotional experience (e.g. Phan, Wager, Tayler,Liberzon, 2002; Wicker et al., 2003), facial imitation (e.g. Carr,

acoboni, Dubeau, Mazziotta, & Lenzi, 2003) and language taskse.g. Moro et al., 2001; Price, 2000; Wise, Greene, Buchel, & Scott,999). Our results suggest that both regions (STG and insula) areore involved in the processing of isolated speech than in the pro-

essing of speech accompanied by meaningful gestures, possiblyeflecting a gesture related reduction of processing demand forpeech comprehension. The specific activation enhancement andeduction occurred exclusively in the SG condition which suggestshat underlying processes are highly interdependent. We assumehat successful integration of speech and gesture information (pos-ibly accompanied by activation increase in contrast to the isolatedonditions) is responsible for advantages in speech comprehensionpossibly accompanied by activation reductions in comparison tohe isolated sentence condition). Further investigations focusing onhe functional connectivity between related brain areas could testhis assumption.

Taken together, our results demonstrate interaction of speechnd gestures in a twofold manner: as an additional activation mostikely reflecting the integration of both modalities (left posterior

TG, IFG, BA6 and right STS), and as a gesture related activationeduction possibly reflecting a reduction of processing demand forpeech comprehension (left STG and insula). Further investigationsf specific mechanisms involved in the processing of abstractpeech contents and metaphoric gestures could directly compareifferent gesture types, e.g. iconic and metaphoric co-verbalestures.

cknowledgements

This research project is supported by a grant from the Inter-isciplinary Center for Clinical Research “BIOMAT.” (VV N68) andhe START programme within the Faculty of Medicine at the RWTHachen University.

We thank Pit Aretz and Uli Heuter from the “Audio Visual Mediaenter” of the Faculty of Medicine at the RWTH Aachen UniversityAVMZ) for supporting the recording, adaptation and evaluation of

he video material, Jochen Weber for his programming efforts andssistance, Georg Eder and André Schüppen for fMRI data acqui-ition, René Vohn and Thilo Kellerman for SPM assistance and thehole IZKF service team for their allocation of necessary technical

nd personnel requirements.

F

F

gia 47 (2009) 169–179 177

eferences

hrens, K., Liu, H. L., Lee, C. Y., Gong, S. P., Fang, S. Y., & Hsu, Y. Y. (2007). Functional MRIof conventional and anomalous metaphors in Mandarin Chinese. Brain Language,100, 163–171.

llison, T., Puce, A., & McCarthy, G. (2000). Social perception from visual cues: Roleof the STS region. Trends Cognitive Sciences, 4, 267–278.

eauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. (2004). Unravelingmultisensory integration: Patchy organization within human STS multisensorycortex. Nature Neurosciences, 7, 1190–1192.

eauchamp, M. S., Lee, K. E., Argall, B. D., & Martin, A. (2004). Integration of auditoryand visual information about objects in superior temporal sulcus. Neuron, 41,809–823.

eattie, G., & Shovelton, H. (1999). Mapping the range of information contained in theiconic hand gestures that accompany spontaneous speech. Journal of Languageand Social Psychology, 184, 438–462.

eattie, G., & Shovelton, H. (2002). What properties of talk are associated withthe generation of spontaneous iconic hand gestures? British Journal of SocialPsychology, 41, 403–417.

eeman, M. (1998). Coarse semantic coding and discourse comprehension. In M.Beeman & C. Chiarello (Eds.), Right hemisphere language comprehension: Perspec-tives from cognitive neuroscience (pp. 255–284). Mahwah, NJ: Lawrence ErlbaumAssociates.

eeman, M. J. (2005). Bilateral brain processes for comprehending natural language.Trends Cognitive Sciences, 911, 512–518.

onda, E., Petrides, M., Ostry, D., & Evans, A. (1996). Specific involvement of humanparietal systems and the amygdala in the perception of biological motion. Journalof Neuroscience, 16, 3737–3744.

uccino, G., Binkofski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, V., Seitz, R. J., Zilles,K., Rizzolatti, G., & Freund, H. J. (2001). Action observation activates premotorand parietal areas in a somatotopic manner: An fMRI study. European Journal ofNeurosciences, 13, 400–404.

uckner, R. L., Goodman, J., Burock, M., Rotte, M., Koutstaal, W., Schacter, D. L.,Rosen, B., & Dale, A. M. (1998). Function-anatomic correlates of object prim-ing in humans revealed by rapid presentation event-related fMRI. Neuron, 20,285–296.

allan, D. E., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2001). Multimodalcontribution to speech perception revealed by independent component analy-sis: A single-sweep EEG case study. Brain Research: Cognitive Brain Research, 10,349–353.

allan, D. E., Jones, J. A., Munhall, K., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E.(2003). Neural processes underlying perceptual enhancement by visual speechgestures. Neuroreport, 14, 2213–2218.

allan, D. E., Jones, J. A., Munhall, K., Kroos, C., Callan, A. M., & Vatikiotis-Bateson, E.(2004). Multisensory integration sites identified by perception of spatial waveletfiltered visual speech gesture information. Journal of Cognitive Neurosciences, 16,805–816.

alvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S.(1999). Response amplification in sensory-specific cortices during crossmodalbinding. Neuroreport, 10, 2619–2623.

alvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functionalmagnetic resonance imaging of crossmodal binding in the human heteromodalcortex. Current Biology, 10, 649–657.

alvert, G. A. (2001). Crossmodal processing in the human brain: Insights fromfunctional neuroimaging studies. Cerebral Cortex, 11, 1110–1123.

arr, L., Iacoboni, M., Dubeau, M. C., Mazziotta, J. C., & Lenzi, G. L. (2003). Neuralmechanisms of empathy in humans: a relay from neural systems for imitationto limbic areas. Proceedings of the National Academy of Sciences United States ofAmerica, 100, 5497–5502.

haminade, T., Meltzoff, A. N., & Decety, J. (2005). An fMRI study of imitation: Actionrepresentation and body schema. Neuropsychologia, 43, 115–127.

orballis, M. C. (2003). From mouth to hand: Gesture, speech, and the evolution ofright-handedness. Behavioral Brain Sciences, 26, 199–208.

ostantini, M., Galati, G., Ferretti, A., Caulo, M., Tartaro, A., Romani, G. L., & Agli-oti, S. M. (2005). Neural systems underlying observation of humanly impossiblemovements: An fMRI study. Cerebral Cortex, 15, 1761–1767.

ecety, J., Grezes, J., Costes, N., Perani, D., Jeannerod, M., Procyk, E., Grassi, F., & Fazio,F. (1997). Brain activity during observation of actions. Infuence of action contentand subject’s strategy. Brain, 120, 1763–1777.

emonet, J. F., Thierry, G., & Cardebat, D. (2005). Renewal of the neurophysiology oflanguage: Functional neuroimaging. Physiological Reviews, 85, 49–95.

viatar, Z., & Just, M. A. (2006). Brain correlates of discourse processing: An fMRIinvestigation of irony and conventional metaphor comprehension. Neuropsy-chologia, 44, 2348–2359.

rens, M. A., Van Opstal, A. J., & Van der Willigen, R. F. (1995). Spatial and temporal fac-tors determine auditory–visual interactions in human saccadic eye movements.Perceptual Psychophysiology, 57, 802–816.

ried, I., Katz, A., McCarthy, G., Sass, K. J., Williamson, P., Spencer, S. S., & Spencer, D. D.(1991). Functional organization of human supplementary motor cortex studied

by electrical stimulation. Journal of Neuroscience, 11, 3656–3666.

riederici, A. D., Opitz, B., & von Cramon, D. Y. (2000). Segregating semantic andsyntactic aspects of processing in the human brain: An fMRI investigation ofdifferent word types. Cerebral Cortex, 10, 698–705.

riston, K. J., Fletcher, P., Josephs, O., Holmes, A., Rugg, M. D., & Turner, R. (1998).Event-related fMRI: Characterizing differential responses. Neuroimage, 7, 30–40.

1 sychol

F

G

G

G

G

G

G

G

G

G

H

H

H

H

H

H

H

H

J

K

K

K

L

L

L

L

L

L

L

M

M

M

M

M

M

N

N

N

N

N

O

O

Ö

P

P

P

P

P

R

R

R

R

R

S

S

S

S

S

S

S

S

78 T. Kircher et al. / Neurop

riston, K. J., Penny, W. D., & Glaser, D. E. (2005). Conjunction revisited. Neuroimage,25, 661–667.

auger, P. W. (1952). The effect of gesture and the presence or absence of the speakeron the listening comprehension of eleventh and twelfth grade high school pupils.Speech Monographs, 19, 116–117.

entilucci, M., Bernardis, P., Crisi, G., & Dalla, V. R. (2006). Repetitive transcranialmagnetic stimulation of Broca’s area affects verbal responses to gesture obser-vation. Journal of Cognitive Neurosciences, 18, 1059–1074.

oldin Meadow, S., & Momeni Sandhofer, C. (1999). Gestures convey substantiveinformation about a child’s thoughts to ordinary listeners. Developmental Sci-ences, 21, 67–74.

oldenberg, G., Hermsdörfer, J., & Laimgruber, K. (2001). Imitation of gestures bydisconnected hemispheres. Neuropsychologia, 39, 1432–1443.

raham, J. A., & Argyle, M. (1975). A cross-cultural study of the communication ofextra-verbal meaning by gestures. International Journal of Psychology, 10, 57–67.

rant, K. W., Walden, B. E., & Seitz, P. F. (1998). Auditory–visual speech recognition byhearing-impaired subjects: Consonant recognition, sentence recognition, andauditory–visual integration. Journal of the Acoustical Society of America, 103,2677–2690.

rezes, J., Armony, J. L., Rowe, J., & Passingham, R. E. (2003). Activations related to“mirror” and “canonical” neurones in the human brain: An fMRI study. Neuroim-age, 18, 928–937.

rezes, J., Costes, N., & Decety, J. (1999). The effects of learning and intention on theneural network involved in the perception of meaningless actions. Brain, 122(Pt10), 1875–1887.

rezes, J., & Decety, J. (2001). Functional anatomy of execution, mental simula-tion, observation, and verb generation of actions: A meta-analysis. Human BrainMapping, 12, 1–19.

anlon, R., Brown, J., & Gerstman, L. (1990). Enhancement of naming in nonfluentaphasia through gesture. Brain and Language, 38, 298–314.

ari, R., Forss, N., Avikainen, S., Kirveskari, E., Salenius, S., & Rizzolatti, G. (1998).Activation of human primary motor cortex during action observation: A neuro-magnetic study. Proceedings of the National Academy of Sciences United States ofAmerica, 95, 15061–15065.

ein, G., Doehrmann, O., Muller, N. G., Kaiser, J., Muckli, L., & Naumer, M. J. (2007).Object familiarity and semantic congruency modulate responses in corticalaudiovisual integration areas. Journal of Neuroscience, 27, 7881–7887.

ershenson, M. (1962). Reaction time as a measure of intersensory facilitation. Jour-nal of Experimental Psychology, 63, 289–293.

ocking, J., & Price, C. J. (2008). The role of the posterior superior temporal sulcus inaudiovisual processing. Cerebral Cortex, doi:10.1093/cercor/bhn007

olle, H., & Gunter, T. C. (2007). The role of iconic gestures in speech disambiguation:ERP evidence. Journal of Cognitive Neurosciences, 19, 1175–1192.

olle, H., Gunter, T. C., Ruschemeyer, S. A., Hennenlotter, A., & Iacoboni, M. (2008).Neural correlates of the processing of co-speech gestures. Neuroimage, 39,2010–2024.

ughes, H. C., Reuter, L. P., Nozawa, G., & Fendrich, R. (1994). Visual–auditoryinteractions in sensorimotor processing: Saccades versus manual responses.Journal of Experimental Psychology: Human Perception and Performance, 20,131–153.

eannerod, M. (2001). Neural simulation of action: A unifying mechanism for motorcognition. Neuroimage, 14, 103–109.

elly, S. D., Barr, D. J., Church, R. B., & Lynch, K. (1999). Offering a hand to pragmaticunderstanding: The role of speech and gesture in comprehension and memory.Journal of Memory and Language, 40, 577–592.

elly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speechand gesture comprehension. Brain and Language, 89, 253–260.

ircher, T. T., Leube, D. T., Erb, M., Grodd, W., & Rapp, A. M. (2007). Neural correlatesof metaphor processing in schizophrenia. Neuroimage, 34, 281–289.

aurienti, P. J., Burdette, J. H., Wallace, M. T., Yen, Y. F., Field, A. S., & Stein, B. E.(2002). Deactivation of sensory-specific cortex by crossmodal stimuli. Journal ofCognitive Neurosciences, 14, 420–429.

ee, S. S., & Dapretto, M. (2006). Metaphorical vs. literal word meanings: fMRIevidence against a selective role of the right hemisphere. NeuroImage, 29,536–544.

evanen, S., Uutela, K., Salenius, S., & Hari, R. (2001). Cortical representation of signlanguage: Comparison of deaf signers and hearing non-signers. Cerebral Cortex,11, 506–512.

evelt, W. J. M., Richardson, G., & la Heij, W. (1985). Pointing and voicing in deicticexpressions. Journal of Memory and Language, 24, 133–164.

ewis, J. W., Brefczynski, J. A., Phinney, R. E., Janik, J. J., & DeYoe, E. A. (2005). Distinctcortical pathways for processing tool versus animal sounds. Journal of Neuro-sciences, 25, 5148–5158.

ewis, J. W. (2006). Cortical networks related to human use of tools. The Neuroscien-tist, 12, 211–231.

otze, M., Heymans, U., Birbaumer, N., Veit, R., Erb, M., Flor, H., & Halsband, U. (2006).Differential cerebral activation during observation of expressive gestures andmotor acts. Neuropsychologia, 44, 1787–1795.

cGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264,

746–748.

cNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago, IL:University of Chicago Press.

eister, I. G., Boroojerdi, B., Foltys, H., Sparing, R., Huber, W., & Topper, R. (2003).Motor cortex hand area and speech: Implications for the development of lan-guage. Neuropsychologia, 41, 401–406.

S

S

ogia 47 (2009) 169–179

oro, A., Tettamanti, M., Perani, D., Donati, C., Cappa, S. F., & Fazio, F. (2001). Syntaxand the brain: Disentangling grammar by selective anomalies. NeuroImage, 13,110–118.

orrell, L. K. (1968). Temporal characteristics of sensory interaction in choice reac-tion times. Journal of Experimental Psychology, 77, 14–18.

ummery, C. J., Shallice, T., & Price, C. J. (1999). Dual-process model in semanticpriming: A functional imaging perspective. NeuroImage, 9, 516–525.

ichols, T., Brett, M., Andersson, J., Wager, T., & Poline, J. B. (2005). Valid conjunctioninference with the minimum statistic. NeuroImage, 25, 653–660.

ishitani, N., & Hari, R. (2000). Temporal dynamics of cortical representation foraction. Proceedings of the National Academy of Sciences United States of America,97, 913–918.

ishitani, N., Schurmann, M., Amunts, K., & Hari, R. (2005). Broca’s region: Fromaction to language. Physiology, 20, 60–69.

obe, S. (2000). Where do most spontaneous representational gestures actuallyoccur with respect to speech? In D. McNeill (Ed.), Language and Gesture (pp.186–198). Cambridge: Cambridge University Press.

oppeney, U., Josephs, O., Hocking, J., Price, C. J., & Friston, K. J. (2008). The effect ofprior visual information on recognition of speech and sounds. Cerebral Cortex,18, 598–609.

ldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburghinventory. Neuropsychologia, 9, 97–113.

lson, I. R., Gatenby, J. C., & Gore, J. C. (2002). A comparison of bound andunbound audio-visual information processing in the human cerebral cortex.Brain Research: Cognitive Brain Research, 14, 129–138.

zyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration ofsemantic information from speech and gesture: Insights from event-relatedbrain potentials. Journal of Cognitive Neuroscience, 19, 605–616.

ekkola, J., Ojanen, V., Autti, T., Jaaskelainen, I. P., Mottonen, R., Tarkiainen, A., & Sams,M. (2005). Primary auditory cortex activation by visual speech: An fMRI studyat 3 T. NeuroReport, 16, 125–128.

han, K. L., Wager, T., Tayler, S. F., & Liberzon, I. (2002). Functional neuroanatomyof emotion: A meta-analysis of emotion activation studies in PET and fMRI.NeuroImage, 16, 331–348.

oldrack, R. A., Wagner, A. D., Prull, M. W., Desmond, J. E., Glover, G. H., & Gabrieli,J. D. (1999). Functional specialization for semantic and phonological processingin the left inferior prefrontal cortex. Neuroimage, 10, 15–35.

ulvermüller, F. (2005). Brain mechanisms linking language and action. NatureReview Neurosciences, 6, 576–582.

rice, C. J. (2000). The anatomy of language: Contributions from functional neu-roimaging. Journal of Anatomy, 197, 335–359.

app, A. M., Leube, D. T., Erb, M., Grodd, W., & Kircher, T. T. J. (2004). Neural correlatesof metaphor processing. Cognitive Brain Reseach, 20, 395–402.

issman, J., Eliassen, J. C., & Blumstein, S. E. (2003). An event-related fMRI inves-tigation of implicit semantic priming. Journal of Cognitive Neuroscience, 15,1160–1175.

izzolatti, G., Fogassi, L., & Gallese, V. (2001). Neurophysiological mechanisms under-lying the understanding and imitation of action. Nature Reviews Neuroscience, 2,661–670.

ose, M., & Douglas, J. (2001). The differential facilitatory effects of gestureand visualisation processes on object naming in aphasia. Aphasiology, 15,977–990.

oskies, A. L., Fiez, J. A., Balota, D. A., Raichle, M. E., & Petersen, S. E.(2001). Task-dependent modulation of regions in the left inferior frontalcortex during semantic processing. Journal of Cognitive Neuroscience, 13,829–843.

achs, O., Weis, S., Krings, T., Huber, W., & Kircher, T. (2008). Categorical and the-matic knowledge representation in the brain: Neural correlates of taxonomicand thematic conceptual relations. Neuropsychologia, 46, 409–418.

achs, O., Weis, S., Zellagui, N., Huber, W., Zvyagintsev, M., Mathiak, K., & Kircher, T.(2008). Automatic processing of semantic relations in fMRI: Neural activationduring semantic priming of taxonomic and thematic categories. Brain Research,1218, 194–205.

chacter, D. L., & Buckner, R. L. (1998). Priming and the brain. Neuron, 20,185–195.

ekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003). Auditory–visual speech per-ception examined by fMRI and PET. Neuroscience Research, 47, 277–287.

kipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain andLanguage, 101, 260–277.

lotnick, S. D., Moo, L. R., Segal, J. B., & Hart, J., Jr. (2003). Distinct prefrontal cortexactivity associated with item memory and source memory for visual shapes.Brain Research: Cognitive Brain Research, 17, 75–82.

tein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade, L. (1989). Behaviouralindices of multisensory integration: Orientation to visual cues is affected byauditory stimuli. Journal of Cognitive Neuroscience, 1, 12–24.

traube, B., Green, A., Weis, S., Chatterjee, A., & Kircher, T. (2008). Memory effectsof speech and gesture binding: Cortical and hippocampal activation in rela-tion to subsequent memory performance. Journal of Cognitive Neuroscience,

doi:10.1162/jocn.2009.21053

tringaris, A. K., Medford, N. C., Giampietro, V., Brammer, M. J., & David, A. S. (2007).Deriving meaning: Distinct neural mechanisms for metaphoric, literal, and non-meaningful sentences. Brain and Language, 100, 150–162.

umby, W., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise.Journal of the Acoustical Society of America, 26, 212–215.

ycholo

T

V

W

W

W

W

W

W

W

W

T. Kircher et al. / Neurops

alairach, J., & Tournoux, P. (1988). Co-planar stereotaxic atlas of the human brain.New York: Thieme Medical.

alenzeno, L., Alibali, M. W., & Klatzkya, R. (2003). Teacher’s gestures facilitate stu-dents learning: A lesson in symmetry. Contemporary Educational Psychology, 28,187–204.

heatley, T., Weisberg, J., Beauchamp, M. S., & Martin, A. (2005). Automatic primingof semantically related words reduces activity in the fusiform gyrus. Journal ofCognitive Neuroscience, 17, 1871–1885.

iggs, C. L., & Martin, A. (1998). Properties and mechanisms of perceptual priming.

Current Opinion in Neurobiology, 8, 227–233.

illems, R. M., Ozyurek, A., & Hagoort, P. (2007). When language meetsaction: The neural integration of gesture and speech. Cerebral Cortex, 17,2322–2333.

illems, R. M., & Hagoort, P. (2007). Neural evidence for the interplay betweenlanguage, gesture, and action: A review. Brain and Language, 101,278–289.

W

W

gia 47 (2009) 169–179 179

ilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004). Listening to speechactivates motor areas involved in speech production. Nature Neuroscience, 7,701–702.

icker, B., Keysers, C., Plailly, J., Royet, J. P., Gallese, V., & Rizzolatti, G. (2003). Bothof us disgusted in my insula: The common neural basis of seeing and feelingdisgust. Neuron, 40, 655–664.

ise, R. J., Greene, J., Buchel, C., & Scott, S. K. (1999). Brain regions involved inarticulation. Lancet, 353, 1057–1061.

right, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., & McCarthy, G. (2003).

Polysensory interactions along temporal regions evoked by audiovisual speech.Cerebral Cortex, 13, 1034–1043.

u, Y. C., & Coulson, S. (2005). Meaningful gestures: Electrophysiological indices oficonic gesture comprehension. Psychophysiology, 42, 654–667.

u, Y. C., & Coulson, S. (2007). Iconic gestures prime related concepts: An ERP study.Psychonomic Bulletin & Review, 14, 57–63.