Delay Activity and Sensory-Motor Translation During Planned Eye or Hand Movements to Visual or...

Delay Activity and Sensory-Motor Translation During Planned Eye or HandMovements to Visual or Tactile Targets

E. Macaluso,1,2 C. D. Frith,3 and J. Driver2,3

1Neuroimaging Laboratory, Santa Lucia Foundation, Rome, Italy; 2UCL Institute of Cognitive Neuroscience and Departmentof Psychology, University College London; and 3Wellcome Centre for Neuroimaging, Instituteof Neurology, London, United Kingdom

Submitted 20 February 2007; accepted in final form 21 September 2007

Macaluso E, Frith CD, Driver J. Delay activity and sensory-motortranslation during planned eye or hand movements to visual or tactiletargets. J Neurophysiol 98: 3081–3094, 2007. First published Sep-tember 26, 2007; doi:10.1152/jn.00192.2007. To perform eye or handmovements toward a relevant location, the brain must translate sen-sory input into motor output. Recent studies revealed segregationbetween circuits for translating visual information into saccadic ormanual movements, but less is known about translation of tactileinformation into such movements. Using human functional magneticresonance imaging (fMRI) in a delay paradigm, we factorially crossedsensory modality (vision or touch) and motor effector (eyes or hands)for lateralized movements (gaze shifts to left or right or pressing a leftor right button with the corresponding left or right hand located there).We investigated activity in the delay-period between stimulation andresponse, asking whether the currently relevant side (left or right)during the delay was encoded according to sensory modality, upcom-ing motor response, or some interactive combination of these. Delayactivity mainly reflected the motor response subsequently required.Irrespective of visual or tactile input, we found sustained activity inposterior partial cortex, frontal-eye field, and contralateral visualcortex when subjects would later make an eye movement. For delaysprior to manual button-press response, activity increased in contralat-eral precentral regions, again regardless of stimulated modality. Pos-terior superior temporal sulcus showed sustained delay activity, irre-spective of sensory modality, side, and response type. We concludethat the delay activations reflect translation of sensory signals intoeffector-specific motor circuits in parietal and frontal cortex (plus animpact on contralateral visual cortex for planned saccades), regardlessof cue modality, whereas posterior STS provides a representation thatgeneralizes across both sensory modality and motor effector.

I N T R O D U C T I O N

Representation of external space and performance of spa-tially selective actions require many combined neural compu-tations. These will include encoding of the position of sensorystimuli, in relation to current posture, as well as translation ofthe locations indicated via senses into planning of a movementtoward a specific location. Although in everyday life we caneffortlessly direct gaze or move our hands to objects, brainstructures that encode locations according to sensory input ormotor output may be segregated anatomically. Moreover, wecan acquire information about the position of an event or anobject through different sensory modalities (e.g., vision andtouch) and can spatially direct different parts of our body onthe basis of information from these different modalities (e.g.,

shifting gaze and moving our hands to explore a seen and feltobject that we are holding). The existence of distinct sensorymodalities and motor modalities that can all encode spatialpositions raises questions of whether and how the brain com-bines these different types of spatial representation, to achievecoherent perception of space and to control different types ofspatially selective movement (Andersen et al. 1997; Colby andDuhamel 1996).

One possible architecture for integrating or translating be-tween such spatial representations would be first to combinesensory information from different modalities into a commonand unique representation of sensory space and then feed sucha “supramodal” representation into different effector motorsystems. Other possible architectures might include multiple,segregated representations of multisensory space, separatelyconnected to each specific effector system. Intensive electro-physiological work on neurons in parietal cortex of awakebehaving monkeys (Andersen et al. 1997; Batista andAndersen 2001; Colby and Goldberg 1999; Duhamel et al.1992; Gottlieb et al. 1998; Quian et al. 2006; Snyder et al.1998; Wurtz et al. 1982) has revealed the existence of distinctregions in and around the intra-parietal sulcus that may serve assensori-motor interfaces, each selectively or predominantlyconnected with a specific effector system. The lateral intrapa-rietal region (LIP) has been associated with planning andexecution of eye movements and primarily with visual re-sponses (e.g., Ben Hamed et al. 2001; Snyder et al. 1997)although also showing some auditory responses in trainedanimals (Grunewald et al. 1999; Mullette-Gillman et al. 2005).A more medial region of the intraparietal cortex (variouslytermed parietal reach region, PRR, and/or medial intraparietalarea, MIP) is thought to play a role in pointing and reachingmovements with the hands (Cohen et al. 2002; Johnson et al.1996; Kalaska et al. 1997). This region, as for several otherintraparietal regions involved in spatial representation andmovements (e.g., area AIP in the anterior part of the intrapa-rietal sulcus, Murata et al. 2000; and area 5 in the superiorparietal lobule, Kalaska 1996; Rushworth et al. 1997), has alsobeen shown to receive afferent input from several distinctsensory modalities, including touch as well as vision (Colbyand Duhamel 1991). Moreover, the anatomical connectivity ofthese different intraparietal regions mirrors their apparent func-tional specificity: LIP connects primarily with dorsal regions ofpremotor cortex (PM) involved in eye-movement control,

Address for reprint requests and other correspondence: E. Macaluso, Neu-roimaging Laboratory, Fondazione Santa Lucia, Via Ardeatina, 306-00179Roma, Italy (E-mail: [email protected]).

The costs of publication of this article were defrayed in part by the paymentof page charges. The article must therefore be hereby marked “advertisement”in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

J Neurophysiol 98: 3081–3094, 2007.First published September 26, 2007; doi:10.1152/jn.00192.2007.

30810022-3077/07 $8.00 Copyright © 2007 The American Physiological Societywww.jn.org

whereas PRR/MIP and AIP project to premotor regions in-volved in arm and hand movement control, in dorsocaudal PMand ventrorostral PM, respectively (Jones and Powell 1970;Luppino et al. 1999; Tanne-Gariepy et al. 2002; Wise et al.1997). According to this anatomo-functional organization, spa-tial information concerning several sensory modalities may bebrought together in distinct parietal-premotor circuits that ul-timately specialize in different aspects of spatial movement.

Some recent human neuroimaging studies have adaptedclassical electrophysiological paradigms to test for possibleanalogies (or differences) between the organization of parietalcortex and related structures in human and non-human pri-mates (Astafiev et al. 2003; Bremmer et al. 2001; Culham andValyear 2006; Desouza et al. 2000; Grefkes et al. 2004;Kawashima et al. 1994; Prado et al. 2005; Schluppeck et al.2005; Simon et al. 2002). Many of these human studiestypically employ visual cues specifying a location for subse-quent eye (saccadic) or finger (e.g., pointing) movement. Over-all, neuroimaging results from such studies have highlightedsome segregation of saccade- versus manual-related activa-tions, although some overlap (or some degree of activationduring the other task) has also been found (Astafiev et al. 2003;Connolly et al. 2003; Medendorp et al. 2005a; Simon et al.2002). Overlapping responses across different effector systems(see also Wurtz et al. 1982 for related data from monkeys),when found, may indicate that location encoding in the af-fected area is primarily associated with sensory or attentionalrepresentations there rather than strictly motor output (Astafievet al. 2003; Colby and Goldberg 1999; Macaluso et al. 2003a).

One well-established approach to such issues, for bothelectrophysiological work in nonhuman primates as well as forneuroimaging studies in humans, has utilized delayed-responseparadigms (Calton et al. 2002; Desouza et al. 2000; Duhamelet al. 1992; Schluppeck et al. 2006; Snyder et al. 1997). In suchdelay paradigms, a variable period separates the sensory stim-ulus specifying the target location from the subsequent overtmovement, triggered by a later go signal. Much recent interesthas focused on the processes occurring during the delay perioditself, when there is no sensory stimulus present at the cuedlocation and as yet no overt motor behavior, although the targetlocation for the upcoming movement has already been identi-fied.

Location-specific delay activity has been found both inapparently saccade-related (Duhamel et al. 1992) and appar-ently manual-related (Snyder et al. 1997) regions of parietalcortex. Nonetheless, it should be noted that such delay activityin specific regions linked to particular effector circuits is alsotypically coupled with strong responses to the initial sensory(visual) onset, leading increasingly to the perspective that suchregions should be viewed as sensori-motor interfaces (Gottliebet al. 2005; Schluppeck et al. 2006; Snyder et al. 1998). Forexample, in a recent human fMRI study on delay-activityduring preparation for saccadic eye movements, Schluppeckand colleagues (2006) showed that in medial and posteriorintraparietal areas (IPS1, IPS2, and V7) activity increasedduring the delay period, but this sustained effect was relativelysmall compared with the initial response to the visual cue andthe subsequent response to the overt movement (saccade).Further this study found spatially specific activation (greateractivity for contralateral than ipsilateral target/saccades) for thevisual cue and during the delay period, but no side-specific

effects were found for saccade execution. These findings sug-gest (in accord with much monkey electro-physiological data)(e.g., Colby et al. 1996; Snyder et al. 1997) that locationencoding in these apparently saccade-related parietal regionsmight not be fully segregated from sensory processing, withthe same regions participating both in the “sensory” encodingof the relevant location, as well as the beginnings of motorpreparation (see also Beurze et al. 2007; Connolly et al. 2000;Gottlieb and Goldberg 1999; Gottlieb et al. 2005; Medendorpet al. 2005b; Zhang and Barash 2000).

Such findings, in humans as in monkeys, that even thoseparietal regions that seems linked primarily to one or other typeof upcoming movement can also show strong sensory-relatedresponses, opens the issue of whether providing location cuesin different sensory modalities (e.g., within touch not onlyvision) will modulate activity in these regions. Although sev-eral studies have examined this issue for the monkey brain(Cohen and Andersen 2000; Mullette-Gillman et al. 2005;Schlack et al. 2005), until recently there have been rather fewerhuman fMRI studies of the issue (though see Macaluso et al.2003a) because most of the human neuroimaging studies ex-amining different types of movements and/or delay periodshave used only visual cues (e.g., Beurze et al. 2007; Connollyet al. 2003; Medendorp et al. 2005a; Schluppeck et al. 2005,2006). If different sensory modalities can trigger activation ofregions related to particular types of motor circuit, the newquestion then arises of whether delay activity in differentregions may change as a function of the cue modality or ifinstead delay activity will solely reflect the type of upcomingmovement. We tested this here in a human fMRI study of delayactivity.

We investigated possible effects and interactions concerningthe modality of the cue (visual or tactile stimuli in the left orright hemifield to specify the relevant spatial location forsubsequent movement) and also concerning the motor effectorused for the upcoming overt response at the end of the delay.Our design separated cue- and response-related activity fromthe critical sustained activity occurring during the delay periodbetween sensory cue and motor response movement (e.g.,Schluppeck et al. 2006). For motor responses, we used eithersaccades or lateralized manual responses but included a vari-able delay period between stimulus cue and motor executionwith our focus being on the delay activity observed. Themanual task involved button presses with the left or right handon the corresponding target side. Thus although left versusright eye movements differed solely in term of end spatialposition (i.e., with the same effector—the eyes being used forboth left and right movements), our manual task did notdissociate effector (left or right hand) from spatial position(target location in left or right hemifield) (see also Beurze et al.2007, who addressed the issue of position vs. effector speci-ficity). Thus the present manual task of spatially compatiblebutton pressing, after the delay after a visual or tactile stimuluscue, may require a step of effector selection (left or right hand)(see also Thoenissen et al. 2002) as well as the locationselection involved in the oculomotor task also. Nonetheless,because the present paradigm factorially crossed cue modality(visual or tactile) with movement type (oculomotor or manual),this allowed us to determine whether during the delay phase therelevant position would be encoded according to the modalityof the cue (visual or tactile), the type of upcoming overt

3082 E. MACALUSO, C. D. FRITH, AND J. DRIVER

J Neurophysiol • VOL 98 • NOVEMBER 2007 • www.jn.org

response (saccadic or manual), or some interactive combina-tion of these two factors. Moreover, other potential designs(such as placing one hand in a central start position, analogousto central fixation for the eyes) would have prevented us fromplacing tactile stimuli at identical locations to the visual stimulion either side. We return to such issues in our DISCUSSION. Fornow, the present design allowed us to assess whether brainactivity as measured with fMRI, for the delay period, woulddepend on the stimulus modality, the response type (oculomo-tor or manual) or some interaction between these.

M E T H O D S

Participants

Nine right-handed healthy volunteers (6 males) participated (meanage: 26 yr). They gave written informed consent in accord with localethics.

Paradigm

The experimental design was a 2 � 2 � 2 factorial; with cuemodality (vision/touch) and side (left/right) as orthogonal event-related factors and movement type (eyes/hand) as an orthogonalblocked factor. On each trial, a visual cue or tactile cue on the left orright indicated the relevant position for the subsequent movement.After a variable delay (see following text), an auditory “go-signal”instructed the subjects to execute the movement for the cued side. Eyemovements to left or right or hand movements (button presses withleft or right hand) were blocked within each fMRI run, so subjectsalways knew whether the eyes or hands were currently relevant butdid not know which target side they should be directed to prior to thestimulus cue on each trial. The delay between the sensory cue and thesubsequent go signal for movement initiation was varied (range:0.3–12.8 s) to allow us to isolate preparatory activity in this delay(separated from the preceding stimulus and subsequent action) (see

also Schluppeck et al. 2006) for the different combinations of cuemodality and movement type; (i.e., eye to vision, eye to touch, handto vision, and hand to touch). Our aim was to assess whether brainactivity in this delay period would reflect the cue-modality (vision vs.touch), the type of upcoming movement (hand vs. eye), or someinteractive combination thereof and in which brain regions.

Stimuli and task

Subjects lay in the scanner with their hands on a plastic tablepositioned above the hips, where we placed LEDs for visual stimu-lation on either side; piezo-electric components (T220_H3BS_304,Piezo Systems, Cambridge, MA) to deliver unseen tactile stimulationto either hand; and electronic switches to record button presses witheither hand when these were made (see Fig. 1A). Thus on each side,we could deliver visual or tactile cue stimuli (either of these incorrespondence with the index finger of one hand) and could alsorecord motor responses (i.e., key presses performed with either of thetwo thumbs, which were occluded to avoid any visual stimulationassociated with manual responses and any eye-movements towardeither side). The visual scene was filmed with a shielded cameraplaced outside the head-only magnetic resonance (MR) bore (SIE-MENS, Allegra) and projected onto a screen at the back of the scanner(see Fig. 1A, left). The subjects viewed the visual scene through amirror system that also allowed monitoring of eye-position with aremote infra-red eye-tracker (see following text). From inside thescanner, the scene appeared to the subjects as if they were lookingdown at their own hands resting on a table (see Fig. 1A, right) whilethey looked straight ahead into the mirror system. The soft illumina-tion of the MR room permitted continuous viewing of the scenethroughout the experiment. On the table, the index fingers (and theLEDs) rested at 10 cm to the left or right of the central fixation point.The distance between the camera and the table was adjusted so thatthe index fingers were at 7.8° visual angle in the projected image, thusmaintaining a realistic size scaling for the seen hands.

Before the beginning of each fMRI run, the subjects were instructedverbally on whether they had to respond with eye movements or with

FIG. 1. A: schematic illustration of theexperimental setup. A camera attached infront of the MR-bore filmed the visual scenearound the subject’s hands that was thensimultaneously projected onto a screen at theback of the scanner (see illustration on theleft). The subjects viewed the visual scene asif in front of them (as the filmed imageindeed was) via a mirror system. The scenecomprised a central fixation point, light-emitting diodes (LEDs) for presentation ofthe visual cues, plus the top part of the hands(see schematic illustration on the right). Theunseen tactile cues were delivered to thefingerpad of index fingers, and the occludedthumbs were used for button-press re-sponses. B: mean (and therefore smooth)horizontal eye-position traces against time inthe response phase are plotted for each sub-ject, according to the modality of the cue(tactile or visual), the target side (left orright) and the type of motor response (man-ual or saccades; see legend). For clarity, alleye-position traces are aligned at saccadeonset.

3083EYES OR HAND MOVEMENTS TO VISUAL OR TACTILE TARGETS


manual button presses for the upcoming block of trials. Irrespective ofeffector type, each trial began with the illumination of a centralfixation point. After 1,500 ms this central fixation point was turnedoff, and either a visual or a tactile cue was presented on the left or onthe right side for 200 ms. Cue modality and position were randomizedand unpredictable. The lateralized sensory cue indicated the targetposition for subsequent movement, but subjects had to withhold theirmotor response until the auditory go signal. After a variable delay (seefollowing text), an auditory tone (frequency of 500 Hz; duration of200 ms) instructed the subject to execute the movement for therelevant location (go signal). In two fMRI runs, subjects responded bymanually pressing the button on the relevant side, and in the other twofMRI runs, they made a saccade toward the relevant position (i.e.,toward the index finger of one or the other hand, at the same locationsas the possible visual target) and back to the central fixation point,with the order of tasks counterbalanced.

Each fMRI run consisted of 28 trials (intertrial interval � 16 s).There were 14 visual trials (7 on the left and 7 on the right side) and14 tactile trials (7 left and 7 right) in random order. We usedcue-to-go-signal delay periods of between 0.3 and 12.8 s. To reducethe correlation between cue-related activity, movement-related activ-ity, and preparatory/delay activity, we used long delays for 57% of thetrials (864–12,800 ms, uniform distribution), middle-range delays for29% of the trials (447–863 ms), and short delays for 14% of the trials(300–446 ms). The high variability of possible delays also ensuredthat subjects could not anticipate the exact time of the go signal andtherefore should prepare for whenever the go signal might occur. Theorder of the four fMRI runs (2 runs with eye movements, and 2 runswith hand movements) was counterbalanced across subjects. Each runlasted for 8 min. Prior to the fMRI session, all subjects receivedapproximately ten minutes of practice inside the scanner.

Magnetic resonance imaging

A Siemens Allegra (Siemens Medical Systems, Erlangen, Ger-many) operating at 3 T and equipped for echo-planar imaging ac-quired functional MR images. A quadrature volume head coil wasused for radio frequency transmission and reception. Thirty-two slicesof functional MR images were acquired using blood-oxygenation-level-dependent imaging (3 � 3 mm, 2.5-mm thick, 50% distancefactor, TR � 2.08 s, TE � 30 ms), covering the entirety of the cortex.

Data analysis

We used SPM2 (http://www.fil.ion.ucl.ac.uk) for data preprocess-ing and statistical analyses. We acquired 920 fMRI volumes for eachparticipant. After discarding the first four volumes of each run, allimages were corrected for head movement. Slice-acquisition delayswere corrected using the middle slice as reference. All images werenormalized to the standard SPM2 EPI template, resampled to 2-mmisotropic voxel size, and spatially smoothed using an isotropic Gauss-ian kernel of 10 mm full-width half-maximum (FWHM) in accordwith the SPM approach. Time series at each voxel for each subjectwere high pass filtered at 220 s and prewhitened by means of anautoregressive model AR(1).

Statistical inference was based on a random effects approach(Penny and Holmes 2004), and assigned corrected P values as de-scribed later. The SPM analysis comprised two steps. First, for eachsubject, the data were best-fitted at every voxel using a combinationof effects of interest. These represented the three phases of the trial:sensory cue, variable-length delay, and motor response. Within eachfMRI run (either with eye or hand responses), these three phases(sensory, delay, motor) were modeled separately according to cuemodality (vision or touch) and relevant side (left or right hemifield).Thus each fMRI-time series was fitted with 12 effects of interest (3phases � 4 trial-types), plus some confounding effects of no interestthat were modeled separately to remove their influence. These were

rare trial phases when subjects moved their eyes when they shouldinstead have maintained central fixation (see following text). Linearcontrasts were used to average responses for the 12 effects of interestin the two fMRI runs requiring the same motor response (i.e., eyes orhands). This procedure resulted in 24 contrast-images per subject,representing the four possible sensory-motor combinations (eye tovision, eye to touch, hand to vision, hand to touch), for eitherhemifield (left or right), and for the three trial phases (sensory, delay,and motor).

In the second stage (group analyses), we modeled eight conditions(4 sensory-motor combinations on each side) separately in three SPMgroup analyses, one for each trial phase. Finally, within each suchSPM analysis, linear compounds were used to compare the eightconditions, now using between-subjects variance (rather than betweenscans). Correction for nonsphericity (Friston et al. 2002) was used toaccount for possible differences in error variance across conditionsand any nonindependent error terms for the repeated measures. Inthese group random-effects analyses, we retained as statisticallysignificant only activations surviving correction for multiple compar-isons (P corr. � 0.05; see next section for more details).

Preliminary analyses assessed activity in the sensory cue phase andin the motor response phase. In the sensory cue phase, we tested forany modality-dependent activation, plus any common activations thatgeneralized over modalities (conjunction analysis) (Friston et al.1999; Price and Friston 1997). To assess this, we compared activityduring sensory cueing versus rest (e.g., visual cue vs. rest) with awhole-brain corrected threshold (P corr. � 0.05). Furthermore, toensure the specificity of any modality-dependent activation duringsensory stimulation, we included the additional constraint that suchactivity during cueing for one modality also had to be greater than forthe other modality (e.g., visual vs. tactile cue; P uncorr. � 0.001).Note that adding these additional constraints (i.e., requiring that aparticular activation should pass a further uncorrected test also) canonly make our analyses more conservative than the basic correctedcontrast. Likewise, when testing for commonalities across vision/touch (whole-brain-corrected conjunction analyses), we also furtherrequired each of the individual modalities to activate at P uncorr. �0.001.

An analogous approach was used to assess brain activation in themotor response phase. Accordingly, we tested for any activationspecific for a movement type (saccade or manual), plus any commonactivation that generalized over movement types (using conjunctionanalyses). We compared activity during motor-response versus rest(whole-brain-corrected threshold: P corr. � 0.05) plus added thefurther constraint that activity for one movement type also had to begreater than for the other movement-type (e.g., saccade vs. manualresponses; P uncorr. � 0.001). When testing for commonalities acrosseye/hand (whole-brain-corrected conjunction analyses), we addition-ally required each of the individual movement-types to activate atP uncorr. � 0.001.

For the critical analyses of the preparatory delay phase, again wetested for modality or effector-dependent delay activity and for anycommon activations that generalized across modality (and/or effector)versus rest (whole-brain P corr. � 0.05). To confirm modality- and/oreffector-specific activation, we added the further constraint that ac-tivity for one modality (or effector) also had to be greater than for theother modality/effector (P uncorr. � 0.001) to count as specific in thissense.

Further, we tested for any side-dependent activations (left minusright targets or vice versa), separately for the sensory, delay, or motorphases described in the preceding text within those regions alreadyhighlighted by the comparisons mentioned in the preceding text (i.e.,already characterized as modality specific or generalizing acrossmodality; and as effector specific or generalizing across effectors). Forthese side-dependent contrasts, correction for multiple comparisons(P corr. � 0.05) considered only the volumes already activated for the



main comparisons (i.e., modality/effector-dependent or common ac-tivations).

Finally, for completeness only (and to allow some comparison withour prior work) (Macaluso et al. 2003a), we also tested for side-dependent activations in the anterior intraparietal sulcus (aIPS). In aprevious study that had used a similar design but without the criticaldelay period introduced here, we had found that this region wasactivated contralaterally regardless of stimulus modality (vision ortouch) or effector (hand vs. eye) when immediate movements weremade (Macaluso et al. 2003a). In the current experiment, we couldassess whether any such side-dependent activations in the aIPS wouldbe specific to the sensory cue phase, delay phase, or motor responsephase. For this a priori defined aIPS region, we assigned corrected Pvalues using as the volume of interest a 10-mm sphere (small volumecorrection) (Worsley et al. 1996) that was centered on the intraparietalpeak found in Macaluso et al. (2003a).

Eye tracking

In all subjects but one, eye position was monitored using an ASLeye-tracking system with remote optics that was custom-adapted foruse in the scanner (Applied Science Laboratories, Bedford, MA;Model 504, sampling rate � 60 Hz). Eye-position traces were exam-ined in a 14.5-s time window, beginning with the cue onset until theend of the trial. For each subject the eye-tracking system was cali-brated before fMRI scanning. The subject performed a series ofsaccades from central fixation to the left and to the right LEDs,visiting each lateral position twice (duration of each fixation � 1,500ms). The calibration served to convert the (arbitrary) units of theeye-tracking system to degrees of visual angle. The eye-position dataacquired during fMRI scanning were baseline adjusted using theaverage horizontal position in a 500-ms precue window. We identifiedlosses of fixation as changes in horizontal eye-position �2° withdurations �100 ms. To ensure that trials containing inappropriatelosses of fixation would not affect our imaging results, these weremodeled separately in the imaging data analysis (see Data analysis).After removal of those trials, no systematic gaze deviation could befound in any of the periods requiring maintenance of central fixation.For trials requiring saccadic responses, the first derivative of the eyeposition (eye velocity) was used to compute saccadic reaction time(i.e., here defined as the time between the auditory go signal and eyevelocity exceeding 100°/s; see Fig. 1B).

R E S U L T S

Behavioral data

Overt responses were measured throughout each scanningsession using an eye-tracking system with remote optics (seepreceding text), plus electronic switches on each side to recordmanual button presses. The percent of misses was 5% in themanual task and 2% in the saccade task. Subjects neverresponded with a button press on saccade trials, but in themanual task, we detected losses of fixation after target onset on18% of the trials. All error trials were excluded from the fMRIanalyses. A within-subject ANOVA with cue modality, move-ment type, and target side as independent factors was used tocompare the reaction times in the eight conditions. This re-vealed a main effect of type of movement [F(1,7) � 49.1; P �0.001] with saccades yielding faster reaction times [422 � 36(SE) ms for saccades and 567 � 23 ms for button presses]. Themain effect of side and the side-by-movement interactionalmost reached significance (P � 0.058 and 0.051, respec-tively) due to faster responses with the right than the left hand[539 � 29 vs. 595 � 22 ms; t(7) � 2.6; P � 0.036], especially

on button-press trial as might be expected in these right-handedsubjects. Overall the reaction times were quite slow (butsimilar to Macaluso et al. 2003a, who used a similar experi-mental setup). A possible reason for this is because of thedemanding fMRI environment in which subjects had to per-form the tasks (i.e., a noisy surrounding where subjects have tolay still for �1 h, including setup time). However, any sucheffect of the environment should presumably affect all condi-tions similarly, thus having no specific influence on our fMRIresults.

fMRI results: sensory-cue-phase and movement-phase

SPM analyses of the sensory-cue phase highlighted sensorycircuits for touch or vision, plus any “common” multisensoryactivation (see Fig. 2A, and Table 1); whereas fMRI analysesof the movement phase revealed motor circuits involved in eyeor hand movement and effector-independent responses (Fig.2B and Table 2).

Analyses of brain activity associated with the sensory cuesrevealed “common” regions responding to both visual andtactile cues in and around the intraparietal sulcus and premotorregions, plus the occipito-temporal junction (see Fig. 2A,magenta). Selective responses for the visual cues were found inventral, lateral, and dorsal occipital cortex, plus posteriorparietal cortex (see Fig. 2A, in cyan), as would be expected. Fortactile cues, we found activation of the postcentral gyrus plusparietal operculum (see Fig. 2A, in yellow), again as expected.

Within both “bimodal” regions (activated in common byboth vision and touch) and “unimodal” (visual or tactile spe-cific), we then tested for any effect of stimulus position (leftminus right and vice versa; see Table 1, rightmost columns).The bimodal regions in frontal, parietal and posterior temporalcortex did not show any significant effect of stimulus positionat our whole-brain corrected threshold. But for completeness,we examined the anterior intraparietal sulcus (aIPS) regionpreviously reported by Macaluso et al. (2003a) as showing acontralateral response that generalized across vision and touch.We considered a spherical region of interest (10 mm radius)centered at the coordinates reported in that study (left hemi-sphere: x, y, z � �38, �36, 54; right hemisphere: x, y, z ��48, �32, 46). This confirmed a significant (P corrected �0.05) main effect of contralateral minus ipsilateral cue side,regardless of cue modality (see also Table 1).

For areas responding selectively for visual cues, greaterresponses for contralateral minus ipsilateral stimulation werefound within the occipital lobe (i.e., in ventral, lateral, anddorsal occipital cortex; see Table 1). The visually selectiveposterior parietal cortex did not show any lateralized effect inour study. A possible reason for this is that—unlike severalstudies that did find some spatiotopic responses in high-ordervisual areas—here we used small low-contrast and brief visualstimuli to signal the relevant position rather than the intensiveperiodic stimulation required for phase-mapping techniques(cf. Hagler and Sereno 2006; Schluppeck et al. 2005). Fortactile cues, contralateral effects were found both in the post-central gyrus and in the parietal operculum as expected.

We highlighted motor circuits by considering activity afterthe auditory “go signal” that prompted movement execution(see Fig. 2B and Table 2). All movement conditions activatedthe medial frontal cortex and superior premotor areas (Fig. 2B,



in blue). In addition, regions along the superior temporal gyruswere activated but probably as an auditory response to the gosignal (common for all movement conditions). Saccadic eyemovements resulted in selective activation of posterior parietalcortex and superior premotor regions. The peaks of the dorsalpremotor activation were located on the lateral side of theprecentral sulcus, thus in a somewhat more lateral location thanthe classical localization of the frontal eye fields (e.g., see Petitet al. 1997, who reported peak activation in the depth of thesulcus but with the activation cluster also extending to thelateral surface as here). Nonetheless, the current finding isconsistent with our previous study that also directly comparedeye versus hand movements revealing nearly identical coordi-nates (Macaluso et al. 2003a). The entirety of occipital cortex,including visual cortex in the calcarine fissure, also showedincreased activity during eye movements, presumably as aconsequence of the changes of retinal input due to saccadicexecution (see Fig. 2B, in red). Hand movements resulted inthe activation of peri-central regions, extending posteriorly tothe anterior part of the superior parietal lobule (see Fig. 2B, ingreen). In addition we also found increased activity in thesupplementary motor areas (partially overlapping with theregion showing activation for both types of movements, cf.transverse section in Fig. 2B), in the temporo-occipital junctionand in the parietal operculum (see Table 2).

Analogously to the sensory cue phase, we then tested themotor response phase data for any effect of movement side,

comparing leftward minus rightward trials and vice versa (seeTable 2, rightmost columns). For hand responses, this showedactivation in the precentral gyrus with greater activity forbutton presses using the contralateral versus ipsilateral hand aswould be expected in primary motor regions. In addition, handresponses activated also the anterior parietal cortex, parietaloperculum, SMA, and temporo-occipital junction. Fronto-pa-rietal regions that activated selectively for eye movements didnot show any significant effect of the target side, possiblybecause each saccade to the target was followed by a saccadein the opposite direction, re-orienting gaze to the centralfixation point (see also Macaluso et al. 2003a). Primary visualcortex in the calcarine fissure showed greater activity foripsilateral than contralateral saccades, possibly due to changesof retinal input during the eye movements. Auditory andmedial frontal regions that activated in common for both handand eye movements (see preceding text) did not show anyeffect of movement side.

In summary, analyses of brain activity in the sensory cuephase and in the movement phases revealed the expectedeffects according to the sensory stimulation (Fig. 2A and Table1) or the type of movement (Fig. 2B and Table 2). We foundunimodal visual, unimodal tactile, and bimodal visuo-tactileactivations during the sensory cue phase and effector-specificplus some effector-independent activations in the movement-phase. Several of the sensory and motor regions also showed

FIG. 2. Cue- and target-related responses.A: brain areas activating in the cue-phase arerendered on 3-dimensional (3D) views and atransverse section of the SPM template. Ac-tivations are divided according to whetherthe response was modality-specific (visual incyan; tactile in yellow) or was “bimodal”(i.e., common for vision and touch; in ma-genta). B: regions responding in the targetphase. Activations are displayed in differentcolors according to the specificity of theresponses: effector specific (saccade in red;manual in green) vs. common responses inblue. It should be noted that an auditorysignal was used as a go signal for the motorresponse, thereby yielding also bilateral ac-tivation of the auditory cortex in common foreye and hand movements. All statisticalmaps are thresholded at P corr � 0.05 (clus-ter level), plus additional constraints to en-suring “specificity” or “commonality” for therelevant effect (see METHODS) with the ex-ception of the map relating to the saccade-specific response (B, in red; thresholded atP uncorr � 0.001 for display purposes, tofully show the bilateral activation of dorsalpremotor cortex).



some preference for stimulation or movement for the contralat-eral side (see Tables 1 and 2, rightmost columns).

fMRI results: during the critical delay-phase

The central aim of the current study was to investigateactivity in the cue to movement time interval (delay phase) andto assess the role of the cue modality and/or movement type forsuch delay activity. Importantly, we found no significant dif-ferences between delay activity following a visual or a tactilecue. By contrast, the type of movement that was prepareddramatically influenced brain activity in the delay phase. Thusthe type of upcoming movement but not a sensory-specificlocation appeared to be maintained during the delay.

Preparing to perform an eye movement toward the cuedlocation resulted in selective activation of the posterior parietalcortex, superior premotor cortex and ventral occipital visualareas (see Figs. 3 and 4A). The signal plots in Figs. 3 and 4show the level of activation in posterior parietal cortex (Fig.3A), superior premotor cortex (Fig. 3B), and ventral occipitalcortex (Fig. 4A) for the three trial phases [cue (C), delay (D),and target (T)] as a function of the cue-modality (touch orvision), motor effector (hands or eyes), and side (left or right).Figure 3A shows that in posterior intraparietal cortex, activityincreased selectively during preparation for an eye movement,irrespective of cue modality (see red bars). There was atendency for intraparietal activity to be higher for saccadictargets on the contralateral side compared with ipsilateral side.

This reached statistical significance for a region in the righthemisphere, but the left hemisphere did not reach full correctedstatistical significance and so is reported here for completenessonly (see also Table 3, 1st section, rightmost columns) to avoidany misleading impression of strong hemispheric asymmetriesin intraparietal cortex. In the superior premotor cortex, we didnot observe such a tendency for contralateral activity levels tobe larger than ipsilateral (see signal plots in Fig. 3B).

Figure 4A shows activity in ventral occipital regions wherethe effect of saccade target position was stronger and delayactivity was observed only when subjects prepared for aneye-movement to a contralateral location (see red bars, in Fig.4A and Table 3, 1st section). Note that unlike the ipsilateralactivity observed during saccade execution (see Table 2), herevisual areas activated contralaterally during saccade prepara-tion (i.e., specifically during the delay, prior to an actualsaccade). Hence this preparatory activity is unlikely to repre-sent changes of retinal input due to small eye movements thatsurvived our exclusion criteria (see METHODS) but may insteadreflect visual enhancement of the target location for an upcom-ing saccade. Critically, we thus show a modulation of activityin visual cortex that was selective for the type of upcomingmovement (oculomotor vs. manual) but that generalized acrossboth visual and tactile cues.

Analyses of delay activity preceding button-press responses(hand-movement trials) revealed selective activation of motorregions in the precentral gyrus, plus supplementary motor areas

TABLE 1. Cue-related activity (sensory responses)

Main Effects Spatial Effects

Coordinates z Score P-Corr. Coordinates z Score P-Corr.

Visual responsesL posterior parietal cortex �16 �76 52 5.7 �0.001 — — —R posterior parietal cortex 34 �62 50 6.7 �0.001 — — —L lateral occipital cortex �48 �80 0 7.1 �0.001 �46 �86 �2 5.3 �0.001R lateral occipital cortex 52 �64 2 7.6 �0.001 50 �72 4 4.2 0.008L ventral occipital cortex �42 �76 �20 6.7 �0.001 �26 �80 �18 6.1 �0.001R ventral occipital cortex 44 �56 �22 5.3 0.002 26 �68 �18 3.6 0.084L dorsal occipital cortex �26 �78 22 5.1 0.005 �24 �84 32 4.5 0.002R dorsal occipital cortex 32 �76 26 5.9 �0.001 26 �82 22 3.3 0.158

Tactile responsesL post-central gyrus �48 �36 46 5.6 �0.001 �53 �26 54 4.3 0.005R post-central gyrus 60 �20 46 4.5 0.073 50 �20 60 6.0 �0.001L parietal operculum �62 �22 18 7.1 �0.001 �40 �20 8 4.3 0.005R parietal operculum 56 �20 12 7.0 �0.001 46 �18 14 4.4 0.004

Common responses for vision and touchL occipito-temporal junct. �54 �74 25 7.5 �0.001 — — —R occipito-temporal junct. 54 �64 0 7.8 �0.001 — — —L posterior parietal cortex �32 �58 50 7.5 �0.001 — — —R posterior parietal cortex 34 �56 46 6.2 �0.001 — — —L superior premotor cortex �52 0 44 5.6 0.001 — — —R superior premotor cortex 44 0 56 7.3 �0.001 — — —Pre-SMA 2 10 56 6.2 �0.001 — — —L anterior IPS* — — — �44 �24 50 3.7 0.007R anterior IPS* — — — 38 �26 54 4.0 0.003

Cue-related activity (sensory responses). Main effects: anatomical location, peak coordinates, z values, and corrected P values for regions activated in the cuephase. Areas are divided according to whether the activation was modality-specific (visual, tactile) or bimodal (i.e. common for vision and touch, last section).P values are corrected for multiple comparisons considering the whole brain as the volume of interest. Spatial effects: within areas responding in the cue phase,we assessed any effect of stimulus-side comparing left cues vs. right cues and vice-versa. All areas with a significant effect of side showed greater activity forstimulation of the contralateral versus ipsilateral hemifield. For these side- specific effects, P values are corrected for multiple comparisons considering all voxelsshowing a main effect of the stimulation (modality-specific or bimodal), as the volumes of interest. *We also tested for side-specific responses in the anteriorintraparietal sulcus, now using a priori regions-of-interest, taken from prior work, to assign corrected P values (see METHOD). This highlighted contralateralresponses, irrespective of cue modality. SMA, supplementary motor area; IPS, intraparietal area.



(see Fig. 4B and Table 3, 2nd section). The medial activationdid not show any selectivity according to relevant side,whereas the precentral regions showed greater activity forpreparation of movement with the contralateral hand (seesignal plots in Fig. 4B, green bars) as would be expected for amanual motor-plan.

Finally, we tested for any brain regions showing increasedpreparatory activity irrespective of both the modality of the cueand the type of upcoming motor response. This revealed robustbilateral activation in the posterior part of the superior temporalsulcus (see Fig. 5 and Table 3, 3rd section). Here activityincreased during the delay phase of all trial types, respondingalso here irrespective of target side (see cyan bars in Fig. 5).

D I S C U S S I O N

Spatial orienting to external events or objects often requirescombination of signals from different modalities (as when wecan both see and feel an object) and also coordination ofdifferent effector systems. Here we asked whether encoding ofa relevant position during a delay period follows primarily thesensory modality signaling the location or the motor systemused for the upcoming lateralized motor response (or someinteraction of these). We examined brain activity in the intervalbetween sensory cue and delayed motor response (delay activ-ity) in the absence of any current sensory stimulation or motorresponse. We found that delay activity clearly reflected thetype of upcoming movement rather than the sensory modality

of the cue stimulus. Activity increased in contralateral precen-tral cortex when subjects prepared for a button press with eitherthe left or the right hand, irrespective of whether the left/rightposition had been cued with a visual or a tactile stimulus.When subjects prepared for an eye movement instead, activityincreased in the posterior parietal cortex, dorsal premotorcortex, and ventral occipital cortex, again irrespective of themodality of the sensory cue. In occipital cortex, the delayactivity was strongly lateralized to the hemisphere contralateralto the relevant location, whereas posterior parietal cortexactivated for both contra- and for ipsilateral locations but withlarger effects during the preparation of saccades to the con-tralateral hemifield. In addition, a region in the superior tem-poral sulcus activated during the delay, irrespective of cuemodality, relevant side, and type of movement. Our findingssuggest that during a delay period intervening between astimulus cue to target location and a later motor response tothat location, the relevant location is coded according to themovement type rather than the cue modality, and—critically—that these effector-specific areas can be cued by spatiallyspecific signals from different sensory modalities.

Our finding of effector-dependent responses during the cue-to-movement delay is in general agreement with previouselectrophysiological studies in non-human primates (Andersenet al. 1997; Barash et al. 1991; Cohen et al. 2002; Kalaska et al.1997; Snyder et al. 2000) and with more recent fMRI experi-ments in humans (Astafiev et al. 2003; Bremmer et al. 2001;

TABLE 2. Target-related activity (motor responses)



Eye movementsL posterior parietal cortex �26 �62 52 6.1 �0.001 — — —R posterior parietal cortex 28 �62 56 5.1 0.006 — — —L superior premotor cortex �52 �4 46 4.4 0.102 — — —R superior premotor cortex 52 0 46 6.4 �0.001 — — —L Occipital cortex* �10 �78 4 �8 �0.001 �10 �76 �12 4.7 0.004R Occipital cortex* 14 �72 6 �8 �0.001 14 �72 �12 4.0 0.054

Hand movementsR pre-central gyrus 34 �24 72 7.1 �0.001 36 �28 72 �8 �0.001L pre-central gyrus �28 �14 66 �8 �0.001

�34 �26 64 �8 �0.001L anterior parietal cortex �32 �44 66 6.3 �0.001R anterior parietal cortex 36 �42 64 5.3 0.002 30 �44 68 7.0 �0.001L parietal operculum �52 �24 12 �8 �0.001 �36 �22 12 3.6 0.132R parietal operculum 62 �24 12 �8 �0.001 44 �20 14 5.8 �0.001L occipito-temporal junct. �52 �76 2 6.9 �0.001 �52 �74 10 4.1 0.027R occipito-temporal junct. 56 �72 �4 6.3 �0.001 �50 �70 �6 4.0 0.038

SMA �2 �4 46 �8 �0.001�6 �20 48 3.6 0.152

8 �10 44 5.3 �0.001Common responses for eye and hand movements

L superior temporal cortex �54 �22 4 5.3 0.036 — — —R superior temporal cortex 64 �26 10 7.2 �0.001 — — —SMA 2 �4 56 6.3 �0.001 — — —L superior premotor cortex �40 �10 60 5.8 �0.001 — — —R superior premotor cortex 46 �10 60 6.0 �0.001 — — —L lateral occipital cortex �50 �82 2 7.1 �0.001 — — —

Target-related activity (motor responses). Main effects: anatomical location, peak coordinates, z values, and corrected P values for regions activated in themovement- phase. Areas are divided according to whether the activation was effector-specific (eye movement; hand movement) or it generalized across effectors(i.e. common activation for eye and hand movements). P values are corrected for multiple comparisons considering the whole brain as the volume of interest.Spatial effects: within areas responding in the movement phase, we tested for any effect of movement-direction (left vs. right and vice- versa). P values arecorrected for multiple comparisons considering all voxels showing a main effect of the movement (effector specific or effector independent), as the volumes ofinterest. whereas for hand movements all side-specific effects were contralateral to the responding hand, during saccade execution the side-effects in visual cortex(*) were ipsilateral to the movement/target direction.



Culham and Valyear 2006; Desouza et al. 2000; Grefkes et al.2004; Kawashima et al. 1994; Prado et al. 2005; Schluppecket al. 2005; Simon et al. 2002). We found preparatory activityin the posterior parietal cortex that was selective for eyemovements. This activation was located in the posterior part ofthe superior parietal gyrus, in an area that may correspond tothe IPS2 region reported by Schluppeck and colleagues (2005),who showed some retinotopic organization there in a detailedstudy using an overt saccade task. In a later study (Schluppecket al. 2006), the same group reported that this area is activeduring saccade preparation with greater activity during prepa-ration of contralateral versus ipsilateral saccades. In the presentstudy, we demonstrate that preparatory activity in this posteriorparietal region is selective for eye movement when comparingeye- versus hand-movement preparation directly for the firsttime in a delay paradigm using both visual and tactile cues. Asin previous studies (Schluppeck et al. 2006; Sereno et al.2001), we also found larger activity during preparation ofcontralateral than ipsilateral saccades, albeit only the righthemisphere reached full statistical significance. A possiblereason for the relatively weak lateralization within posterior

parietal cortex in the current study might be that here we didnot use a phase and/or polar-angle mapping approach thatrequires predictable and extended sequences of saccades(Schluppeck et al. 2005; Sereno et al. 2001). But instead, wedid introduce here for the first time both visual and tactilelateralized cue stimuli to show that either of these can triggereffector-specific preparatory activity in posterior IPS. Thusspatial information from different sensory modalities seems toconverge on posterior parietal cortex that exhibit sustainedactivation selectively while subjects prepare for a saccadic eyemovement.

Together with the activation of the posterior parietal cortex,we also found saccade-specific preparatory activity in dorsalpremotor cortex and in ventral occipital cortex. In premotorcortex, the activity increased irrespective of planned saccadedirection, whereas the occipital activation was contralateral tothe side of the up-coming saccade. The finding of effector-specific modulation within visual occipital cortex suggests alink between motor preparation and visual processing as pre-dicted by premotor theories of spatial selection (Rizzolatti et al.1987). Recent micro-stimulation experiments in monkeys

FIG. 3. Activations in the delay-phase specific for the preparation of eye-movement responses, irrespective of target side. While subjects prepared to performan eye movement, activity increased in the posterior parietal cortex (A) and in the superior premotor cortex (B). Critically, these preparatory effects occurredirrespective of the modality of the cue (visual or tactile), indicating that sensory signals from both modalities can trigger sustained preparatory activity withinthese areas. Premotor cortex activated during preparation of both contra- and ipsilateral saccades. Parietal cortex also activated irrespective of target side, albeitwith stronger responses during preparation of contralateral saccades. The signal plots show activity for all sensory-motor combinations (crossing sensory-modality of the cue with effector type), for the 3 trial-phases (C, cue; D, delay; T, target/response), and separately for left and right hemifield trials. Bars in redhighlight the critical bimodal (common to visual and tactile) but effector-specific activation in the delay phase. Activations are rendered on anatomical sectionsat a threshold of P uncorr. � 0.001.



found that stimulation of oculomotor control structures canchange the response proprieties of single neurons in occipitalvisual cortex (Armstrong et al. 2006; Tolias et al. 2001). Suchinteractions between anatomically distant neural populationsemphasizes the potential for interplay between higher associa-tive regions and lower sensory areas during eye-movementguidance and attention selection (Moore 2006; see alsoAstafiev et al. 2003). Here, we show effector- and side-specificsustained activity in human occipital cortex that also general-ized across the sensory modality of the cue for the preparedsaccade. The finding of these spatially specific crossmodaleffects is consistent with proposals that spatial informationfrom the somatosensory system can reach and modulate spatialrepresentations within visual occipital cortex, possibly viahigher-order control structures in frontal and parietal cortex(Macaluso and Driver 2005; Macaluso et al. 2002), such asthose involved in saccade preparation.

Unlike most previous studies on preparatory activity forsaccadic eye movements (Desouza et al. 2000; Schluppecket al. 2006; Sereno et al. 2001), our paradigm included not onlyleft/right saccades, but also left/right manual button presses

(see also Connolly et al. 2000, 2007; Simon et al. 2002 forstudies that also used more than one effector-system in awithin-subject fMRI design). Due to this, we could show herethat our occipital, premotor, and posterior parietal findings forsaccade preparation are truly effector specific (specific tooculomotor preparation) and cannot be accounted for by anymore general process common to the two spatial-motor tasks(such as general shifts of spatial attention). Nonetheless, weshould note that our manual button-press task differed from thepointing/reaching tasks used in most electrophysiological stud-ies to date (Batista and Andersen 2001; Calton et al. 2002;Johnson et al. 1996; Snyder et al. 1997) and in some neuro-imaging studies (Connolly et al. 2003; Desouza et al. 2000;Prado et al. 2005). We used a simple button-press task withthe left or the right hand (see also Macaluso et al. 2003a).One potentially important difference between traditionalpointing tasks and the current button-press task is that herewe did not dissociate the left/right location of the manualresponse from the selection of one or other hand (i.e., herethe left hand always pressed the left button, and the righthand pressed the right button, within their respective hemis-

FIG. 4. Activations in the delay phase specific for the preparation of eye- or hand-movements, and contralateral to the target side. A: eye-movementpreparatory activity in occipital visual cortex was strongly lateralized to the hemisphere contralateral to the location of the impending eye movement. This sidespecific effect was observed irrespective of the sensory modality of the cue (touch or vision). B: hand-movement preparation was associated with increasedactivation in the contralateral precentral cortex, again irrespective of the cue modality (touch or vision). The signal plots show activity for all sensory-motorcombinations (crossing sensory modality of the cue with effector type), for the 3 trial phases (C, cue; D, delay; T, target/response), and separately for left andright hemifield trials. Bars in red highlight the activation that was selective for preparation of a contralateral eye response, while bars in green highlightcontralateral hand-responses. Activations are rendered on anatomical sections at a threshold of P uncorr. � 0.001 for display purposes.



paces). On hand trials, the sensory cue thus signaled bothrelevant location and relevant hand. Several previous stud-ies addressed the issue of location versus hand/limb selec-tion in manual responses (Beurze et al. 2007; Medendorp etal. 2005a). Double cueing studies can separate in timesignals relating to the selection of the desired end positionand of the responding effector. For example, Beurze andcolleagues used a pointing task with the left or the righthand and found that the posterior parietal cortex and dorsalpremotor cortex participate in selection of both the desiredend location and effector (Beurze et al. 2007). Maximalactivity was observed when both components were speci-fied, i.e., in correspondence with the final, second cue.Furthermore, the IPS showed lateralized responses (con-tralateral greater than ipsilateral) in respect of both theresponding hand and the target location, whereas in precen-

tral cortex (M1) activity reflected effector selection only.Medendorp and colleagues obtained similar results in afMRI experiment, where subjects either pointed with the leftor the right hand, or they saccaded toward lateralized visualtargets (Medendorp et al. 2005a). A retinotopic region of IPScombined responding hand and target side during the pointingtask, whereas M1 activated for movements of the contralateralhand, irrespective of target side. Given these findings, it maybe that effector (left or right hand) may have contributed morethan external position for the preparatory activity in precentralgyrus during the manual task (see Fig. 4B). Future work mightfurther address spatial issues (concerning the respective rolesof target external position, effector side, movement direction,etc.) by involving crossed-reaching responses to visual ortactile targets, thus dissociating not only sensory versus motorpreparation, as here, but also investigating the relative contri-

FIG. 5. Activations in the delay-phase common to the preparation of eye and hand movements. In the posterior part of the superior temporal sulcus, activityincreased during the delay phase, irrespective of both cue modality (touch or vision) and movement effector (eyes or hands). The signal plots show activity forall sensory-motor combinations (crossing sensory modality of the cue with effector type), for the 3 trial phases (C, cue; D, delay; T, target/response) andseparately for left and right hemifield trials. The bars in cyan highlight the bimodal, effector- and side-independent activation in the delay-phase. Activations arerendered on anatomical sections at a threshold of P corr. � 0.05 (cluster level).

TABLE 3. Delay activity (preparatory activity)



Delay activity specific for eye responsesL superior premotor cortex �48 �10 42 4.9 0.014 — — —R superior premotor cortex 58 0 44 4.0 0.403 — — —L posterior parietal cortex �22 �64 56 4.6 0.036 �12 �80 52 2.6 n.sR posterior parietal cortex 24 �62 58 4.4 0.120 22 �66 46 3.8 0.011L ventral occipital cortex �14 �72 0 4.9 0.013 �18 �80 �10 4.4 0.001R ventral occipital cortex 12 �62 �8 4.1 0.273 30 �72 �16 3.7 n.v.

Delay activity specific for hand responsesL pre-central gyrus �26 �28 64 5.7 �0.001 �36 �28 60 3.3 n.v.R pre-central gyrus 30 �26 58 4.6 0.056 34 �20 58 4.4 �0.001

Delay activity independent of cue-modality and motor-effectorL superior temporal sulcus �58 �28 2 7.6 �0.001 — — —R superior temporal sulcus 68 �22 �2 �8 �0.001 — — —

Delay-related activity (preparatory activity). Main effects: anatomical location, peak coordinates, z values, and corrected P values for regions activating duringthe delay phase. We could distinguish between areas showing selective preparatory activity depending on the type of upcoming movement (preparation for aneye movement; preparation for a hand movement); plus areas showing sustained activity irrespective of effector (last section). Critically, all areas activatedirrespective of the modality (vision or touch) that was used to cue the relevant position. P values are corrected for multiple comparisons considering the wholebrain as the volume of interest. Spatial effects: within areas responding during the delay phase, we tested for any effect of relevant location (left vs. right andvice-versa). P values are corrected for multiple comparisons considering all voxels showing effector-specific or effector-independent preparatory activity, as thevolumes of interest. For the right ventral occipital cortex and the left pre-central gyrus, the side-specific effects did not overlap with the main effects (n.v.: notin volume of interest), hence we do not report corrected P values. (n.s.: not significant).



bution of each potential spatial framework to each stage of thesensory-motor spatial translation.

Within our current paradigm, it might be even argued thatthe sensory cues acted as “symbolic” signals instructing sub-jects to move one or the other hand (somewhat similar topresenting verbal cues: “move left” or “move right”) or tosaccade left or right. But although it is indeed possible thatsubjects converted the exogenous lateralized sensory-cue sig-nals into some other form of representation and then issued anendogenous motor command on presentation of the auditory gosignal, we note that in fact we found strong motor-specificactivations during the delay period. Moreover, prior to obtain-ing our actual results, it was entirely possible that subjectsmight maintain an internal representation of the relevant loca-tion in sensory rather than motor terms. Thus in principle,subjects could process the cue and then select a sensoryrepresentation endogenously to represent the critical locationduring the delay (e.g., simply “attending” to the relevant visualor tactile location during the delay). Our results show deci-sively that this does not happen; instead, the activity duringdelay reflects only the motor requirements.

Concerning the IPS, Medendorp et al. (2005) also reportedgreater activity during saccades than pointing, but note thattheir study did not explicitly separate cue, delay and movementphases unlike here. Delay activity during pointing or saccadeswas recently investigated in a fMRI study that dissociatedposition versus effector cuing (Connolly et al. 2007). Thisrevealed increased activity in IPS, even when the instructioncue concerned only the relevant effector system (i.e., notspecifying relevant location). IPS activation was observed bothfor saccadic and pointing preparation but with greater effectsfor saccade than pointing. Our IPS findings during the delayphase (larger activation for saccade task versus manual task, seeFig. 3A) appear consistent with Medendorp’s and Connolly’sfindings (Connolly et al. 2007; Medendorp et al. 2005a),although our button-press task differs from previous pointingtasks as noted earlier. For present purposes, an advantage of thebutton-press task was that it allowed us to present tactile cuestimuli on each side (to the index-finger) at exactly the samelocation as the possible visual stimuli and that always requiredspatially compatible motor responses to the correspondinglocation. Pointing or reaching tasks would have precluded thepossibility of using tactile cues in the same way, which wascritical for the current aim of assessing the possible role ofcue-modality (as well as movement type) during the cue-to-movement delay phase. But these differences in the type ofhand-motor responses between the current study and previouspointing/reaching tasks may explain also why here we foundpreparatory activity in precentral cortex but not within theparietal cortex itself for hand movements. A possible alterna-tive design could have used tactile cues on the left/right handto instruct leftward/rightward pointing movement. Althoughthis design might reveal tactually triggered pointing activity inparietal cortex, it would have required placing visual andtactile cue stimuli at different positions or using different targetlocations for saccade and pointing tasks.

Together with our effector-specific findings (i.e., posteriorparietal, dorsal premotor plus ventral occipital cortex for sac-cade preparation vs. precentral cortex for button-press prepa-ration), we also found sustained delay activity in the superiortemporal sulcus, but now irrespective of movement type. Also

in this region, the delay response was observed irrespective ofthe modality that had cued the relevant location. Severalprevious studies implicated the STS in spatial covert orienting(Macaluso et al. 2001; Yamaguchi et al. 1994; see also Luhet al. 1986 for related studies in non-human primates). UsingPET, we previously reported STS activation when subjectssustained attention either to the left or to the right hemifield(Macaluso et al. 2001) and—as in the current study—irrespective of whether vision or touch was used to definethe currently relevant position. Our present results of sus-tained activity in the delay phase thus appear consistent withthe interpretation that STS may be involved in committingspatial attention to a given spatial location prior to the gosignal for executing a movement, and that this process canhappen supramodally.

Although the primary aim of the current study was toinvestigate preparatory activity in the absence of any sensorystimulation or motor response, we also examined brain activityin the cue phase and the movement phase to highlight patternsof sensory (cue phase) and motor (movement phase) activity.For the movement phase, we observed the expected activationof the contralateral precentral gyrus for button press responses,and posterior parietal plus dorsal premotor regions (FEF) forsaccade execution. In the cue phase, we could discriminatebetween “unimodal” (i.e., specific just for vision or touch)versus bimodal sensory activations. This revealed the expectedunimodal effects in occipital cortex for vision versus postcen-tral gyrus plus parietal operculum for touch. Bimodal re-sponses were observed at the occipito-temporal junction, pos-terior parietal cortex, and premotor regions. Several of theseregions are likely to contain neurons responding to more thanone modality (Bruce et al. 1981; Colby and Duhamel 1991;Graziano et al. 1994), although more general attentional ortask-related processes might also explain some “common,bimodal” responses in fronto-parietal regions (Corbetta andShulman 2002; Gitelman et al. 1999; Mesulam 1998; Yantiset al. 2002). Further, we also replicated (albeit only at a smallvolume corrected level) our previous observation of bimodalbut spatially specific (contralateral) responses in the anteriorparietal sulcus. We had previously reported bimodal effects inthis region for both covert (Macaluso et al. 2002, 2003b) andovert (Macaluso et al. 2003a) spatial orienting, but here wecould show that multisensory effects in anterior IPS relate tothe sensory stimulation or transient covert orienting rather thanto more sustained attentional or motor preparation (because thepresent aIPS result was specific to the sensory-cue phase notthe delay or motor phase).

In summary, sustained delay activity while subjects preparedfor an overt response toward a cued location was found toreflect primarily motor preparation rather than sensory orattentional encoding of the relevant location. We found sepa-rate, effector-specific responses in posterior parietal cortex,frontal-eye-field, and ventral occipital cortex for saccade prep-aration versus contralateral precentral cortex for hand re-sponses. Our findings are in general agreement with single-cell work in monkeys by indicating that sensory input istransformed into nascent motor plans during the delay.Critically, here we show that this generalizes across sensorymodalities with spatial signals from vision and touch feed-ing into effector-specific motor representations in the humanbrain.



R E F E R E N C E S

Andersen RA, Snyder LH, Bradley DC, Xing J. Multimodal representationof space in the posterior parietal cortex and its use in planning movements.Annu Rev Neurosci 20: 303–330, 1997.

Armstrong KM, Fitzgerald JK, Moore T. Changes in visual receptive fieldswith microstimulation of frontal cortex. Neuron 50: 791–798, 2006.

Astafiev SV, Shulman GL, Stanley CM, Snyder AZ, Van E, Corbetta M.Functional organization of human intraparietal and frontal cortex for attend-ing, looking, and pointing. J Neurosci 23: 4689–4699, 2003.

Barash S, Bracewell RM, Fogassi L, Gnadt JW, Andersen RA. Saccade-related activity in the lateral intraparietal area. II. Spatial properties. J Neu-rophysiol 66: 1109–1124, 1991.

Batista AP, Andersen RA. The parietal reach region codes the next plannedmovement in a sequential reach task. J Neurophysiol 85: 539–544, 2001.

Ben Hamed S, Duhamel JR, Bremmer F, Graf W. Representation of thevisual field in the lateral intraparietal area of macaque monkeys: a quanti-tative receptive field analysis. Exp Brain Res 140: 127–144, 2001.

Beurze SM, De Lange FP, Toni I, Medendorp WP. Integration of target andeffector information in the human brain during reach planning. J Neuro-physiol 97: 188–199, 2007.

Bremmer F, Schlack A, Shah NJ, Zafiris O, Kubischik M, Hoffmann K,Zilles K, Fink GR. Polymodal motion processing in posterior parietal andpremotor cortex: a human fMRI study strongly implies equivalencies be-tween humans and monkeys. Neuron 29: 287–296, 2001.

Bruce C, Desimone R, Gross CG. Visual properties of neurons in a polysen-sory area in superior temporal sulcus of the macaque. J Neurophysiol 46:369–384, 1981.

Calton JL, Dickinson AR, Snyder LH. Non-spatial, motor-specific activationin posterior parietal cortex. Nat Neurosci 5: 580–588, 2002.

Cohen YE, Andersen RA. Reaches to sounds encoded in an eye-centeredreference frame. Neuron 27: 647–652, 2000.

Cohen YE, Batista AP, Andersen RA. Comparison of neural activity pre-ceding reaches to auditory and visual stimuli in the parietal reach region.Neuroreport 13: 891–894, 2002.

Colby CL, Duhamel JR. Heterogeneity of extrastriate visual areas andmultiple parietal areas in the macaque monkey. Neuropsychologia 29:517–537, 1991.

Colby CL, Duhamel JR. Spatial representations for action in parietal cortex.Cog Brain Res 5: 105–115, 1996.

Colby CL, Duhamel JR, Goldberg ME. Visual, presaccadic, and cognitiveactivation of single neurons in monkey lateral intraparietal area. J Neuro-physiol 76: 2841–2852, 1996.

Colby CL, Goldberg ME. Space and attention in parietal cortex. Annu RevNeurosci 22: 319–349, 1999.

Connolly JD, Andersen RA, Goodale MA. FMRI evidence for a “parietalreach region” in the human brain. Exp Brain Res 153: 140–145, 2003.

Connolly JD, Goodale MA, Cant JS, Munoz DP. Effector-specific fields formotor preparation in the human frontal cortex. Neuroimage 34: 1209–1219,2007.

Connolly JD, Goodale MA, Desouza JF, Menon RS, Vilis T. A comparisonof frontoparietal fMRI activation during anti-saccades and anti-pointing.J Neurophysiol 84: 1645–1655, 2000.

Corbetta M, Shulman GL. Control of goal-directed and stimulus-drivenattention in the brain. Nat Rev Neurosci 3: 215–229, 2002.

Culham JC, Valyear KF. Human parietal cortex in action. Curr OpinNeurobiol 16: 205–212, 2006.

Desouza JF, Dukelow SP, Gati JS, Menon RS, Andersen RA, Vilis T. Eyeposition signal modulates a human parietal pointing region during memory-guided movements. J Neurosci 20: 5835–5840, 2000.

Duhamel JR, Colby CL, Goldberg ME. The updating of the representationof visual space in parietal cortex by intended eye movements. Science 255:90–92, 1992.

Friston KJ, Glaser DE, Henson RN, Kiebel S, Phillips C, Ashburner J.Classical and Bayesian inference in neuroimaging: applications. Neuroim-age 16: 484–512, 2002.

Friston KJ, Holmes AP, Price CJ, Buchel C, Worsley KJ. MultisubjectfMRI studies and conjunction analyses. Neuroimage 10: 385–396, 1999.

Gitelman DR, Nobre AC, Parrish TB, LaBar KS, Kim YH, Meyer JR,Mesulam M. A large-scale distributed network for covert spatial attention:further anatomical delineation based on stringent behavioural and cognitivecontrols. Brain 122: 1093–1106, 1999.

Gottlieb J, Goldberg ME. Activity of neurons in the lateral intraparietal areaof the monkey during an antisaccade task. Nat Neurosci 2: 906–912, 1999.

Gottlieb JP, Kusunoki M, Goldberg ME. The representation of visualsalience in monkey parietal cortex. Nature 391: 481–484, 1998.

Gottlieb J, Kusunoki M, Goldberg ME. Simultaneous representation ofsaccade targets and visual onsets in monkey lateral intraparietal area. CerebCortex 15: 1198–1206, 2005.

Graziano MS, Yap GS, Gross CG. Coding of visual space by premotorneurons. Science 266: 1054–1057, 1994.

Grefkes C, Ritzl A, Zilles K, Fink GR. Human medial intraparietal cortexsubserves visuomotor coordinate transformation. Neuroimage 23: 1494–1506, 2004.

Grunewald A, Linden JF, Andersen RA. Responses to auditory stimuli inmacaque lateral intraparietal area. I. Effects of training. J Neurophysiol 82:330–342, 1999.

Hagler DJ Jr, Sereno MI. Spatial maps in frontal and prefrontal cortex.Neuroimage 29: 567–577, 2006.

Johnson PB, Ferraina S, Bianchi L, Caminiti R. Cortical networks for visualreaching: physiological and anatomical organization of frontal and parietallobe arm regions. Cereb Cortex 6: 102–119, 1996.

Jones EG, Powell TP. An anatomical study of converging sensory pathwayswithin the cerebral cortex of the monkey. Brain 93: 793–820, 1970.

Kalaska JF. Parietal cortex area 5 and visuomotor behavior. Can J PhysiolPharmacol 74: 483–498, 1996.

Kalaska JF, Scott SH, Cisek P, Sergio LE. Cortical control of reachingmovements. Curr Opin Neurobiol 7: 849–8, 1997.

Kawashima R, Roland PE, O’Sullivan BT. Fields in human motor areasinvolved in preparation for reaching, actual reaching, and visuomotor learning:a positron emission tomography study. J Neurosci 14: 3462–3474, 1994.

Luh KE, Butter CM, Buchtel HA. Impairments in orienting to visual stimuliin monkeys following unilateral lesions of the superior sulcal polysensorycortex. Neuropsychologia 24: 461–470, 1986.

Luppino G, Murata A, Govoni P, Matelli M. Largely segregated parieto-frontal connections linking rostral intraparietal cortex (areas AIP and VIP)and the ventral premotor cortex (areas F5 and F4). Exp Brain Res 128:181–187, 1999.

Macaluso E, Driver J. Multisensory spatial interactions: a window onto func-tional integration in the human brain. Trends Neurosci 28: 264–271, 2005.

Macaluso E, Driver J, Frith CD. Multimodal spatial representations engagedin human parietal cortex during both saccadic and manual spatial orienting.Curr Biol 13: 990–999, 2003a.

Macaluso E, Eimer M, Frith CD, Driver J. Preparatory states in crossmodalspatial attention: spatial specificity and possible control mechanisms. ExpBrain Res 149: 62–74, 2003b.

Macaluso E, Frith CD, Driver J. Multimodal mechanisms of attention relatedto rates of spatial shifting in vision and touch. Exp Brain Res 137: 445–454,2001.

Macaluso E, Frith CD, Driver J. Directing attention to locations and tosensory modalities: multiple levels of selective processing revealed withPET. Cereb Cortex 12: 357–368, 2002.

Medendorp WP, Goltz HC, Crawford JD, Vilis T. Integration of target andeffector information in human posterior parietal cortex for the planning ofaction. J Neurophysiol 93: 954–962, 2005a.

Medendorp WP, Goltz HC, Vilis T. Remapping the remembered targetlocation for anti-saccades in human posterior parietal cortex. J Neurophysiol94: 734–740, 2005b.

Mesulam MM. From sensation to cognition. Brain 121: 1013–1052, 1998.Moore T. The neurobiology of visual attention: finding sources. Curr Opin

Neurobiol 16: 159–165, 2006.Mullette-Gillman OA, Cohen YE, Groh JM. Eye-centered, head-centered,

and complex coding of visual and auditory targets in the intraparietal sulcus.J Neurophysiol 94: 2331–2352, 2005.

Murata A, Gallese V, Luppino G, Kaseda M, Sakata H. Selectivity for theshape, size, and orientation of objects for grasping in neurons of monkeyparietal area AIP. J Neurophysiol 83: 2580–2601, 2000.

Penny W, Holmes A. Random effects analysis. In: Human Brain Function,edited by Frackowiak RSJ, Friston K, Frith C, Dolan RJ, Price C, Zeki S,Ashburner J, Penny W. San Diego, CA: Elsevier, 2004, p. 843–851.

Petit L, Clark VP, Ingeholm J, Haxby JV. Dissociation of saccade-relatedand pursuit-related activation in human frontal eye fields as revealed byfMRI. J Neurophysiol 77: 3386–3390, 1997.

Prado J, Clavagnier S, Otzenberger H, Scheiber C, Kennedy H, PereninMT. Two cortical systems for reaching in central and peripheral vision.Neuron 48: 849–858, 2005.

Price CJ, Friston KJ. Cognitive conjunction: a new approach to brainactivation experiments. Neuroimage 5: 261–270, 1997.



Quian QR, Snyder LH, Batista AP, Cui H, Andersen RA. Movementintention is better predicted than attention in the posterior parietal cortex.J Neurosci 26: 3615–3620, 2006.

Rizzolatti G, Riggio L, Dascola I, Umilta C. Reorienting attention across thehorizontal and vertical meridians: evidence in favor of a premotor theory ofattention. Neuropsychologia 25: 31–40, 1987.

Rushworth MF, Nixon PD, Passingham RE. Parietal cortex and movement.II. Spatial representation. Exp Brain Res 117: 311–323, 1997.

Schlack A, Sterbing-D’Angelo SJ, Hartung K, Hoffmann KP, Bremmer F.Multisensory space representations in the macaque ventral intraparietal area.J Neurosci 25: 4616–4625, 2005.

Schluppeck D, Curtis CE, Glimcher PW, Heeger DJ. Sustained activity intopographic areas of human posterior parietal cortex during memory-guidedsaccades. J Neurosci 26: 5098–5108, 2006.

Schluppeck D, Glimcher P, Heeger DJ. Topographic organization for de-layed saccades in human posterior parietal cortex. J Neurophysiol 94:1372–1384, 2005.

Sereno MI, Pitzalis S, Martinez A. Mapping of contralateral space inretinotopic coordinates by a parietal cortical area in humans. Science 294:1350–1354, 2001.

Simon O, Mangin JF, Cohen L, Le Bihan D, Dehaene S. Topographicallayout of hand, eye, calculation, and language-related areas in the humanparietal lobe. Neuron 33: 475–487, 2002.

Snyder LH, Batista AP, Andersen RA. Coding of intention in the posteriorparietal cortex. Nature 386: 167–170, 1997.

Snyder LH, Batista AP, Andersen RA. Change in motor plan, without achange in the spatial locus of attention, modulates activity in posteriorparietal cortex. J Neurophysiol 79: 2814–2819, 1998.

Snyder LH, Batista AP, Andersen RA. Intention-related activity in theposterior parietal cortex: a review. Vision Res 40: 1433–1441, 2000.

Tanne-Gariepy J, Rouiller EM, Boussaoud D. Parietal inputs to dorsalversus ventral premotor areas in the macaque monkey: evidence for largelysegregated visuomotor pathways. Exp Brain Res 145: 91–103, 2002.

Thoenissen D, Zilles K, Toni I. Differential involvement of parietal andprecentral regions in movement preparation and motor intention. J Neurosci22: 9024–9034, 2002.

Tolias AS, Moore T, Smirnakis SM, Tehovnik EJ, Siapas AG, Schiller PH.Eye movements modulate visual receptive fields of V4 neurons. Neuron 29:757–767, 2001.

Wise SP, Boussaoud D, Johnson PB, Caminiti R. Premotor and parietalcortex: corticocortical connectivity and combinatorial computations. AnnuRev Neurosci 20: 25–42, 1997.

Worsley KJ, Marrett S, Neelin P, Vandal AC, Friston KJ, Evans AC. Aunified statistical approach for determining significant signals in images ofcerebral activation. Hum Brain Map 4: 58–73, 1996.

Wurtz RH, Goldberg ME, Robinson DL. Brain mechanisms of visualattention. Sci Am 246: 124–135, 1982.

Yamaguchi S, Tsuchiya H, Kobayashi S. Electroencephalographic activ-ity associated with shifts of visuospatial attention. Brain 117: 553–562,1994.

Yantis S, Schwarzbach J, Serences JT, Carlson RL, Steinmetz MA, PekarJJ, Courtney SM. Transient neural activity in human parietal cortex duringspatial attention shifts. Nat Neurosci 5: 995–1002, 2002.

Zhang M, Barash S. Neuronal switching of sensorimotor transformations forantisaccades. Nature 408: 971–975, 2000.



Delay Activity and Sensory-Motor Translation During Planned Eye or Hand Movements to Visual or...

Documents

Transcript of Delay Activity and Sensory-Motor Translation During Planned Eye or Hand Movements to Visual or...