Unfamiliar Walking Movements Are Detected Early in the Visual Stream: An fMRI Study

Vincenzo Maffei1,, Maria Assunta Giusti1,3, Emiliano Macaluso2, Francesco Lacquaniti1,3,4 and Paolo Viviani1,3

1Laboratory of Neuromotor Physiology, 2Laboratory of Neuroimaging, IRCCS Santa Lucia Foundation, 00179 Rome, Italy, 3Centreof Space BioMedicine and 4Department of Systems Medicine, University of Rome Tor Vergata, 00133 Rome, Italy

Address correspondence to Vincenzo Maffei, PhD, Laboratory of Neuromotor Physiology, IRCCS Santa Lucia Foundation, via Ardeatina 306,00179 Rome, Italy. Email: v.maffei@hsantalucia.itVincenzo Maffei and Maria Assunta Giusti contributed equally to the work.

Two experiments investigated the network involved in the visual per-ception of walking. Video clips of forward and backward walk (realwalk direction) were shown either as recorded, or reversed in time(rendering). In Experiment 1 (identification task), participants wereasked to indicate whether or not the stimulus was time-reversed. InExperiment 2 (free-viewing), participants viewed the video clips pas-sively. Identification accuracy was good with the more familiarscene, that is, when the visual walk was in the direction of thefacing orientation, and at chance level in the opposite case. In bothexperiments, the temporo-occipital junction (TOJ) was activatedmore strongly by unfamiliar than familiar scenes. Only in Experiment1 intraparietal, superior temporal, and inferior temporal regions werealso activated. TOJ activation signals the detection in unfamiliarscenes of a mismatch between facing orientation and visual move-ment direction. We argue that TOJ response to a mismatch preventsthe further processing of the visual input required to identify tem-poral inversions. When no mismatch is detected (familiar stimuli),TOJ would, instead, be involved in the kinematic analysis that makessuch identification possible. The study demonstrates that unfamiliarwalking movements are detected earlier than so far assumed alongthe visual movement processing stream.

Keywords: action observation, biological motion, fMRI, predictive coding

Introduction

We perceive human movements with great accuracy andspeed. The visual factors responsible for this remarkable per-formance have been investigated extensively both behaviorallyand with brain imaging techniques (for a review, see Grosbraset al. 2012). Recently, growing attention has been paid to therole of nonvisual factors, such as expectations, contextual con-straints, rationality, and motor competence. These attempts toplace movement within a broader context rely increasingly onthe use of more complex and realistic visual stimuli than point-light (PL) displays (Rizzolatti and Craighero 2004). It has beensuggested (Frith and Frith 2003, 2006; Brass et al. 2007; deLange et al. 2008) that visual movement provides access to theunderlying intentionality of the action by involving corticalareas where mental states are attributed to the agent. Seeingpeople engaged in motor tasks activates an action observationnetwork (AON; Gallese and Goldman 1998; Gallese et al.2004), which extends beyond the temporo-occipital cortex(Han et al. 2013) and includes premotor and parietal areas(Caspers et al. 2010). It has also been argued that the AON sup-ports a processes of resonance and simulation by which motorcompetence would be brought to bear for movement under-standing (Rizzolatti and Craighero 2004; Kilner et al. 2009;Oosterhof et al. 2010). However, the neuronal underpinnings

of such hypothetical process are still debated (e.g., Lingnauet al. 2009). Several brain imaging studies have focused onwhole-body movements. In particular, it has been shown thatthe activation of AON components when viewing dance move-ments is modulated by the motor expertise of the observer(Calvo-Merino et al. 2005, 2006), movement naturalness (Crosset al. 2012), and training (Cross et al. 2006; Cross, Hamilton,et al. 2009; Cross, Kraemer, et al. 2009).

The human brain is sensitive to the temporal order withwhich events unfold in the real world (Hasson et al. 2008).However, to our knowledge, no previous study has investi-gated how the cortical network involved in the perception ofhuman movements takes into account the arrow of time.Several complex human movements have a prevalent timearrow dictating the sequence with which their componentsusually follow each other. In some cases, prevalence is absol-ute because inverting the usual sequence is barred by the inter-action between the body and the environment, as in mostfetching gestures, or by internal constraints of the motorsystem, as in articulatory gestures. Walking is instead a case ofrelative prevalence, which is well suited for investigating themechanisms leading to the identification of the time arrow. Onthe one side, the sequence of stance and swing phases of thelegs that produces a forward displacement of the body caneasily be inverted to produce a backward displacement. Onthe other side, backward walking is not the time-symmetricversion of forward walking. Kinematic measurements demon-strated subtle but consistent differences between the 2 actions,which include the time course of the ankle joint angle (Thor-stensson 1986; Winter et al. 1989), step length (Grasso et al.1998), the variability of movement kinematics (Katzavelis et al.2010), and the correlation between gait cycle, step length, andspeed (Grasso et al. 1998). Therefore, even when the transla-tional component of the movement is suppressed or is ambigu-ous, these asymmetric kinematic cues could still be used todiscriminate forward and backward walk. In principle, dis-crimination may be achieved with a purely visual analysis ofthe cues. However, because walking is an over-trained actionwith strong idiosyncratic features and a robust motor represen-tation, discriminating whether a sequence of leg movementscorresponds to a forward or backward walk may also involvean interaction between visual inputs and the motor compe-tence of the observer.

Previous studies on the perception of walk have mostly uti-lized PL stimuli in which the translational component of theaction is eliminated. However, a recent study (Viviani et al.2011) showed that this component plays a major role in theability to identify the time arrow of the display. The stimuliwere video clips of actors walking either forward or backward.Participants were shown both types of recordings played back

© The Author 2014. Published by Oxford University Press. All rights reserved.For Permissions, please e-mail: journals.permissions@oup.com

Cerebral Cortexdoi:10.1093/cercor/bhu008

Cerebral Cortex Advance Access published February 13, 2014 at Fondazione S. L

ucia IRC

CS on February 17, 2014

http://cercor.oxfordjournals.org/D

ownloaded from

either as recorded or after a time reversal and were asked toidentify the play-back mode. The results showed that theability to detect time reversals was strongly influenced by thedirection of the body movement on the display. When thebody appeared to move in the more familiar forward direction(forward walk rendered normally or time-reversed backwardwalk), response accuracy was well above chance. In contrast,responses were at chance level when the body appeared tomove in the unusual backward direction. This unexpectedresult suggested that familiarity has a gating effect. Apparently,visual inputs describing the rather unusual scene in which theactor moves backward cannot be processed adequately todetect a temporal reversal.

Based on this finding, and using the same stimuli as inViviani et al. (2011), we designed a brain imaging study toaddress 3 issues. The first issue is where along the visualstream brain activity correlates with the familiarity of thestimuli. An action can be perceptually familiar because we seeit often performed, and also familiar from the motor point ofview because we perform it frequently. If motor familiarity iscrucial for identifying the arrow of time, the 2 directions of thebody on the display should elicit different patterns of activationin premotor areas, irrespective of whether the time arrow hasbeen preserved or reversed. If instead visual familiarity iscrucial for correct identification, a differential pattern of acti-vation should emerge already at some earlier stage of themotion processing stream. In particular, differences may beexpected in the superior temporal sulcus (STS). STS is a keycomponent of AON that responds to several aspects of biologi-cal movement (cf. Puce and Perrett 2003) and is involved inseveral higher-order cognitive functions such as detecting theappropriateness of the action (Pelphrey et al. 2004), action rec-ognition (Iacoboni 2005; Molnar-Szakacs et al. 2005), actionunderstanding (Thioux et al. 2008; Herrington et al. 2011), andintentional attunement (Gallese 2006). A computational modelby Giese and Poggio (2003) predicted that forward walk elicitsa stronger STS activity than backward walk. In addition, it hasbeen argued (Jastorff and Orban 2009) that the body of theactor is linked to the action he/she is performing already in thetemporo-occipital region (extra-striate body area, EBA; fusi-form body areas) by integrating shape and kinematic cues.

If one can identify an area (or areas) that appears to beresponsible for the gating effect of the visual body direction,the next issue is whether unfamiliar stimuli (such as a bodymoving backward on the display) increase or decrease theactivity in these areas with respect to familiar stimuli. The pre-vailing view, adopted by Giese and Poggio (2003), holds thatthe AON should respond more energetically to familiar, thanto unfamiliar, actions (Press 2011). Recently, this view hasbeen challenged by Cross et al. (2012) who showed that parie-tal, premotor, and temporo-occipital areas respond morestrongly to robot-like motions than to natural biological move-ments. Clearly, these conflicting results lead to differentinterpretations of the cortical responses. A more energeticresponse to familiar actions is in keeping with the general viewthat motor perceptual interactions are mediated by a resonancemechanism whereby incoming information and visual/motorcompetence compatible with that information reinforce mu-tually (Decety and Chaminade 2003; Aglioti et al. 2008). Incontrast, a more energetic response to unfamiliar action maybe interpreted as a startling reaction to a mismatch (Kilneret al. 2007a, 2007b; Cross et al. 2012).

The final issue concerns the identification of the time arrow.Provided that the visual direction of the walk is the familiarforward one, how is then a proficient performance achieved?Viviani et al. (2011) argued that identification involves motorcompetence. Clearly, an association between proficient per-formance and a pattern of differential activation within theAON in response to normal and reversed stimuli cannot beconstrued as evidence that identification is causally dependenton activity in these areas (Mahon and Caramazza 2008). Itwould however, be in keeping with the notion that motorexpertise is conducive to a finer evaluation of the kinematicfeatures of the stimuli (see above). Conversely, should normaland reversed stimuli fail to activate differentially any AON com-ponent, the most natural inference would be that correctidentification is based mainly on visual processes either in thetemporo-occipital cortex (Jastorff and Orban 2009) or in theSTS (Grossman and Blake 2002).

Two event-related brain imaging experiments investigatedthe 3 issues above. In Experiment 1, the same two-alternativeforced choice (2AFC) task as in Viviani et al. (2011) estimatedthe ability to detect a time reversal in 4 types of displays inwhich the actual direction of the walk (forward/backward)was crossed with the rendering of the recording (normal/time-reversed). In addition to components related to the visual pro-cessing, the pattern of cortical activations emerging from thisexperiment is likely to include also components related to thetask (attention, task difficulty, and response selection). Tofactor out the potential confounding effects of the latter com-ponents, in Experiment 2, the same stimuli as in Experiment 1were shown to a different sample of observers in a free-viewing condition in which no identification judgment was re-quired.

Methods

ParticipantsFourteen young adults (8 females, 6 males; age range: 22–28) partici-pated in Experiment 1. A different sample of 15 young adults (8 females,7 males; age range: 22–28) participated in Experiment 2. All participantswere right-handed and right-foot dominant according to the Edinburghhandedness inventory. Participants were healthy, free of psychotropic,or vasoactive medication, with no past history of psychiatric or neuro-logical diseases. They all had normal or corrected-to-normal (contactlenses) visual acuity. After receiving the instructions, participants gavetheir written informed consent to procedures approved by the Insti-tutional Review Board of Fondazione Santa Lucia. However, they werenot aware of the ultimate goal of the experiments. Experimental proto-cols complied with the Declaration of Helsinki on the use of humansubjects in research.

StimuliEight young adults (4 females, 4 males, age range: 21–24) who did notparticipate in the imaging experiments volunteered as actors for recor-ding the stimuli. They were selected with the criterion of minimizinganthropometric differences in the lower part of the body. Identity andsex cues were also minimized by asking actors to wear black leg wears.Actors were informed that recordings of their walking movementswould be used as experimental stimuli, but were not aware of the goalof the study.

Actors were asked to walk at their spontaneous pace along a 7-mlong platform. Each actor performed the exercise 12 times. In 6 trials,the walk was in the forward direction (FW). In the other 6, actorswalked backward direction (BW). Half of each set of walks was fromthe left to the right side of the platform (LR), and the other half was

2 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

from the right to the left side (RL). There were 3 repetitions for eachcondition. Movements were recorded in color with a Sony HandycamNP-F330 at 25 frames per second. The camera was level with the plat-form at a distance of 3 m. With the selected focal length, the field ofview (FOV) included the central 4-m portion of the platform. The back-ground was uniformly gray. The walking action began before enteringthe FOV and continued after disappearing.

Normalized video clips [96 frames = 3.84 s, field size: 26° (H) × 9.5° (V)]were obtained by editing the original recordings with Adobe Premiere7.0. Video clips showed only the lower part of the body from waist tofeet and a portion of the platform. They included at least 2 completestepping cycles. The experimental stimuli were the video clips showneither as recorded (N: normal) or after a time reversal (R: reverse).Altogether, there were 4 [actor] × 2 [sex] × 3 [repetition] × 2 [FW/BW] × 2 [LR/RL] × 2 [N/R] = 192 different stimuli. Each stimuluswas presented twice to each participant in a different pseudorandomorder.

Procedure and TaskThe total number of trials (2 × 192 = 384) was divided into 5 runs(4 blocks of 77 trials and 1 block of 76 trials) with the constraint thatsuccessive trials never involved the same actor and same walk direc-tions. Within runs, intertrial intervals were randomized with a long-tailed (geometric) distribution (Hagberg et al. 2001). The mean onsetasynchrony was 5.68 s (minimum: 4.68 s and maximum: 11.38 s). Arun lasted about 8 min. Runs were administered in a single session andwere separated by brief pauses.

In Experiment 1, participants were told that the stimuli representedeither a forward or a backward walk. They were also informed that, insome cases, the recordings were reversed in time, so that whatappeared to be a forward walk could either be a true forward walk orthe result of inverting a backward walk. The task (2AFC) was to indi-cate whether the movement was displayed as recorded or reversed intime. Responses were entered by pressing with the right index finger 1of 2 buttons marked N (normal) and R (reverse), respectively. Therewas no time limit, but participants were encouraged to answer as soonas possible, even before the end of the stimulus. Between trials, thedisplay was uniformly gray and participants fixated a central point. Noconstraints were imposed on oculomotor behavior during the presen-tation of the stimuli. Before the experimental session, participantswere administered 48 warm-up trials, which included at least oneexample for each combination of actor sex, walk direction, and render-ing. The results of these trials were not analyzed.

In Experiment 2, the presentation schedule was as in Experiment 1,but participants were only asked to watch the video clips, withoutmaking any overt judgment about the stimuli (free-viewing). UnlikeExperiment 1, participants were not informed that half of the stimuliwere time-reversed versions of a real recording.

Stimulus PresentationParticipants laid supine in the MR scanner with the head immobilizedwith foam cushioning and wore ear plugs and headphones to suppressambient noise. A digital projector (NEC LT158, 60-Hz refresh rate) sentthe stimuli through an inverted telephoto lens onto a semi-opaque Plexi-glas screen mounted vertically inside the scanner bore, behind the par-ticipant’s head. The back-projected image was then viewed via a mirrormounted on the head coil positioned at about 4.5 cm from the eyes. Theeye-to-screen equivalent distance was 57 cm, and the angular size of theprojected image was 16° × 5.8°. Responses were acquired with anMR-compatible response box (fORP, Current Designs) sampled at 1 kHz.Eye movements were recorded with an ASL 504 eye-tracking system(Bedford, MA, USA) and sampled at 60 Hz for off-line processing.

Processing of Eye Movement DataEye movement traces included a large initial saccade from the centralfixation point to the screen edge where the actor entered the scene, fol-lowed by a smooth pursuit phase interspersed with catch-up saccades.Only the horizontal component of the movement was processed with acustomized analysis program. First, the program extracted the trace

portion from the end of the initial saccade to either the response(Experiment 1) or to the end of the video clip (Experiment 2). Then,catch-up saccades were detected and eliminated by realigning thesmooth components and filling the gap with a linear interpolation. Foreach participant, conditions (FW/N, FW/R, BW/N, and BW/R), and walkorientation (LR/RL), we computed an average eye movement trace bypooling the results over actors and repetitions (the end of the averagetrace was clipped to the shortest length). Finally, we estimated theaverage gain of the smooth pursuit component by dividing the slope ofthe [gaze position/time] linear regression by the average walking speedin each recording (computed separately for FWand BW).

fMRI Data AcquisitionFunctional imaging data were acquired by a Siemens MagnetomAllegra 3-T head-only scanning system (Siemens Medical Systems,Erlangen, Germany), equipped with a quadrature volume RF head coil.Whole-brain blood oxygen level-dependent (BOLD) echo-planarimaging (EPI) functional data were acquired with a 3T-optimizedgradient-echo pulse sequence (repetition time = 2.47 s, echo time = 30ms, flip angle = 70°, FOV = 192 mm, fat suppression). Thirty-eightimage slices were acquired in the ascending order (64 × 64 voxels, 3 ×3 × 2.5 mm, distance factor 50%). For each participant, a total of 204volumes of data were acquired for each of the 5 runs.

Image PreprocessingFor both experiments, the 204 × 5 = 1020 available fMRI volumes foreach participant were processed using the SPM8 software (WellcomeDepartment of Cognitive Neurology, University College London;implemented in MATLAB 7.4). The first 4 volumes of each run werediscarded to allow for stabilization of longitudinal magnetization. Theremaining 1000 volumes underwent the following preprocessingsteps: (1) Realignment of all images to compensate for head motion;(2) correction of the slice acquisition delays using the middle slice as areference; (3) normalization to the MNI EPI template and re-samplingto 2-mm isotropic voxel size; (4) spatial smoothing with an isotropicGaussian kernel with a 8-mm full-width at half maximum.

Data Analysis for Each ExperimentStatistical analysis was performed in 2 steps (Penny et al. 2003). First,for every participant, the onset and duration of each condition weremodeled by a general linear model (GLM). The design matrix includedmultiple regressors, each modeling 1 of the 4 conditions, and 6additional covariates modeling head motion. Voxel time-series wereprocessed to remove autocorrelations using a first-order autoregressivemodel and high-pass filtering (128-s cut-off). For each participant, theresulting parameter estimate images were sorted separately for eachexperimental condition (FW/N, BW/N, FW/R, and BW/R).

In the second step of the analysis, for both Experiments 1 and 2, weapplied a random-effect (RFX) GLM to the individual parameter esti-mates obtained from the first step. The statistical threshold was set toP-corr < 0.05 (FWE correction) at the cluster level (cluster size esti-mated at P-uncorr <0.001), considering the whole brain as the volumeof interest. The localization of the activation clusters was based onDuvernoy’s (1991) anatomical atlas of the human brain.

We tested whether additional information emerged when the analy-sis of cortical activity is repeated with a new design matrix by groupingtrials according to the response. For instance, a FW/N trial the responseto which was “Reversed” was reckoned as FW/R, and a BW/R trial theresponse to which was “Normal” was reckoned as FW/N. Four images,corresponding to the new grouping criterion, were calculated for eachparticipant at the first level. The images were then again subjected to asecond-level RFX analysis.

The analysis of the smooth pursuit component of eye movementsrevealed only in Experiment 1 an effect of the experimental conditionson the gain (see Results). Thus, for this experiment, we tested whethereye movements contributed to the pattern of activity. To this end, inthe first-level statistical analysis of the BOLD data, we added trial bytrial the individual pursuit gain as a covariate. This control was per-formed only on the participants (N = 8) in which the estimate of the

Cerebral Cortex 3

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

Table 1Experiment 1: Probability of correct rendering identification for each combination of experimental factors

Male actors Female actors

N R N R

FW 0.842 [0.783–0.902] 0.391 [0.260–0.523] 0.897 [0.857–0.937] 0.426 [0.264–0.587]BW 0.562 [0.460–0.665] 0.692 [0.502–0.757] 0.530 [0.411–0.648] 0.714 [0.566–0.862]

X2 = 74.30, P< 0.001 X2 = 111.23, P<0.001

Note: Results averaged over participants, orientation of the walk, and repetitions. In brackets, the 0.95 confidence intervals.N: normal rendering; R: reverse rendering; FW: real forward walk; BW: real backward walk.

Figure 1. Experiment 1. Oculomotor smooth pursuit in one representative participant for all 4 indicated types of stimuli and both orientations of the walk (left to right/right to left).Solid dots: pursuit trajectory averaged over all repetitions. Also shown the linear regressions through the data points used for computing the average pursuit gain (shown inset withthe 95% confidence limits). Light lines: 95% confidence bars.

4 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

gain was robust in at least two-thirds of the trials. The resulting imagefor each condition was then subjected to a second-level RFX analysis.

Comparison Between Experiments 1 and 2First, we verified whether the regions active in Experiment 1 were alsoactive in Experiment 2. Accordingly, we considered as region of inter-est (ROI) spheres (8 mm radius) centered on the local maxima of theclusters detected in Experiment 1, and tested the corresponding con-trast in Experiment 2 (main effects and interactions). Altogether, 11ROIs were considered (Table 7). Secondly, for completeness, we testedwhether the effects of interest within these ROIs were significantlydifferent between experiments. To this end, a set of unpaired t-testscompared the contrasts of interest (main effects and interactions) inExperiments 1 and 2. Note that in this second step, the contrastweights for ROI selection and for ROI test were not independent. AllROI analyses were performed using the MARSBARS toolbox for SPM8,and P-values were Bonferroni-corrected for multiple comparison.However, the correction was not applied to the analysis of noninde-pendent data where we focused on single areas.

Results

Experiment 1

Identification PerformanceIn agreement with the literature (Grasso et al. 1998), statisticalanalysis of step size (cf. Viviani et al. 2011) showed a signifi-cant difference between backward (110.2 cm) and forward(132.6 cm) movements, but no effect of actor sex or orientationof the walk (LR/RL). In subsequent analyses, response prob-abilities were computed by pooling repetitions and LR/RLtrials. Table 1 reports the probability P {C } of a correct responseaveraged over all participants as a function of the experimentalfactors. Statistical analysis (mixed-model analyses of variance[ANOVAs], 4 [actor] × 2 [sex] × 2 [FW/BW] × 2 [N/R] with randomfactor actor nested within sex, arcsin transformation) showedthat neither real walking direction (FW/BW: F1,427 = 2.04, P =0.154) nor sex (F1,427 = 2.71, P = 0.100) had a significant effecton identification performance. In contrast, the effect of render-ing was highly significant (N/R; F1,427 = 57.77, P < 0.001), andso was the interaction between real walk direction and render-ing (F1,427 = 186.76, P < 0.001). For each participant, the inter-action was confirmed by performing a χ2 test on the 2 × 2contingency tables for correct responses. For all the partici-pants, the test revealed that real walk direction and renderinginteracted significantly. Identification performance was highlyaccurate in the condition FW/N (pooling over sex, P {C } =0.870), decreased but remained well above chance level (P {C }= 0.672) in the condition BW/R, and collapsed to chance levelwhen the actor appeared to move backward (BW/N: P {C } =0.546; FW/R: P {C } = 0.408).

Eye MovementsAll participants spontaneously adopted a pursuit oculomotorstrategy to keep the walking figure near the center of the visualfield. Figure 1 illustrates with data from one representative par-ticipant the pursuit component of eye movements. Pursuiteffectiveness was estimated by the pursuit gain (see Methods).Because pursuit gain was <1 (Table 2), a strategy of saccadicrecapture was often adopted to maintain the gaze on the figure.Statistical analysis (three-way ANOVAs for repeated measures,2 [FW/BW] × 2 [LR/RL] × 2 [N/R], Greenhouse-Geisser correc-tion) showed that the pursuit gain was affected neither by the

orientation of the walk (LR/RL: F1,13 < 0.01, P = 0.993) nor therendering (N/R: F1,13 = 3.351, P = 0.090). Instead, the realdirection of the walk was a significant factor (FW/BW: F1,13 =

Table 3Experiment 1: Main effect of all moving stimuli versus baseline

Anatomical areas Cluster size MNI peak coordinates Z-score

x y z (mm)

Left temporo-occipital junction 22 562 −50 −76 2 6.17Right temporo-occipital junction 42 −76 −12 6.11Right middle occipital gyrus 34 −88 12 6.02Left post-central gyrus 8474 −42 −34 54 5.86Left precentral gyrus −36 0 66 5.38Left superior parietal lobe −34 −46 60 5.23Left inferior frontal gyrus 541 −34 18 −4 5.66

−28 22 10 4.36Left supplementary motor area 1786 −2 16 50 5.56Left superior frontal gyrus −14 16 38 4.55

−12 24 38 4.14Right inferior frontal gyrus 3072 40 32 12 5.34

36 20 −6 5.1552 14 30 4.57

Note: Anatomical areas, cluster sizes, peak coordinates in MNI space, and Z-scores for significantactivations.

Table 2Experiment 1: Smooth pursuit gain averaged over participants and repetitions for each combinationof experimental factors

Left to right Right to left

N R N R

FW 0.682 0.651 0.662 0.588BW 0.711 0.705 0.746 0.755

Left to right/right to left: orientation of the walk; FW: real forward walk; BW: real backward walk;N: normal rendering; R: reverse rendering.

Figure 2. Experiment 1. Main effect of all moving stimuli versus baseline(P-corr < 0.05).

Cerebral Cortex 5

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

9.583, P = 0.009), the gain being on average lower for forward(FW: 0.646) than for backward (BW: 0.729) walk.

fMRI ResultsThe analysis of BOLD signals was carried out twice using adesign matrix in which trials were grouped according to eitherthe actual conditions or the responses (see Methods). The 2grouping criteria yielded similar results. In the following, wewill detail only the results corresponding to the former criterion.A preliminary global analysis (Table 3) charted the corticalregions activated by the stimuli, irrespective of the parametersmanipulated experimentally (main effect of all moving stimulivs. baseline). As shown in Figure 2, significant activations werefound in temporo-occipital, parietal, and frontal (premotor andprefrontal) areas, which are generally included in the AON.

The detailed analysis of the cortical activations was pat-terned after the identification performance, which, as detailedabove, was strongly dependent on the interaction between realwalk direction and rendering. The direction of the body displa-cement on the display (hereafter, “visual walk”) was desig-nated as the first main effect and tested by the contrast [(FW/N) + (BW/R)] versus [(FW/R)] + (BW/N)]. The time arrow inthe playback of the video clip (hereafter, “rendering”) was

designated as the second main effect and tested by the contrast[(FW/N) + (BW/N)] versus [(BW/R) + (FW/R)]. Finally, the inter-action between visual walk and rendering, which correspondsto the real walking direction (hereafter, “real walk”), was testedby the contrast [(FW/N)− (BW/R)] versus [(BW/N)− (FW/R)].

Figure 3 summarizes the results for the visual walk factor.The contrast [(FW/R) + (BW/N)] > [(FW/N) + (BW/R)] (back-ward visual walk > forward visual walk), which reflects theeffect of watching a backward walking, demonstrated signifi-cant bilateral activations of the temporo-occipital junction(TOJ). Activation was also detected bilaterally in the intraparie-tal sulci (IPS) and in the right STS. Table 4 reports the coordi-nates of the peaks of activation for all clusters. The oppositecontrast [(FW/N) + (BW/R)] > [(FW/R) + (BW/N)] (forwardvisual walk > backward visual walk) failed to detect any signifi-cant activation. Significant differences signaled by the GLManalysis may be due to BOLD responses with similar peak am-plitude but different time course. We excluded this possiblesource of confound by showing that the time course of theresponses was actually very similar. These comparisons are re-ported in Supplementary Results.

We found no evidence of learning in either behavioral orfMRI data. The rate of correct identifications did not change

Figure 3. Experiment 1. Effect of visual walk (backward > forward). Bar plots: Activity changes relative to the baseline for each stimulus type (FW: forward real walk, BW:backward real walk; forward/backward: direction of the visual walk). Ordinates: Percentual signal change of estimated activity. Error bars indicate SE. Clusters of activation (inparenthesis the coordinates of the local maxima) are superposed to coronal sections of the canonical MNI template.

6 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

significantly across the 5 sessions (three-way ANOVA, 2 [FW/BW] × 2 [N/R] × 5 [session], arcsin transformation; session: F4,52-= 2.539, P = 0.073). As for the BOLD signal, we considered boththe contrasts [FW/N] > [BW/R] and [BW/N] > [FW/R]. When thewhole brain was defined as the volume of interest, no significantdifference was detected between sessions 1 and 5 (regression onall sessions with weights: [−2 −1 0 1 2], P-corr > 0.3). The sameanalysis was then repeated separately on the 8 ROIs (8 mmsphere) centered on the peaks of the main effect of backward[(FW/R) + (BW/N)] > [(FW/N) + (BW/R)] and normal [(FW/N) +(BW/N)] > [(FW/R) + (BW/R)] rendering (see Table 4). No sig-nificant temporal trends emerged in any ROI (P-uncorr > 0.1).

The contrast [(FW/N) + (BW/N)] > [(BW/R) + (FW/R)] (normalrendering > reverse rendering) demonstrated significant activityin the left fusiform gyrus (FG), right precuneus (PrCu), andanterior cingulate cortex (ACC). Because estimated activity inthe former 2 areas was negative, Figure 4 illustrates the resultsonly in the case of ACC. No significant activation emerged fromthe opposite contrast [(BW/R) + (FW/R)] > [(FW/N) + (BW/N)].

For the interaction between visual walk and rendering,which corresponds to the direction of the real walk, both con-trasts [(FW/N)− (BW/R)] > [(BW/N)− (FW/R)] and [(BW/N)−(FW/R)] > [(FW/N)− (BW/R)] yielded significant results (Table 4).The first interaction (forward real walk > backward real walk)showed a significant cluster of activity in the left orbito-frontal

cortex and in the posterior cingulate gyrus (Table 4). However,as in the case of the rendering contrast (see above), the esti-mated activity in these areas was negative. The reverse contrast(backward real walk > forward real walk) revealed a significantcluster of activation in right TOJ (Fig. 5). Note that 60% of thevoxels in this cluster were also within the correspondingcluster detected by the (backward visual walk > forward visualwalk) contrast (cf. Fig. 3). In fact, within the ROIs identified bythe contrast (backward real walk > forward real walk), wefound a significant activation also for the contrast (backwardvisual walk > forward visual walk) (t = 5.01, P < 0.001). Thus,the 2 contrasts identified overlapping areas within the sameTOJ region.

Finally, because smooth pursuit gain was found to dependon the experimental conditions, we tested on a subset of theparticipants whether eye movements contributed to thepattern of cortical activity (see Methods). Taking the wholebrain as the search volume of interest and adding the smoothpursuit gain as covariates, the main effect (backward visualwalk > forward visual walk) was again significant in the rightTOJ [50 −74 14], right IPS [28 −42 60], right STS [54 −30 6](P-corr < 0.05), and left TOJ [−50 −70 2] (P-uncorr < 0.001) inspite of the reduced data set. The interaction (backward realwalk > forward real walk) was significant (P-uncorr < 0.001) inthe right TOJ [54 −74 10]. Thus, the control confirmed that eyemovements did not represent a significant confound.

Experiment 2

Eye MovementsAlso in free-viewing conditions, several participants showed aspontaneous tendency to pursue the walking figure withsmooth eye movements. However, there were some significantdifferences with respect to Experiment 1. First, because no

Figure 4. Experiment 1. Effect of rendering (normal > reverse). FW: forward realwalk; BW: backward real walk; forward/backward: direction of the visual walk. Sameformat as Figure 3.

Table 4Peaks of cluster activations in Experiment 1

Anatomical areas Cluster size MNI peak coordinates Z-score

x y z

Backward visual walkLeft temporo-occipital junction 951 −52 −74 8 6.74Right temporo-occipital junction 1173 48 −64 2 6.90Left intraparietal sulcus 367 −34 −36 54 4.43Right intraparietal sulcus 425 32 −32 40 5.06Right superior temporal sulcus 321 68 −34 14 4.67

Normal renderingLeft fusiform gyrus 274 −38 −30 −26 4.21Right precuneus 743 4 −40 60 3.82Anterior cingulate cortex 295 4 18 34 3.85

Forward real walkLeft orbito-frontal cortex 2633 −8 36 −12 4.59Left posterior cingulate gyrus 403 −20 −50 38 4.04

Backward real walkRight temporo-occipital junction 261 58 −64 −6 3.97

Note: whole-brain analysis

Figure 5. Experiment 1. Effect of real walk (interaction of the main effects shown inFigs 3 and 4). Backward > forward contrast. FW: forward real walk; BW: backward realwalk; forward/backward: direction of the visual walk. Same format as Figure 3.

Table 5Experiment 2: Smooth pursuit gain averaged over participants and repetitions for each combinationof experimental factors

Left to right Right to left

N R N R

FW 0.342 0.373 0.323 0.338BW 0.387 0.358 0.388 0.331

Left to right/right to left: orientation of the walk; FW: forward real walk; BW: backward real walk;N: normal rendering; R: reverse rendering.

Cerebral Cortex 7

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

response was required, the pursuit continued to the end of thevideo clip. Secondly, in 5 participants, saccadic scanning wasthe dominant oculomotor behavior, and a smooth pursuitcomponent could not be reliably isolated. Thirdly, even in theremaining 10 participants, saccadic recapture was more promi-nent than in the condition where a response was required bythe task. As shown in Table 5, in all conditions, the averagepursuit gain computed over these 10 participants was signifi-cantly lower than in Experiment 1 (cf. Table 2). Statisticalanalysis (three-way ANOVAs for repeated measures, 2 [FW/BW] × 2 [LR/RL] × 2 [N/R], Greenhouse-Geisser correction)showed that the pursuit gain was affected neither by the orien-tation of the walk (LR/RL: F1,9 < 0.420, P = 0.533) nor the ren-dering (N/R: F1,9 = 0.339, P = 0.575). Unlike Experiment 1, theactual direction of the walk did not affect the pursuit gain (FW/BW: F1,9 = 1.432, P = 0.262).

fMRI ResultsThe contrast [(FW/R) + (BW/N)] > [(FW/N) + (BW/R)] (back-ward visual walk > forward visual walk) revealed a strong bilat-eral activation of TOJ (Fig. 6) similar to that revealed byExperiment 1 (cf. Fig. 3). Table 6 reports the cluster size andthe coordinates of the activation peaks for both clusters. About70% of the voxels in these clusters were also within the corre-sponding clusters detected by the (backward visual walk >forward visual walk) contrast in Experiment 1 (cf. Fig. 3). Nosignificant activation was detected by the contrasts defined bythe direction of the real walk. As in Experiment 1, no signifi-cant cluster of activity was detected by the reverse contrast(forward visual walk > backward visual walk). No consistentbrain activations were associated with the contrast [(FW/N) +(BW/N)] > [(BW/R) + (FW/R)] (normal rendering > reverse ren-dering). Instead, the opposite contrast [(FW/R) + (BW/R)] >[(BW/N) + (FW/N)] (reverse rendering > normal rendering) re-vealed a cluster of activity in the right TOJ, not present inExperiment 1.

The same pattern of cortical activity emerged whether ornot the population sample had a homogeneous oculomotorbehavior. We performed an additional second-level analysisof the data after excluding the 5 participants who, instead ofpursuing the moving body, explored the stimuli mainly withsaccades (see above). The whole-brain analysis detectedonly a significant activation in the bilateral TOJ for the con-trast (backward visual walk > forward visual walk; peaks at[50 −64 6], [−46 −72 6]). The analysis was repeated on theROIs (8-mm radius spheres) centered on the peaks, as

reported in Table 6. The main effect (backward visual walk >forward visual walk) was significant in both left and right TOJ(for all ROIs P < 0.001). The interaction [(BW/N)− (FW/R)] >[(FW/N)− (BW/R)] was significant only in right TOJ (for allROIs P < 0.05). No other activation emerged from this newanalysis.

Finally, we explored, in further detail, the effects of themoving stimuli on the more rostral components of the AON.We considered 4 ROIs in the premotor/prefrontal regions cen-tered on the local maxima (see Fig. 2) that are nearest to thecoordinates reported in the meta-analysis study by Grosbraset al. (2012): Superior frontal gyrus ([−24 2 58]), inferior pre-central gyrus (IPG) ([−50 10 32]), and left and right inferiorfrontal gyrus (IFG) ([52 22 20], [−56 6 22]). Within theseROIs, we calculated the main effects (forward visual walk >backward visual walk) and (normal rendering > reversed ren-dering) and the interactions (real walk) for both Experiments 1and 2. Only for Experiment 1, significance was reached for themain effect (forward visual walk > backward visual walk) inthe IPG (P < 0.05, corrected for the number of ROIs) and forthe interaction in the right IFG (P = 0.015, corrected for thenumber of ROIs).

Comparing Experiments 1 and 2The effect of the task was investigated by testing the patternsof activation in Experiment 2 considering the 11 ROIs definedby the clusters activated in Experiment 1 (Table 7; see Methodsfor ROIs definition). For the contrast (backward visual walk >forward visual walk), the test involved 5 ROIs. In the 2 ROIswithin the left and right TOJ, the activity was statistically indis-tinguishable in the task (Experiment 1) and free-viewing(Experiment 2) conditions (univariate t-test; Table 7). Thesimilarity between experiments of the activity in this area is

Figure 6. Experiment 2. Effect of visual walk (backward > forward). FW: forward real walk; BW: backward real walk; forward/backward: direction of the visual walk. Same formatas Figure 3.

Table 6Peaks of cluster activations in Experiment 2

Anatomical areas Cluster size MNI peak coordinates Z-score

x y z

Backward visual walkLeft temporo-occipital junction 915 −48 −74 10 5.62Right temporo-occipital junction 1286 50 −64 8 6.90

Reverse renderingRight temporo-occipital junction 465 50 −64 10 5.06

Note: whole-brain analysis

8 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

confirmed by the analysis of the contrast (backward real walk> forward real walk) (univariate t-test; Table 7).

The role of TOJ was investigated further by testing simplecontrasts in Experiment 2 within the ROIs identified in Exper-iment 1. The tests showed a pattern in keeping with the behav-ioral results of Experiment 1 (Table 1). Stimuli showing anactor moving backward on the display activated TOJ morestrongly than those showing an actor moving forward: ([BW/N > FW/N]: P = 0.001; [FW/R > BW/R]: P = 0.02; [BW/N >BW/R]:P = 0.0793; [FW/R > FW/N]: P < 0.001). Moreover, the activityin the condition BW/R was higher than in the condition FW/N(P = 0.0001). In contrast, conditions FW/R and BW/N were in-distinguishable (P = 0.358). Thus, whether or not an identifi-cation response was required, TOJ responded differentially tothe visual direction of the stimuli. Moreover, within forwardvisual walk stimuli, the activity in TOJ discriminated betweenrendering modes.

In the other ROIs associated in Experiment 1 with the con-trasts (backward visual walk > forward visual walk), (normal >reverse), and (backward real walk > forward real walk), thefree-viewing condition of Experiment 2 failed to confirm thedifferential activations found previously. The differencebetween experiments within these ROIs was confirmed statisti-cally (univariate t-test, Table 7).

Discussion

Video clips of walking actors were shown in 2 brain imagingexperiments. Four types of displays resulted from crossing thedirection of the real walk with the rendering of the video clip.Identification performance in Experiment 1 (Table 1) con-firmed the results of Viviani et al. (2011). Accuracy in decidingwhether a stimulus was the original video clip or its time-reversed version was at chance level in the 2 conditions (FW/Rand BW/N) where the actor’s body moved backward on thedisplay (backward visual walk). Accuracy improved signifi-cantly in the other 2 conditions (forward visual walk), beingvery good when a real forward walk was rendered normally(FW/N; P {C } = 0.870) and well above chance level when a realbackward walk was time-reversed (BW/R; P {C } = 0.703).

The main fMRI result was that TOJ responded bilaterallymore vigorously to backward visual walk (when it is imposs-ible to identify the rendering mode) than to forward visualwalk, irrespective of whether backward walk was real, or itresulted from time-reversing a forward walk. The inverserelation between TOJ activation and identification performancewas emphasized further by the analysis of simple effects. Notonly did identifiable stimuli (FW/N and BW/R) activate TOJless than indistinguishable ones (BW/N and FW/R), but alsothe cortical response was weaker in the easier condition (FW/N) than in the more difficult one (BW/R). TOJ activity was verysimilar both when the task required an identification response(Experiment 1) and in the free-viewing condition (Experiment 2).Instead, the areas that are generally considered as componentsof the AON reacted differently in the 2 experiments. In Exper-iment 1, sensitivity to the rendering mode emerged mostclearly in the anterior cingulate cortex (ACC), but only whenthe visual walk was in the FW and identification responseswere fairly accurate.

Effect of Visual Walk Direction on TOJThe first issue to be addressed is the functional significance ofTOJ activity in the context of our experiment. The finding thatTOJ responds differently to stimuli that do or do not complywith the usual walking direction is incompatible with oneinfluential model for the recognition of biological motion(Giese and Poggio 2003), postulating that differences in corti-cal response to the forward and backward walk should emergeno earlier than STS. Our results are instead in keeping withmore recent evidence on the role of the early stages of visualprocessing in body perception. Jastorff and Orban (2009) sug-gested that an integration of the actor’s body with the actionhe/she is performing takes place already in the temporo-occipital cortex. Moreover, de Lange et al. (2008) showed thatactions implemented in unusual manners elicit a strongeractivity than ordinary actions in the lateral temporo-occipitalcortex, where the context in which body parts are viewedwould be taken into account. It is already known that an areaof the lateral temporo-occipital cortex (EBA) responds selec-tively to visual stimuli representing body movements or bodyparts (Downing et al. 2001; Peelen et al. 2006), is activated bylimb movements (Astafiev et al. 2004), links a portrayed move-ment to the body scheme (Jastorff and Orban 2009), and dis-tinguishes self- from other-generated movements (David et al.2007). Along similar lines, TOJ responses to forward and back-ward visual walk may be taken to indicate that this area is se-lectively sensitive to the global familiarity of the action.

Neural Response to FamiliarityIt is generally agreed that activity in the AON increases inresponse to stimuli conforming to the motor experience of theobserver (Calvo-Merino et al. 2005, 2006; Aglioti et al. 2008;Cross, Hamilton, et al. 2009; Cross, Kraemer, et al. 2009). More-over, Giese and Poggio (2003) argued that a video clip ofwalking played normally should elicit significantly moreactivity than that played in reverse. We found instead that TOJresponds more strongly to an action (backward walk) that werarely execute and see other people executing. A similar con-tradiction has been noted by Cross et al. (2012), who hypoth-esized a U-shaped relation between cortical activity and thedegree of familiarity of the action being displayed, so that both

Table 7Analysis of ROIs

Anatomical areas No task Task > no taskP-corr P-uncorr

Visual walk: backward > forwardLeft temporo-occipital junction <0.001 0.272Right temporo-occipital junction <0.001 0.619Left intraparietal sulcus 0.79 0.020Right intraparietal sulcus 0.79 0.004Right superior temporal sulcus 0.08 0.031

Rendering: normal > reverseLeft fusiform gyrus 0.537 0.002Anterior cingulate cortex 0.987 <0.001Right precuneus 0.939 <0.001

Interaction (real walk: forward > backward)Left orbito-frontal cortex 0.437 0.003Left posterior cingulate gyrus 0.338 <0.001

Interaction (real walk: backward > forward)Right temporo-occipital junction 0.013 0.087

No task: P-values (Bonferroni-corrected for the number of ROIs) for data in Experiment 2 within theROIs detected in Experiment 1 (Table 4). Task > no task: comparison between Experiments 1 and2. Uncorrected P-values.

Cerebral Cortex 9

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

very familiar and unfamiliar actions would result in high BOLDsignals. These authors argued that, within a Bayesian perspec-tive, the non-monotonic relationship between familiarity andcortical activity is compatible with the notion of predictionerror (Kilner et al. 2007a, 2007b). Specifically, one may assumethat, upon seeing the onset of an action, we entertain priorexpectations about its future course, which are all the morespecific that the action appears to be a familiar one. Therefore,large prediction errors (i.e., sharp discrepancies between ex-pected and actual course of action) would occur both whenthe action is very unfamiliar and the associated prior expec-tations are poorly defined, and when the action is very familiar,but its actual course diverges from the expected one. In eithercase, the level of activation would correlate with the predictionerror. Seeing someone walking is a very familiar visual experi-ence. Thus, if we adopt the above line of reasoning, the strongactivation of TOJ for backward visual walk suggests that thisarea is signaling a strong discrepancy between expected andactual input. Of course, the main discrepancy arises when thevisual direction of the body displacement is opposite to thenatural one. However, kinematics may be a further source ofdiscrepancy. As mentioned in the Introduction, the kinematicsof a backward walk is significantly different from that of aforward walk reversed in time. Thus, in the BW/R condition,the body is actually moving in the expected FW, but the kin-ematics is somewhat at variance with the visual direction. Onthe one hand, this may explain why identification performanceis less good with BW/R than FW/N stimuli. Let us assumethat the forward visual direction biases responses towardthe “as recorded” option. Then, the perfect match of FW/Nstimuli with the expected kinematic template would furtherprime this (correct) response. Instead, the assumed bias is inconflict with the mismatch detected in BW/R stimuli, whichfavors a “reversed” (correct) response. On the other hand, vio-lations of kinematic expectations may also account for theresult of the interaction (backward real walk > forward realwalk), showing that, when the visual walk is forward, TOJ isactivated more strongly by backward (BW/R) than by forward(FW/N) real walk (see lower panel in Fig. 5). Thus, even whenthe visual direction is familiar, TOJ signals that the kinematicsdiverges from the one expected in the case of a real forwardwalk.

The demonstration that TOJ was activated differentially bythe direction of the visual walk lends itself to at least 2interpretations. Both interpretations share the premise thatTOJ reacts to backward visual walk by signaling a mismatchbetween the stimulus and an internal model embodying the apriori expectations of the observer. The interpretations divergesomewhat insofar as the consequences of the mismatch areconcerned. On the one hand, it can be surmised (Friston 2005)that the brain actually uses the mismatch as an error signal tobroaden the scope of the internal model, by progressivelyincluding kinematic configurations not previously experi-enced. In other words, the error signal would provide the basisfor perceptual learning. On the other hand, as suggested byViviani et al. (2011), the mismatch may prevent the extractionof the kinematic cues allowing the observer to detect temporalinversions. If so, a failure to pick up and feed forward thesecues should have no consequence in free-viewing conditions.In contrast, when a response is requested it should be expectedthat areas downstream from the TOJ react differently to the di-rection of the visual walk.

It is not easy to assess conclusively the merits of these 2interpretations. The fact that we detected no change across ses-sions in either performance or cortical response speaks againstthe hypothesis that the error signal is used to support learning.Moreover, in 3 areas downstream from the TOJ (IPS bilaterallyand right STS), the response was stronger to backward than toforward visual walk (Fig. 3), suggesting a spread of the mis-match detected by the TOJ. However, because the time con-stant of the learning process may exceed the relatively short(40 min) duration of our experiments, one cannot exclude thatthe 2 interpretations are both valid. On a short time scale, theincongruence between expected and actual kinematics mayindeed prevent any further processing of the stimuli. At thesame time, with a prolonged exposure to the unusual (back-ward) visual walk, the error signal might drive an update ofthe expectations and a corresponding improvement of theperformance.

Two related factors may provide an alternative account ofthe modulation of cortical activity by the direction of the visualwalk. One can argue that identifying the rendering modeis both more difficult and more attention demanding for theunusual backward visual walk than for familiar forward visualwalk. Thus, the increased BOLD signal in the former case maysimply reflect both the increased difficulty of the task and thecorresponding higher level of attention required to cope with it.In fact, this hypothesis is in agreement with the observed IPSresponses, which in Experiment 1 were higher to backwardthan to forward visual walk (Fig. 3) and disappeared in Exper-iment 2. In contrast, in the TOJ, the BOLD signal was essentiallythe same (Table 7), irrespective of whether the identificationtask was imposed (Experiment 1) or not (Experiment 2). There-fore, task difficulty and attention levels do not seem to provide aviable alternative to the assumption that TOJ activity signals amismatch between the actual and expected visual direction.

The activation of STS and IPS in the context of the decisionprocess is compatible with the role of these structures. STS isinvolved in the understanding of dynamic stimuli (Puce andPerett 2003; Herrington et al. 2011). It has been suggested thatSTS codes walking direction (Oram and Perrett 1996) and isactivated when the observer predictions on the unfolding ofthe action are violated (Grèzes et al. 2003; Jastorff et al. 2011).Also, the posterior parietal area is involved in the processing ofthe visual properties of movements and in the generation ofvisuo-motor transformations (Decety and Grèzes 1999). In par-ticular, the IPS is supposed to be instrumental for bringing theobserver motor competence to bear for understanding the goalof an action (Fogassi et al. 2005), and for discriminating amongdifferent actions (Jastorff et al. 2010). The fact that both STSand IPS were activated more energetically when renderingidentification is close to impossible (FW/R and BW/N) maythen reflect the difficulty arising with unfamiliar stimuli, whenthe decision process is not supported by adequate discriminalinformation (Liew et al. 2011).

Is Perceptual Identification Mediated by MotorCompetence?In our previous study (Viviani et al. 2011), we discussed why thedifference in step size between FW and BW is unlikely to be acritical cue for deciding whether or not the stimuli are time-reversed. We argued that discriminal cues could instead be pro-vided by the interplay between perceived kinematics and

10 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

implicit motor competence. Parietal and premotor areas encodeboth performed and observed actions in a way that is fairlyunique for different actions (Rizzolatti and Craighero 2004;Kilner et al. 2007a, b; Oosterhof et al. 2010). If indeed theforward-moving body triggers a motoric simulation of the action,identification may then result from a congruence check betweenperceived and expected kinematics. The results did not supportthis suggestion. When a response was not required, forward andbackward visual walk did not activate differentially any of theregions that, according to recent meta-analyses (Caspers et al.2010; Grosbras et al. 2012), belong to the AON. Moreover, noneof these regions reacted differently to normal and time-reversedstimuli. Conversely, the 3 areas that in Experiment 1 were acti-vated more strongly by normal than by time-reversed stimuli,namely the left FG, the PrCu, and the ACC (Fig. 4), are notknown to be involved in the interplay between motoric compe-tence and perception. Instead, the available evidence invites theinference that time-reversals are detected mostly by perceptualprocesses, rather early within the visual stream.

Effect of Rendering ModeAs mentioned above, the (normal rendering vs. reverse render-ing) contrast detected activity in the FG, PrCu, and ACC. FGhas a role in the identification of biological motion (Bondaet al. 1996; Vaina et al. 2001; Grossman and Blake 2002;Beauchamp et al. 2003). The observed PrCu sensitivity to therendering mode (Table 4) is consistent with previous work(Hasson et al. 2008), suggesting that this area is involved in theaccumulation of information over a time window required toperceive the coherence of visual events (Bischoff-Grethe et al.2000; Hasson et al. 2008). However, the functional significanceof these 2 areas in the present context is unclear because theywere actually deactivated. Finally, the anterior cingulate cortex(ACC) has a role in monitoring conflicts between competinginputs (Botvinick et al. 1999; Carter et al. 1999) decisionmaking, and action selection (Rushworth et al. 2007). Thedorsal region of the ACC is activated by cognitive demandingtasks in which conflicting evidence may lead to erroneousresponses (Bush et al. 2000). The difference between ACCactivity elicited by normal and reversed stimuli was largerwhen visual walk was forward than backward. Insofar as ACCis directly connected to both premotor and motor areas (Hata-naka et al. 2003), this pattern of activations may correspond tothe effect of the rendering mode on one relay station prior tothe motor response.

Summary and ConclusionsWe have shown that, as face recognition is crucially dependenton the familiar (upright) spatial orientation, so correct percep-tion of walk kinematics is crucially dependent on the familiar(forward) visual direction of walk. In both cases, visual famili-arity dictates whether or not stimuli details are accuratelyprocessed. It has been claimed that different components ofthe AON deal with different aspects of the action (intention,rationality, kinematics, and activated body part). In particular,familiarity is supposed to be a rather abstract aspect of theaction evaluated within parietal and motor areas (Calvo-Merinoet al. 2006). The new significant contribution of this study is thedemonstration that familiar and unfamiliar walking directionsare instead discriminated fairly early along the visual stream,mainly in the TOJ. Because a similar pattern of activity in this

area was observed in both the task and free-viewing conditions,TOJ appears to perform a preliminary automatic analysis of thestimuli. More specifically, we suggest that unfamiliar stimulievoke an error signal that interferes with the further processingof the kinematics and is ultimately responsible for the pooridentification performancewith these stimuli.

TOJ may also have a role when identification of the renderingmode is feasible. Asked whether a walk has been shown as re-corded or reversed in time, participants in Experiment 1 werequite accurate when the body moved forward. To respond“Normal” to FW/N trials, the participant had to decide that noelement in the display was in conflict with the visual direction.To respond “Reversed” to BW/R trials, the observer must havedetected at least one incongruence with respect to the visualdirection. Albeit different, both decision criteria necessarilyrequire the availability of kinematic templates describing the ex-pected time course of the stance and swing phase in forwardwalk. The fMRI results of Experiment 1 cannot adjudge conclus-ively the outstanding issue of where this information becomesavailable. Indeed, the (backward visual direction > forwardvisual direction) contrast of Figure 3 showed activations inseveral areas—including parietal and prefrontal ones—that aregenerally included in the AON. However, the (backward realwalk > forward real walk) contrast (Fig. 5) demonstrated that,when the visual direction is forward, TOJ is activated morestrongly by backward (BW/R) than by forward (FW/N) realwalk. This may be taken to indicate that, as suggested above,correct responses to BW/R trials are based on a mismatchbetween incoming information and a kinematic walk template,and that the mismatch is detected as early as TOJ. The involve-ment in the identification process of other areas downstreamfrom TOJ could not be demonstrated conclusively. Further re-search focusing on the 2 conditions conducive to correct identi-fication is necessary to investigate this issue.

Supplementary MaterialSupplementary material can be found at: http://www.cercor.oxfordjournals.org/.

NotesWe thank the anonymous referees for their comments and suggestions.Conflict of Interest: The authors declare no competing financial interests.

Funding

This work was supported by the Italian Ministry of Health(RF-10.057 grant), Italian Ministry of University and Research(PRIN grant), and Italian Space Agency (CRUSOE and COREAgrants).

ReferencesAglioti SM, Cesari P, Romani M, Urgesi C. 2008. Action anticipation and

motor resonance in elite basketball players. Nat Neurosci.11:1109–1116.

Astafiev SV, Stanley CM, Shulman GL, Corbetta M. 2004. Extrastriatebody area in human occipital cortex responds to the performanceof motor actions. Nat Neursosci. 7:542–548.

Beauchamp MS, Lee KE, Haxby JV, Martin A. 2003. fMRI responses tovideo and point-light displays of moving humans and manipulableobjects. J Cogn Neurosci. 15:991–1001.

Cerebral Cortex 11

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

Bischoff-Grethe A, Proper SM, Mao H, Daniels KA, Berns GS. 2000.Conscious and unconscious processing of nonverbal predictabilityin Wernicke’s area. J Neurosci. 20:1975–1981.

Bonda E, Petrides M, Ostry D, Evans A. 1996. Specific involvement ofhuman parietal systems and the amygdala in the perception of bio-logical motion. J Neurosci. 16:3737–3744.

Botvinick M, Nystrom LE, Fissel K, Carter CS, Cohen JD. 1999. Conflictmonitoring versus selection-for-action in anterior cingulate cortex.Nature. 402:179–181.

Brass M, Schmitt RM, Spengler S, Gergely G. 2007. Investigating actionunderstanding: inferential processes versus action simulation. CurrBiol. 17:2117–2121.

Bush G, Luu P, Posner MI. 2000. Cognitive and emotional influences inanterior cingulate cortex. Trends Cogn Sci. 4:215–222.

Calvo-Merino B, Glaser DE, Grèzes J, Passingham RE, Haggard P.2005. Action observation and acquired motor skills: an fMRI studywith expert dancers. Cereb Cortex. 15:1243–1249.

Calvo-Merino B, Grèzes J, Glaser DE, Passingham RE, Haggard P.2006. Seeing or doing? Influence of visual and motor familiarity inaction observation. Curr Biol. 16:1905–1910.

Carter CS, Botvinick MM, Cohen JD. 1999. The contribution of theanterior cingulate cortex to executive processes in cognition. RevNeurosci. 10:49–57.

Caspers S, Zilles K, Laird AR, Eickoff SB. 2010. ALE meta-analysis ofaction observation and imitation in the human brain. Neuroimage.50:1148–1167.

Cross ES, Hamilton AF, Grafton ST. 2006. Building a motor simulation denovo: observation of dance by dancers. Neuroimage. 31:1257–1267.

Cross ES, Hamilton AF, Kraemer DJ, Kelley WM, Grafton ST. 2009. Dis-sociable substrates for body motion and physical experience in thehuman action observation network. Eur J Neurosci. 30:1383–1392.

Cross ES, Kraemer DJM, Hamilton AF, Kelley WM, Grafton ST. 2009.Sensitivity of the action observation network to physical and obser-vational learning. Cereb Cortex. 19:315–326.

Cross ES, Liepelt R, Hamilton AF de C, Parkinson J, Ramsey R, StadlerW, Prinz W. 2012. Robotic movement preferentially engages theaction observation network. Hum Brain Mapp. 33:2238–2254.

David N, Cohen MX, Newen A, Bewernick BH, Shah NJ, Fink GR,Vogeley K. 2007. The extrastriate cortex distinguishes between theconsequences of one’s own and others’ behavior. Neuroimage.36:1004–1014.

Decety J, Chaminade T. 2003. When the self represents the other: anew cognitive neuroscience view on psychological identification.Conscious Cogn. 12:577–596.

Decety J, Grèzes J. 1999. Neural mechanisms subserving the percep-tion of human actions. Trends Cogn Sci. 3:172–178.

de Lange FP, Spronk M, Willems RM, Toni I, Bekkering H. 2008. Comp-lementary systems for understanding action intentions. Curr Biol.18:454–457.

Downing PE, Jiang Y, Shuman M, Kanwisher N. 2001. A cortical areaselective for visual processing of the human body. Science.293:2470–2473.

Duvernoy HM. 1991. The human brain: surface, three-dimensional sec-tional anatomy and MRI. New York: Springer.

Fogassi L, Ferrari PF, Gesierich B, Rozzi S, Chersi F, Rizzolatti G. 2005.Parietal lobe: from action organization to intention understanding.Science. 308:662–667.

Friston K. 2005. A theory of cortical responses. Philos Trans R SocLond B Biol Sci. 360:815–836.

Frith CD, Frith U. 2006. How we predict what other people are goingto do. Brain Res. 1079:36–46.

Frith U, Frith CD. 2003. Development and neurophysiology of menta-lizing. Philos Trans R Soc Lond B Biol Sci. 358:459–473.

Gallese V. 2006. Intentional attunement: a neurophysiological perspectiveon social cognition and its disruption in autism. Brain Res. 1079:15–24.

Gallese V, Goldman A. 1998. Mirror neurons and the simulation theoryof mindreading. Trends Cogn Sci. 2:493–501.

Gallese V, Keysers C, Rizzolatti G. 2004. A unifying view of the basis ofsocial cognition. Trends Cogn Sci. 8:396–403.

Giese MA, Poggio T. 2003. Neural mechanisms for the recognition ofbiological movements. Nat Rev Neurosci. 4:179–192.

Grasso R, Bianchi L, Lacquaniti F. 1998. Motor patterns for humangait: backward versus forward locomotion. J Neurophysiol. 80:1868–1885.

Grèzes J, Frith CD, Passingham RE. 2003. Inferring false beliefs fromthe actions of oneself and others: an fMRI study. Neuroimage.21:744–750.

Grosbras MH, Beaton S, Eickhoff SB. 2012. Brain regions involved inhuman movement perception: a quantitative voxel-based meta-analysis. Hum Brain Mapp. 33:431–454.

Grossman ED, Blake R. 2002. Brain areas active during visual percep-tion of biological motion. Neuron. 35:1167–1175.

Hagberg GE, Zito G, Patria F, Sanes JN. 2001. Improved detection ofevent-related functional MRI signals using probability functions.Neuroimage. 14:1193–1205.

Han Z, Bi Y, Chen J, Chen Q, He Y, Caramazza A. 2013. Distinctregions of right temporal cortex are associated with biological andhuman-agent motion: functional magnetic resonance imaging andneuropsychological evidence. J Neurosci. 33:15442–15453.

Hasson U, Yang E, Vallines I, Heeger DJ, Rubin N. 2008. A hierarchy oftemporal receptive windows in human cortex. J Neurosci.28:2539–2550.

Hatanaka N, Tokuno H, Hamada I, Inase M, Ito Y, Imanishi M, Hasega-wa N, Akazawa T, Nambu A, Takada M. 2003. Thalamocortical andintracortical connections of monkey cingulate motor areas. J CompNeurol. 462:121–138.

Herrington JD, Nymberg C, Schultz RT. 2011. Biological motion taskperformance predicts superior temporal sulcus activity. BrainCogn. 77:372–381.

Iacoboni M. 2005. Understanding others: imitation, language, andempathy. In perspectives on imitation: from neuroscience to socialscience. Massachusetts: MIT Press. p. 77–100.

Jastorff J, Begliomini C, Fabbri-Destro M, Rizzolatti G, Orban GA. 2010.Coding observed motor acts: different organizational principles inthe parietal and premotor cortex of humans. J Neurophysiol.104:128–140.

Jastorff J, Clavagnier S, Gergely G, Orban GA. 2011. Neural mechan-isms of understanding rational actions: middle temporal gyrus acti-vation by contextual violation. Cereb Cortex. 21:318–329.

Jastorff J, Orban GA. 2009. Human functional magnetic resonanceimaging reveals separation and integration of shape and motioncues in biological motion processing. J Neurosci. 29:7315–7329.

Katzavelis D, Mukherjee M, Decker L, Stergiou N. 2010. Variability oflower extremity joint kinematics during backward walking in avirtual environment. Nonlin Dyn Psychol Life Sci. 14:165–178.

Kilner JM, Friston KJ, Frith CD. 2007a. The mirror-neuron system: aBayesian perspective. Neuroreport. 18:619–623.

Kilner JM, Friston KJ, Frith CD. 2007b. Predictive coding: an account ofthe mirror neuron system. Cogn Process. 8:159–166.

Kilner JM, Neal A, Weiskopf N, Friston KJ, Frith CD. 2009. Evidence ofmirror neurons in human inferior frontal gyrus. J Neurosci.29:10153–10159.

Liew SL, Han S, Aziz-Zadeh L. 2011. Familiarity modulates mirrorneuron and mentalizing regions during intention understanding.Hum Brain Mapp. 32:1986–1997.

Lingnau A, Gesierich B, Caramazza A. 2009. Asymmetric fMRI adap-tation reveals no evidence for mirror neurons in humans. Proc NatlAcad Sci. 106:9925–9930.

Mahon BZ, Caramazza A. 2008. A critical look at the embodied cogni-tion hypothesis and a new proposal for grounding conceptualcontent. J Physiol (Paris). 102:59–70.

Molnar-Szakacs I, Iacoboni M, Koski L, Mazziotta JC. 2005. Functionalsegregation within pars opercularis of the inferior frontal gyrus:evidence from fMRI studies of imitation and action observation.Cereb Cortex. 15:986–994.

Oosterhof NN, Wiggett AJ, Diedrichsen J, Tipper SP, Downing PE.2010. Surface-based information mapping reveals crossmodalvision-action representations in human parietal and occipito-temporal cortex. J Neurophysiol. 104:1077–1089.

Oram MW, Perrett DI. 1996. Integration of form and motion in theanterior superior temporal polysensory area (STPa) of the macaquemonkey. J Neurophysiol. 76:109–129.

12 Early Detection of Unfamiliar Walking Movements • Maffei et al.

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from

Peelen MV, Wiggett AJ, Downing PE. 2006. Patterns of fMRI activitydissociate overlapping functional brain areas that respond to bio-logical motion. Neuron. 49:815–822.

Pelphrey KA, Morris JP, McCarthy G. 2004. Grasping the intentions ofothers: the perceived intentionality of an action influences activityin the superior temporal sulcus during social perception. J CognNeurosci. 16:1706–1716.

Penny WD, Holmes AP, Friston KJ. 2003. Random effects analysis. In:Frackowiak RSJ, Friston KJ, Frith C, Dolan R, Price CJ, Zeki S, Ash-burner J, Penny WD. editors. Human brain function. 2nd ed. SanDiego: Academic Press. p. 843–850.

Press C. 2011. Action observation and robotic agents: learning andanthropomorphism. Neurosci Biobehav Rev. 35:1410–1418.

Puce A, Perrett D. 2003. Electrophysiology and brain imaging ofbiological motion. Philos Trans R Soc Lond B Biol Sci. 358:435–445.

Rizzolatti G, Craighero L. 2004. The mirror-neuron system. Annu RevNeurosci. 27:169–192.

Rushworth MFS, Behrens TEJ, Rudebeck PH, Walton ME. 2007. Con-trasting roles for cingulate and orbitofrontal cortex in decisions andsocial behavior. Trends Cogn Sci. 11:168–176.

Thioux M, Gazzola V, Keysers C. 2008. Action understanding: how,what and why. Curr Biol. 18:R431–R434.

Thorstensson A. 1986. How is the normal locomotor program modifiedto produce backward walking? Exp Brain Res. 61:664–668.

Vaina LM, Solomon J, Chowdhury S, Sinha P, Belliveau JW. 2001. Func-tional neuroanatomy of biological motion perception in humans.Proc Natl Acad Sci USA. 98:11656–11661.

Viviani P, Figliozzi F, Campione GC, Lacquaniti F. 2011. Detecting tem-poral reversals in human locomotion. Exp Brain Res. 214:93–103.

Winter RF, Pluck N, Yang JF. 1989. Backward walking: a simple rever-sal of forward walking? J Motor Behav. 21:291–305.

Cerebral Cortex 13

at Fondazione S. Lucia IR

CC

S on February 17, 2014http://cercor.oxfordjournals.org/

Dow

nloaded from