Eye Movements During Mental Time Travel Follow a Diagonal Line

Running head: EYE MOVEMENTS DURING MENTAL TIME TRAVEL 1

Eye Movements During Mental Time Travel Follow a Diagonal Line

Matthias Hartmann1,2, Corinna S. Martarelli1, Fred W. Mast1, and Kurt Stocker3

Author Note

1 Department of Psychology, University of Bern, Fabrikstrasse 8, CH-3012 Bern,

Switzerland; 2 Division of Cognitive Sciences, University of Potsdam, Germany; 3

Department of Psychology, University of Zurich, Binzmühlestrasse 14/1 CH-8050 Zürich,

Switzerland.

Correspondence regarding this article should be addressed to:

Matthias Hartmann

Division of Cognitive Sciences

University of Potsdam

Karl-Liebknecht-Strasse 24-25 House 14

D-1446 Potsdam OT Golm, Germany

E-mail: [email protected]

EYE MOVEMENTS DURING MENTAL TIME TRAVEL 2

Abstract

Recent research showed that past events are associated with the back and left side, whereas

future events are associated with the front and right side of space. These spatial-temporal

associations have an impact on our sensorimotor system: Thinking about one’s past and future

leads to subtle body sways in the sagittal dimension of space (Miles, Nind, & Macrae, 2010).

In this study we investigated whether mental time travel leads to sensorimotor correlates in

the horizontal dimension of space. Participants were asked to mentally displace themselves

into the past or future while measuring their spontaneous eye movements on a blank screen.

Eye gaze was directed more rightward and upward when thinking about the future than when

thinking about the past. Our results provide further insight into the spatial nature of temporal

thoughts, and show that not only body, but also eye movements follow a (diagonal) “time

line” during mental time travel.

Keywords: mental time travel, eye movements, mental time line, spatial-temporal

association, future, past, embodied cognition, metaphors, mental number line


1. Introduction

The passing of time is not directly graspable by the human senses. The metaphorical

mapping view assumes that abstract concepts are expressed in entities of concrete domains

(e.g., Barsalou, 2008; Boroditsky, 2000; Gallese & Lakoff, 2005; Lakoff & Johnson, 1980).

In line with this view, humans often use the domain of space, which is thought to be concrete,

in order to understand and structure the more abstract concept of time (Boroditsky, 2000;

Casasanto & Boroditsky, 2008). Especially, future and past events are often described by

means of spatially distinct locations. For example, events in the past lie behind us, whereas

we are looking forward to events in the future. In Western cultures, two spatial-temporal

associations have been identified: past-back and future-front association (Casasanto & Jasmin,

2012; Hartmann & Mast, 2012; Miles, Karpinska, Lumsden, & Macrae, 2010; Miles, Nind, &

Macrae, 2010; Ulrich et al., 2012; Walker, Bergen, & Núñez), as well as a past-left and

future-right association (Casasanto & Jasmin, 2012; Ouellet, Santiago, Funes, & Lupianez,

2010; Saj, Fuhrman, Vuilleumier, & Boroditsky, 2014; Santiago, Lupianez, Perez, & Funes,

2007; Torralbo, Santiago, & Lupianez, 2006; Ulrich & Maienborn, 2010; Weger & Pratt,

2008). While the past-back and future-front association is in line with the metaphorical

language use, there are no equivalent expressions in language associating past with the left

and future with the right side of space (Casasanto & Jasmin, 2012; Cienki, 1998; Radden,

2004). It has been argued that temporal associations with the horizontal space can be

explained by reading and writing direction from left-to-right in Western cultures (e.g., Bergen

& Chan Lau, 2012; Casasanto & Bottini, 2014; Fuhrman & Boroditsky, 2007; Santiago et al.,

2007; Torralbo et al., 2006). The “mental time line” has been established as an analogous

concept to the “mental number line”: Not only numbers are represented in an ascending order

from left to right (e.g., Dehaene, Bossini, & Giraux, 1993; Fischer, Castel, Dodd, & Pratt,

2003; Hartmann, Grabherr, & Mast, 2012; Hubbard, Piazza, Pinel, & Dehaene, 2005;


Lourenco & Longo, 2010; Wood, Willmes, Nuerk, & Fischer, 2008), but also time flies from

left to right (e.g., Santiago et al., 2007).

Most empirical evidence for the past-left and future-right association comes from

temporal categorization tasks where future and past-related stimuli need to be categorized as

future or past in two-alternative forced-choice reaction time tasks. Results show that left-sided

responses are faster for past than for future-related stimuli, whereas the opposite is true for

right-sided responses (e.g., Santiago et al., 2007; Torralbo et al., 2006; Ulrich & Maienborn,

2010). Because these tasks require participants to categorize stimuli to the past or future, the

concepts of past and future as well as the response options left and right are explicitly part of

the task representation. As a consequence, these results do not provide direct evidence that

past events are inherently represented in the left, and future events in the right side of space.

Rather, these congruence effects can also be explained by stimulus-response compatibility

such as the polarity correspondence account (Proctor & Cho, 2006). This account would

suggest that past and left are coded with a negative, and future and right with a positive

polarity, thus leading to faster responses when theses polarities are congruent. Thus, shorter

response times in the congruent conditions can be explained without making the assumption

of a direct link between the two concepts (past-left, future-right). In fact, when the same

stimuli needed to be categorized as word or non-word (also with a left and right key), the

past-left and future-right advantage disappeared (Flumini & Santiago, 2013; see also Ulrich &

Maienborn, 2010). The nature of the past-left and future-right association and the conditions

under which it occurs remain unclear. It is an open question whether the mental time line is

automatically activated during temporal processing (Flumini & Santiago, 2013), and the

polarity correspondence account points to the possibility that the horizontal mental time line is

merely an epiphenomenon of temporal categorization tasks.

Mental time travel is an alternative paradigm that allows investigating spatial-temporal

associations by avoiding stimulus-response compatibility effects. In episodic mental time


travel, participants are asked to mentally displace themselves into their subjective past or

future (i.e., remembering vs. autobiographical temporal imagining; see Stocker, 2012;

Tulving, 2002). Mental time travel has been used to study the past-back and future-front

association. For example, Miles, Nind et al. (2010) found that participants leaned slightly

forward when they imagined how their life circumstances would look like in the future, and

their body swayed slightly backward when thinking about their life circumstances in the past.

Miles, Karpinska et al. (2010) also analyzed the future and past-related contents of

spontaneous mind-wandering during a vigilance task. More future-related thoughts came to

participants’ minds when the task included visually induced forward motion (vection),

whereas backward motion induced more past-related thoughts. Thus, thinking about the past

and future seems to be intertwined with backward and forward motion, following the

metaphorical language use (see also Hartmann & Mast, 2012).

In this study, we investigated the horizontal mental time line by asking participants to

mentally displace themselves into the past and future (i.e., episodic mental time travel). If

mental time travel has a behavioral correlate in the horizontal dimension of space, this would

strengthen the evidence that the horizontal mental time line is more than an epiphenomenon

of temporal categorization tasks. We used eye movements as an indicator of the location of

spatial attention (e.g., Corbetta et al., 1998; Sheliga, Riggio, & Rizzolatti, 1994). Participants

were asked to mentally displace themselves into their subjective past and future while

spontaneous eye movements on a blank screen were measured (see Loetscher, Bockisch,

Nicholls, & Brugger, 2010 for a similar approach in the domain of numerical cognition). If

past-related events were inherently associated with the left, and future-related events with the

right side of space, we would expect more rightward directed eye gaze behavior when

participants think about the future than when they think about the past. This paradigm also

allows for assessing whether there is a vertical component in the spatial-temporal association.

The mental number line not only runs from left to right but also from bottom to top (Grade,


Lefèvre, & Pesenti, 2012; Hartmann, Gashaj, Stahnke, & Mast, in press; Hartmann et al.,

2012; Ito & Hatta, 2004; Loetscher et al., 2010; Schwarz & Keus, 2004; Winter & Matlock,

2013). Given the analogy between the mental number line and the mental time line (Arzy,

Adi-Japha, & Blanke, 2009; Bonato, Zorzi, & Umilta, 2012), it is conceivable that past is

associated with the lower, and future with the upper space, which could also influence eye

gaze behavior during mental time travel.

In addition to the implicit measurement of spatial-temporal associations by means of

eye movements, we also asked participants to indicate their explicit associations between

future and past and the horizontal and vertical space. We were interested to see whether there

is a general agreement about explicit temporal associations with the horizontal and vertical

space and whether there is a match between the explicit and implicit measurements.

2. Method

2.1 Participants

Nineteen right-handed undergraduate students from the University of Bern

participated in this study for course credit (13 women, mean age: 21.9, range: 19-28 years).

Participants gave written informed consent prior to the study, and the study was approved by

the local Ethics Committee. All participants had normal or corrected-to-normal visual acuity.

2.2 Eye Movement Recording

Eye movements were recorded by using an SMI RED tracking system (SensoMotoric

Instruments,Teltow, Germany). Data were registered with a sampling rate of 50 Hz, a spatial

resolution of 0.1° and a gaze position accuracy of 0.5°. The eye-tracking device is contact-

free and determines the gaze by combining the cornea reflex with the pupil location, via an

infrared light sensitive video camera. The stimuli were presented on a 17-inch screen (1280 x

1024 pixel) using Experiment Center Software and eye data were recorded with I-View X

Software, both developed by SensoMotoric Instruments (SensoMotoric Instruments, Teltow,

Germany). Our analyses were based on eye gaze fixations. Fixations were extracted using Be-


Gaze software (SensoMotoric Instruments, Teltow, Germany) and were defined as sum of the

gaze stream on the x- and y-axes below 100 pixels and a fixation duration exceeding 80 ms

(SensoriMotor Instruments, 2009). Blink events were subtracted from the original gaze stream

by the software.

2.3 Task and Procedure

Participants were seated in front of the computer screen. The distance between

participants and the screen was approximately 70 cm. Task instructions appeared on the

screen: “In the following task, you are asked to mentally displace yourself in time to your

personal past (future). Please think for one minute about your personal life circumstances one

year back in the past (how your personal life circumstances may look like one year in the

future). Think about what you did on a typical day one year ago (what you will be doing on a

typical day one year from now in the future). Try to remember (imagine) the episodes in as

much detail as possible (e.g., how does the location look like, what other people are there

etc.).”Instructions were presented in black (Arial, font size = 16) on a grey background.

In order to direct participant’s attention away from eye movements, we introduced a

cover story. Participants were told that the study is about the relationship between pupil size

and cognitive processes. Thus, participants should think that their pupil sizes, rather than their

eye movements, were recorded. Participants were also told that eye movements do not

influence the pupil size (in order to prevent participants from actively suppressing their eye

movements). After task instruction, a 5-point calibration and validation procedure was

performed (only error values below 0.8° were accepted), followed by a fixation cross. When

participants were ready to perform the task, they pressed the space bar and the fixation cross

disappeared. After a minute, the recording stopped and another instruction appeared on the

screen to inform participants that they now have to mentally displace themselves into the

future. Half of the participants (n = 9) started with the past, followed by the future task


instruction, and vice versa for the other half of participants. After mental time travel,

participants were asked to rate the valence of their future and past-related thoughts on a 9-

point Likert scale (1 = very negative to 9 = very positive; see Miles, Nind, et al., 2010), and

also how difficult it was to travel into the past and future (1 = very difficult to 9 = very easy).

They were also asked to guess the hypothesis of this study. Finally, they indicated their

explicit horizontal and vertical spatial association for the concepts future and past. To this

end, they received two horizontal and two vertical oriented 7-point Likert scales where the

center of the line indicated no spatial association, and the left, right, upper, and lower

endpoints indicate a strong association with this spatial direction (see Figure 2 b). The scales

can be found in the supplemental materials. At the end of the experiment, participants were

asked to report 2 examples of their thoughts during mental time travel. This allows us to

assess whether participants followed task instructions.

3. Results

When interrogated about the research hypothesis, most participants referred to our

cover story and thought that the hypothesis was about differences in the pupil size. One

participant guessed our hypothesis about the past-left and future-right association. Data of this

participant was excluded from the analysis. As in the study by Miles, Nine, et al. (2010), there

was no difference in the valence ratings between future and past-related thoughts, and future

and past mental time travels were perceived as equally difficult (for both F < 1). None of the

participants showed any difficulty to report two examples of their thoughts, suggesting that

they were all able to follow the task instructions. They reported thoughts about their future

and past housing situation, work/study environment, or about their friendships/relationships

3.1 Analysis of eye gaze during mental time travel

Fixations that were outside of the screen were excluded from the data (2.3%). For the

remaining fixations, the mean horizontal and vertical eye gaze position was computed for

each participant. In order to capture the time course of mental time travel, the eye gaze


position was averaged for six time windows: Fixations that started between 0-10, 10-20, 20-

30, 30-40, 40-50, and 50-60 s after mental time travel onset. On average, 19.2 fixations (SEM

= 0.58) were recorded in each time window (10 s) for future and past mental time travel The

mean horizontal and vertical eye gaze positions for future and past mental time travel per time

window are presented in Figure 1a and 1b. Two separate repeated measures analyses of

variance with the variables mental time travel (future, past) and time window (1, 2, 3, 4, 5, 6)

were computed for the horizontal and vertical screen position.

3.1.1 Horizontal association

For the horizontal gaze position, the ANOVA revealed a significant main effect of

mental time travel, F(1, 17) = 6.24, p = .023, ηp2 = .27. When thinking about the future, the

mean fixation position was 1.04° (48 pixels) more rightward than when thinking about the

past (see Figure 2a). Mental time travel interacted by trend with time window, F(5, 85) =

2.13, p = .070, ηp2 = .11. As shown in Figure 1a, the differences in horizontal eye position

increased with progressing time. Pairwise comparisons (t-tests) between future and past

mental time travel for each time window revealed the following p-values (uncorrected): Time

window 1 = .182, Time window 2 = .800, Time window 3 = .032, Time window 4 = .003,

Time window 5 = .035, Time window 6 = .095. Only the difference in Time window 4

survives Bonferroni corrections for multiple comparisons (significance threshold after

correction: p ≤ .008). There was no main effect of time window (F < 1).

3.1.2 Vertical association

For the vertical gaze position, the ANOVA revealed a significant main effect of

mental time travel, F(1, 17) = 4.71, p = .045, ηp2 = .22. When thinking about the future, the

mean fixation position was 1.08° (50 pixels) more upward than when thinking about the past

(see Figure 2a). No other effects were significant (F < 1).

Figure 1a and b suggest that the differences in gaze position between future and past

mental time travel were more pronounced in the second half of the task. In this time period,


16 out of 18 participants directed their gaze relatively more rightward during future than past

mental time travel, and 12 more upward during future than past mental time travel. From the

12 participants who showed a future-up association, 10 of them also showed a future-right

association. Thus, when thinking about the future, the majority of participants moved their

gaze more rightward and upward at the same time, suggesting the existence of a diagonal

mental time line rather than two independent (horizontal and vertical) lines.

3.2 Explicit spatial-temporal associations

3.2.1 Horizontal association. From the total of 18 participants, 14 participants associated

past with the left space, one with the right space, and three indicated no spatial association.

Similarly, 15 participants associated future with the right space, and three indicated no spatial

association.

3.2.2 Vertical association. From the total of 18 participants, 11 participants associated past

with the lower space, four with the upper space, and three indicated no spatial association.

Similarly, 11 participants associated future with the upper space, two with the lower space,

and five indicated no spatial association.

The mean explicit associations of future and past with the horizontal and vertical space

are illustrated in Figure 2b. For the horizontal spatial association, paired t-test revealed that

mean scores for past were significantly lower than for future, t(17) = 7.07, p < .001. The same

was true for the vertical spatial association, t(17) = 2.39, p = .029. Thus, participants

explicitly associate the past with the lower left, and the future with the top right space.

3.3 Implicit versus Explicit Associations

On the group level, there is a correspondence between explicit and implicit spatial-

temporal associations: In both measurements, future events are associated more rightward and

upward in space than past events (see Figure 2). Next, we analyzed whether the explicit and

implicit measurements of spatial-temporal associations correspond on an individual level. To

this end, we correlated the strengths of horizontal and vertical spatial-temporal association for


the implicit and explicit measurements. For the implicit measurement, we computed the

differences in the mean horizontal and vertical eye gaze position for past and future mental

time travel (Mhorizontal position future – Mhorizontal position past; Mvertical position future – Mvertical position past). For

the explicit measurement, we computed the analog differences in the score values

(Scorehorizontal position future – Scorehorizontal position past; Scorevertical position future – Scorevertical position past).

In both cases, a positive value indicates a past-left and future-right (or a past-down and future-

up) association, whereas a negative value indicates a past-right and future-left (or a past-up

and future-down) association. Thus, a positive correlation between the values of the implicit

and explicit measurements could be expected. However, there was no correlation between the

two measurements, neither for the horizontal (p = .300) nor the vertical (p = .159) association,

as revealed by Spearman-Rho tests.

3.4 Pupil Size

The previous analysis showed that thinking about the past and future is correlated with

different gaze positions. Pupil size is an indicator of cognitive load (e.g., Hartmann & Fischer,

2014; Kahneman & Beatty, 1966), and this additional analysis should clarify whether

cognitive load, which could potentially also interfere with attention orienting processes (e.g.,

Lavie, 2005; Lepsien, Griffin, Devlin, & Nobre, 2005), was different during the two

conditions. Be-Gaze software (SensoMotoric Instruments, Teltow, Germany) provides the

average pupil size for each fixation in the fixation report. The same ANOVA we described

above was conducted, with the average pupil size instead of gaze position as dependent

variable. There was no main effect of mental time travel, F(1, 17) = 1.51, p = .236, ηp2 = .08,

and no interaction with time window, F(5, 85) = 0.49, p = .782, ηp2 = .03. There was only a

main effect of time window, F(5, 85) = 7.72, p < .001, ηp2 = .31, showing a reduction of pupil

diameter from the beginning to the end of the task (pupil size in mm: M1 = 2.97, M2 = 2.94,

M3 =2.90, M4 = 2.84, M5 = 2.83, M6 = 2.83). Thus, these results suggest that cognitive load


continuously decreased during the task (probably due to familiarization with the task) and that

there is no difference with respect to the cognitive load for future and past mental time travel.

4. Discussion

The aim of this study was to investigate whether mental time travel has an observable

sensorimotor correlate in the horizontal dimension of space. We showed that participants shift

their gaze more to the right (and to the upper space) when they think about the future than

when they think about the past. Thus, thinking about the future and the past is associated with

a shift in spatial attention on the horizontal and vertical axis. Importantly, our findings

demonstrate that the horizontal spatial-temporal association is not merely an epiphenomenon

of temporal classification tasks where past and future-related stimuli are categorized by means

of left and right-sided responses. Rather, spatial-temporal associations seem to reflect a more

fundamental representation of knowledge that can influence our behavior independent of

specific task characteristics. This view is in line with other recent findings. For example, Saj

et al. (2014) found that patients suffering from spatial neglect do not only neglect the left side

of space but also past events. Moreover, past-left and future-right associations can also be

observed in spontaneous gestures (Casasanto & Jasmin, 2012; Cooperrider & Núñez, 2009) or

when people are asked to order temporal sequences onto space (Bergen & Lau, 2012;

Fuhrman & Boroditsky, 2007).

An important finding of this study is the association between past and the (relatively)

lower, and between future and the (relatively) upper space. As outlined in the introduction, the

vertical spatial-temporal association could be explained by the analogy between the mental

time line and the mental number line that also runs from the bottom to the top in ascending

order (Grade et al., 2012; Hartmann, Gashaj, Stahnke, & Mast, 2014; Hartmann et al., 2012;

Ito & Hatta, 2004; Loetscher et al., 2010; Schwarz & Keus, 2004; Winter & Matlock, 2013).

Here, we discuss further possible explanations that could contribute to this finding. First, in

the English language, there is also a “future is up” metaphor (Lakoff & Johnson, 1980).


Expressions such as “the next event is coming up” use vertical information to express time

(Radden, 2004). We are not aware of similar examples in the German language, but it is

conceivable that the experience with the English language of our German-speaking

participants was sufficient to establish a vertical temporal-spatial association. Second, an

association between future and the upper space is in line with the universal experience of

growth: As time goes by, humans and plants (or buildings during the construction period)

typically become larger – and larger size is associated with the upper space (e.g., Barsalou,

2008; Lakoff & Johnson, 1980; Sell & Kaschak, 2012). Third, it is also possible that the

vertical association that we found in this study reflects the association between the future and

the front, since the front part of a sagittal line (running from back to front) is likely to appear

higher up in space with increasing distance from the observer (Ooi, Wu, & He, 2001; Yang &

Purves, 2003). However, the origin of the vertical spatial-temporal association remains an

open issue.

A further important point of our results is the fact that, when thinking about the future,

the majority of participants moved their gaze more rightward and upward at the same time.

This suggests that time is not necessarily represented in a clear horizontal or vertical mental

line but rather on a diagonal line. Similarly, Loetscher et al. (2010) showed that participants

looked to the bottom left when they had a small number in mind, and to the top right when

they had a large number in mind. Thus, as for spatial-numerical associations, it seems that

also for spatial-temporal associations the concept of a horizontal and vertical line should be

extended to a “mental time space” (Miles, Betka, Pendry, & Macrae, 2010).

In this study, the difference in horizontal gaze direction between future and past

mental time travel was developed over time, and was strongest between 20-50 s after task

onset. Interestingly, the forward and backward body sways when thinking about the past and

future seems to develop sooner; differences are evident already 4 s after task onset (see Miles,

Nind, et al., 2010; Figure 1). These differences between our and Miles, Nind, et al.’s study


might be based on methodological issues. (For example, we asked participants to travel one

year through time, whereas Miles, Nind, et al. (2010) asked their participants to mentally

travel four years through time.) However, we also want to point out the possibility that there

are differences between the recruitment of the horizontal and sagittal (front-back) spatial

frame of references because they differ with respect to the strength of temporal associations.

First, the past-back and future-front (but not the past-left and future-right) association is in

line with the metaphorical use of space when expressing time in language. Second, the back-

front mental timeline runs through the “self” (because past events are represented in the back

and future events are represented in the front of the observer). This is not necessarily true for

the horizontal mental time line. The horizontal mental time line can also be conceptualized as

running from left to right in front of the observer where the “self” is not the center of the line

(Nunez & Cooperrider, 2013). A possible result of these differences might be that the past-

back and future-front association is stronger and more accessible and is therefore activated

faster than the past-left and future-right association. This possible delay in the activation of

the horizontal spatial frame of reference might account for the absence of the past-left and

future-right association in other studies where the temporal range of a trial is within seconds

(Flumini & Santiago, 2013; Stocker, Hartmann, Martarelli, & Mast, submitted).

Spatial-temporal associations that we observed via spontaneous eye movements

(implicit measurement) were congruent with those explicitly indicated by participants. Indeed,

participants also reported a past-left (and down) and future-right (and up) association when

explicitly asked. However, the explicit and implicit measurements were not correlated,

suggesting dissociations between the two measurements on an individual level. Such

dissociations have also been shown in other research areas (e.g., Asendorpf, Banse, & Mücke,

2002; Hannula et al., 2010; Lucidi & Thevenot, 2014; Priftis, Zorzi, Meneghello, Marenzi, &

Umilta, 2006; Sherman, Rose, Koch, Presson, & Chassin, 2003; Willingham, Nissen, &

Bullemer, 1989). For example, in the domain of spatial-numerical associations, the hand with


which people start counting (left vs. right) observed in natural behavior does often not match

with the explicit response when the same individuals are asked to indicate the side of start

counting in a questionnaire (Lucidi & Thevenot, 2014), suggesting that conscious access to

higher-order spatial representations may be limited.

Besides spatial-temporal associations, other factors might have contributed to our

findings. Future and past thinking involves different cognitive and neuronal processes (e.g.,

Abraham, Schubotz, & von Cramon, 2008; Addis, Pan, Vu, Laiser, & Schacter, 2009; Addis,

Wong, & Schacter, 2007). Indeed, thinking about the past relies on memory retrieval, whereas

thinking about the future relies more strongly on imagining processes. Although some studies

emphasize the view that remembering the past and imagining the future involve, to a large

extent, the same brain networks (see Schacter, Addis, & Buckner, 2007), we cannot rule out

that differences in the neuronal processes underlying remembering and imagining contributed

to the different eye gaze behavior that we found. Moreover, thinking about one’s past versus

future might differ with respect to other psychological variables such as cognitive load,

arousal, or valence. All these factors have been shown to be related to spatial attention and

might thus also influence eye movements (e.g., Bareham, Manly, Pustovaya, Scott, &

Bekinschtein, 2014; de la Vega, de Filippis, Lachmair, Dudschig, & Kaup, 2012; Lavie, 2005;

Lepsien et al., 2005; Meier & Robinson, 2004). We showed that pupil size, as indicator of

cognitive load (e.g., Hartmann & Fischer, 2014; Kahneman & Beatty, 1966) did not differ

between the two conditions, and participants’ subjective ratings confirmed that thinking about

the future and past was perceived as equally difficult. Similarly, there was no difference in

participants’ valence ratings of future and past thoughts. Regarding arousal, it has recently

been shown that spatial attention shifts to the right when people are transitioning in and out of

sleep (Bareham et al., 2014). This was clearly not the case during our (two times) one-minute

session. For all these reasons, we believe that the spatial-temporal association is a valid

explanation for our results.


Yet another issue about the methodology used in this study is the lack of a reliable

control over participant’s task compliance during the one-minute mental time travel period.

Other studies used a trial-by-trial approach, instructing participants to think about a future or

past event in response to a cue, to press a button as soon as they had a clear image of a

specific event in mind, and to report that event in detail immediately afterwards

(D'Argembeau & Van der Linden, 2004). While such an account provides a better control

about task compliance, it might have interfered with natural eye movement behavior. We

therefore preferred no to interrupt participants during a one-minute mental time travel

episode. Mean response times in D’Argembeau and Van der Linden’s (2004) study were

between 18-36 s, suggesting that this amount of time is needed in order to generate a mental

representation of a future and past event. Interestingly, this duration corresponds exactly to

the time windows 3 and 4 (20-40 s) for which we found the largest difference for the

horizontal gaze position. This correspondence, together with the fact that participants had no

difficulty to report their future and past thoughts after the experiment, suggests that they

reliably followed task instructions.

4.1 Conclusion

To conclude, we showed that eye movements follow a diagonal mental time line

(running from bottom left to the top right) when thinking about past and future events. Our

results extend previous findings showing temporal associations with the sagittal axis of space

(Miles, Nind, et al. 2010). We showed that not only body movements but also eye movements

spontaneously follow the spatial structure of temporal representations. This is in line with the

conceptualization of time in language where gaze direction is often used to express time (“We

are looking forward to the summer holidays vs. we are looking back on the summer

holidays”; see Stocker, 2014). Our results widen the growing body of research highlighting

the importance of sensorimotor systems in the representation of abstract knowledge, and show

that eye movements can be used as on-line measurement of the spatial character of cognition.


Acknowledgments

This research was funded by the Swiss National Science Foundation

(P2BEP1_152104) and the Cogito Foundation, Wollerau, Switzerland (R-135/12).


References

Abraham, A., Schubotz, R. I., & von Cramon, D. Y. (2008). Thinking about the future versus

the past in personal and non-personal contexts. Brain Research, 1233, 106-119. doi:

10.1016/j.brainres.2008.07.084

Addis, D. R., Pan, L., Vu, M. A., Laiser, N., & Schacter, D. L. (2009). Constructive episodic

simulation of the future and the past: distinct subsystems of a core brain network

mediate imagining and remembering. Neuropsychologia, 47(11), 2222-2238. doi:

10.1016/j.neuropsychologia.2008.10.026

Addis, D. R., Wong, A. T., & Schacter, D. L. (2007). Remembering the past and imagining

the future: common and distinct neural substrates during event construction and

elaboration. Neuropsychologia, 45(7), 1363-1377. doi:

10.1016/j.neuropsychologia.2006.10.016

Arzy, S., Adi-Japha, E., & Blanke, O. (2009). The mental time line: an analogue of the mental

number line in the mapping of life events. Consciousness and Cognition, 18(3), 781-

785. doi: 10.1016/j.concog.2009.05.007

Asendorpf, J. B., Banse, R., & Mücke, D. (2002). Double dissociation between implicit and

explicit personality self-concept: the case of shy behavior. Journal of Personality and

Social Psychology, 83(2), 380.

Bareham, C. A., Manly, T., Pustovaya, O. V., Scott, S. K., & Bekinschtein, T. A. (2014).

Losing the left side of the world: rightward shift in human spatial attention with sleep

onset. Scientific Reports, 4, 5092. doi: 10.1038/srep05092

Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617-645. doi:

10.1146/annurev.psych.59.103006.093639

Bergen, B. K., & Chan Lau, T. T. (2012). Writing direction affects how people map space

onto time. Frontiers in Psychology, 3, 109. doi: 10.3389/fpsyg.2012.00109


Bergen, B. K., & Lau, T. T. C. (2012). Writing direction affects how people map space onto

time. Frontiers in Psychology, 3. doi: 10.3389/Fpsyg.2012.00109

Bonato, M., Zorzi, M., & Umilta, C. (2012). When time is space: evidence for a mental time

line. Neuroscience & Biobehavioral Reviews, 36(10), 2257-2273. doi:

10.1016/j.neubiorev.2012.08.007

Boroditsky, L. (2000). Metaphoric structuring: understanding time through spatial metaphors.

Cognition, 75(1), 1-28. doi: S0010-0277(99)00073-6

Casasanto, D., & Boroditsky, L. (2008). Time in the mind: using space to think about time.

Cognition, 106(2), 579-593. doi: 10.1016/j.cognition.2007.03.004

Casasanto, D., & Bottini, R. (2014). Mirror reading can reverse the flow of time. Journal of

Experimental Psychology: General, 143(2), 473-479. doi: 10.1037/a0033297

Casasanto, D., & Jasmin, K. (2012). The hands of time: Temporal gestures in English

speakers. Cognitive Linguistics, 23(4), 643. doi: 10.1515/cog-2012-0020

Cienki, A. (1998). Metaphoric gestures and some of their relations to verbal metaphoric

expressions. Discourse and Cognition: Bridging the gap, 189-204.

Cooperrider, K., & Núñez, R. (2009). Across time, across the body: Transversal temporal

gestures. Gesture, 9(2), 181-206.

Corbetta, M., Akbudak, E., Conturo, T. E., Snyder, A. Z., Ollinger, J. M., Drury, H. A., . . .

Shulman, G. L. (1998). A common network of functional areas for attention and eye

movements. Neuron, 21(4), 761-773.

D'Argembeau, A., & Van der Linden, M. (2004). Phenomenal characteristics associated with

projecting oneself back into the past and forward into the future: influence of valence

and temporal distance. Consciousness and Cognition, 13(4), 844-858. doi:

10.1016/j.concog.2004.07.007


de la Vega, I., de Filippis, M., Lachmair, M., Dudschig, C., & Kaup, B. (2012). Emotional

valence and physical space: limits of interaction. Journal of Experimental Psychology:

Human Perception and Performance, 38(2), 375-385. doi: 10.1037/a0024979

Dehaene, S., Bossini, S., & Giraux, P. (1993). The Mental Representation of Parity and

Number Magnitude. Journal of Experimental Psychology: General, 122(3), 371-396.

Fischer, M. H., Castel, A. D., Dodd, M. D., & Pratt, J. (2003). Perceiving numbers causes

spatial shifts of attention. Nature Neuroscience, 6(6), 555-556. doi: 10.1038/nn1066

nn1066

Flumini, A., & Santiago, J. (2013). Time (also) flies from left to right... if it is needed! In M.

Knauff, M. Pauen, N. Sebanz & I. Wachmuz (Eds.), Proceedings of the 36th Annual

Conference of the Cognitive Science Society (pp. 2315-2320). Austin, TX.

Fuhrman, O., & Boroditsky, L. (2007). Mental time-lines follow writing direction: Comparing

English and Hebrew speakers. Paper presented at the Proceedings of the 29th annual

conference of the cognitive science society.

Gallese, V., & Lakoff, G. (2005). The Brain's concepts: the role of the Sensory-motor system

in conceptual knowledge. Cognitive Neuropsychology, 22(3), 455-479. doi:

10.1080/02643290442000310

Grade, S., Lefèvre, N., & Pesenti, M. (2012). Influence of gaze observation on random

number generation. Experimental Psychology, 60, 122-130.

Hannula, D. E., Althoff, R. R., Warren, D. E., Riggs, L., Cohen, N. J., & Ryan, J. D. (2010).

Worth a glance: using eye movements to investigate the cognitive neuroscience of

memory. Frontiers in Human Neuroscience, 4, 166. doi: 10.3389/fnhum.2010.00166

Hartmann, M., & Fischer, M. H. (2014). Pupillometry: the eyes shed fresh light on the mind.

Current Biology, 24(7), R281-282. doi: 10.1016/j.cub.2014.02.028


Hartmann, M., Gashaj, V., Stahnke, A., & Mast, F. (2014). There is more than "more is up":

Hand and foot responses reverse the vertical association of number magnitudes.

Journal of Experimental Psychology: Human Perception and Performance, 40(4), 14.

doi: 10.1037/a0036686

Hartmann, M., Grabherr, L., & Mast, F. W. (2012). Moving along the mental number line:

Interactions between whole-body motion and numerical cognition. Journal of

Experimental Psychology: Human Perception and Performance, 38(6), 1416-1427.

doi: 10.1037/a0026706

Hartmann, M., & Mast, F. (2012). Moving along the mental time line influences the

processing of future related words. Consciousness and Cognition, 21, 1558-1562.

Hubbard, E. M., Piazza, M., Pinel, P., & Dehaene, S. (2005). Interactions between number

and space in parietal cortex. Nature Review Neuroscience, 6(6), 435-448.

SMI - SensoMotoric Instruments (2009). Begaze 2.2 Manual. Teltow, Germany.

Ito, Y., & Hatta, T. (2004). Spatial structure of quantitative representation of numbers:

evidence from the SNARC effect. Memory & Cognition, 32(4), 662-673.

Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154, 3.

Lakoff, G., & Johnson, M. (1980). Conceptual metaphor in everyday language. The Journal

of Philosophy, 77(8), 453-486.

Lavie, N. (2005). Distracted and confused?: selective attention under load. Trends in

Cognitive Sciences, 9(2), 75-82. doi: 10.1016/j.tics.2004.12.004

Lepsien, J., Griffin, I. C., Devlin, J. T., & Nobre, A. C. (2005). Directing spatial attention in

mental representations: Interactions between attentional orienting and working-

memory load. Neuroimage, 26(3), 733-743. doi: 10.1016/j.neuroimage.2005.02.026

Loetscher, T., Bockisch, C. J., Nicholls, M. E., & Brugger, P. (2010). Eye position predicts

what number you have in mind. Current Biology, 20(6), R264-265. doi:

10.1016/j.cub.2010.01.015


Lourenco, S. F., & Longo, M. R. (2010). General magnitude representation in human infants.

Psychological Science, 21(6), 873-881. doi: 10.1177/0956797610370158

Lucidi, A., & Thevenot, C. (2014). Do not count on me to imagine how I act: behavior

contradicts questionnaire responses in the assessment of finger counting habits.

Behavioral Research Methods. doi: 10.3758/s13428-014-0447-1

Meier, B. P., & Robinson, M. D. (2004). Why the sunny side is up: association between affect

and vertical position. Psychological Science, 15(4), 243-247. doi: 10.1111/j.0956-

7976.2004.00659.x

Miles, L. K., Betka, E., Pendry, L. F., & Macrae, C. N. (2010). Mapping temporal constructs:

Actions reveal that time is a place. The Quarterly Journal of Experimental

Psychology, 63(11), 2113-2119.

Miles, L. K., Karpinska, K., Lumsden, J., & Macrae, C. N. (2010). The meandering mind:

vection and mental time travel. PLoS One, 5(5), e10825. doi:

10.1371/journal.pone.0010825

Miles, L. K., Nind, L. K., & Macrae, C. N. (2010). Moving through time. Psychological

Science, 21(2), 222-223. doi: 10.1177/0956797609359333

Nunez, R., & Cooperrider, K. (2013). The tangle of space and time in human cognition.

Trends in Cognitive Sciences, 17(5), 220-229. doi: 10.1016/j.tics.2013.03.008

Ooi, T. L., Wu, B., & He, Z. J. (2001). Distance determined by the angular declination below

the horizon. Nature, 414(6860), 197-200. doi: 10.1038/35102562

Ouellet, M., Santiago, J., Funes, M. J., & Lupianez, J. (2010). Thinking about the future

moves attention to the right. Journal of Experimental Psychology: Human Perception

and Performance, 36(1), 17-24. doi: 10.1037/a0017176


Priftis, K., Zorzi, M., Meneghello, F., Marenzi, R., & Umilta, C. (2006). Explicit versus

implicit processing of representational space in neglect: dissociations in accessing the

mental number line. Journal of Cognitive Neuroscience, 18(4), 680-688. doi:

10.1162/jocn.2006.18.4.680

Proctor, R. W., & Cho, Y. S. (2006). Polarity correspondence: A general principle for

performance of speeded binary classification tasks. Psychological Bulletin, 132(3),

416-442. doi: 10.1037/0033-2909.132.3.416

Radden, G. (2004). The metaphor TIME AS SPACE across languages. Zeitschrift für

Interkulturellen Fremdsprachenunterricht, 8, 226-239.

Saj, A., Fuhrman, O., Vuilleumier, P., & Boroditsky, L. (2014). Patients With Left Spatial

Neglect Also Neglect the “Left Side” of Time. Psychological Science, 25(1), 207-214.

Santiago, J., Lupianez, J., Perez, E., & Funes, M. J. (2007). Time (also) flies from left to right.

Psychonomic Bulletin and Review, 14(3), 512-516.

Schacter, D. L., Addis, D. R., & Buckner, R. L. (2007). Remembering the past to imagine the

future: the prospective brain. Nature Review Neuroscience, 8(9), 657-661. doi:

10.1038/nrn2213

Schwarz, W., & Keus, I. M. (2004). Moving the eyes along the mental number line:

comparing SNARC effects with saccadic and manual responses. Percept Psychophys,

66(4), 651-664.

Sell, A. J., & Kaschak, M. P. (2012). The comprehension of sentences involving quantity

information affects responses on the up-down axis. Psychonomic Bulletin & Review,

19(4), 708-714. doi: 10.3758/s13423-012-0263-5

Sheliga, B. M., Riggio, L., & Rizzolatti, G. (1994). Orienting of attention and eye movements.

Experimental Brain Research, 98(3), 507-522.


Sherman, S. J., Rose, J. S., Koch, K., Presson, C. C., & Chassin, L. (2003). Implicit and

explicit attitudes toward cigarette smoking: The effects of context and motivation.

Journal of Social and Clinical Psychology, 22(1), 13-39. doi:

10.1521/jscp.22.1.13.22766

Stocker, K. (2012). The time machine in our mind. Cognitive Science, 36, 385-420. doi:

10.1111/j.1551-6709.2011.01225.x

Stocker, K. (2014). Mental perspective in multiple-event memory and foresight. Journal of

Cognitive Science, 15, 57-96.

Stocker, K., Hartmann, M., Martarelli, C., & Mast, F. W. (submitted). Eye movements reveal

mental scanning through time.

Torralbo, A., Santiago, J., & Lupianez, J. (2006). Flexible conceptual projection of time onto

spatial frames of reference. Cognitive Science, 30(4), 745-757. doi:

10.1207/s15516709cog0000_67

Tulving, E. (2002). Episodic memory: from mind to brain. Annual Review of Psychology, 53,

1-25. doi: 10.1146/annurev.psych.53.100901.135114

Ulrich, R., Eikmeier, V., de la Vega, I., Ruiz Fernandez, S., Alex-Ruf, S., & Maienborn, C.

(2012). With the past behind and the future ahead: back-to-front representation of past

and future sentences. Memory and Cognition, 40(3), 483-495. doi: 10.3758/s13421-

011-0162-4

Ulrich, R., & Maienborn, C. (2010). Left-right coding of past and future in language: the

mental timeline during sentence processing. Cognition, 117(2), 126-138. doi:

10.1016/j.cognition.2010.08.001

Walker, E. J., Bergen, B. K., & Núñez, R. Later events lie behind her, but not behind you:

Compatibility effects for temporal sequences along the sagittal axis depend on


perspective. Retrieved from:

http://mindmodeling.org/cogsci2013/papers/0660/paper0660.pdf

Weger, U. W., & Pratt, J. (2008). Time flies like an arrow: space-time compatibility effects

suggest the use of a mental timeline. Psychonomic Bulletin and Review, 15(2), 426-

430.

Willingham, D. B., Nissen, M. J., & Bullemer, P. (1989). On the development of procedural

knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition,

15(6), 1047.

Winter, B., & Matlock, T. (2013). More is up… and right: Random number generation along

two axes. Paper presented at the Proceedings of the 35th Annual Conference of the

Cognitive Science Society. Austin, TX: Cognitive Science Society.

Wood, G., Willmes, K., Nuerk, H. C., & Fischer, M. H. (2008). On the cognitive link between

space and number: A meta-analysis of the SNARC effect. Psychology Science

Quarterly, 50, 489-525.

Yang, Z., & Purves, D. (2003). A statistical explanation of visual space. Nature Neuroscience,

6(6), 632-640. doi: 10.1038/nn1059


Figure 1

Figure 1. Mean horizontal (a) and vertical (b) eye gaze position (in ° of visual angle) for

future and past mental time travel separate for the time windows 0-10, 10-20, 20-30, 30-40,

40-50, and 50-60 s from task onset. Positive values represent gaze position in the right

(horizontal) and upper (vertical) screen half. Asterisk indicates a significant difference

between future and past mental time travel in Time window 4 (30-40 s). Error bars depict +/-

1 SEM.

-2

-1

0

1

2

3

4

1 2 3 4 5 6

Hor

izon

tal S

cree

n Po

sitio

n (°

)

Time Window

Future

Past

a.

*

-2

-1

0

1

2

3

4

1 2 3 4 5 6

Vert

ical

Scr

een

Posi

tion

(°)

Time Window

Future

Past

b.

E3,

E2,

E1,

0,

1,

2,

3,

E3, E2, E1, 0, 1, 2, 3,

Future,

Past,

a.

E3,

E2,

E1,

0,

1,

2,

3,

E3, E2, E1, 0, 1, 2, 3,

Future,

Past,

b.


Figure 2

Figure 2. Implicit (a) and explicit (b) measurements of spatial-temporal associations. Implicit

measurements are the mean horizontal and vertical eye gaze positions (in ° of visual angle;

averaged over the whole recording session) and explicit measurements are spatial-temporal

associations indicated on horizontal and vertical 7-point Likert scales (b). Positive values

represent an association with the right (horizontal) and upper (vertical) space. Error bars

depict +/- 1 SEM.

-2

-1

0

1

2

3

4

1 2 3 4 5 6

Hor

izon

tal S

cree

n Po

sitio

n (°

) Time Window

Future

Past

a.

*

-2

-1

0

1

2

3

4

1 2 3 4 5 6

Vert

ical

Scr

een

Posi

tion

(°)

Time Window

Future

Past

b.

E3,

E2,

E1,

0,

1,

2,

3,

E3, E2, E1, 0, 1, 2, 3,

Future,

Past,

a.

E3,

E2,

E1,

0,

1,

2,

3,

E3, E2, E1, 0, 1, 2, 3,

Future,

Past,

b.

Eye Movements During Mental Time Travel Follow a Diagonal Line

Documents

Transcript of Eye Movements During Mental Time Travel Follow a Diagonal Line