Eye tracking and stimulated recall

interviewsNathaniel Owen

University of Leicester

Background: Comparative study of reading components of IELTS

and TOEFL iBT• Participants:

– Six postgraduate students (L1 Chinese)

– Data collected: most recent English test scores (all IELTS; reading and speaking scores); date of administration; academic programme

  Session 1 (Test 1)

Session 2 (Test 2)

Participant 1 IELTS 1 (Q1-13) TOEFL 1 (Q1-13)Participant 2 IELTS 2 (Q14-26) TOEFL 2 (Q14-26)Participant 3 IELTS 3 (Q27-40) TOEFL 3 (Q27-38)Participant 4 TOEFL 1 (Q1-13) IELTS 1 (Q1-13)Participant 5 TOEFL 2 (Q14-26) IELTS 2 (Q14-26)Participant 6 TOEFL 3 (Q27-38) IELTS 3 (Q27-40)

Research Question 1• What observable test-taking strategies and cognitive processes may be reported for a video-stimulus approach to immediate retrospective verbal protocol analysis?

• For today:– Can a video-stimulus approach distinguish

between levels of processing in Khalifa and Weir’s (2009) model of reading?

Eye tracking and cognitive processing in L2 reading – where are we?

• McCray (2014)Higher-level test takers more likely to use global (more complex) processes (banked gapfill tasks)

• Bax (2013)successful for identifying metacognitive strategies displayed by the participants

successful for identifying and differentiating between low level processes (lexical access/syntactic parsing)

• Evidence for the efficacy of eye tracking to reveal and distinguish between higher level processes is implied by the absence (or limited duration) of fixations on specific lexical items (Bax, 2013, McCray, 2015)

Stimulated-recall interviews – Issues and advantages

For complete discussion, see Gass and Mackey (2000)

Methodology: Video stimulus – “what were you thinking at that moment?”

RQ1 StimuliMethodology: Test, annotation and response stimuli

Toulmin (2003) diagram of logical argumentation for stimulated-recall interview data using video stimulus

Participants’ verbalisations

Claim (cognitive process)Since

Unless

Qualifier

• Verbal statements reflect underlying thought processes related directly to specific moments of action

• Video of specific moments played directly to participant (strength of stimulus)

• Concrete stimuli (relatability)• Video replayed to participant immediately after test completion (memory)

• Participants are able to verbalise the thought processes that occurred to them at specific moments of the test (veridicality)

On account of

• Verbalisations are metacognitive (verbalisations are reflective)

• Abstract/unrelatable stimuli• Verbalisation reveals test-wise approach to item completion (construct-irrelevant action)

• Verbalisation inaccurate (contradicts action) (veridicality)

• Verbalisations are partially complete (only account for part of the test takers’ actions

(D) (C)

(W)

(B)

So

(A)

(R)

• Verbalisations reflect current thought processes or test-wise item completion

Creating a text-level representation: construct an organised representation of a single text

[P7]     

Building a mental model: integrating new information; enriching the proposition

[P6]   

Inferencing:At sentence/clause levelAt word level

 [P5s/c]

[P5w]

Establishing propositional meaning: At clause levelAt sentence level

  

[P4c][P4s]

Syntactic parsing

[P3]

Lexical Access

[P2]

Khalifa and Weir (2009)

[P5s/c] Inferring at the clause/sentence levelExtract from paragraph 2: As a general rule, the higher a mountain is, the more recently it was formed; for example, the high mountains of the Himalayas are only about 50 million years old. Lower mountains tend to be older, and are often the eroded relics of much higher mountain chains. About 400 million years ago, when the present-day continents of North America and Europe were joined, the Caledonian mountain chain was the same size as the modern Himalayas. Today, however, the relics of the Caledonian orogeny (mountain-building period) exist as the comparatively low mountains of Greenland, the northern Appalachians in the United States, the Scottish Highlands, and the Norwegian coastal plateau.

Q29. Which of the following can be inferred from paragraph 2 about the mountains of the Himalayas?

– Their current height is not an indication of their age.– *At present, they are much higher than the mountains of the Caledonian range.– They were a uniform height about 400 million years ago.– They are not as high as the Caledonian mountains were 400 million years ago.

“I think this sentence is correct from the information that I have found…as a general rule, the higher a mountain is, the more recently it was formed. So, because the age of the Himalayas is younger, so it’s higher than the Caledonian range” [Participant 6, 06:02:01:27-33].

[P2]

[P4s] Therefore… [P5s]

[P6] Building a mental modelG It follows from the above that sparing use of energy reserves should tend to extend life. Extreme high performance sports may lead to optimal cardiovascular performance, but they quite certainly do not prolong life. Relaxation lowers metabolic rate, as does adequate sleep and in general an equable and balanced personality. Each of us can develop his or her own ‘energy saving programme’ with a little self-observation, critical self-control and, above all, logical consistency. Experience will show that to live in this way not only increases the life span but is also very healthy. This final aspect should not be forgotten.

Yes/No/Not given

40 Conserving energy may help to extend a human’s life.

Participant correctly answers ‘Yes’:“It’s about conserving energy, so it’s mentioned in paragraph G. I remember the content, if you stay calm, you can extend your life, so energy can help you to extend your life, so I put ‘yes’ here.” [Participant 6, 06:02:03:27-40]

[P2]

[P4s][P4s]

Inferencing at the clause/sentence level [P5s/c] vs building a mental model [P6]

“In many of the test items (5 of 10), no significant dif ferences were identified in the eye movement behaviour of those who were correct on each item and those who were not. The implication of this is that in these cases the more successful students did not use any one cognitive process significantly more than the less successful students in coming to the correct answer with these items, but perhaps used a variety of processes and strategies, or other faculties related for example to memory or lexical knowledge, none of which was either predominant or else was traceable in the eye-tracking record” (Bax, 2013: 20).

Inferencing [P5s/c] Strategic action

Building a mental model [P6]

[TOEFL] Q29: Selects option 2

[IELTS] Q40: Selects ‘Yes’

Identifies the purpose of the question

2 6 Marks/notes key noun phrase(s) in the questions stem or options

Reads question stem(s) and/or option(s) carefully

5 11 Careful local reading (text)

Careful local reading (text)

11 27 Compares question stem/option to a portion of the text

Recommendations• Strength of eye tracking lies in explanatory power of unobservable cognitive processes (the processing core) rather than already observable metacognitive strategic decision-making

• Stimulated-recall interviews offer: Holistic/cumulative data to complement reductionist approach of eye-tracking.

Good reading is “constructively responsive” (Pressley and Afflerbach, 1995: 2) – that is, good readers always adjust their processing in relation to the text that they are reading;

Fixation (frequency and duration) affected by working memory

Nature of stimulus (and ‘overloading’). The nature and quality of the stimulus is important – it needs to be relatable to the test taker’s experience of taking the test (to put the test taker ‘in the moment’) and as close to test administration as possible

References• Bax, S. (2013). The cognitive processing of candidates during reading tests: Evidence from eye-tracking. Language Testing, 30(4), 441–465.

• Bowles, M. (2010). The think-aloud controversy in second language research. New York: Routledge. Oxford.

• Ericsson, K., & Simon, H. (1993). Protocol Analysis: Verbal Reports as Data (2nd ed.). Boston: MIT Press.

• Gass, S. M. & Mackey, A. (2000). Stimulated recall methodology in second language research. Mahwah, NJ: Lawrence Erlbaum Associates.

• Khalifa, H., and Weir, C.J. (2009) Examining Reading: Research and practice in assessing second language reading, Studies in Language Testing 29. Cambridge: UCLES/CUP.

• McCray, G. (2014). Cognitive processing in banked-gapfill items: Linking eye-tracking traces with testing theory. BAAL Testing, Evaluation and Assessment SIG. Presentation 2nd May, 2014.

• Oxford, R.L., (1996). Language learning strategies around the world: Cross-cultural perspectives. Manoa: University of Hawaii Press.

• Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates.

• Toulmin, S. E. (2003). The uses of argument (Updated ed.). Cambridge: Cambridge University Press. (1st ed. 1958).

Thank you!

Any questions?