978-1-4577-0557-1/12/$26.00 ©2012 IEEE

1

Monitoring Expertise Development during Simulated UAV

Piloting Tasks using Optical Brain Imaging

Hasan Ayaz1,Murat P. Çakır

2,Kurtuluş İzzetoğlu

1,Adrian Curtin

1,Patricia A. Shewokis

1,Scott C. Bunce

3,Banu Onaral

1

1Drexel University, 3141 Chestnut St. Philadelphia PA USA, +1 (215) 895 2215, (ha45,ki25,abc48,pas38,bo26)@drexel.edu

2Middle East Technical University, Informatics Institute, Ankara, Turkey, +90 (312) 210 7706, perit@ii.metu.edu.tr

3Penn State University, M.S. Hershey Medical Center, Hershey, PA USA, +1 (717) 531-4127, sbunce@hmc.psu.edu

Abstract— An accurate assessment of mental workload and

expertise level would help improve operational safety and

efficacy of human computer interaction for aerospace

applications. The current study utilized functional near-

infrared spectroscopy (fNIR) to investigate the relationship of

the hemodynamic response in the anterior prefrontal cortex to

changes in mental workload, level of expertise, and task

performance during learning of simulated unmanned aerial

vehicle (UAV) piloting tasks. Results indicated that fNIR

measures are correlated to task performance and subjective

self-reported measures; and contained additional information

that allowed categorizing learning phases. Level of expertise

does appear to influence the hemodynamic response in the

dorsolateral/ventrolateral prefrontal cortices. Since fNIR

allows development of portable and wearable instruments, it

has the potential to be deployed in future learning

environments to personalize the training regimen and/or assess

the effort of human operators in critical multitasking settings.

TABLE OF CONTENTS

1. INTRODUCTION ................................................. 1 2. MENTAL WORKLOAD AND EXPERTISE ............. 2 3. OPTICAL BRAIN IMAGING ................................ 3 3. METHODS .......................................................... 4 4. RESULTS ............................................................ 5 5. DISCUSSION ....................................................... 7 6. SUMMARY ......................................................... 7 REFERENCES ......................................................... 8 BIOGRAPHIES ...................................................... 10

1. INTRODUCTION

The efficiency and safety of many complex human-machine

systems are closely related to the cognitive workload and

situational awareness of their operators [1, 2]. An ideal

human-machine system would be informed about the

current cognitive workload level of its personnel and/or be

designed to keep the necessary workload at an optimum

level. Hence, it could help prevent potential overload and

minimize errors. Importantly, behavioral measures alone

may not be sufficiently sensitive to index overload because

operators may extend extra effort to maintain system

performance, but this could come at a cost that is reflected

only in neural measures. Significant progress has been made

over the last decade in understanding the neural bases of

cognitive processes and behavior. The advent of new and

improved brain imaging tools that allow monitoring of brain

activity in the field and operational environments is

expected to allow better identification of neurophysiological

markers that can index impending overload or fatigue before

performance measures can.

Deployment of portable neuroimaging technologies to real

time settings could help assess cognitive and motivational

states of human operators and close the loop for advanced

human-machine interaction for optimized learning and

training. There is considerable evidence that

neurophysiological and psychophysiological variables

respond to cognitive demand in a predictable manner [3-9].

Functional Magnetic Resonance Imaging (fMRI) is

currently considered the “gold standard” for measuring

functional brain activation. However, fMRI is expensive to

operate and requires massive installations and a large

infrastructure to operate. In addition, fMRI confines

participants to restricted positions, is highly sensitive to

motion artifacts and exposes participants to loud noises.

Direct measures of central nervous system function such as

electroencephalography (EEG) and event-related potentials

(ERPs) have been particularly strong candidates for

accurate, objective measures of operator workload [10-13].

Increasing task difficulty, for instance, is known to be

associated with EEG changes such as increased power in the

beta bandwidth, increased theta activity at frontal sites and

the suppression of alpha activity [11, 13]. Although EEG

has many excellent qualities for monitoring mental

workload, including excellent temporal resolution, it is

limited in its capacity for spatial resolution.

In this study, we utilized a field-deployable functional

optical imaging method for monitoring the prefrontal cortex

using functional near-infrared spectroscopy (fNIR) [14-20].

fNIR is a safe, non-invasive, affordable and portable

neuroimaging technology that can be used to monitor

hemodynamic changes that occur in the brain, i.e., blood

oxygenation and blood volume, during select cognitive

tasks. These qualities pose fNIR as an ideal candidate for

monitoring cognitive activity-related hemodynamic changes

not only in laboratory settings but also under ecologically

valid conditions – real world environments.

For the current study, 13 healthy individuals with no prior

flight simulator experience volunteered for a training

program of unmanned aerial vehicle (UAV) piloting.

2

Throughout the 9 day program, participants practiced turn-

to-approach tasks 10 times per day while their behavioral

data and anterior prefrontal cortex (PFC) brain activation (as

measured by fNIR) were recorded.

Preliminary results indicated significant improvement in the

behavioral performance measures and a corresponding

reduction in total hemoglobin concentration changes,

consistent with the scaffolding-storage framework [21] and

our previous results [20]. In this paper, we have further

demonstrated that fNIR and behavioral data together

provides a better indicator of training. Development of these

neuro-behavioral models could lead to optimized and

personalized training regimens and/or reliable assessment of

the effort human operators in critical multitasking

environments such as aerospace settings.

2. MENTAL WORKLOAD AND EXPERTISE

There is no singular definition of mental workload [7].

There are at least two major theoretical approaches to the

construct: 1) mental workload may be defined such that a

given task’s requirements are viewed as an independent,

external variable with which the working participants have

to cope more or less efficiently; or 2) mental workload may

be defined in terms of an interaction between task

requirements and human capabilities or resources [22].

Both approaches can offer essential and well-founded

contributions to different problems. In either of these

paradigms, the definition of workload involves the

“objective” effects of task difficulty on the participant, and

the participant’s effort involved in maintaining performance.

Workload is an intervening variable between task and

environmental demands and the operator’s performance,

which is defined by the relationship between task demand

and the participant’s supply of neural resources, i.e., the

portion of an operator’s limited mental capacities actually

required to perform a particular task. In other words,

workload can be defined in terms of some “objective”

criteria for task difficulty or in terms of the participant’s

capacities to perform the identified task. As such, workload

can be differentiated from performance. Two people

performing the same task can have identical performance,

yet one operator may have significant cognitive resources

free to allocate to concurrent tasks, whereas the second

operator may be just on the brink of performance failure.

The difference between the required capacity and the

available capacity of an individual can be referred to as the

mental or cognitive reserve.

There are four basic methodologies for the measurement of

mental workload: 1) primary task performance, 2)

subjective ratings (e.g., NASA Task Load Index

(TLX)[23]), 3) central and peripheral measures of

physiological response, and 4) secondary task techniques

[24]. Primary task performance measures are critical in that

they define the task that needs to be accomplished.

However, they cannot predict failure, and are only sensitive

to changes in workload at the limits of mental capacity. If

operators can compensate for increased workload by

increasing effort, then the primary task measure is

insensitive. Finally, task performance measures do not

provide any direct evidence about the subject’s degree of

effort or level of arousal. Subjective ratings provide an

important measure of the operator’s perceived load, and

have good face validity. However, they are intrusive to

collect while on task, and are often dissociated from actual

performance and potential failure. Secondary task

techniques can give some sense of cognitive or mental

reserve, but they are often intrusive, and cannot reasonably

be employed during actual UAV missions. Physiological

measures (such as skin conductance, electrooculogram,

electrocardiogram and respiration) can be unobtrusive, and

provide an “objective” measure of workload. However,

many of the typical physiological systems measure

autonomic responses, i.e., the fight or flight system, which

assesses stress, whereas some of the difference in workload

may more appropriately be assessed using a measure of

cognitive workload, rather than stress.

Central Nervous System Measures of Workload & Expertise

Physiological measures of central nervous system (CNS)

functioning hold the most promise for operational

monitoring of UAV pilots, as they provide continuous and

unobtrusive monitoring of the operator and do not interfere

with an operator's work as do secondary-task performance

or subjective measures of workload. Moreover, CNS

measures can provide information on cognitive functioning

as well as measures of stress or emotion.

Commonly employed techniques such as EEG, ERPs,

magnetoencephalography (MEG), positron emission

tomography (PET), single positron emission computed

tomography (SPECT), and fMRI have dramatically

increased our understanding of a broad range of cognitive

and emotional states. Methods that directly measure the

summation of neural function, such as MEG, EEG, and

ERPs, allow researchers to monitor the direct consequences

of brain electromagnetic activity with temporal resolution

on the order of milliseconds. However, these technologies

also have limited spatial resolution [17], and are susceptible

to electromagnetic artifacts in the field. In contrast, indirect

methods such as PET, SPECT and fMRI, monitor the

hemodynamic and metabolic changes associated with neural

activity with impressive spatial resolution, but are limited in

temporal resolution and are associated with neuronal

activity through a poorly understood neurovascular coupling

function [25]. In addition, PET and SPECT do not allow for

continuous or repeated measurements because they require

the use of radioactive isotopes, which also limits their use in

children.

A given individual will develop expertise more quickly with

increased time on task and/or by utilizing the most

appropriate and effective mode of practice for the task to be

learned. Native ability may contribute to the speed of an

individual’s progression, and may assist in determining

individual differences in enhanced development of expertise

in a given course of training. The literature dealing with the

3

effect of practice on the functional anatomy of task

performance is extensive and complex. Practice and the

development of expertise have been studied across a range

of motor, visuomotor, perceptual and cognitive tasks, and

from disparate research perspectives. To summarize briefly ,

four main patterns of practice-related activation change can

be distinguished [3]. Practice can lead to 1) an increase in

activation in the brain areas involved in task performance,

2) a decrease in those areas, or, 3) a functional redistribution

of brain activity, in which the level of activation in some

initially recruited areas decrease, whereas activation in

initially non-recruited areas increase, and 4) a functional

reorganization of brain activity, i.e., the pattern of activation

increases and decreases occur in distinct brain areas as well

as the initial areas.

The majority of studies examining task practice have found

decreases in the extent or intensity of activations,

particularly in the attentional and control areas [3]. This

finding is true whether the task is primarily motor (e.g., a

golf swing, [4]) or primarily cognitive in nature (e.g., the

Tower of London problem; [26]). Decreases in activation

represent a contraction of the neural representation of the

stimulus [27] or a more precise functional circuit [28] . This

finding provides an important overlap with the literature on

expertise. There is considerable evidence that experts tend

to be show lower brain activity relative to novices,

particularly in the prefrontal areas (e.g., [4]). Both practice

and the development of expertise (the latter of which

includes individual differences in ability) typically involve

decreased activation across attentional and control areas,

freeing these neural resources to attend to other incoming

stimuli or task demands. As such, measuring activation in

these attentional and control areas relative to task

performance can provide an index of level of expertise.

fNIR has been widely used to monitor activation across

attentional and control areas. Importantly, there is evidence

from fNIR technology-based studies showing that

differences in workload can be assessed in the dorsolateral

and ventrolateral prefrontal cortices during performance of

complex tasks and that the level of expertise development

can be objectively monitored [20, 29].

3. OPTICAL BRAIN IMAGING

Near infrared spectroscopy has been increasingly used in

human brain activation studies in adults and children since it

was first described by Jobsis (1977) as an optical method for

non-invasively assessing cerebral oxygenation changes [30].

The technique eventually evolved into a useful tool for

neuroimaging studies for measuring functional brain

activity. Functional Near-Infrared Spectroscopy (fNIR) is a

multi-wavelength optical method that monitors changes in

local cerebral oxygenation by measuring concentration

changes of both deoxygenated hemoglobin (deoxy-Hb) and

oxygenated hemoglobin (oxy-Hb) [14-16]. Various types of

brain activities, such as motor and cognitive activities, have

been studied using fNIR which has become increasingly

popular for neuroimaging studies due to its portability and

relatively low cost [18]. It measures hemodynamic changes

in the brain similar to fMRI but unlike it, fNIR is quiet (no

operating sound), provides higher temporal resolution and

participants are not restricted to a confined space or are not

required to lie down.

Principles of fNIR and Modified Beer Lambert Law

Typically, an optical apparatus consists of a light source by

which the tissue is radiated and a light detector that receives

light after it has interacted with the tissue. Most biological

tissues are relatively transparent to light in the near infrared

range, with the range of light between 700 to 900 nm.

Photons that enter tissue undergo two different types of

interaction: absorption and scattering. Within the near

infrared range of light, the two primary absorbers are oxy-

Hb (HbO2) and deoxy-Hb (Hb). During a cognitive activity,

the change in concentration of these main absorbers

provides information about brain functions.

Figure 1 Absorption spectrum of main chromophores in

tissue: Low absorption between 700-900nm provides an

optical window to the tissue

By measuring optical density (OD) changes at two

wavelengths, the relative change of oxy-Hb and deoxy-Hb

versus time can be obtained using the modified Beer–

Lambert Law. OD at a specific input wavelength (λ) is the

logarithmic ratio of input light intensity (Iin) and output

(detected) light intensity. OD is also related to the

concentration (c) and absorption coefficient ( ) of

chromophores, the corrected distance (d) of the light source

and detector, plus a constant attenuation factor (G).

(

) (1)

Having the same Iin at two different time instances and

detected light intensity during baseline (Irest) and during

performance of the task (Itest), the difference in OD is

(

)

( 2)

4

Measuring the OD at two different wavelengths gives

[

] [

] [

] (3)

This equation set can be solved for concentrations if the 2x2

matrix is non-singular. Typically, the two wavelengths are

chosen i) within 700-900nm where the absorption of oxy-

Hb and deoxy-Hb are dominant as compared to other tissue

chromophores, and ii) below and above the isosbestic point

(~805nm where absorption spectrums of deoxy- and oxy-Hb

cross each other) to focus the changes in absorption to either

deoxy-Hb or oxy-Hb, respectively.

3. METHODS

Data Acquisition

Throughout all experiments, the prefrontal cortex of each

participant was monitored using a continuous wave fNIR

system first described by Chance et al. (1998), further

developed at Drexel University (Philadelphia, PA),

manufactured and supplied by fNIR Devices LLC

(Potomac, MD; www.fnirdevices.com). The system was

composed of three modules: a flexible headpiece (sensor

pad), which holds light sources and detectors to enable a fast

placement of all 16 optodes; a control box for hardware

management; and a computer that runs the data acquisition

(Fig. 2).

Figure 2 fNIR sensor pad that measures from 16

locations over the forehead and system overview.

The sensor has a temporal resolution of 500 milliseconds

per scan with 2.5 cm source-detector separation allowing for

approximately 1.25 cm penetration depth. The light emitting

diodes (LED) were activated one light source at a time and

the four surrounding photodetectors around the active

source were sampled. The positioning of the light sources

and detectors on the sensor pad yielded a total of 16 active

optodes (channels) and was designed to monitor dorsal and

inferior frontal cortical areas underlying the forehead [31].

COBI Studio software (Drexel University) was used for data

acquisition and visualization [19]. During the UAV tasks, a

serial cable between the fNIR data acquisition computer and

stimulus presentation computer was used to transfer time

synchronization signals for marking the onset of sessions

and stimuli. All flight path and self-reported subjective

measures were recorded for each participant, each day.

Participants

Thirteen college students between the ages of 21 to 28 with

no prior flight simulator experience volunteered for this

study. Prior to the study, all participants signed informed

consent forms.

Experiment Protocol

Participants practiced approach scenarios while piloting a

virtual unmanned aerial vehicle (UAV) in a flight simulator.

The scenarios were designed to expose novice subjects to

realistic and critical tasks for a UAV ground operator

piloting an aircraft. During the turn-to-approach task, the

pilots flew through several waypoints on an approach to

land at an airfield. Subjects were told to fly as smoothly as

possible, learn the optimal paths, cope with crosswinds, and

operate within certain speed and bank angle constraints.

The flight simulator system was designed in house and

consisted of a custom desktop computer with high

performance graphics capabilities, semi-immersive angled

triple-monitor display, Thrustmaster HOTAS Cougar

joystick-and-throttle system and a CH Pro Pedals rudder

pedal system [32]. The scenarios were rendered on

Microsoft Flight Simulator X. FS Recorder, an add-on for

Flight Simulator X, was integrated to record behavioral data

during the simulated flights.

The experimental protocol involved a total of nine sessions

per subject, one session per day. The first session (day 1)

was introductory, allowing subjects to become acquainted

with the flight simulator; with the requirement that by the

end of this session, they needed to demonstrate a basic

understanding of flight simulator controls.

Study data were collected during the following eight

practice sessions. Practice sessions consisted of ten

repetitions of the approach to turn scenario (See Figure 3), a

total of ten flights per session, for a total of 80 trials per

participant over the 8 days.

Participants provided subjective mental effort and

performance evaluation using the NASA Task Load Index

(TLX) questionnaire [23]. Each session lasted 2 to 3 hours,

with no more than one session per day.

5

Figure 3 Pilot’s view of UAV flight simulator rendered

scene (left) and bird-s eye view of turn-to-approach task

waypoints marked on flight path map (right)

Data Analysis

For each participant, raw fNIR data (16 optodes×2

wavelengths) were low-pass filtered with a finite impulse

response, linear phase filter with order 20 and cut-off

frequency of 0.1 Hz to attenuate the high frequency noise,

respiration and cardiac cycle effects [19, 33]. Saturated

channels (if any), in which light intensity at the detector was

higher than the analog-to-digital converter limit were

excluded. fNIR data epochs for the rest and task periods

were extracted from the continuous data using time

synchronization markers. Blood oxygenation and volume

changes within each 16 optodes were calculated using the

modified Beer-Lambert Law for task periods with respect to

rest periods at beginning of each task with fnirSoft [34].

The main effect for practice level was tested using one-way

repeated measures analysis of variance (ANOVA), with

Subject and Practice Level designated as fixed effects.

Geisser–Greenhouse (G–G) correction was used when

violations of sphericity occurred in the omnibus tests.

Tukey's post hoc tests were used to determine the locus of

the main effects with a 0.05 significance criterion. Three

practice levels were defined for each participant: the

beginner phase included days 2 through 4, intermediate

phase included days 5 through 7 and the advanced phase

included days 8 and 9. Number Cruncher Statistical

Software (NCSS) 2007 (www.ncss.com) was used for the

statistical tests. Effect size indices of d* were calculated

[35] and were used to aid in the interpretation of the results.

4. RESULTS

Self-reported Ratings

The NASA TLX index results for the UAV tasks were

analyzed using one-way repeated measures ANOVA. The

results indicated a significant main effect of practice level

(beginner/intermediate/advanced conditions) for mental

demand (F2,24=12.37, p < 0.01), effort (F2,24=11.50, p <

0.01) and frustration (F2,24 = 6.40, p < 0.01). Both mental

demand and perceived effort followed a monotonic decrease

from beginner to advanced phase (see Figure 4). Tukey’s

post hoc tests revealed that mental demand (q0.05/2, 24 = 3.53,

p < 0.05) for the beginner phase was significantly higher

than the other conditions. The size of the effect was large (d

= -0.89 and -1.12 when the beginning phase is compared to

the intermediate and advanced phases, respectively. Results

for effort (q0.05/2, 24 = 3.53, p < 0.05) showed the beginner

phase required higher effort than the other conditions with

large effect sizes (d = -0.88, -1.07). Similar to mental

demand and effort, the frustration demand (q0.05/2, 24 = 3.53,

p < 0.05) was higher in the beginner phase than in the

intermediate or advanced phases and the effect sizes ranged

from d = -0.37 to -0.85).

Similarly, using each day instead of phases

(beginner/intermediate/advanced) indicated similar trends.

There was a significant main effect of days with mental

demand (F7,84=4.47, p < 0.01), effort (F7,84=5.88, p < 0.01)

and frustration (F7,84 = 5.69, p < 0.01). Tukey’s post hoc

tests confirmed that days 2&3 and were significantly

different than days 7,8 and 9 (q0.05/2, 84 = 4.39, p < 0.05) .

Figure 4 Self-reported ratings. Mental demand (top

row), effort (middle row) and frustration (bottom row).

Comparison of phases (left column), and days (right

column). Error bars indicate Standard Error of Mean

(SEM)

Behavioral Measures

Behavioral performance was calculated as root-mean-square

(RMS) deviation from the 4th

-order polynomial fits to the

path for latitude and longitude [20]. There was a significant

monotonic decrease in mean deviation from the optimal

path when comparing practice level (beginner through

advanced) for altitude (F2,24=2.32, p < 0.01) , longitude

6

(F2,24=3.14, p < 0.01), latitude (F2,24=3.36, p < 0.01) and

bank angle deviation (F2,24=4.03, p < 0.01), see Figure 5.

Post hoc analyses confirmed that an error at the beginner

phase was significantly higher than the other two phases

(q0.05/2, 24 = 3.53, p < 0.05) resulting in moderate to large

effect sizes (d = -0.62 to -1.03).

Figure 5 Behavioral measures. Altitude (top row),

longitude (2nd row), latitude (3rd row) and bank angle

(bottom row). Comparison of phases (left column), and

days (right column). Error bars indicate SEM.

Similarly, the deviation from optimal path gradually

lessened across day. The following parameters were

significant: altitude (F7,84=2.56, p < 0.01), longitude

(F7,84=2.90, p < 0.01), latitude (F7,84=2.50, p < 0.01) and

bank angle deviations (F7,84=4.40, p < 0.01), see Figure 5.

Post hoc analyses confirmed individual difference across

days. Most specifically, errors at days 2, 3 were

significantly greater than the error at the rest of the days

(q0.05/2, 84 = 4.39, p < 0.05) for all parameters.

fNIR Measures

The average total hemoglobin (HbT) concentration changes

throughout the practice levels (beginner /intermediate

/advanced) were analyzed using one-way repeated measures

ANOVA. Based on findings from our previous study, the

region of interest was identified as optode #2, which is close

to AF7 in the International 10-20 System and is located at

the left PFC (inferior frontal gyrus). The response was

significant (F2,24= 1.96, p < 0.01), and indicated a monotonic

decrease trend as expected from our previous results and as

seen in self-reported and behavioral measures (see Figure

6). The effect size d = -0.19 represents a small effect when

assessing the difference between advanced and beginner

stages of learning. When the results were analyzed using

days, the response was significant (F7, 84= 1.87, p < 0.01),

and yielded an interesting aspect of the learning process that

results in different trends at beginning, intermediate and

advanced phases. In the beginner phase, there was an

increase in brain activation until the intermediate phase was

reached. During the advanced phase, there is a decreasing

trend across days. The effect size across comparing days 2,

3, and 4 with day 9 are d= -0.13, -0.32, and -1.91,

respectively. The mean effect size comparing the initial

days of training to the last day of training is d -0.79.

Figure 6 fNIR measures. Comparison of phases (left)

and days (right). Error bars indicate SEM.

An efficiency graph can be a useful visual tool to assess the

impact of learning on performance [36]. The efficiency

graph in Figure 7 visualizes the overall results by plotting

normalized HbT (total hemoglobin concentration) changes

that represent mental effort against the inverse-normalized

bank angle values that model behavioral performance.

In this efficiency graph, the fourth quadrant represents low

efficiency, where minimum performance is achieved with

maximum effort. The second quadrant represents high

efficiency where maximum performance is achieved with

minimal effort. The diagonal y=x is the neutral axis, where

efficiency (E) is zero and effort and performance are equal.

Results indicate that subjects have achieved a transition

from a low efficiency mode to a high efficiency mode

throughout the 9-day long learning process.

7

Figure 7 Efficiency graph indicating transition from low

efficiency to high efficiency throughout the learning

process.

5. DISCUSSION

The purpose of this study was to examine the impact of

practice, or relative levels of expertise, on

neurophysiological measures of the hemodynamic response

in the prefrontal cortex during a complex cognitive task.

The complex, but interpretable, pattern of prefrontal cortical

response in the UAV tasks are consistent with a

‘scaffolding-storage’ framework of functional

reorganization during the development of expertise [21].

The results indicated an expected trend in self-reported

measures, behavioral performance and brain activation

when overall beginner, intermediate and advanced phases

were compared. All measures indicate an improvement in

performance and a decrease in brain activation across

sessions consistent with our previous practice and learning

related fNIR results [20, 37-39]. When the level of analysis

is shifted from phases to days, our results showed that

behavioral measures provided more detailed trend

information than subjective self-reported measures, which

suggests that subjective self-reports have a lower resolution

as compared to objective behavioral measures. Moreover,

brain activation changes as measured by fNIR provided

additional fine-grained information by revealing a different

trend in each learning phase. At the beginner’s level,

behavioral performance improved each day at the cost of

increased brain activation suggesting that increased effort

was required to learn the required skill set. In the

intermediate phase, a higher level of performance could be

maintained with less activation. Finally, in the advanced

phase, an even higher level of performance was achieved

with a decreasing trend in the activation.

It is possible that there may be additional factors that

contribute to an increase in workload and may recruit areas

of the brain that were not being monitored in the current

study. More information could be gained by monitoring

parietal areas in addition to the frontal cortex, or by

combining fNIR with EEG. Alternatively, signal processing

algorithms for fNIR may not be optimized at this time, or

the hemodynamic response itself may not be sufficiently

sensitive to pick up subtle changes in workload. This is an

area for future research.

6. SUMMARY

A field deployable optical brain imaging technology (fNIR)

holds potential for research studies and clinical applications

that require quantitative measurements of hemodynamic

changes during brain activation under ambulant conditions

in natural environments. As such, fNIR has been already

deployed in many field settings for objective measurements

of cognitive states and expertise development which will,

among other advantages, allow for dynamic interventions in

the training process, and help to assure robust performance

under adverse circumstances. fNIR application areas

include, but are not limited to, brain computer interface for

cognitive enhancement, neurological and gaming

applications, education, training and cognitive aging.

The study described here provides important albeit

preliminary information about fNIR measures of the PFC

hemodynamic response and its relationship to mental

workload, expertise, and performance, in a complex

multitasking environment. Level of expertise does appear to

influence the hemodynamic response in the

dorsolateral/ventrolateral prefrontal cortices, at least for

some complex tasks. Since fNIR technology allows the

development of mobile, non-intrusive and miniaturized

devices, it has the potential to be deployed in future learning

and training environments to personalize the training

regimen and/or to assess the effort of human operators in

critical multitasking environments.

Acknowledgements

Authors would like to thank Joshua Harrison and Justin

Menda for data acquisition and handling. This study was

funded in part under a U.S. Army Medical Research

Acquisition Activity; Cooperative Agreement W81XWH-

092-0104. The content of the information herein does not

necessarily reflect the position or the policy of the U.S.

Government or the U.S. Army and no official endorsement

should be inferred.

8

REFERENCES

[1] R. Parasuraman and G. Wilson, "Putting the brain

to work: Neuroergonomics past, present, and

future," Human factors, vol. 50, p. 468, 2008.

[2] R. Parasuraman and M. Rizzo, Neuroergonomics:

The brain at work: Oxford University Press, USA,

2007.

[3] A. Kelly and H. Garavan, "Human functional

neuroimaging of brain changes associated with

practice," Cerebral Cortex, vol. 15, p. 1089, 2005.

[4] J. G. Milton, S. S. Small, and A. Solodkin, "On the

road to automatic: dynamic aspects in the

development of expertise," Journal of clinical

neurophysiology, vol. 21, p. 134, 2004.

[5] A. M. Owen, K. M. McMillan, A. R. Laird, and E.

Bullmore, "N-back working memory paradigm: a

meta-analysis of normative functional

neuroimaging studies," Human Brain Mapping,

vol. 25, pp. 46-59, May 2005.

[6] J. D. Cohen, W. M. Perlstein, T. S. Braver, L. E.

Nystrom, D. C. Noll, J. Jonides, and E. E. Smith,

"Temporal dynamics of brain activation during a

working memory task," Nature, vol. 386, pp. 604-

608, 1997.

[7] B. Cain, "A review of the mental workload

literature," DTIC Document 2007.

[8] E. E. Smith and J. Jonides, "Working Memory: A

View from Neuroimaging," Cognitive Psychology,

vol. 33, pp. 5-42, 1997.

[9] S. H. Fairclough and L. Venables, "Prediction of

subjective states from psychophysiology: A

multivariate approach," Biological Psychology,

vol. 71, pp. 100-110, 2006.

[10] A. Gevins, M. Smith, L. McEvoy, and D. Yu,

"High-resolution EEG mapping of cortical

activation related to working memory: effects of

task difficulty, type of processing, and practice,"

Cerebral Cortex, vol. 7, p. 374, 1997.

[11] A. Gevins and M. Smith, "Neurophysiological

measures of cognitive workload during human-

computer interaction," Theoretical Issues in

Ergonomics Science, vol. 4, pp. 113-131, 2003.

[12] R. Stevens, T. Galloway, and C. Berka, "EEG-

related changes in cognitive workload, engagement

and distraction as students acquire problem solving

skills," User Modeling 2007, pp. 187-196, 2009.

[13] W. Klimesch, "EEG alpha and theta oscillations

reflect cognitive and memory performance: a

review and analysis," Brain Research Reviews, vol.

29, pp. 169-195, 1999.

[14] B. Chance, Z. Zhuang, C. UnAh, C. Alter, and L.

Lipton, "Cognition-activated low-frequency

modulation of light absorption in human brain,"

Proc Natl Acad Sci U S A, vol. 90, pp. 3770-4, Apr

15 1993.

[15] A. Villringer, J. Planck, C. Hock, L. Schleinkofer,

and U. Dirnagl, "Near infrared spectroscopy

(NIRS): a new tool to study hemodynamic changes

during activation of brain function in human

adults," Neurosci Lett, vol. 154, pp. 101-4, May 14

1993.

[16] A. Villringer and B. Chance, "Non-invasive optical

spectroscopy and imaging of human brain

function," Trends Neurosci, vol. 20, pp. 435-42,

Oct 1997.

[17] G. Strangman, D. A. Boas, and J. P. Sutton, "Non-

invasive neuroimaging using near-infrared light,"

Biol Psychiatry, vol. 52, pp. 679-93, Oct 1 2002.

[18] M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A.

Devaraj, B. Onaral, and K. Pourrezaei, "Functional

near-infrared neuroimaging," IEEE Trans Neural

Syst Rehabil Eng, vol. 13, pp. 153-9, Jun 2005.

[19] H. Ayaz, P. A. Shewokis, A. Curtin, M. Izzetoglu,

K. Izzetoglu, and B. Onaral, "Using MazeSuite and

Functional Near Infrared Spectroscopy to Study

Learning in Spatial Navigation," J Vis Exp, p.

e3443, 2011.

[20] H. Ayaz, P. A. Shewokis, S. Bunce, K. Izzetoglu,

B. Willems, and B. Onaral, "Optical brain

monitoring for operator training and mental

workload assessment," Neuroimage, vol. 59, pp.

36-47, 2012.

[21] S. E. Petersen, H. Van Mier, J. A. Fiez, and M. E.

Raichle, "The effects of practice on the functional

anatomy of task performance," Proceedings of the

National Academy of Sciences, vol. 95, p. 853,

1998.

[22] P. Hancock and M. H. Chignell, "Toward a Theory

of Mental Work Load: Stress and Adaptability in

Human-Machine Systems," Proc. IEEE SMC 1986,

pp. 378-383, 1986.

[23] S. Hart and L. Staveland, "Development of NASA-

TLX (Task Load Index): Results of empirical and

theoretical research," Human mental workload, vol.

1, pp. 139–183, 1988.

[24] N. Meshkati, P. A. Hancock, M. Rahimi, and S. M.

Dawes, "Techniques in mental workload

assessment," in Evaluation of human work: A

practical ergonomics methodology J. R. Wilson

and E. N Corlett, Eds., 2nd ed Philadelphia, PA,

US: Taylor & Francis,, 1995, pp. 749-782.

[25] H. Obrig, R. Wenzel, M. Kohl, S. Horst, P. Wobst,

J. Steinbrink, F. Thomas, and A. Villringer, "Near-

infrared spectroscopy: does it function in

functional activation studies of the adult brain?,"

International Journal of Psychophysiology, vol. 35,

pp. 125-142, 2000.

[26] M. Beauchamp, A. Dagher, J. Aston, and J. Doyon,

"Dynamic functional changes associated with

cognitive skill learning of an adapted version of the

Tower of London task," Neuroimage, vol. 20, pp.

1649-1660, 2003.

[27] R. A. Poldrack, A. D. Wagner, M. W. Prull, J. E.

Desmond, G. H. Glover, and J. D. E. Gabrieli,

"Functional Specialization for Semantic and

Phonological Processing in the Left Inferior

Prefrontal Cortex* 1," Neuroimage, vol. 10, pp. 15-

9

35, 1999.

[28] H. Garavan, D. Kelley, A. Rosen, S. M. Rao, and

E. A. Stein, "Practice-related functional activation

changes in a working memory task," Microscopy

Research and Technique, vol. 51, pp. 54-63, 2000.

[29] K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei,

and B. Chance, "Functional Optical Brain Imaging

Using Near-Infrared During Cognitive Tasks,"

International Journal of Human-Computer

Interaction, vol. 17, pp. 211-227, 2004.

[30] F. F. Jobsis, "Noninvasive, infrared monitoring of

cerebral and myocardial oxygen sufficiency and

circulatory parameters," Science, vol. 198, pp.

1264-7, Dec 23 1977.

[31] H. Ayaz, M. Izzetoglu, S. M. Platek, S. Bunce, K.

Izzetoglu, K. Pourrezaei, and B. Onaral,

"Registering fNIR data to brain surface image

using MRI templates," Conf Proc IEEE Eng Med

Biol Soc, pp. 2671-4, 2006.

[32] J. Menda, J. T. Hing, H. Ayaz, P. A. Shewokis, K.

Izzetoglu, B. Onaral, and P. Oh, "Optical Brain

Imaging to Enhance UAV Operator Training,

Evaluation, and Interface Development," Journal

of Intelligent & Robotic Systems, vol. 61, pp. 423-

443, 2010.

[33] H. Ayaz, M. Izzetoglu, P. A. Shewokis, and B.

Onaral, "Sliding-window Motion Artifact

Rejection for Functional Near-Infrared

Spectroscopy," Conf Proc IEEE Eng Med Biol Soc,

pp. 6567-70, 2010.

[34] H. Ayaz, "Functional Near Infrared Spectroscopy

based Brain Computer Interface," PhD Thesis,

School of Biomedical Engineering Science &

Health Systems, Drexel University, Philadelphia,

2010.

[35] P. J. Potvin and R. W. Schutz, "Statistical power

for the two-factor repeated measures ANOVA,"

Behavior Research Methods, vol. 32, pp. 347-356,

2000.

[36] R. Clark, F. Nguyen, and J. Sweller, Efficiency in

learning: Evidence-based guidelines to manage

cognitive load. San Fransico, CA: Pfeiffer, An

Imprint of Wiley, 2006.

[37] P. Shewokis, H. Ayaz, M. Izzetoglu, S. Bunce, R.

Gentili, I. Sela, K. Izzetoglu, and B. Onaral, "Brain

in the Loop: Assessing Learning Using fNIR in

Cognitive and Motor Tasks," in Foundations of

Augmented Cognition. Directing the Future of

Adaptive Systems. vol. 6780, D. Schmorrow and C.

Fidopiastis, Eds., ed: Springer Berlin / Heidelberg,

2011, pp. 240-249.

[38] S. Bunce, K. Izzetoglu, H. Ayaz, P. Shewokis, M.

Izzetoglu, K. Pourrezaei, and B. Onaral,

"Implementation of fNIRS for Monitoring Levels

of Expertise and Mental Workload," in

Foundations of Augmented Cognition. Directing

the Future of Adaptive Systems. vol. 6780, D.

Schmorrow and C. Fidopiastis, Eds., ed: Springer

Berlin / Heidelberg, 2011, pp. 13-22.

[39] H. Ayaz, P. Shewokis, S. Bunce, M. Schultheis,

and B. Onaral, "Assessment of Cognitive Neural

Correlates for a Functional Near Infrared-Based

Brain Computer Interface System," in Foundations

of Augmented Cognition. Neuroergonomics and

Operational Neuroscience, D. Schmorrow, Ed., ed,

2009, pp. 699-708.

10

BIOGRAPHIES

Hasan Ayaz is an Assistant Research

Professor at Drexel University, School

of Biomedical Engineering, Science and

Health Systems, Philadelphia, PA. He

received his BSc. in Electrical and

Electronics Engineering at Bogazici

University, Istanbul, Turkey, with high

honors and MSc. and PhD degrees from

Drexel University where he developed enabling software for

functional Near Infrared Spectroscopy (fNIR) based brain

monitoring instruments. This technology was licensed by

fNIR Devices LLC and has been distributed to over 50

research labs by Biopac Systems, Inc. As an extension, he

worked on a portable-handheld medical device

(InfraScanner) that utilizes fNIR to detect hematoma in head

trauma patients. InfraScanner is currently pending FDA

approval but has been deployed overseas and is already

saving lives in Europe, Africa and Asia. InfraScanner

related awards include DMD (Design of Medical Devices)

Conference 2011 top presenter and EID (Excellence in

Design) Gold Award. Dr. Ayaz’s research interests include

Neuroengineering applications of human computer

interaction and Neuroergonomics, specifically, i)

development of noninvasive brain computer interfaces (BCI)

for communication and augmented interactivity in

simulation and gaming settings, ii) optical brain imaging

for assessment of cognitive workload and expertise

development of operators such as such as air traffic

controllers (ATC) and unmanned aerial vehicle (UAV)

ground operators.

Murat Perit Çakır is an Assistant

Professor in Middle East Technical

University (METU), Ankara, Turkey. He

received BSc from METU in

Mathematics and MSc degree from

University of Pennsylvania in Computer

Science and PhD degree from Drexel University in

Information Science & Technology. Dr. Cakir’s research

interests include Computer-Supported Collaborative

Learning, Human-Computer Interaction, Interaction

Analysis, Groupware Design, Immersive Learning

Environments, and Cognitive Neuroscience of Learning.

Kurtuluş İzzetoğlu received his Ph.D.

degree from Drexel University, School

of Biomedical Engineering, Science and

Health Systems, Philadelphia, PA.

During his Ph.D. studies, he worked on

the functional near-infrared (fNIR)

spectroscopy system to identify

neuromarkers during change in the cognitive state of mental

engagement at both high and low levels of neural activation.

He is currently a Research Assistant Professor in the School

of Biomedical Engineering at Drexel University and has

also been working as a project engineer in various research

projects for several years. His research interests include

functional brain imaging, medical sensor development and

biomedical signal processing. His current research projects

mainly focus on novel algorithms and techniques to deploy

the fNIR system for various application areas, such as depth

of anesthesia monitor and cognitive workload assessment.

He is currently working on various funded projects that are

involved in 1) cognitive workload assessment of unmanned

air vehicle (UAV) operators, 2) sensor and algorithm

development for depth of anesthesia monitoring, 3)

cognitive baselining and index developments, 4) cognitive

workload assessment of air traffic controllers.

Adrian Curtin is a graduate research

assistant at Drexel University. Mr.

Curtin is currently pursuing his BS-MS

degree in Biomedical Engineering.

Patricia A. Shewokis received a Ph.D.

in the Psychology of Motor Behavior

from the University of Georgia in

1993. She is a Professor with joint

appointments in the College of Nursing

and Health Professions and School of

Biomedical Engineering, Science and

Health Systems at Drexel University, Philadelphia, PA. Her

active research program focuses on: (a) the processes and

mechanisms involved in the acquisition, retention and

transfer of cognitive and motor skills; b) brain-computer

interface research employing biofeedback in learning

paradigms; c) brain in the loop assessments of learning;

and d) statistical and methodological assessments of

learning. Currently, Dr Shewokis is the Research Director

of the Cognitive Neuroengineering and Quantitative

Research (CONQUER) Collab0rative at Drexel.

Scott C. Bunce received a B.A. in

Philosophy and Biology from Wheaton

College in 1984, an M.A. in Personality

Psychology from the University of

Michigan in 1990, and Ph.D.’s in

Clinical and Personality Psychology

from the University of Michigan in

1993. Dr. Bunce is currently Associate Professor and

Interim Director of Behavioral Neuroimaging in Psychiatry

at Penn State Milton S. Hershey Medical Center and the

Penn State College of Medicine. He also holds an adjunct

appointment in the School of Biomedical Engineering,

Science & Health Systems at Drexel University. In his

previous appointment as Assistant Professor of Psychiatry

at MCP Hahnemann/Drexel University College of Medicine,

Dr. Bunce served as Director of the Clinical Neuroscience

Research Unit, a laboratory that investigated a range of

topics related to affective and cognitive neuroscience using

noninvasive measures of brain function. While at Drexel,

Dr. Bunce helped develop functional near-infrared optical

technology as a member of the School of Biomedical

Engineering’s Optical Imaging Team.

11

Banu Onaral is H. H. Sun Professor

of Biomedical Engineering and

Electrical Engineering at Drexel

University, Philadelphia, PA. She

holds a Ph.D. [1978] in Biomedical

Engineering from the University of

Pennsylvania and BSEE [1973] and

MSEE [1974] in Electrical Engineering from Bogazici

University, Istanbul, Turkey. Dr. Onaral joined the faculty

of the Department of Electrical and Computer Engineering

and the Biomedical Engineering and Science Institute in

1981. Since 1997, she has served as the founding Director

of the School of Biomedical Engineering Science and Health

Systems. Her academic focus, both in research and

teaching, is centered on information engineering with

special emphasis on complex systems and biomedical signal

processing in ultrasound and optics. She has led major

research and development projects sponsored by the

National Science Foundation (NSF), National Institutes of

Health (NIH), Office of Naval Research (ONR), DARPA and

Department of Homeland Security (DHS). She has

supervised a large number of graduate students to degree

completion and has an extensive publication record in

biomedical signals and systems. She is the recipient of a

number of faculty excellence awards, including the 1990

Lindback Distinguished Teaching Award of Drexel

University, the EDUCOM Best educational Software award,

and the NSF Faculty Achievement Award.