978-1-4577-0557-1/12/$26.00 ©2012 IEEE
1
Monitoring Expertise Development during Simulated UAV
Piloting Tasks using Optical Brain Imaging
Hasan Ayaz1,Murat P. Çakır
2,Kurtuluş İzzetoğlu
1,Adrian Curtin
1,Patricia A. Shewokis
1,Scott C. Bunce
3,Banu Onaral
1
1Drexel University, 3141 Chestnut St. Philadelphia PA USA, +1 (215) 895 2215, (ha45,ki25,abc48,pas38,bo26)@drexel.edu
2Middle East Technical University, Informatics Institute, Ankara, Turkey, +90 (312) 210 7706, [email protected]
3Penn State University, M.S. Hershey Medical Center, Hershey, PA USA, +1 (717) 531-4127, [email protected]
Abstract— An accurate assessment of mental workload and
expertise level would help improve operational safety and
efficacy of human computer interaction for aerospace
applications. The current study utilized functional near-
infrared spectroscopy (fNIR) to investigate the relationship of
the hemodynamic response in the anterior prefrontal cortex to
changes in mental workload, level of expertise, and task
performance during learning of simulated unmanned aerial
vehicle (UAV) piloting tasks. Results indicated that fNIR
measures are correlated to task performance and subjective
self-reported measures; and contained additional information
that allowed categorizing learning phases. Level of expertise
does appear to influence the hemodynamic response in the
dorsolateral/ventrolateral prefrontal cortices. Since fNIR
allows development of portable and wearable instruments, it
has the potential to be deployed in future learning
environments to personalize the training regimen and/or assess
the effort of human operators in critical multitasking settings.
TABLE OF CONTENTS
1. INTRODUCTION ................................................. 1 2. MENTAL WORKLOAD AND EXPERTISE ............. 2 3. OPTICAL BRAIN IMAGING ................................ 3 3. METHODS .......................................................... 4 4. RESULTS ............................................................ 5 5. DISCUSSION ....................................................... 7 6. SUMMARY ......................................................... 7 REFERENCES ......................................................... 8 BIOGRAPHIES ...................................................... 10
1. INTRODUCTION
The efficiency and safety of many complex human-machine
systems are closely related to the cognitive workload and
situational awareness of their operators [1, 2]. An ideal
human-machine system would be informed about the
current cognitive workload level of its personnel and/or be
designed to keep the necessary workload at an optimum
level. Hence, it could help prevent potential overload and
minimize errors. Importantly, behavioral measures alone
may not be sufficiently sensitive to index overload because
operators may extend extra effort to maintain system
performance, but this could come at a cost that is reflected
only in neural measures. Significant progress has been made
over the last decade in understanding the neural bases of
cognitive processes and behavior. The advent of new and
improved brain imaging tools that allow monitoring of brain
activity in the field and operational environments is
expected to allow better identification of neurophysiological
markers that can index impending overload or fatigue before
performance measures can.
Deployment of portable neuroimaging technologies to real
time settings could help assess cognitive and motivational
states of human operators and close the loop for advanced
human-machine interaction for optimized learning and
training. There is considerable evidence that
neurophysiological and psychophysiological variables
respond to cognitive demand in a predictable manner [3-9].
Functional Magnetic Resonance Imaging (fMRI) is
currently considered the “gold standard” for measuring
functional brain activation. However, fMRI is expensive to
operate and requires massive installations and a large
infrastructure to operate. In addition, fMRI confines
participants to restricted positions, is highly sensitive to
motion artifacts and exposes participants to loud noises.
Direct measures of central nervous system function such as
electroencephalography (EEG) and event-related potentials
(ERPs) have been particularly strong candidates for
accurate, objective measures of operator workload [10-13].
Increasing task difficulty, for instance, is known to be
associated with EEG changes such as increased power in the
beta bandwidth, increased theta activity at frontal sites and
the suppression of alpha activity [11, 13]. Although EEG
has many excellent qualities for monitoring mental
workload, including excellent temporal resolution, it is
limited in its capacity for spatial resolution.
In this study, we utilized a field-deployable functional
optical imaging method for monitoring the prefrontal cortex
using functional near-infrared spectroscopy (fNIR) [14-20].
fNIR is a safe, non-invasive, affordable and portable
neuroimaging technology that can be used to monitor
hemodynamic changes that occur in the brain, i.e., blood
oxygenation and blood volume, during select cognitive
tasks. These qualities pose fNIR as an ideal candidate for
monitoring cognitive activity-related hemodynamic changes
not only in laboratory settings but also under ecologically
valid conditions – real world environments.
For the current study, 13 healthy individuals with no prior
flight simulator experience volunteered for a training
program of unmanned aerial vehicle (UAV) piloting.
2
Throughout the 9 day program, participants practiced turn-
to-approach tasks 10 times per day while their behavioral
data and anterior prefrontal cortex (PFC) brain activation (as
measured by fNIR) were recorded.
Preliminary results indicated significant improvement in the
behavioral performance measures and a corresponding
reduction in total hemoglobin concentration changes,
consistent with the scaffolding-storage framework [21] and
our previous results [20]. In this paper, we have further
demonstrated that fNIR and behavioral data together
provides a better indicator of training. Development of these
neuro-behavioral models could lead to optimized and
personalized training regimens and/or reliable assessment of
the effort human operators in critical multitasking
environments such as aerospace settings.
2. MENTAL WORKLOAD AND EXPERTISE
There is no singular definition of mental workload [7].
There are at least two major theoretical approaches to the
construct: 1) mental workload may be defined such that a
given task’s requirements are viewed as an independent,
external variable with which the working participants have
to cope more or less efficiently; or 2) mental workload may
be defined in terms of an interaction between task
requirements and human capabilities or resources [22].
Both approaches can offer essential and well-founded
contributions to different problems. In either of these
paradigms, the definition of workload involves the
“objective” effects of task difficulty on the participant, and
the participant’s effort involved in maintaining performance.
Workload is an intervening variable between task and
environmental demands and the operator’s performance,
which is defined by the relationship between task demand
and the participant’s supply of neural resources, i.e., the
portion of an operator’s limited mental capacities actually
required to perform a particular task. In other words,
workload can be defined in terms of some “objective”
criteria for task difficulty or in terms of the participant’s
capacities to perform the identified task. As such, workload
can be differentiated from performance. Two people
performing the same task can have identical performance,
yet one operator may have significant cognitive resources
free to allocate to concurrent tasks, whereas the second
operator may be just on the brink of performance failure.
The difference between the required capacity and the
available capacity of an individual can be referred to as the
mental or cognitive reserve.
There are four basic methodologies for the measurement of
mental workload: 1) primary task performance, 2)
subjective ratings (e.g., NASA Task Load Index
(TLX)[23]), 3) central and peripheral measures of
physiological response, and 4) secondary task techniques
[24]. Primary task performance measures are critical in that
they define the task that needs to be accomplished.
However, they cannot predict failure, and are only sensitive
to changes in workload at the limits of mental capacity. If
operators can compensate for increased workload by
increasing effort, then the primary task measure is
insensitive. Finally, task performance measures do not
provide any direct evidence about the subject’s degree of
effort or level of arousal. Subjective ratings provide an
important measure of the operator’s perceived load, and
have good face validity. However, they are intrusive to
collect while on task, and are often dissociated from actual
performance and potential failure. Secondary task
techniques can give some sense of cognitive or mental
reserve, but they are often intrusive, and cannot reasonably
be employed during actual UAV missions. Physiological
measures (such as skin conductance, electrooculogram,
electrocardiogram and respiration) can be unobtrusive, and
provide an “objective” measure of workload. However,
many of the typical physiological systems measure
autonomic responses, i.e., the fight or flight system, which
assesses stress, whereas some of the difference in workload
may more appropriately be assessed using a measure of
cognitive workload, rather than stress.
Central Nervous System Measures of Workload & Expertise
Physiological measures of central nervous system (CNS)
functioning hold the most promise for operational
monitoring of UAV pilots, as they provide continuous and
unobtrusive monitoring of the operator and do not interfere
with an operator's work as do secondary-task performance
or subjective measures of workload. Moreover, CNS
measures can provide information on cognitive functioning
as well as measures of stress or emotion.
Commonly employed techniques such as EEG, ERPs,
magnetoencephalography (MEG), positron emission
tomography (PET), single positron emission computed
tomography (SPECT), and fMRI have dramatically
increased our understanding of a broad range of cognitive
and emotional states. Methods that directly measure the
summation of neural function, such as MEG, EEG, and
ERPs, allow researchers to monitor the direct consequences
of brain electromagnetic activity with temporal resolution
on the order of milliseconds. However, these technologies
also have limited spatial resolution [17], and are susceptible
to electromagnetic artifacts in the field. In contrast, indirect
methods such as PET, SPECT and fMRI, monitor the
hemodynamic and metabolic changes associated with neural
activity with impressive spatial resolution, but are limited in
temporal resolution and are associated with neuronal
activity through a poorly understood neurovascular coupling
function [25]. In addition, PET and SPECT do not allow for
continuous or repeated measurements because they require
the use of radioactive isotopes, which also limits their use in
children.
A given individual will develop expertise more quickly with
increased time on task and/or by utilizing the most
appropriate and effective mode of practice for the task to be
learned. Native ability may contribute to the speed of an
individual’s progression, and may assist in determining
individual differences in enhanced development of expertise
in a given course of training. The literature dealing with the
3
effect of practice on the functional anatomy of task
performance is extensive and complex. Practice and the
development of expertise have been studied across a range
of motor, visuomotor, perceptual and cognitive tasks, and
from disparate research perspectives. To summarize briefly ,
four main patterns of practice-related activation change can
be distinguished [3]. Practice can lead to 1) an increase in
activation in the brain areas involved in task performance,
2) a decrease in those areas, or, 3) a functional redistribution
of brain activity, in which the level of activation in some
initially recruited areas decrease, whereas activation in
initially non-recruited areas increase, and 4) a functional
reorganization of brain activity, i.e., the pattern of activation
increases and decreases occur in distinct brain areas as well
as the initial areas.
The majority of studies examining task practice have found
decreases in the extent or intensity of activations,
particularly in the attentional and control areas [3]. This
finding is true whether the task is primarily motor (e.g., a
golf swing, [4]) or primarily cognitive in nature (e.g., the
Tower of London problem; [26]). Decreases in activation
represent a contraction of the neural representation of the
stimulus [27] or a more precise functional circuit [28] . This
finding provides an important overlap with the literature on
expertise. There is considerable evidence that experts tend
to be show lower brain activity relative to novices,
particularly in the prefrontal areas (e.g., [4]). Both practice
and the development of expertise (the latter of which
includes individual differences in ability) typically involve
decreased activation across attentional and control areas,
freeing these neural resources to attend to other incoming
stimuli or task demands. As such, measuring activation in
these attentional and control areas relative to task
performance can provide an index of level of expertise.
fNIR has been widely used to monitor activation across
attentional and control areas. Importantly, there is evidence
from fNIR technology-based studies showing that
differences in workload can be assessed in the dorsolateral
and ventrolateral prefrontal cortices during performance of
complex tasks and that the level of expertise development
can be objectively monitored [20, 29].
3. OPTICAL BRAIN IMAGING
Near infrared spectroscopy has been increasingly used in
human brain activation studies in adults and children since it
was first described by Jobsis (1977) as an optical method for
non-invasively assessing cerebral oxygenation changes [30].
The technique eventually evolved into a useful tool for
neuroimaging studies for measuring functional brain
activity. Functional Near-Infrared Spectroscopy (fNIR) is a
multi-wavelength optical method that monitors changes in
local cerebral oxygenation by measuring concentration
changes of both deoxygenated hemoglobin (deoxy-Hb) and
oxygenated hemoglobin (oxy-Hb) [14-16]. Various types of
brain activities, such as motor and cognitive activities, have
been studied using fNIR which has become increasingly
popular for neuroimaging studies due to its portability and
relatively low cost [18]. It measures hemodynamic changes
in the brain similar to fMRI but unlike it, fNIR is quiet (no
operating sound), provides higher temporal resolution and
participants are not restricted to a confined space or are not
required to lie down.
Principles of fNIR and Modified Beer Lambert Law
Typically, an optical apparatus consists of a light source by
which the tissue is radiated and a light detector that receives
light after it has interacted with the tissue. Most biological
tissues are relatively transparent to light in the near infrared
range, with the range of light between 700 to 900 nm.
Photons that enter tissue undergo two different types of
interaction: absorption and scattering. Within the near
infrared range of light, the two primary absorbers are oxy-
Hb (HbO2) and deoxy-Hb (Hb). During a cognitive activity,
the change in concentration of these main absorbers
provides information about brain functions.
Figure 1 Absorption spectrum of main chromophores in
tissue: Low absorption between 700-900nm provides an
optical window to the tissue
By measuring optical density (OD) changes at two
wavelengths, the relative change of oxy-Hb and deoxy-Hb
versus time can be obtained using the modified Beer–
Lambert Law. OD at a specific input wavelength (λ) is the
logarithmic ratio of input light intensity (Iin) and output
(detected) light intensity. OD is also related to the
concentration (c) and absorption coefficient ( ) of
chromophores, the corrected distance (d) of the light source
and detector, plus a constant attenuation factor (G).
(
) (1)
Having the same Iin at two different time instances and
detected light intensity during baseline (Irest) and during
performance of the task (Itest), the difference in OD is
(
)
( 2)
4
Measuring the OD at two different wavelengths gives
[
] [
] [
] (3)
This equation set can be solved for concentrations if the 2x2
matrix is non-singular. Typically, the two wavelengths are
chosen i) within 700-900nm where the absorption of oxy-
Hb and deoxy-Hb are dominant as compared to other tissue
chromophores, and ii) below and above the isosbestic point
(~805nm where absorption spectrums of deoxy- and oxy-Hb
cross each other) to focus the changes in absorption to either
deoxy-Hb or oxy-Hb, respectively.
3. METHODS
Data Acquisition
Throughout all experiments, the prefrontal cortex of each
participant was monitored using a continuous wave fNIR
system first described by Chance et al. (1998), further
developed at Drexel University (Philadelphia, PA),
manufactured and supplied by fNIR Devices LLC
(Potomac, MD; www.fnirdevices.com). The system was
composed of three modules: a flexible headpiece (sensor
pad), which holds light sources and detectors to enable a fast
placement of all 16 optodes; a control box for hardware
management; and a computer that runs the data acquisition
(Fig. 2).
Figure 2 fNIR sensor pad that measures from 16
locations over the forehead and system overview.
The sensor has a temporal resolution of 500 milliseconds
per scan with 2.5 cm source-detector separation allowing for
approximately 1.25 cm penetration depth. The light emitting
diodes (LED) were activated one light source at a time and
the four surrounding photodetectors around the active
source were sampled. The positioning of the light sources
and detectors on the sensor pad yielded a total of 16 active
optodes (channels) and was designed to monitor dorsal and
inferior frontal cortical areas underlying the forehead [31].
COBI Studio software (Drexel University) was used for data
acquisition and visualization [19]. During the UAV tasks, a
serial cable between the fNIR data acquisition computer and
stimulus presentation computer was used to transfer time
synchronization signals for marking the onset of sessions
and stimuli. All flight path and self-reported subjective
measures were recorded for each participant, each day.
Participants
Thirteen college students between the ages of 21 to 28 with
no prior flight simulator experience volunteered for this
study. Prior to the study, all participants signed informed
consent forms.
Experiment Protocol
Participants practiced approach scenarios while piloting a
virtual unmanned aerial vehicle (UAV) in a flight simulator.
The scenarios were designed to expose novice subjects to
realistic and critical tasks for a UAV ground operator
piloting an aircraft. During the turn-to-approach task, the
pilots flew through several waypoints on an approach to
land at an airfield. Subjects were told to fly as smoothly as
possible, learn the optimal paths, cope with crosswinds, and
operate within certain speed and bank angle constraints.
The flight simulator system was designed in house and
consisted of a custom desktop computer with high
performance graphics capabilities, semi-immersive angled
triple-monitor display, Thrustmaster HOTAS Cougar
joystick-and-throttle system and a CH Pro Pedals rudder
pedal system [32]. The scenarios were rendered on
Microsoft Flight Simulator X. FS Recorder, an add-on for
Flight Simulator X, was integrated to record behavioral data
during the simulated flights.
The experimental protocol involved a total of nine sessions
per subject, one session per day. The first session (day 1)
was introductory, allowing subjects to become acquainted
with the flight simulator; with the requirement that by the
end of this session, they needed to demonstrate a basic
understanding of flight simulator controls.
Study data were collected during the following eight
practice sessions. Practice sessions consisted of ten
repetitions of the approach to turn scenario (See Figure 3), a
total of ten flights per session, for a total of 80 trials per
participant over the 8 days.
Participants provided subjective mental effort and
performance evaluation using the NASA Task Load Index
(TLX) questionnaire [23]. Each session lasted 2 to 3 hours,
with no more than one session per day.
5
Figure 3 Pilot’s view of UAV flight simulator rendered
scene (left) and bird-s eye view of turn-to-approach task
waypoints marked on flight path map (right)
Data Analysis
For each participant, raw fNIR data (16 optodes×2
wavelengths) were low-pass filtered with a finite impulse
response, linear phase filter with order 20 and cut-off
frequency of 0.1 Hz to attenuate the high frequency noise,
respiration and cardiac cycle effects [19, 33]. Saturated
channels (if any), in which light intensity at the detector was
higher than the analog-to-digital converter limit were
excluded. fNIR data epochs for the rest and task periods
were extracted from the continuous data using time
synchronization markers. Blood oxygenation and volume
changes within each 16 optodes were calculated using the
modified Beer-Lambert Law for task periods with respect to
rest periods at beginning of each task with fnirSoft [34].
The main effect for practice level was tested using one-way
repeated measures analysis of variance (ANOVA), with
Subject and Practice Level designated as fixed effects.
Geisser–Greenhouse (G–G) correction was used when
violations of sphericity occurred in the omnibus tests.
Tukey's post hoc tests were used to determine the locus of
the main effects with a 0.05 significance criterion. Three
practice levels were defined for each participant: the
beginner phase included days 2 through 4, intermediate
phase included days 5 through 7 and the advanced phase
included days 8 and 9. Number Cruncher Statistical
Software (NCSS) 2007 (www.ncss.com) was used for the
statistical tests. Effect size indices of d* were calculated
[35] and were used to aid in the interpretation of the results.
4. RESULTS
Self-reported Ratings
The NASA TLX index results for the UAV tasks were
analyzed using one-way repeated measures ANOVA. The
results indicated a significant main effect of practice level
(beginner/intermediate/advanced conditions) for mental
demand (F2,24=12.37, p < 0.01), effort (F2,24=11.50, p <
0.01) and frustration (F2,24 = 6.40, p < 0.01). Both mental
demand and perceived effort followed a monotonic decrease
from beginner to advanced phase (see Figure 4). Tukey’s
post hoc tests revealed that mental demand (q0.05/2, 24 = 3.53,
p < 0.05) for the beginner phase was significantly higher
than the other conditions. The size of the effect was large (d
= -0.89 and -1.12 when the beginning phase is compared to
the intermediate and advanced phases, respectively. Results
for effort (q0.05/2, 24 = 3.53, p < 0.05) showed the beginner
phase required higher effort than the other conditions with
large effect sizes (d = -0.88, -1.07). Similar to mental
demand and effort, the frustration demand (q0.05/2, 24 = 3.53,
p < 0.05) was higher in the beginner phase than in the
intermediate or advanced phases and the effect sizes ranged
from d = -0.37 to -0.85).
Similarly, using each day instead of phases
(beginner/intermediate/advanced) indicated similar trends.
There was a significant main effect of days with mental
demand (F7,84=4.47, p < 0.01), effort (F7,84=5.88, p < 0.01)
and frustration (F7,84 = 5.69, p < 0.01). Tukey’s post hoc
tests confirmed that days 2&3 and were significantly
different than days 7,8 and 9 (q0.05/2, 84 = 4.39, p < 0.05) .
Figure 4 Self-reported ratings. Mental demand (top
row), effort (middle row) and frustration (bottom row).
Comparison of phases (left column), and days (right
column). Error bars indicate Standard Error of Mean
(SEM)
Behavioral Measures
Behavioral performance was calculated as root-mean-square
(RMS) deviation from the 4th
-order polynomial fits to the
path for latitude and longitude [20]. There was a significant
monotonic decrease in mean deviation from the optimal
path when comparing practice level (beginner through
advanced) for altitude (F2,24=2.32, p < 0.01) , longitude
6
(F2,24=3.14, p < 0.01), latitude (F2,24=3.36, p < 0.01) and
bank angle deviation (F2,24=4.03, p < 0.01), see Figure 5.
Post hoc analyses confirmed that an error at the beginner
phase was significantly higher than the other two phases
(q0.05/2, 24 = 3.53, p < 0.05) resulting in moderate to large
effect sizes (d = -0.62 to -1.03).
Figure 5 Behavioral measures. Altitude (top row),
longitude (2nd row), latitude (3rd row) and bank angle
(bottom row). Comparison of phases (left column), and
days (right column). Error bars indicate SEM.
Similarly, the deviation from optimal path gradually
lessened across day. The following parameters were
significant: altitude (F7,84=2.56, p < 0.01), longitude
(F7,84=2.90, p < 0.01), latitude (F7,84=2.50, p < 0.01) and
bank angle deviations (F7,84=4.40, p < 0.01), see Figure 5.
Post hoc analyses confirmed individual difference across
days. Most specifically, errors at days 2, 3 were
significantly greater than the error at the rest of the days
(q0.05/2, 84 = 4.39, p < 0.05) for all parameters.
fNIR Measures
The average total hemoglobin (HbT) concentration changes
throughout the practice levels (beginner /intermediate
/advanced) were analyzed using one-way repeated measures
ANOVA. Based on findings from our previous study, the
region of interest was identified as optode #2, which is close
to AF7 in the International 10-20 System and is located at
the left PFC (inferior frontal gyrus). The response was
significant (F2,24= 1.96, p < 0.01), and indicated a monotonic
decrease trend as expected from our previous results and as
seen in self-reported and behavioral measures (see Figure
6). The effect size d = -0.19 represents a small effect when
assessing the difference between advanced and beginner
stages of learning. When the results were analyzed using
days, the response was significant (F7, 84= 1.87, p < 0.01),
and yielded an interesting aspect of the learning process that
results in different trends at beginning, intermediate and
advanced phases. In the beginner phase, there was an
increase in brain activation until the intermediate phase was
reached. During the advanced phase, there is a decreasing
trend across days. The effect size across comparing days 2,
3, and 4 with day 9 are d= -0.13, -0.32, and -1.91,
respectively. The mean effect size comparing the initial
days of training to the last day of training is d -0.79.
Figure 6 fNIR measures. Comparison of phases (left)
and days (right). Error bars indicate SEM.
An efficiency graph can be a useful visual tool to assess the
impact of learning on performance [36]. The efficiency
graph in Figure 7 visualizes the overall results by plotting
normalized HbT (total hemoglobin concentration) changes
that represent mental effort against the inverse-normalized
bank angle values that model behavioral performance.
In this efficiency graph, the fourth quadrant represents low
efficiency, where minimum performance is achieved with
maximum effort. The second quadrant represents high
efficiency where maximum performance is achieved with
minimal effort. The diagonal y=x is the neutral axis, where
efficiency (E) is zero and effort and performance are equal.
Results indicate that subjects have achieved a transition
from a low efficiency mode to a high efficiency mode
throughout the 9-day long learning process.
7
Figure 7 Efficiency graph indicating transition from low
efficiency to high efficiency throughout the learning
process.
5. DISCUSSION
The purpose of this study was to examine the impact of
practice, or relative levels of expertise, on
neurophysiological measures of the hemodynamic response
in the prefrontal cortex during a complex cognitive task.
The complex, but interpretable, pattern of prefrontal cortical
response in the UAV tasks are consistent with a
‘scaffolding-storage’ framework of functional
reorganization during the development of expertise [21].
The results indicated an expected trend in self-reported
measures, behavioral performance and brain activation
when overall beginner, intermediate and advanced phases
were compared. All measures indicate an improvement in
performance and a decrease in brain activation across
sessions consistent with our previous practice and learning
related fNIR results [20, 37-39]. When the level of analysis
is shifted from phases to days, our results showed that
behavioral measures provided more detailed trend
information than subjective self-reported measures, which
suggests that subjective self-reports have a lower resolution
as compared to objective behavioral measures. Moreover,
brain activation changes as measured by fNIR provided
additional fine-grained information by revealing a different
trend in each learning phase. At the beginner’s level,
behavioral performance improved each day at the cost of
increased brain activation suggesting that increased effort
was required to learn the required skill set. In the
intermediate phase, a higher level of performance could be
maintained with less activation. Finally, in the advanced
phase, an even higher level of performance was achieved
with a decreasing trend in the activation.
It is possible that there may be additional factors that
contribute to an increase in workload and may recruit areas
of the brain that were not being monitored in the current
study. More information could be gained by monitoring
parietal areas in addition to the frontal cortex, or by
combining fNIR with EEG. Alternatively, signal processing
algorithms for fNIR may not be optimized at this time, or
the hemodynamic response itself may not be sufficiently
sensitive to pick up subtle changes in workload. This is an
area for future research.
6. SUMMARY
A field deployable optical brain imaging technology (fNIR)
holds potential for research studies and clinical applications
that require quantitative measurements of hemodynamic
changes during brain activation under ambulant conditions
in natural environments. As such, fNIR has been already
deployed in many field settings for objective measurements
of cognitive states and expertise development which will,
among other advantages, allow for dynamic interventions in
the training process, and help to assure robust performance
under adverse circumstances. fNIR application areas
include, but are not limited to, brain computer interface for
cognitive enhancement, neurological and gaming
applications, education, training and cognitive aging.
The study described here provides important albeit
preliminary information about fNIR measures of the PFC
hemodynamic response and its relationship to mental
workload, expertise, and performance, in a complex
multitasking environment. Level of expertise does appear to
influence the hemodynamic response in the
dorsolateral/ventrolateral prefrontal cortices, at least for
some complex tasks. Since fNIR technology allows the
development of mobile, non-intrusive and miniaturized
devices, it has the potential to be deployed in future learning
and training environments to personalize the training
regimen and/or to assess the effort of human operators in
critical multitasking environments.
Acknowledgements
Authors would like to thank Joshua Harrison and Justin
Menda for data acquisition and handling. This study was
funded in part under a U.S. Army Medical Research
Acquisition Activity; Cooperative Agreement W81XWH-
092-0104. The content of the information herein does not
necessarily reflect the position or the policy of the U.S.
Government or the U.S. Army and no official endorsement
should be inferred.
8
REFERENCES
[1] R. Parasuraman and G. Wilson, "Putting the brain
to work: Neuroergonomics past, present, and
future," Human factors, vol. 50, p. 468, 2008.
[2] R. Parasuraman and M. Rizzo, Neuroergonomics:
The brain at work: Oxford University Press, USA,
2007.
[3] A. Kelly and H. Garavan, "Human functional
neuroimaging of brain changes associated with
practice," Cerebral Cortex, vol. 15, p. 1089, 2005.
[4] J. G. Milton, S. S. Small, and A. Solodkin, "On the
road to automatic: dynamic aspects in the
development of expertise," Journal of clinical
neurophysiology, vol. 21, p. 134, 2004.
[5] A. M. Owen, K. M. McMillan, A. R. Laird, and E.
Bullmore, "N-back working memory paradigm: a
meta-analysis of normative functional
neuroimaging studies," Human Brain Mapping,
vol. 25, pp. 46-59, May 2005.
[6] J. D. Cohen, W. M. Perlstein, T. S. Braver, L. E.
Nystrom, D. C. Noll, J. Jonides, and E. E. Smith,
"Temporal dynamics of brain activation during a
working memory task," Nature, vol. 386, pp. 604-
608, 1997.
[7] B. Cain, "A review of the mental workload
literature," DTIC Document 2007.
[8] E. E. Smith and J. Jonides, "Working Memory: A
View from Neuroimaging," Cognitive Psychology,
vol. 33, pp. 5-42, 1997.
[9] S. H. Fairclough and L. Venables, "Prediction of
subjective states from psychophysiology: A
multivariate approach," Biological Psychology,
vol. 71, pp. 100-110, 2006.
[10] A. Gevins, M. Smith, L. McEvoy, and D. Yu,
"High-resolution EEG mapping of cortical
activation related to working memory: effects of
task difficulty, type of processing, and practice,"
Cerebral Cortex, vol. 7, p. 374, 1997.
[11] A. Gevins and M. Smith, "Neurophysiological
measures of cognitive workload during human-
computer interaction," Theoretical Issues in
Ergonomics Science, vol. 4, pp. 113-131, 2003.
[12] R. Stevens, T. Galloway, and C. Berka, "EEG-
related changes in cognitive workload, engagement
and distraction as students acquire problem solving
skills," User Modeling 2007, pp. 187-196, 2009.
[13] W. Klimesch, "EEG alpha and theta oscillations
reflect cognitive and memory performance: a
review and analysis," Brain Research Reviews, vol.
29, pp. 169-195, 1999.
[14] B. Chance, Z. Zhuang, C. UnAh, C. Alter, and L.
Lipton, "Cognition-activated low-frequency
modulation of light absorption in human brain,"
Proc Natl Acad Sci U S A, vol. 90, pp. 3770-4, Apr
15 1993.
[15] A. Villringer, J. Planck, C. Hock, L. Schleinkofer,
and U. Dirnagl, "Near infrared spectroscopy
(NIRS): a new tool to study hemodynamic changes
during activation of brain function in human
adults," Neurosci Lett, vol. 154, pp. 101-4, May 14
1993.
[16] A. Villringer and B. Chance, "Non-invasive optical
spectroscopy and imaging of human brain
function," Trends Neurosci, vol. 20, pp. 435-42,
Oct 1997.
[17] G. Strangman, D. A. Boas, and J. P. Sutton, "Non-
invasive neuroimaging using near-infrared light,"
Biol Psychiatry, vol. 52, pp. 679-93, Oct 1 2002.
[18] M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A.
Devaraj, B. Onaral, and K. Pourrezaei, "Functional
near-infrared neuroimaging," IEEE Trans Neural
Syst Rehabil Eng, vol. 13, pp. 153-9, Jun 2005.
[19] H. Ayaz, P. A. Shewokis, A. Curtin, M. Izzetoglu,
K. Izzetoglu, and B. Onaral, "Using MazeSuite and
Functional Near Infrared Spectroscopy to Study
Learning in Spatial Navigation," J Vis Exp, p.
e3443, 2011.
[20] H. Ayaz, P. A. Shewokis, S. Bunce, K. Izzetoglu,
B. Willems, and B. Onaral, "Optical brain
monitoring for operator training and mental
workload assessment," Neuroimage, vol. 59, pp.
36-47, 2012.
[21] S. E. Petersen, H. Van Mier, J. A. Fiez, and M. E.
Raichle, "The effects of practice on the functional
anatomy of task performance," Proceedings of the
National Academy of Sciences, vol. 95, p. 853,
1998.
[22] P. Hancock and M. H. Chignell, "Toward a Theory
of Mental Work Load: Stress and Adaptability in
Human-Machine Systems," Proc. IEEE SMC 1986,
pp. 378-383, 1986.
[23] S. Hart and L. Staveland, "Development of NASA-
TLX (Task Load Index): Results of empirical and
theoretical research," Human mental workload, vol.
1, pp. 139–183, 1988.
[24] N. Meshkati, P. A. Hancock, M. Rahimi, and S. M.
Dawes, "Techniques in mental workload
assessment," in Evaluation of human work: A
practical ergonomics methodology J. R. Wilson
and E. N Corlett, Eds., 2nd ed Philadelphia, PA,
US: Taylor & Francis,, 1995, pp. 749-782.
[25] H. Obrig, R. Wenzel, M. Kohl, S. Horst, P. Wobst,
J. Steinbrink, F. Thomas, and A. Villringer, "Near-
infrared spectroscopy: does it function in
functional activation studies of the adult brain?,"
International Journal of Psychophysiology, vol. 35,
pp. 125-142, 2000.
[26] M. Beauchamp, A. Dagher, J. Aston, and J. Doyon,
"Dynamic functional changes associated with
cognitive skill learning of an adapted version of the
Tower of London task," Neuroimage, vol. 20, pp.
1649-1660, 2003.
[27] R. A. Poldrack, A. D. Wagner, M. W. Prull, J. E.
Desmond, G. H. Glover, and J. D. E. Gabrieli,
"Functional Specialization for Semantic and
Phonological Processing in the Left Inferior
Prefrontal Cortex* 1," Neuroimage, vol. 10, pp. 15-
9
35, 1999.
[28] H. Garavan, D. Kelley, A. Rosen, S. M. Rao, and
E. A. Stein, "Practice-related functional activation
changes in a working memory task," Microscopy
Research and Technique, vol. 51, pp. 54-63, 2000.
[29] K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei,
and B. Chance, "Functional Optical Brain Imaging
Using Near-Infrared During Cognitive Tasks,"
International Journal of Human-Computer
Interaction, vol. 17, pp. 211-227, 2004.
[30] F. F. Jobsis, "Noninvasive, infrared monitoring of
cerebral and myocardial oxygen sufficiency and
circulatory parameters," Science, vol. 198, pp.
1264-7, Dec 23 1977.
[31] H. Ayaz, M. Izzetoglu, S. M. Platek, S. Bunce, K.
Izzetoglu, K. Pourrezaei, and B. Onaral,
"Registering fNIR data to brain surface image
using MRI templates," Conf Proc IEEE Eng Med
Biol Soc, pp. 2671-4, 2006.
[32] J. Menda, J. T. Hing, H. Ayaz, P. A. Shewokis, K.
Izzetoglu, B. Onaral, and P. Oh, "Optical Brain
Imaging to Enhance UAV Operator Training,
Evaluation, and Interface Development," Journal
of Intelligent & Robotic Systems, vol. 61, pp. 423-
443, 2010.
[33] H. Ayaz, M. Izzetoglu, P. A. Shewokis, and B.
Onaral, "Sliding-window Motion Artifact
Rejection for Functional Near-Infrared
Spectroscopy," Conf Proc IEEE Eng Med Biol Soc,
pp. 6567-70, 2010.
[34] H. Ayaz, "Functional Near Infrared Spectroscopy
based Brain Computer Interface," PhD Thesis,
School of Biomedical Engineering Science &
Health Systems, Drexel University, Philadelphia,
2010.
[35] P. J. Potvin and R. W. Schutz, "Statistical power
for the two-factor repeated measures ANOVA,"
Behavior Research Methods, vol. 32, pp. 347-356,
2000.
[36] R. Clark, F. Nguyen, and J. Sweller, Efficiency in
learning: Evidence-based guidelines to manage
cognitive load. San Fransico, CA: Pfeiffer, An
Imprint of Wiley, 2006.
[37] P. Shewokis, H. Ayaz, M. Izzetoglu, S. Bunce, R.
Gentili, I. Sela, K. Izzetoglu, and B. Onaral, "Brain
in the Loop: Assessing Learning Using fNIR in
Cognitive and Motor Tasks," in Foundations of
Augmented Cognition. Directing the Future of
Adaptive Systems. vol. 6780, D. Schmorrow and C.
Fidopiastis, Eds., ed: Springer Berlin / Heidelberg,
2011, pp. 240-249.
[38] S. Bunce, K. Izzetoglu, H. Ayaz, P. Shewokis, M.
Izzetoglu, K. Pourrezaei, and B. Onaral,
"Implementation of fNIRS for Monitoring Levels
of Expertise and Mental Workload," in
Foundations of Augmented Cognition. Directing
the Future of Adaptive Systems. vol. 6780, D.
Schmorrow and C. Fidopiastis, Eds., ed: Springer
Berlin / Heidelberg, 2011, pp. 13-22.
[39] H. Ayaz, P. Shewokis, S. Bunce, M. Schultheis,
and B. Onaral, "Assessment of Cognitive Neural
Correlates for a Functional Near Infrared-Based
Brain Computer Interface System," in Foundations
of Augmented Cognition. Neuroergonomics and
Operational Neuroscience, D. Schmorrow, Ed., ed,
2009, pp. 699-708.
10
BIOGRAPHIES
Hasan Ayaz is an Assistant Research
Professor at Drexel University, School
of Biomedical Engineering, Science and
Health Systems, Philadelphia, PA. He
received his BSc. in Electrical and
Electronics Engineering at Bogazici
University, Istanbul, Turkey, with high
honors and MSc. and PhD degrees from
Drexel University where he developed enabling software for
functional Near Infrared Spectroscopy (fNIR) based brain
monitoring instruments. This technology was licensed by
fNIR Devices LLC and has been distributed to over 50
research labs by Biopac Systems, Inc. As an extension, he
worked on a portable-handheld medical device
(InfraScanner) that utilizes fNIR to detect hematoma in head
trauma patients. InfraScanner is currently pending FDA
approval but has been deployed overseas and is already
saving lives in Europe, Africa and Asia. InfraScanner
related awards include DMD (Design of Medical Devices)
Conference 2011 top presenter and EID (Excellence in
Design) Gold Award. Dr. Ayaz’s research interests include
Neuroengineering applications of human computer
interaction and Neuroergonomics, specifically, i)
development of noninvasive brain computer interfaces (BCI)
for communication and augmented interactivity in
simulation and gaming settings, ii) optical brain imaging
for assessment of cognitive workload and expertise
development of operators such as such as air traffic
controllers (ATC) and unmanned aerial vehicle (UAV)
ground operators.
Murat Perit Çakır is an Assistant
Professor in Middle East Technical
University (METU), Ankara, Turkey. He
received BSc from METU in
Mathematics and MSc degree from
University of Pennsylvania in Computer
Science and PhD degree from Drexel University in
Information Science & Technology. Dr. Cakir’s research
interests include Computer-Supported Collaborative
Learning, Human-Computer Interaction, Interaction
Analysis, Groupware Design, Immersive Learning
Environments, and Cognitive Neuroscience of Learning.
Kurtuluş İzzetoğlu received his Ph.D.
degree from Drexel University, School
of Biomedical Engineering, Science and
Health Systems, Philadelphia, PA.
During his Ph.D. studies, he worked on
the functional near-infrared (fNIR)
spectroscopy system to identify
neuromarkers during change in the cognitive state of mental
engagement at both high and low levels of neural activation.
He is currently a Research Assistant Professor in the School
of Biomedical Engineering at Drexel University and has
also been working as a project engineer in various research
projects for several years. His research interests include
functional brain imaging, medical sensor development and
biomedical signal processing. His current research projects
mainly focus on novel algorithms and techniques to deploy
the fNIR system for various application areas, such as depth
of anesthesia monitor and cognitive workload assessment.
He is currently working on various funded projects that are
involved in 1) cognitive workload assessment of unmanned
air vehicle (UAV) operators, 2) sensor and algorithm
development for depth of anesthesia monitoring, 3)
cognitive baselining and index developments, 4) cognitive
workload assessment of air traffic controllers.
Adrian Curtin is a graduate research
assistant at Drexel University. Mr.
Curtin is currently pursuing his BS-MS
degree in Biomedical Engineering.
Patricia A. Shewokis received a Ph.D.
in the Psychology of Motor Behavior
from the University of Georgia in
1993. She is a Professor with joint
appointments in the College of Nursing
and Health Professions and School of
Biomedical Engineering, Science and
Health Systems at Drexel University, Philadelphia, PA. Her
active research program focuses on: (a) the processes and
mechanisms involved in the acquisition, retention and
transfer of cognitive and motor skills; b) brain-computer
interface research employing biofeedback in learning
paradigms; c) brain in the loop assessments of learning;
and d) statistical and methodological assessments of
learning. Currently, Dr Shewokis is the Research Director
of the Cognitive Neuroengineering and Quantitative
Research (CONQUER) Collab0rative at Drexel.
Scott C. Bunce received a B.A. in
Philosophy and Biology from Wheaton
College in 1984, an M.A. in Personality
Psychology from the University of
Michigan in 1990, and Ph.D.’s in
Clinical and Personality Psychology
from the University of Michigan in
1993. Dr. Bunce is currently Associate Professor and
Interim Director of Behavioral Neuroimaging in Psychiatry
at Penn State Milton S. Hershey Medical Center and the
Penn State College of Medicine. He also holds an adjunct
appointment in the School of Biomedical Engineering,
Science & Health Systems at Drexel University. In his
previous appointment as Assistant Professor of Psychiatry
at MCP Hahnemann/Drexel University College of Medicine,
Dr. Bunce served as Director of the Clinical Neuroscience
Research Unit, a laboratory that investigated a range of
topics related to affective and cognitive neuroscience using
noninvasive measures of brain function. While at Drexel,
Dr. Bunce helped develop functional near-infrared optical
technology as a member of the School of Biomedical
Engineering’s Optical Imaging Team.
11
Banu Onaral is H. H. Sun Professor
of Biomedical Engineering and
Electrical Engineering at Drexel
University, Philadelphia, PA. She
holds a Ph.D. [1978] in Biomedical
Engineering from the University of
Pennsylvania and BSEE [1973] and
MSEE [1974] in Electrical Engineering from Bogazici
University, Istanbul, Turkey. Dr. Onaral joined the faculty
of the Department of Electrical and Computer Engineering
and the Biomedical Engineering and Science Institute in
1981. Since 1997, she has served as the founding Director
of the School of Biomedical Engineering Science and Health
Systems. Her academic focus, both in research and
teaching, is centered on information engineering with
special emphasis on complex systems and biomedical signal
processing in ultrasound and optics. She has led major
research and development projects sponsored by the
National Science Foundation (NSF), National Institutes of
Health (NIH), Office of Naval Research (ONR), DARPA and
Department of Homeland Security (DHS). She has
supervised a large number of graduate students to degree
completion and has an extensive publication record in
biomedical signals and systems. She is the recipient of a
number of faculty excellence awards, including the 1990
Lindback Distinguished Teaching Award of Drexel
University, the EDUCOM Best educational Software award,
and the NSF Faculty Achievement Award.
Top Related