A Device for the Detection of Cognitive Brain Functions in Completely Paralyzed or Unresponsive...

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 52, NO. 2, FEBRUARY 2005 211

A Device for the Detection of Cognitive BrainFunctions in Completely Paralyzed or

Unresponsive PatientsThilo Hinterberger*, Barbara Wilhelm, Jürgen Mellinger, Boris Kotchoubey, and Niels Birbaumer

Abstract—Unresponsive patients with remaining cognitive abil-ities may be able to communicate with a brain-computer interface(BCI) such as the Thought Translation Device (TTD). Before ini-tiating TTD learning, which may imply considerable effort, it isimportant to classify the patients’ state of awareness and their re-maining cognitive abilities. A tool for detection of cognitive activity(DCA) in the completely paralyzed was developed and integratedinto the TTD which is a psychophysiological system for direct braincommunication. In the present version, DCA entails five event-re-lated brain-potential (ERP) experiments and investigates the ca-pability of a patient to discriminate, e.g., between semantically re-lated and unrelated concepts and categories. ERPs serve as an in-dicator of the patients’ cortical information processing. Data fromfive severely brain-injured patients in persistent vegetative state di-agnosed as unresponsive and five healthy controls are presented toillustrate the methodology. Two patients showing the highest re-sponsiveness were selected for TTD training. The DCA integratedin the TTD allows screening of cognitive abilities and direct braincommunication in the patients’ home.

Index Terms—Brain-computer-interface (BCI), event-relatedbrain potentials (ERP), cognition, locked-in syndrome.

I. INTRODUCTION

THE Thought Translation Device (TTD) is a brain-com-puter interface (BCI) developed to assist completely para-

lyzed patients with communication using their electric brain po-tentials only [1], [2]. It was described in patients suffering fromend-stage Amyotrophic Lateral Sclerosis (ALS) for communi-cation and control of the environment [1]–[4] and has alreadyproven its usefulness for improvement of the quality of life inthe completely paralyzed. Most of the patients who communi-cate with the TTD use their slow cortical potentials (SCPs) (i.e.,slow event-related electroencephalogram (EEG) changes below1 Hz) to indicate binary decisions by producing positive or neg-

Manuscript received November 24, 2003; revised May 31, 2004. This workwas supported in part by the Deutsche Forschungsgesellschaft (DFG) underGrant SFB 550 (B5) and in part by the National Institute of Health (NIH). As-terisk indicates corresponding author.

*T. Hinterberger is with the Institute of Medical Psychology and BehavioralNeurobiology, University of Tübingen, Tübingen 72074, Germany (e-mail:[email protected]).

B. Wilhelm J. Mellinger, and B. Kotchoubey are with the Instituteof Medical Psychology and Behavioral Neurobiology, University ofTübingen, Tübingen 72074, Germany (e-mail: [email protected];[email protected]; [email protected]).

N. Birbaumer is with the Institute of Medical Psychology and BehavioralNeurobiology, University of Tübingen, Tübingen 72074, Germany and with theCenter of Cognitive Neuroscience, University of Trento, I-38100 Trento, Italy(e-mail: [email protected]).

Digital Object Identifier 10.1109/TBME.2004.840190

ative potential shifts. These binary decisions are used to selector reject a set of letters or words presented on a screen. Beforeinitiating verbal communication with the TTD patients requiretraining for generation of voluntary SCP shifts with more than75% correct responses. The standard training and spelling para-digm consists of trials with a fixed duration of 4 to 8 s. Each trialis subdivided into a 2-s preparation phase followed by a feed-back phase lasting until 0.5 s before the end of the trial. Thefeedback of the brain response (SCP) is provided by the move-ment of a yellow circle whose vertical position reflects the SCPamplitude or by a tone increasing or decreasing in pitch. An up-ward movement indicates negative SCP changes; a downwardmovement indicates positive ones. During the final 0.5 s the de-cision or result of the current trial is calculated on the basis ofthe preceding feedback signal, and presented to the user, accom-panied by a reinforcing smiley face in case of a correct response.During training the computer indicates the required SCP po-larity by lighting one of two goals on the screen and/or a voiceat the beginning of a trial.

Most of the ALS patients initiate training in a state in whichthey are still able to control some muscles that can be usedfor communication. After a successful BCI training, they canswitch to SCP-based communication in case of a complete lossof muscle control. However, ALS patients without any volun-tary muscle activity nor communication abilities present addi-tional problems, as well as other neurological patients with se-vere brain injury whose behavior does not express signs of cog-nitive abilities. During training we do not know if the patientscontinue to keep their vigilance and understand the task. Lack oftraining success in a completely locked-in person may be due tothese factors. Assessing a patient’s cognitive abilities is, thus,crucial in order to judge the capacity to learn self control andregulation of brain responses. Therefore, a tool for detection ofcognitive brain activity (DCA) was developed that can be ap-plied to nonresponsive patients before starting a laborious feed-back communication training. The present paper describes theprinciples of how the DCA tool embedded in the TTD softwareworks.

The DCA system consists of a set of auditory event-relatedpotential (ERP) experiments. The ERPs should reflect differentlevels of complexity of cognitive processing. There is consider-able controversy as to what degree these processing operationsmanifested in ERP components are impaired in ALS patients.Compared to healthy controls, ALS patients as a group oftenshow cognitive impairment in working memory, attention, re-sponse inhibition, naming, and other functions [5], [6]. A more

0018-9294/$20.00 © 2005 IEEE

212 IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 52, NO. 2, FEBRUARY 2005

subtle individual analysis reveals, however, largely intact cog-nitive functions in some ALS patients even in the final stage ofthe disease [7]. The use of ERP-based test paradigms for evalua-tion of awareness and cognitive abilities in comatose, vegetativestate, and locked-in patients has been demonstrated elsewhere[8]. For the present study, these paradigms were implementedin a small and mobile EEG system which allows to easily runthe experiments, and analyze the data immediately at a patient’sbedside. Therefore, an additional tool for DCA was integratedinto the TTD software constituting a single-PC system runningon all Windows platforms [9].

As shown in [8], there are very many different possibilitiesto use various ERP experiments for cognitive assessment inseverely brain damaged patients. The present, first version ofDCA employs only a small subsample of potentially informa-tive testing procedures. Accordingly, only preliminary data ob-tained in such patients are reported to show how this tool can beapplied and implemented. However, as soon as the DCA moduleis installed, its further modification is extremely easy. Hence,the power of the DCA will increase by adding other promisingERP experiments described extensively in the literature [5], [7],[8]. An experienced TTD assistant can add any new ERP exper-iment within hours.

Here, we report the methodological and technical realizationof the DCA. After an introduction to the TTD and its moduleswe report how the DCA was implemented into this device. Sec-tion III gives a methodological description of five typical exper-iments.

II. THE THOUGHT TRANSLATION DEVICE (TTD)

The current version of the TTD constitutes a BCI for verbalbrain-computer communication where the entire functionalityis maintained in one executable program. It performs data ac-quisition, storage, on-line filtering and classification of the on-going EEG, and provides a communication interface which al-lows for letter selection or navigating through the world-wideweb with the classified brain responses [9], [10]. A modularobject-oriented structure was chosen to add and change filtercomponents and applications flexibly. The class structure, pa-rameter handling and the EEG-data format were implementedaccording to the so called “BCI2000 standard” (G. Schalk, J.Mellinger, D. McFarland, and T. Hinterberger) [11], [12]. In theTTD, all modules can be arbitrarily connected to each other aseach output signal of a filter may serve as an input signal foranother filter which can be configured by the user [13]. Fig. 1shows a schematic of the signal flow between the modules asused for training self-regulation of SCP and communication, forthe experiments of the DCA, and finally to perform the statis-tical analysis.

A. Basic Modules

1) Data Acquisition: The EEG signal is recorded from an8 channel EEG amplifier (EEG8, Contact Precision Instru-ments, Inc.) with a bandwidth from 0.01 Hz to 40 Hz. Forthe experiments reported here we were using a PC (Athlon,700 MHz) equipped with a 16-bit analog-to-digital converter(DAS-1602/16, Measurement Computing). This converter is

Fig. 1. Shows the schematic flow diagram for three operation modes ofthe TTD: The TTD-mode (b) for training patients in self-regulation of brainpotentials and verbal communication, the mode for assessment of cognitiveactivity (DCA) (c), and the statistical analysis modules (d). The top rows(a) represent the basic modules which are used in all operational modes.The bottom boxes show the interfaces displaying all relevant information foroperating the TTD (presentation of stimuli and feedback for communicationwith brain signals, left) and for executing the DCA. A statistical analysison Fourier transformed data as depicted on the right is implemented but notreported here in detail.

also available for the PCMCIA slot supporting a portablesystem on a single notebook-PC. For off-line analysis, thesignal is read from the file and treated as on-line data.

Data are sampled at a rate of 256/s and stored in the BCI2000file format. Each sample of data is marked with a number ofstatus bits (“states”) containing continuously changing informa-tion about the systems’ states. States indicate, e.g., the currentstimulus type, or whether feedback is provided, or whether anartifact is detected. Each module can support and change states(see [12] for details); 16 times/s a block of 16 samples is passedto the processing modules resulting in one feedback value. Thus,feedback is updated 16 times/s. This default parameter can bechanged and does not influence the real-time behavior of thesystem when choosing the timing structure of an experiment inmultiples of the duration of a block.

2) Spatial Filtering: The spatial filter performs a multipli-cation of the input channels with a weight matrix. Each outputchannel is computed as a linear combination of the input chan-nels, bipolar derivations are possible even with a unipolar am-plifier setup. This filter allows off-line analysis with arbitraryreference settings, and provides the possibility of artifact cor-rection (see below) by subtracting a proportion of an indepen-dently measured non-EEG signal (e.g., eye movements). Thespatial filter can also be used to reduce the number of signalchannels [Fig. 1(a)].

B. Modules for Communication Training

Fig. 1(b) illustrates the functional modules of the TTD for thetraining of brain-computer communication.

1) SCP Feedback: The SCPs are extracted with a movingaverage of a 500-ms window implemented as a finite impulseresponce (FIR) filter with equal coefficients. Afterwards, a com-pensation of the time constant of the amplifier is applied to theSCP signal calculating a quasi-dc recording of a single trial[9]. From the slow-wave filtered signal, a baseline is subtracted

HINTERBERGER et al.: DETECTION OF COGNITIVE BRAIN FUNCTIONS IN COMPLETELY PARALYZED OR UNRESPONSIVE PATIENTS 213

which is defined as the mean SCP amplitude before the feed-back starts. The feedback cursor will, therefore, always start itsmovement from a center position on the screen.

During on-line feedback, the SCPs are corrected for verticaleye-movement artifacts. Instead of continually subtracting afixed fraction of the electroocculography (EOG) signal fromthe SCP amplitude, correction will take place if EOG andSCP interfere constructively, and feedback is suppressed if theweighted EOG exceeds the SCP amplitude [13].

2) Response Generation: The standard response criterion isdefined by the sign of the average SCP amplitude during thefeedback interval of a trial. On the screen of the patient, thecriterion is defined by two target areas displayed on the screeninto which the feedback cursor has to be moved by the brain re-sponse. A bottom target defines the cortical positivity response,the top target the cortical negativity response. This simple re-sponse classification can be adapted to a well trained patient’sindividual brain response by applying a running classifier, whichperforms, e.g., a linear discriminant analysis to the feedbackdata [14].

3) The Spelling Device: For communication, each target isassociated with a function such as the selection or rejection of aset of letters presented inside the target, or spelled out by a voice.Single letters can be selected with binary decisions (positive ornegative SCP responses) by navigating through a dichotomic de-cision tree: The alphabet is split in half, the selected part is againsplit until a single letter remains and is selected in consecutivetrials [3], [4].

C. Modules for Acquisition of Cortical Activity

The experiments for the DCA are ERP paradigms with audi-tory stimuli. The stimuli are stored as MIDI-sequences or wav-files and played by the TTD with a timing precision of about 10ms under Windows 98 and about 5 ms under Windows 2000 ona 700-MHz machine. To ensure a clear evoked response whenaveraging ERPs of different spoken words, the words’ onsetsneed to be synchronized in the wav-files. Further, words withone syllable are preferred. For this operation mode no on-linesignal processing is necessary. Only the EEG data acquisitionmodule, the storage routine, and the modules for sequencingcontrol and stimulus presentation are used. However, the TTDallows for on-line statistical analysis during the experiment, too[Fig. 1(c)].

D. Analysis Modules for DCA

All analysis is performed with the TTD software. A signalflow chart is depicted in Fig. 1(d). The TTD offers three methodsfor analysis of the ERPs: 1) A qualitative visual inspection of theaveraged waveforms; 2) a statistical test that proves the signifi-cance of a component; and 3) a quantification of the ERP givenby the differences of the standardized means. This is also calledthe effect size of an ERP differentiation or amplitude. Initially,two pre-processing steps have to be carried out as follows.

1) Artifact Correction: The first step consists of the correc-tion of the EEG for eye-blink and eye-movement artifacts. The

TABLE IEOG CORRECTION FACTORS FOR DIFFERENT ELECTRODE LOCATIONS OF THE

SUBJECTS OF THE PILOT STUDY AND THE RESIDUAL EYE-BLINK AND

EYE-MEVEMENT ARTIFACTS IN THE EEG-SIGNAL (0–40 Hz) AND THE

SCP SIGNAL (0–1 Hz)

TTD estimates the correction factors for each EEG-channel sep-arately based on a correlation analysis between large EOG am-plitude changes and the EEG-signal. For an estimation of cor-rection factors for eye-blinks, the EEG and vertical EOG areband-pass filtered within the range from 1 to 10 Hz (i.e., thefrequency range of eye-blinks) using an FIR filter with 127 co-efficients. The subsequent detection algorithm reacts to EOGchanges with a slope steeper than 400 V/s. The slope of theEEG-values for each channel is correlated with the slope of theEOG signal. All these correction factors are averaged separatelyfor each channel, resulting in the average correction factors foreye-blinks. After the correction factors are determined for eachsubject individually, each sample of the unfiltered EEG signalis corrected by subtraction of the EOG multiplied by the cor-rection factors. Table I gives a list of average correction factorsobtained in the pilot study (see below) which are in line with theliterature [15]. This correction method requires that the artifactsoccur simultaneously to the eye-blinks and show equal shapes.The simultaneous occurrence was verified with the data of oneexperiment. The efficiency of this method was tested with dataof the pilot study described below. An analysis of the residualeye-blink artifacts in the EEG gave a reduction of 60–90% ofthe peak depending on the electrode position. Table I shows theresidual artifacts in % of the uncorrected artifact. The methodreveals an efficiency of more than 90% for SCPs. However, fordistant electrodes the waveform (not its peak time) of the ar-tifact signal in the EEG channels is slightly different from theEOG. An additional delayed component of the artifact at thePz electrode could only be corrected with 40%. Since the in-fluence of the EOG signal on distant electrodes is only about5% this correction algorithm was considered to be sufficient fordata analysis. Additionally, the TTD displays the trial averagesof the EOG which helps the user to judge the results individ-ually. The inclusion of advanced correction methods utilizingpattern recognition methods might be an issue for further im-provement of the system.

2) Averaging: Let us consider an ERP experiment con-sisting of trials with the stimulus condition . The EEGtrace of each trial n comprises the EEG amplitudes , wheret is the sample number at a time in the trial. As the analysiscan be applied to each channel independently no channel indexis used. Let be the time of the stimulus onset. To analyzethe ERP signal, a baseline value considering the final 200 ms(containing samples) before stimulus onset (depending


on the experimental design) is subtracted from the EEG traceof a trial. Let be the baseline corrected signal

(1)

All baseline corrected trials are then averaged separately foreach condition

(2)

The average curves of a trial condition are displayed bythe TTD. Averaged waveforms and the corresponding standarddeviation serve as data pool for further analysis specific foreach experiment.

3) Significance Test: To proof the existence of an ERP com-ponent or the statistical significance between the average curvesof two conditions and a statistical test is necessary. To testtwo means of different conditions for being significantlydifferent a t-test can be calculated for a certain time assuminga normal distribution of the values between trials

(3)

As the EEG is a continuously changing curve and the neigh-boring and show high correlations to asimple correction with multiple t-tests over time cannot becalculated. One solution is the consideration of the covariancesover time leading to the Hotelling t-test. However, for a hugenumber of time coefficients this test requires a reduction of thenumber of coefficients which can be realized, e.g., by wavelettransform [7], [8]. Alternatively, a different method common inthe statistical analysis of high-dimensional physiological datanot only for EEG, but also for the analysis of ERPs in the MEGor activation areas in functional magnetic resonance imagingwas used. It applies a t-test according to (3) for each indi-vidual time . When searching for ERPs of a defined polaritya one-tailed test should be chosen ( for ).A time region of interest has to be definedin which we expect the ERP component. For the existence ofa component in a time range with samplesa certain minimum number (cluster-size) of consecutivesignificant values have to be found. This number depends onthe correlation between the neighboring EEG values (autocor-relation), the size of the time window, such as minimally onthe number of trials . The application of this method to ERPs,the calculation of the correlation, and a table with the resulting

for a certain significance level is given by Guthrie andBuchwald [16]. They give an autocorrelation of EEG curves of0.9 which we have verified successfully in our own data. Theresult of this test leads to a binary decision for the existence of acomponent. The TTD applies this test fully automatized to theEEG data sets and gives a list of significant clusters to the user.

4) ERP Quantification: A significance test as a “digital” cri-terion is useful for scientific purposes and may help to arrive

rapidly at a decision whether a patient possesses remaining cog-nitive capacity. However, for diagnostic purposes an “analog”quantification of the ERP peaks may be useful, too.

For the quantification of ERP peaks or peak differences a timerange has to be defined in which the ERP is expected to occuras described above. In this range, the most significant peak dif-ference with the polarity of the expected component has to beidentified. The amplitude difference is then normalized to thestandard deviation and averaged in a defined range around thispeak (see Section III-B) according to (4).depends on the average duration of a component

(4)

The standardized mean which may also be called aneffect size gives a quantitative measure of a component’s size.

III. THE DETECTION OF COGNITIVE BRAIN ACTIVITY (DCA)

This section describes five ERP experiments included in thefirst version of the device for DCA. An overview of the exper-iments and the expected ERP components is given on the leftof Fig. 2(a). Stimulation paradigms used in these experimentsare depicted in the Subsection A. Methods of analysis of theobtained ERP waveforms are presented in the Section III-B. Fi-nally, Section III-C describes participants of the pilot experi-ments.

A. Assessment of Cognitive Evoked Brain Potentials

The TTD allows for a broad variety of diagnostic experi-mental procedures using cognitive auditory ERP paradigms.Five such procedures are implemented in the present versionwith the aim to obtain information about attentional, semanticand intentional processing capacities. Patients who are ableto discriminate between the different conditions and withsignificant differences in the corresponding ERPs should beable in achieving communication performance with a BCI.They also may prove to have a better prognosis for improve-ment. Kotchoubey et al. [17], [18] have shown that no stricthierarchical order in cognitive complexity exists in severelybrain-damaged patients, e.g., a patient without a differentiatedERP response in a simple auditory oddball paradigm can never-theless exhibit consistent responses to more complex harmonicstimuli. Therefore, a set of experimental situations was selectedwith an increasing level of cognitive complexity.

1) Oddball Paradigm: The oddball task evaluates simpleauditory discrimination and working memory. Three hundredstimuli were presented with a frequency of 1/s. Here, 85% ofthe stimuli consisted of a high-pitched piano tone, and in 15%of the stimuli a low-pitched piano tone of a different timbre waspresented as an odd stimulus. The subjects were instructed topay attention to the stimuli. The usual response in this paradigmis a larger positive ERP wave to rare stimuli with a latencyof 300–400 ms referred to as P300. Neurological patientsoften exhibit prolonged latencies up to 500 ms. The P300 iswell established in the literature and supposedly represents a


Fig. 2. Five event ERP-experiments are displayed. (a) The left column shows the average response curves for each experiment from one healthy subject as anexample. The vertical arrows indicate which part of the curves was chosen for the analysis of the ERP components. (b) On the right the results of the quantitativeERP analysis are displayed in units of standardized amplitudes for the pilot study including five healthy and five patients. (c) The cumulative results are shown forthe statistical test in numbers of significantly detected components and for the quantitative analysis by the summed effect sizes over all components. At least 6 ofthe 8 tests revealed significance for the healthy persons whereas the patients showed significant results in only four or less experiments. The quantitative resultsare qualitatively comparable to the significance tests.


TABLE IISHOWS THE CRITERIA FOR THE ANALYSIS OF THE ERP COMPONENTS

memory updating process [19]–[21]. An early negative peak isexpected with a latency of about 100 ms (N100).

2) Semantic Oddball: A verbal oddball experiment wascarried out to assess semantic categorization [22]. A total of255 trials was presented, 180 of which were pseudowords(from a set of 60 different pseudowords), 45 were high-frequentcommon nouns (from a set of 15 words) without emotional con-tent, and the remaining 30 were personally relevant words (firstand second name of the subject or a near relative). The trialswere presented with an interval of 4 s. We used the patient’sname as a salient stimulus [23] in an attempt to improve thelikelihood of an ERP in severely disabled patients. Similar tothe standard oddball, a parietal positive deflection is expectedin response to the rare, meaningful words with a longer latency,such as 500 ms or more (referred to as P500).

3) Word Matching: 120 pairs of one-syllable, high-frequentwords were presented. 60 pairs had obvious semantic relation-ship (e.g., plum-pear) and 60 pairs permitted no association witheach other (e.g., spoon-wall). All words began with a conso-nant. Word pairs were presented every 4 s, and the onset-to-onset interval between the words in a pair was 1.7 s. The firstword serves as a “prime” building a context for the second word(“target”). A contingent negative variation (CNV) [24] is ex-pected between the two words after the N100/P300-complex tothe first word. The second word elicits an N100 followed by anegative N400 only to nonmatching words [25], [26] whereason the semantically congruent words a P600 can be expected[27]. Semantic associative cognitive abilities can be illustratedwith this test.

4) Semantic Congruence: The detection of semantic con-gruence can be extended to the assessment of the meaning of en-tire sentences. This task is even more complex than the previousone. Here, the meaning of an entire sentence must be under-stood and kept in memory. 100 sentences were presented, halfof which terminated with a semantically congruent one-syllableword and the other half with a semantically incongruent one-syl-lable word. The onset of the final word served as the trigger. TheN400 ERP component to the incongruent end word is expectedwith a longer latency than in the word matching experiment.This situation indicates semantic context comprehension.

5) Movement Intention: Subjects were presented with 150trials of 5-s duration. Each trial started with the word “left” or“right” which instructed the subjects to imagine a left or righthand movement according to the verbal requirement. Left andright trials were presented in a randomized order and with equalprobabilities. A lateralized negative SCP (lateralized readiness

potential) is expected over the hand motor areas, reflecting theactivation of the left or right motor cortex during preparation ofright-hand and left-hand movements, respectively [28].

B. Analysis of Specific ERP Components

Three types of ERP can be distinguished which have to betreated differently in the analysis. 1) Early components with alatency below 200 ms, 2) Mid-late components showing a peakbetween 200 and 700 ms, and the late components from 700 mson.

N100: The most predominant early endogenous componentin cortical EEG is the N100, a negative ERP with a la-tency of 75–150 ms. This negativity is expected for alltypes of stimuli and might show a higher amplitude torare and meaningful stimuli but does not necessarilyhave to [8]. As the ERPs for impaired patients some-times show longer latencies, we defined our range fordetection of the N100 from 75 ms to 200 ms. To obtaina quantitative measure for this preconscious processingactivity of auditory stimuli the mean t-values of allconsecutive significant ( , one-tailed) sampleswith a cluster size of more than (see [16])served as measure for significance. Because this com-ponent does not necessarily distinguish between con-ditions, its amplitude was statistically tested for beingdifferent from the baseline potential. This componentwas determined in the standard oddball experiment andthe word matching paradigm.

P300: The P300 in the standard oddball paradigm is a com-ponent discriminative between two stimulus types.Therefore, a t-test of two independent variables wasapplied. The result of the Cz-Mastoid electrodes is pre-sented. Consecutive significant samples are searchedin a range between 200 and 500 ms after stimulusonset according to Table II. As ERPs can show longerlatencies in patients the time range was extended to500 ms. The first level significance test should exhibitat least consecutive samples. In thesecond level analysis an average amplitude within

ms around the peak of the longest cluster or ifnot available the most positive peak between 200 and500 ms was identified. The standardized average con-stitutes the quantitative measure of this component.

N400: The N400 in the semantic word matching paradigmand in the sentence paradigm is identified between 400


and 800 ms after the onset of the second word (se-mantic matching) or the onset of the final word in a sen-tence (semantic congruence). The analysis was donesimilarly to the P300 but with the values according toTable II.

P500: In the semantic oddball paradigm a late positive com-ponent is observed predominantly in parietal regionsfor the meaningful words such as the name comparedto pseudowords. Therefore, the analysis is shownfrom the Pz electrode position with the parameters ofTable II.

SCPshift:

An SCP is a long-term ( s) potential shift withoutshowing a short peak as the other components do.Therefore, the mean amplitude difference betweentwo conditions in a defined range served as one vari-able to which a t-test can be applied (see Table II).

CNV: The CNV is an SCP shift between a warning stimulusand an executive stimulus lasting for more than 700ms [15]. In the Word Matching experiment, the firstword elicits the CNV independently from the task. Itis measured by the mean amplitude 600–1000 ms afteronset of the first word referenced to the baseline whichwas taken from 200 to 0 ms before presentation of thefirst word.

The analyzed ERPs are illustrated in the left column of Fig. 2.For each experiment, an example of one healthy person is pre-sented. Altogether, nine types of ERPs were analyzed in fiveexperiments.

C. Pilot Study for the DCA

The DCA method was applied in five patients with suspectedpersistent vegetative state, aged from 19 to 70, and five healthypersons. The patients’ mean age was 47 [standard deviation(SD) 14] whereas the healthy subjects’ mean age was 39 (SD17). All subjects were native French speakers. Therefore, verbalstimuli information was recorded in French by a native Frenchspeaker. Meanwhile, analogous stimulation is also available inGerman and Hebrew. Each experiment was carried out onceusing the same equipment with each participant.

IV. RESULTS

A. Visual Inspection of ERPs

The TTD realizes on-line averaging and display of the ERPcomponents during the experiment. The exported data of theOddball test are displayed in Fig. 3 for all ten subjects. TheN100 and P300 components in the curves of the healthy subjectsexhibit high amplitudes, especially to the rare stimuli. Thesecomponents can be seen clearly in one patient only (no. 8).

B. Significance Test

The first level significance test revealed significant resultsfor the components according to Table III. Nine different EEGfeatures were extracted from the five experiments according toFig. 2(a). The healthy participants revealed significance in 6–9features, whereas the patients achieved significance in 0–4 fea-tures only.

Fig. 3. The averaged oddball EEG traces at Cz referenced to the mastoidsare displayed as responses to the rare (N = 45) and the frequent tones(N = 255). The left column shows five healthy controls and the rightcolumn the five patients. The most predominant event-related components arethe N100 and the P300. Both components are well developed in the healthycontrols but not in our patients.

1) Standard Oddball Paradigm: The simple oddball re-vealed a significant P300 for each healthy subject with a meant-value in the significant region of 2.2–8.0 ( ,but none of the patients reached significance with sufficientconsecutive significant samples. The N100 was highly signifi-cant for all healthy subjects revealing average t-values between4.0 and 9.1 in the significant region. Patients 7and 8 showed a significant N100 with and

.2) Semantic Oddball: The response to a subject’s name in

the semantic oddball experiment also led to a significant ERP(P500 referenced to the average of the pseudowords) in eachhealthy subject with large significant areas of more than 300-msduration . Of the five patients,one (Patient 8) showed a significant effect in a range of 590 mswith . A significant N100 was foundin all healthy subjects, the first four reached average t-valuesgreater than 5.0 and in three of the patients (7, 8, 9), however,with t-values below 4.0.

3) Word Matching: The CNV after the first word in wordpairs was significant in four healthy controls whereas four of thepatients CNV amplitudes were far from significance. Only Pa-tient 8 showed this preparative component. A significant N400to semantically unrelated word pairs was found in four of thefive controls and in none of the patients.

4) Semantic Congruence: This test with meaningful andmeaningless sentences revealed only two control subjects witha significant N400. One patient (Patient 9) significantly differedin his responses between meaningful and meaningless words,however, his response differed from the typical shape of theN400.


TABLE IIILEFT SIX RESULT COLUMNS: NUMBER OF SIGNIFICANT (P < :05) CONSECUTIVE SAMPLES (1 Sample = 1=256 S) WITH THE POLARITY

OF THE EXPECTED ERP. IN BRACKETS THE MINIMUM NUMBER OF SAMPLES FOR AN OVERALL SIGNIFICANCE (P < :05). RIGHT TWO

RESULT COLUMNS: T-VALUES FOR SIGNIFICANCE TEST OF SLOW ERP COMPONENTS

5) Intentionality: A lateralized readiness potential on the in-tention of hand movement could be significantly observed in allhealthy subjects with the expected polarity. All controls revealedeffect sizes of more than 0.2. Highly significant lateralizationwas found in Patient 7 only whereas the other patients could notproduce any remarkable differentiation of the SCPs. However,this patient actually performed the movements of the right andleft hands, thus, the EEG finding cannot be attributed to move-ment intention itself.

C. ERP Quantification

The standardized amplitudes or differentiations are illus-trated in Fig. 2(b) for all components and all subjects. The effectsizes reflect the results of the significance tests. When setting athreshold at 0.2 to the effect sizes as a decision criterion morethan 90% of the results matched the significance tests.

D. Cumulative Results

A differentiated judgment of the remaining cognitive abili-ties of a patient has to consider the result of each experimentseparately. However, to arrive at an overall measure or successrate of each subject, one can calculate the number of significantcomponents. All healthy persons exceeded the 5% significancelevel in at least six of the eight ERP components, whereas thepatients only revealed zero to four significant ERPs.

The quantitative results can be merged to an overall value bysumming the effect sizes to a cumulated standardized amplitudeas depicted in Fig. 2(c). Comparing the results of both cumula-tive methods all healthy subjects show much higher values thanthe patients. The best results of the patients were obtained in Pa-tients 7 and 8.

V. DISCUSSION

The aim of this report was to present a psychophysiologicalsystem that can easily be used in a nonresponsive patient’s en-vironment to test for cognitive processing abilities. After suc-cessful testing the patient may be trained for direct brain-com-puter communication or other assisted communication devices.

Both tasks are carried out with the same TTD hard- and soft-ware.

This pilot study indicates that the chosen cognitive-psy-chophysiological experiments consistently separate brain-in-jured patients from healthy controls. Importantly, each of thepatients had a pattern of results distinguishable from that of thecontrol subjects. Nevertheless, in accord with previous findings[8], [29] intact cognitive ERPs to complex stimuli were foundin some patients diagnosed as nonresponsive.

Patient 7 showed a clear N100 to mismatch word pairs anda highly significant response to the imagination of left or righthand movements. However, in this experiment he also showedsome systematic muscular reaction. Patient 8 succeeded in 3tests and was close to significance or to the threshold amplitudein 5 of the 9 tests.

This encouraged us to initiate TTD-communication trainingwith those two patients. After 74 training sessions, Patient 8was able to acquire the SCP self regulation skill with an averageprecision of 70%. Training with Patient 7 was terminated after17 sessions without any significant slow wave control.

In this preliminary study, only one EEG channel was ana-lyzed. Additional information can be obtained from multipleelectrodes. The explanatory power of the DCA may furtherincrease by incorporating additional cognitive and emotionaltasks. Whereas only the time domain was analyzed in thepresent study, an analysis in the frequency domain can bedone in a similar way by applying the statistical methods toshort-term Fourier-transformed EEG after averaging the spectraaccording to the experimental condition. This analysis can beused, for example, to obtain differences in the mu-rhythmnecessary for its control [30].

The results encourage continuation of DCA tests with manyunresponsive but also responsive patients. For verification of themethod, the BCI training success can then be correlated with theDCA results. This is a challenging but labor intensive task forfuture work.

These results suggest that the BCI for direct communica-tion of paralyzed neurological patients, and the assessment ofinformation processing capacities of these patients can be in-tegrated into the same software. This will also allow for con-tinuous testing of patients’ cognitive abilities during training.


In combination with external criteria, such as behavioral im-provement or deterioration, the proposed bed-side psychophys-iological system may become a useful and economical tool toassess remaining cognitive functioning in otherwise unrespon-sive patients. This DCA method requires only minimal effort ortraining of the assistant. In the future, the tool will be availablein every TTD (or BCI2000, www.bciresearch.org).

ACKNOWLEDGMENT

The authors would like to thank S. Lang and Vl. Bostanov fortheir assistance in the design of the DCA stimuli. Thay are alsoindebted to Dr. B. Jeanblanc and S. Rudolph (PERLE) from theMaison de la Retraite (Bethel) in Oberhausbergen (France).

REFERENCES

[1] N. Birbaumer, H. Flor, N. Ghanayim, T. Hinterberger, I. Iverson, E.Taub, B. Kotchoubey, A. Kübler, and J. Perelmouter, “A spelling de-vice for the paralyzed,” Nature, vol. 398, pp. 297–298, 1999.

[2] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M.Vaughan, “Brain-computer interfaces for communication and control,”Clin. Neurophysiol., vol. 113, no. 6, pp. 767–791, 2002.

[3] T. Hinterberger, J. Kaiser, A. Kübler, N. Neumann, and N. Birbaumer,“The thought translation device and its applications to the completelyparalyzed,” in Diebner, Druckrey &Weibel: Sciences of the Inter-faces. Tübingen, Germany: Genista-Verlag, 2001, pp. 232–240.

[4] A. Kübler, N. Neumann, J. Kaiser, B. Kotchoubey, T. Hinterberger, andN. Birbaumer, “Brain-computer communication: Self-regulation of slowcortical potentials for verbal communication,” Arch. Phys. Med. Rehab.,vol. 82, pp. 1533–1539, 2001.

[5] H. A. Hanagasi, I. C. H. Gurvit, N. Ermutlu, G. Kaptanoglu, S. Kara-mursel, H. A. Idrisoglu, M. Emre, and T. Demiralp, “Cognitive impair-ment in amyotrophic lateral sclerosis: Evidence from neuropsycholog-ical investigation and event related potentials,” Cogn. Brain Res., vol.14, pp. 234–244, 2002.

[6] K. S. Paulus, I. Magnano, M. R. Piras, M. A. Solinas, G. Solinas, G.F. Sau, and I. Aiello, “Visual and auditory event-related potentials insporadic amyotrophic lateral sclerosis,” Clin. Neurophysiol., vol. 113,pp. 853–861, 2002.

[7] B. Kotchoubey, S. Lang, S. Winter, and N. Birbaumer, “Cognitive pro-cessing in completely paralyzed patients with amyotrophic lateral scle-rosis,” Eur. J. Neurol., vol. 10, pp. 551–558, 2003.

[8] B. Kotchoubey, S. Lang, V. Bostanov, and N. Birbaumer, “Is there amind? electrophysiology of unconscious patients,” News Physiol. Sci.,vol. 17, pp. 38–42, 2002.

[9] T. Hinterberger, “Entwicklung und Optimierung eines Gehirn-Com-puter-Interfaces mit langsamen Hirnpotentialen,” Ph.D. dissertation,Fakultät für Physik der Eberhard-Karls-Universität, Tübingen, Ger-many, 1999.

[10] J. Perelmouter, B. Kotchoubey, A. Kübler, and N. Birbaumer, “Languagesupport program for thought-translation devices,” Automedica, vol. 18,pp. 67–84, 1999.

[11] T. Hinterberger, J. Mellinger, and N. Birbaumer, “The thought transla-tion device: Structure of a multimodal brain-computer communicationsystem,” in Proc. 1st Int. IEEE EMBS Conf. Neural Engineering, 2003,pp. 603–606.

[12] G. Schalk, D. McFarland, T. Hinterberger, N. Birbaumer, and J. R.Wolpaw, “BCI2000: A general-purpose brain-computer interface (BCI)system,” IEEE Trans. Biomed. Eng. (Special Issue on Brain-ComputerInterfaces) of the IEEE Trans. on Neurobiological Engineering, vol. 51,no. 6, pp. 1034–1043, Jun. 2004.

[13] B. Kotchoubey, H. Schleichert, W. Lutzenberger, and N. Birbaumer, “Anew method for self-regulation of slow cortical potentials in a timedparadigm,” Appl. Psychophysiol. Biofeedback, vol. 22, pp. 77–93, 1997.

[14] T. Hinterberger, A. Kübler, J. Kaiser, N. Neumann, and N. Birbaumer,“A brain-computer-interface (BCI) for the locked-in: Comparison of dif-ferent EEG classifications for the thought translation device (TTD),”Clin. Neurophysiol., vol. 114, pp. 416–425, 2003.

[15] B. Rockstroh, T. Elbert, A. Canavan, W. Lutzenberger, and N. Bir-baumer, Slow cortical potentials and behavior. Baltimore, MD:Urban & Schwarzenberg, 1989, p. 23.

[16] D. Guthrie and J. S. Buchwald, “Significance testing of difference po-tentials,” Psychophysiology, vol. 28, pp. 240–244, 1991.

[17] B. Kotchoubey, S. Lang, R. Baales, E. Herb, P. Maurer, G. Metzger, D.Schmalohr, V. Bostanov, and N. Birbaumer, “Brain potentials in humanpatients with extremely severe diffuse brain damage,” Neurosci. Lett.,vol. 301, pp. 37–40, 2001.

[18] B. Kotchoubey, S. Lang, E. Herb, P. Maurer, D. Schmalohr, V. Bostanov,and N. Birbaumer, “Stimulus complexity enhances auditory discrimina-tion in patients with extremely severe brain injuries,” Neurosci. Lett.,vol. 352, pp. 129–132, 2003.

[19] R. Johnson, “A triarchic model of P300 amplitude,” Psychophysiology,vol. 23, pp. 367–384, 1986.

[20] C. C. Duncan-Johnson and E. Donchin, “On quantifying surprise,” Psy-chophysiology, vol. 14, pp. 456–467, 1977.

[21] E. Donchin and M. G. H. Coles, “Is the P300 component a manifestationof context updating?,” Behavioral Brain Sci., vol. 11, pp. 357–374, 1988.

[22] B. Kotchoubey and S. Lang, “Event-related potentials in an auditorysemantic oddball task in humans,” Neurosci. Lett., vol. 310, pp. 93–96,2001.

[23] I. Berlad and H. Pratt, “P300 in response to the subject’s own name,”Electroencephalogr. Clin. Neurophysiol., vol. 96, pp. 472–474, 1995.

[24] B. Rockstroh, T. Elbert, A. Canavan, W. Lutzenberger, and N. Bir-baumer, Slow Cortical Potentials and Behavior. Baltimore, MD:Urban & Schwarzenberg, 1989, p. 99ff.

[25] I. Fischler, D. G. Childers, T. Achaiyapaopan, and N. W. Perry, “Brainpotentials during sentence verification: Automatic aspects of compre-hension,” Biol. Psychol., vol. 21, pp. 83–105, 1985.

[26] S. Lang and B. Kotchoubey, “Learning effects on event-related brainpotentials,” Neuroreport, vol. 11, pp. 3327–3331, 2000.

[27] S. Cansino and B. Téllez-Alanís, “ERP’s elicted by a cognitive incon-gruity paradigm: A semantic memory study,” Neuroreport, vol. 11, pp.977–981, 2000.

[28] B. Rockstroh, T. Elbert, N. Birbaumer, and W. Lutzenberger, “Biofeed-back-produced hemispheric asymmetry of slow cortical potentials andits behavioral effects,” Int. J. Psychophysiol., vol. 9, pp. 151–165, 1990.

[29] B. Kotchoubey, A. Dubischar, H. Mack, J. Kaiser, and N. Birbaumer,“Electrocortical and behavioral effects of chronic immobility on wordprocessing,” Cognitive Brain Research, vol. 17, pp. 188–199, 2003.

[30] M. B. Sterman, “Sensorimotor EEG operant conditioning: Experimentaland clinical effects,” Pavlovian J. Biological Sci., vol. 12, pp. 63–92,1977.

Thilo Hinterberger received the Diploma in physicsfrom the University of Ulm, Ulm, Germany, andthe Ph.D. degree in physics from the Universityof Tübingen, Tübingen, Germany, in 1999 on thedevelopment of a Brain-Computer-Interface, called“Thought Translation Device.”

He is currently a Research Associate with the In-stitute of Medical Psychology and Behavioral Neu-robiology at the University of Tübingen. His primaryresearch interests focus on the further development ofbrain-computer interfaces and their applications but

also on the development of EEG-classification methods and the investigation ofneurophysiological mechanisms during the operation of a BCI using functionalMRI.

Dr. Hinterberger is a member of the Society of Psychophysiological Researchand the Deutsche Physikalische Gesellschaft (DPG).

Barbara Wilhelm studied psychology at the Uni-versity of Würzburg, Würzburg, Germany. Shereceived the diploma in 2000 and is currentlyworking towards the Ph.D. degree at the Institute ofMedical Psychology and Behavioral Neurobiology,University of Tübingen, , Tübingen, , Germany.

Her research interests focus on the application andfurther development of brain computer interfaces andthe investigation of neurophysiological mechanismsduring the operation of a BCI using fMRI/EEG.


Jürgen Mellinger received the degree in physics atthe University of Tübingen, Tübingen, Germany, in1999. He is currently working towards the Ph.D. de-gree at the Institute of Medical Psychology and Be-havioral Neurobiology, University of Tübingen, Ger-many.

His research interests focus on information theo-retical aspects of brain computer interfaces and theirapplications.

Boris Kotchoubey studied medicine and psychologyin Moscow, Russia, and received the Ph.D. from theRussian Academy for Educational Science in 1983.

He is an Associate Professor at the Institute ofMedical Psychology and Behavioral Neurobiology,University of Tübingen, , Tübingen, , Germany. Hismain research interests concern the physiologicalmarkers of human cognitive processing, particularlydisorders of that processing in neurological patients.He

Dr. Kotchoubey is a member of the Society of Psy-chophysiological Research (SPR), British Psychophysiological Society (BPPS),German Society for Psychophysiology and Its Applications (DGPA), Associa-tion of Neuroscientists of the Rhine Valley (Neurex), and the German Associa-tion of University Teachers.

Niels Birbaumer was born 1945. He received thePh.D. degrees in biological psychology, art history,and statistics from the University of Vienna, Vienna,Austria, in 1969.

In 1975-1993, he was Full Professor of Clin-ical and Physiological Psychology, University ofTübingen, Tübingen, Germany. In 1986-1988, hewas Full Professor of Psychology, PennsylvaniaState University, University Park. Since 1993, heis Professor of Medical Psychology and BehavioralNeurobiology with the Faculty of Medicine of the

University of Tübingen and Professor of Clinical Psychophysiology, Universityof Padova, Padua, Italy. Since 2002, he is Director of the Center of CognitiveNeuroscience, University of Trento, Trento, Italy. His research topics includeneuronal basis of learning and plasticity; neurophysiology and psychophysi-ology of pain; and neuroprosthetics and neurorehabilitation. He ha authoredmore than 450 publications in peer-reviewed journals and 12 books.

Among his many awards, Dr. Birbaumer has received the Leibniz-Award ofthe German Research Society (DFG), the Award for Research in Neuromus-cular Diseases, Wilhelm-Wundt-Medal of the German Society of Psychology,and Albert Einstein World Award of Science. He is President of the EuropeanAssociation of Behavior Therapy, a Fellow of the American Psychological As-sociation, a Fellow of the Society of Behavioral Medicine and the American As-sociation of Applied Psychophysiology, and a Member of the German Academyof Science and Literature.

A Device for the Detection of Cognitive Brain Functions in Completely Paralyzed or Unresponsive...

Documents

Transcript of A Device for the Detection of Cognitive Brain Functions in Completely Paralyzed or Unresponsive...