A hybrid BCI speller paradigm combining P300 potential and the SSVEP blocking feature

This article has been downloaded from IOPscience. Please scroll down to see the full text article.

2013 J. Neural Eng. 10 026001

(http://iopscience.iop.org/1741-2552/10/2/026001)

Download details:

IP Address: 202.113.13.11

The article was downloaded on 03/02/2013 at 10:32

Please note that terms and conditions apply.

View the table of contents for this issue, or go to the journal homepage for more

Home Search Collections Journals About Contact us My IOPscience

IOP PUBLISHING JOURNAL OF NEURAL ENGINEERING

J. Neural Eng. 10 (2013) 026001 (13pp) doi:10.1088/1741-2560/10/2/026001

A hybrid BCI speller paradigmcombining P300 potential and the SSVEPblocking featureMinpeng Xu1, Hongzhi Qi1,2,3, Baikun Wan1, Tao Yin2, Zhipeng Liu2

and Dong Ming1,3

1 Department of Biomedical Engineering, Tianjin University, Nankai, Tianjin, People’s Republic ofChina2 Institute of Biomedical Engineering, Chinese Academy of Medical Science and Peking Union MedicalCollege, Beijing, People’s Republic of China

E-mail: qhz@tju.edu.cn and richardming@tju.edu.cn

Received 7 June 2012Accepted for publication 16 November 2012Published 31 January 2013Online at stacks.iop.org/JNE/10/026001

AbstractObjective. Hybrid brain–computer interfaces (BCIs) have been proved to be more effective inmental control by combining another channel of physiologic control signals. Among thosestudies, little attention has been paid to the combined use of a steady-state visual evokedpotential (SSVEP) and P300 potential, both providing the fastest and the most reliable EEGbased BCIs. In this paper, a novel hybrid BCI speller is developed to elicit P300 potential andSSVEP blocking (SSVEP-B) distinctly and simultaneously with the same target stimulus.Approach. Twelve subjects were involved in the study and every one performed offlinespelling twice in succession with two different speller paradigms for comparison: hybridspeller and control P300-speller. Feature analysis was adopted from the view of time domain,frequency domain and spatial distribution; the performances were evaluated by characteraccuracy and information transfer rate (ITR). Main results. Signal analysis of the hybridparadigm shows that SSVEPs are an evident EEG component during the nontarget phase butare dismissed and replaced by P300 potentials after target stimuli. The absence of an SSVEP,called SSVEP-B, mostly appearing in channel Oz, presents a sharp distinction between targetresponses and nontarget responses. The r2 value of SSVEP-B in channel Oz is comparable tothat of P300 in channel Cz. Compared with the control P300-speller, the hybrid spellerachieves significantly higher accuracy and ITR with combined features. Significance. Theresults indicate that the combination of P300 with an SSVEP-B improves target discriminationgreatly; the proposed hybrid paradigm is superior to the control paradigm in spellingperformance. Thus, our findings provide a new approach to improve BCI performances.

(Some figures may appear in colour only in the online journal)

1. Introduction

Brain–computer interfaces (BCIs) are communication systemsthat allow people to send information to a computer orcommands to other electronic devices by only measuring brainactivities, without requiring any peripheral (muscular) activity.

3 Authors to whom any correspondence should be addressed.

Such systems can be considered the only way people whosuffer from severe neuromuscular disease can communicateas they are incapable of any motor functions but are otherwisecognitively intact [1, 2]. To date, among non-invasivefunctional brain monitoring methods, electroencephalography(EEG) provides a preferable solution in most circumstanceswith a high time resolution as well as simple andaffordable recording requirement [3]. A conventional ‘simple’

1741-2560/13/026001+13$33.00 1 © 2013 IOP Publishing Ltd Printed in the UK & the USA

J. Neural Eng. 10 (2013) 026001 M Xu et al

EEG-based BCI relies on the changes in the dynamics of brainoscillations, such as evoked potentials (EPs), event-relatedpotentials (ERPs) and sensory motor rhythm.

In current simple EEG-based BCIs, P300-based BCI andSSVEP-based BCI provide high information transfer rates(ITRs) and have many clinical applications [4–6]. Generally,P300-based BCI speller presents the instruction set on acomputer display and intensifies the commands randomly.The rare intensifications of the user’s intent commands willelicit P300 responses of the brain. Then the BCI systemidentifies and transforms these P300 potentials to commandoutputs. SSVEP has been widely studied in the neuroscienceand engineering [7] fields. SSVEP is a response elicitedby the visual stimulus modulated at a constant repetitiverate, normally higher than 6 Hz, and is characterized asa stationary periodic oscillation with a dominant spectralcontent, consistent with the stimulus temporal frequency [8].Researchers have developed SSVEP-based BCI systems thatcan achieve a peak ITR of 124 bits min−1 [9].

In recent years, BCI paradigms combining two differentphysiologic control signals, called ‘hybrid BCIs’, have beenproposed and studied increasingly as a potent approach toimprove BCI performance [10, 11]. Lots of hybrid BCIscombine at least one BCI channel and another control systembased on EOG, EMG and so on. Specifically, a ‘pure’ hybridBCI improves accuracy and universality by combining twodifferent BCI approaches, such as the SSVEP-ERD system[12–15] and the P300-ERD system [16]. Few studies have beenreported on hybrid BCI paradigms with a combination of theP300 feature and the SSVEP feature. Panicker et al developedan asynchronous P300-SSVEP speller, which accomplishescommand selection with P300 and recognizes the control statesimultaneously with the SSVEP [17]. Generally speaking,current pure hybrid BCIs translate multiple EEG channelsinto multiple commands, i.e. different channels are in chargeof different command sets, rather than a sole command thatmay have a higher accuracy. Given all of that, we think ofdeveloping a hybrid BCI capable of combining different EEGfeatures for the same command.

SSVEP and P300 are two outstanding candidates formental control. Their combination may realize higherefficiency. However, due to the different characteristics ofSSVEP and P300, it demands an elaborate design to buildsuch hybrid BCIs utilizing them together for detecting thesame target (command). SSVEP is a phase-locked signal thatneeds several seconds to find its specified spectral response,while P300 performs the characteristic of a time-locked signalwhose temporal response only last hundreds of milliseconds.It is very hard for a target stimulus to contain both kinds ofinformation simultaneously and efficiently. Thus our researchquestions are as follows:

(1) how to design a hybrid BCI speller that can elicitdifferent EEG features simultaneously with the sametarget stimulus;

(2) whether the combined EEG features can improve BCIefficiency?

To address the first question, we introduce an SSVEPblocking procedure that is synchronized with the production

of ERP. In the proposed speller, all command items flickerat a constant repetitive rate and each item stops its flickeringindividually in a random sequence within a short time. Therepetitive flickering will elicit a background SSVEP signaland the interruption of the intent item will dismiss the SSVEPsignal that is a temporal specified response called SSVEP-blocking (SSVEP-B). Meanwhile, the interruption sequencealso forms an oddball paradigm, thus a P300 component willbe elicited. Thus, two BCI channels, SSVEP-B and P300, areoperated simultaneously to detect the same intent command;we will testify whether the combined features improve BCIefficiency in this paper. This paper is organized as follows.Section 2 addresses the methodology including the hybrid BCIspeller matrix, experimental setup and data analysis. Section 3shows the brain responses to the hybrid BCI as well as thecorresponding accuracies. The discussion and conclusion arestated in sections 4 and 5, respectively.

2. Materials and methods

2.1. Hybrid BCI speller paradigm

SSVEP is an ideal candidate to be used in conjunction withP300 potential, since both can be evoked by visual stimuli.Both are well documented to be reliably evoked in most peoplewithout prior training. Experiments by Panicker et al show thatboth of the two kinds of signals can be elicited at the same timein an individual, without greatly compromising the detectionaccuracy of either [17].

The proposed hybrid paradigm presents a 3 × 3matrix with characters 1–9 on a computer screen. Charactersare intensified individually in a random sequence. Theintensification period of each character is named an event phasein the study. The size and font of the characters will changeduring its event phase, while only the brightness alternatesbetween light and dark at other time. Figures 1(a) and (b) showthe two alternating states when the event phase of character‘2’ occurs in the spelling procedure. For other characters, it istheir flicker phase; only the brightness changes from light todark in a constant frequency to elicit an SSVEP. The evolutionof a character in the time sequence is displayed in figures 1(2)and (5). In the absence of events, the character is in the flickerphase, whose luminance changes in binary states comprisingof a light state and a dark state. When an event occurs, thecorresponding character keeps in low luminance and its sizeand font change as the dotted green box shows. Although theappearance rates of the light state and dark state are equalin the flicker phase, the feeling of the flickering character islight because of the black background. So we chose a lowluminance character in its event phase to produce a contrastfeeling, which may elicit a more evident P300 potential.

2.2. Control speller paradigm

In order to validate the efficiency of the proposed hybridparadigm, we conducted a control experiment for comparison.The control speller paradigm was adapted from a conventionalP300 speller paradigm [18], which has no flicker phase. Tobe fair, the control speller paradigm should be at the same

2

J. Neural Eng. 10 (2013) 026001 M Xu et al

(a) (b) (c) (d ) (e) (f )

Figure 1. The evolution of the character matrix in the hybrid paradigm is shown in time sequences with several special point presentations.All character evolutions are shown as (1), (2) and so on. Character ‘2’ and ‘5’ are specified to explain the changing course. The gray dottedarrow lines across the evolutions introduce the special time point presentations. (a) and (b) show the two alternating states in the event phaseof character ‘2’. Following the event phase, it is a special period with no intensified character shown as (c) and (d). Then a new event ofcharacter ‘5’ will occur whose states are displayed as (e) and (f), but it is also in the flicker phase of character ‘2’. The font of character ‘2’in its event phase and flicker phase are shown in the green dotted box and blue dotted box, respectively.

level with the hybrid paradigm in terms of the character matrixand intensification parameters. Thus a similar character matrixwas used, as figure 2(b) showed. The intensified character‘2’ had the same change of size and font with that of thehybrid paradigm, while other characters kept in a dark state.In this paradigm, a high luminance character was adopted inits event phase for a contrast feeling to the black background.Figure 2(a) shows the evolution of a character in the timesequence. The same time parameters were used in the controlparadigm. The major differences between the two paradigmsare between the flicker phase with changing luminance and theblank phase with constant luminance.

2.3. Stimuli system calibration

SSVEP is sensitive to the stimuli frequency, which requiresa precise system to elicit ideal signals. The stimulationsystem was programmed in E-prime2. The visual stimuliwere presented on an LCD monitor with a refresh rateof 60 Hz. E-Prime is a suite of applications to fulfil thecomputerized experiment needs. It not only provides an easy-to-use environment for computerized experiment design, datacollection and analysis, but also provides millisecond precisiontiming to ensure data accuracy.

Scheduling the related unpredictability of the computeroperating system makes the duration of the stimulus imprecise.Thus it needs calibration of the presentation time andmodification of the E-prime2 program before data recording.In this study, the most important issue is to elicit robust SSVEPsignals as background EEG. It was designed to display eachstimulus picture for 33 ms, which could produce a flickerof about 15.15 Hz. However, the actual stimulus durationwas 33.42 ms on average with a variance of 0.49 ms aftercalibration, which brought a flicker of about 14.96 Hz with avariance of 0.22 Hz. The error of the presentation time wasacceptable to elicit an SSVEP response of the brain. In thisstudy, we rounded 14.96 to 15 Hz for convenience.

2.4. Experimental procedure

Twelve right-handed healthy subjects (23–36 years of age;5 females) participated in this study. All subjects gaveinformed consent after the nature and possible consequences ofthe study were explained. All subjects had no experience witha P300-based BCI system. Every subject performed offlinespelling twice in succession with two different paradigms.They did not receive any training before the experiments andwere absent during the stimuli system calibration. Six of themused the control paradigm first, while the others used the

3

J. Neural Eng. 10 (2013) 026001 M Xu et al

(a)

(b) (c)

Figure 2. The evolution of a character in the control paradigm is shown as (a). (b) is a 3 × 3 character matrix of the control paradigmdisplayed in front of the participants with the character ‘2’ intensifying. (c) shows the 32-electrode locations selected in the experimentaccording to the 10–20 system position.

hybrid paradigm first. The EEG signal was recorded usinga Neuroscan Synamps2 system with a 32-sensor EEG cap,whose located electrode positioning follows the 10–20 system(figure 2(c)). All channels were referenced to the centrallobe and grounded prefrontal lobe signed in figure 2(c), andthen re-referenced to the bilateral mastoid. EEG signals werebandpass-filtered at 0.1–200 Hz, digitized at a rate of 1000 Hzand stored. In pre-processing, EEG signals were first filteredto 1–30 Hz, and then downsampled at 200 Hz for analysis.

In the experiment, all users were required to pay attentionto a specified character on the matrix and to silently count thenumber of times the target character was highlighted (in thecontrol paradigm) or interrupted (in the hybrid paradigm), untila new character was specified for the next selection. Whetherin the control experiment or in the hybrid experiment, thecharacter currently specified for selection was prompted to theuser with the highlight action (when character 2 was specifiedfor selection, it would be like figure 2(b)) during the initial3 s period. The duration of the event phase was 400 ms forboth paradigms and the inter-stimulus interval (ISI: the timeinterval between the end of a stimulus and the beginning of thefollowing one) was 67 ms. Each matrix presented nine differentstimuli to the users and only one stimulus contained the targetcharacter. A cycle of nine character intensifications constituteda round and five rounds constituted a character block. 45character blocks were conducted in each spelling experimentfor each subject. Thus, there were 225 rounds containing 225target trials and 1800 nontarget trials per subject per paradigm.

2.5. Event-related spectral power modulation and inter-trialphase coherence analysis

Event-related spectral perturbation (ERSP) and inter-trialphase coherence (ITC) analysis signals are from the views oftime-frequency domain of spectral power and spectral phaserespectively, which could bring us more detail about SSVEP-

B. Changes of event-related spectral power were analyzed withevent-related spectral perturbation (ERSP) defined as

ERSP( f , t) = 1

n

n∑

k=1

(Fk( f , t)2) (1)

where n is the number of trails and Fk( f , t) is the spectralestimation of trial k at frequency f and time t [19]. Inthis study, ERSP shows the mean spectral power across theinput epochs, which differs from the mean power during the500 ms pre-stimulus baseline period of the same epochs. ITC[20] measures the consistency across the trials of the EEGspectral phase at each frequency and time window, which wasdefined as

ITC( f , t) = 1

n

∣∣∣∣∣

n∑

k=1

Fk( f , t)

|Fk( f , t)|

∣∣∣∣∣ (2)

where | | represents the complex norm. The ITC value rangesfrom 0 to 1. If the test trials show more phase coherence, theITC value is near 1.

2.6. Stepwise linear discriminant analysis

Stepwise linear discriminant analysis (SWLDA) [21] is usedfor classification in the study. It has been reported that SWLDAprovides a good combination of training and performancecharacteristics for practical P300 speller classification [22].SWLDA is an extension of fisher linear discriminant analysis,which performs feature space reduction by selecting suitablefeatures to be incorporated into the discriminant function [23].A combination of forward and backward stepwise analysiswas implemented in SWLDA. The input features are weightedusing ordinary least squares regression (equivalent to FLD) topredict the target class labels. Starting with no initial featuresin the discriminant function, the most statistically significantinput feature for predicting the target label (with p-value < 0.1)is added to the discriminant function. After each new entryto the discriminant function, a backward stepwise analysis is

4

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 3. Global responses of a brain to the hybrid stimuli are displayed with an average of all subjects. The shaded curves show thestandard deviations.

performed to remove the least significant input features, withp-values > 0.15. This process is repeated until the discriminantfunction includes a predetermined number of features or untilno additional features satisfy the entry/removal criteria [22].

3. Results

3.1. Brain responses of SSVEP-B and ERP

In this section, we will show that the brain response evokedby the hybrid target stimulus contains SSVEP-B and ERPsimultaneously; these two responses present a separated time-frequency distribution, which means they can be divided byfilters with different pass bands.

For a primary understanding of brain responses to thehybrid paradigm, global EEG signals of all channels areshown in figure 3. Typical event-related potentials can befound in most channels with a distinct P300. For example,the target response in channel Cz shows an evident positivegoing wave with a latency of about 300 ms, which cannotbe found in a nontarget response. However, besides these

conventional features, channel Oz and its surroundingscontains the most attractive periodic signals of about 15 Hzin the nontarget responses, which are dismissed and replacedby P300 potentials during the target phase. A periodic EEGsignal is the SSVEP elicited by the repetitive visual stimulusat 15 Hz. The absence of a periodic signal following the targetstimulus is called SSVEP-B in this study. From the view ofSSVEP, its amplitude achieves the largest in channel Oz anddecreases with the increasing distance from channel Oz. Inaddition, a latency of about 1.5 periods is used to dismiss theSSVEP in response to the target stimulus.

Since the appearance of P300 and the disappearance ofSSVEP occur in the same time window, the time of 300 msafter stimulus is a key time point to grasp SSVEP-B features.Figure 4 gives the spectral power information of all channelsat the interesting time point. From a comparison of the bottomsub-graphs, there are two evident frequency points, whosepowers are different from target to nontarget. These frequencypoints are marked as A (3 Hz) and C (15 Hz) in the targetsub-graph in accordance with B and D in the nontarget sub-graph. A global analysis of these frequencies is necessary

5

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 4. The grand average spectral powers of all subjects are analyzed at the time point of 300 ms after target or nontarget stimuli. Thefrequency points of 3 Hz (A) and (B) and 15 Hz (C) and (D) are specified to show the global spectral power differences between target andnontarget responses by means of topography representation.

Figure 5. The variations of spectral power of target and nontarget at 15 Hz are displayed in several time points by topography. It shows thegrand average of all subjects.

to investigate the spatial features of the brain responses tothe hybrid paradigm. For a good description, we resort totopography to illustrate the spectral power differences in thecorresponding channels. Compared with B, A shows highervalues globally, especially in the central and frontal areas. Thisphenomenon may be induced by target related P300 potentials.On the contrary, shown by C and D, the target power of 15Hzis lower than the nontarget power in a large scale, especially inthe occipital area. The high values of D may result from SSVEPsignals. But with the loss of repetitive visual stimulus in thetarget phase, SSVEP signals are interdicted and the power ofthe corresponding frequency is depressed, which brings lowvalues of C.

The frequency analysis gives us a preliminary knowledgeof SSVEP-B and a proper frequency angle to know more.Figure 5 gives the discrete time evolution of the spectral powerat 15Hz when the brain responds to the target and nontargetstimuli. For a nontarget response, the spectral powers of allchannels are almost constant within 800 ms after stimuli; thehighest value is in the occipital area. However, the spectralpower values in the target response are variable with timegoing. The occipital area first decreases power and thenrecovers to the previous level. But during 200 to 600 ms, the

spectral powers of channel Oz and its surroundings keep at aconstantly low value, which is the representation of SSVEP-B.

For a better understanding of brain responses to hybridparadigm, the elicited EEG signals should be further analyzedon typical channels. From the view of the previous section,typical P300 potentials can be found in Cz and visual responsesin Oz, so we pay the most attention to the two channelshere. Figure 6 shows the analyzed result of typical EEGsignals of the corresponding channels of subject 9. In theleft column, the temporal waves of channel Cz are shown inthe top row. According to experience, an enhanced positivegoing component with a latency of about 300 ms can befound in the target response, while it does not exist in thenontarget response. The bottom row shows the time frequencyanalysis by means of short time Fourier transforms (STFT).The spectral power of the target response at the range of 0 to6 Hz has an increase from 0 to 400 ms after stimulus. In theright column, temporal waves and STFT analysis of channelOz are displayed respectively. A stable SSVEP of about 15 Hzcan be found in target waves and nontarget waves beforestimulus presentation. However, there is a distinct SSVEP-B inthe target response from about 100 to 600 ms after the stimulus.Therefore, three features (SSVEP feature prior stimuli, P300potential after stimuli as well as the SSVEP blocking feature)

6

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 6. Signal analyses of channels Cz and Oz of subject 9 are presented in columns. The top row displays the temporal waves of targetresponses and nontarget responses. Time–frequency analyses are displayed in the bottom row.

Table 1. r2 values of different conditions of 12 subjects.

r2\subject 1 2 3 4 5 6 7 8 9 10 11 12 mean

P300 of control 0.46 0.56 0.57 0.67 0.63 0.64 0.63 0.46 0.64 0.58 0.58 0.52 0.58P300 of hybrid 0.48 0.67 0.56 0.58 0.68 0.63 0.63 0.52 0.70 0.72 0.58 0.44 0.60SSVEP-B 0.61 0.49 0.61 0.59 0.54 0.64 0.66 0.51 0.72 0.82 0.71 0.63 0.63Combination 0.86 0.82 0.87 0.87 0.87 0.88 0.85 0.82 0.93 0.98 0.92 0.86 0.88

can be found simultaneously in the time frequency analysispresentation.

3.2. Discrimination quantified by r2 value

The r2 value is calculated to compare the discrimination ofthe EEG features under different conditions. The r2 valueis the proportion of the total variance in the dependent variablethat is accounted for by the independent variable. It measureshow the EEG feature depends upon the target presented tothe user [24]. In this section, the P300 feature is representedby the EEG signal filtered to 1–10 Hz at channel Cz in0–800 ms, while the SSVEP-B feature is described as anEEG signal at 15 Hz at channel Oz. Table 1 shows ther2 values of the P300 feature, SSVEP-B feature and theircombination of all subjects. The combined features give thehighest r2 values for all subjects with a range from 0.82 to 0.98,while other individual features are in the range from 0.44 to0.82. It indicates that the additional channel does improvethe discrimination information of EEG signals as expected. Inaddition, the average r2 value is 0.56 in the hybrid P300 and thehighest is 0.69, which is comparable with 0.55 and 0.65 fromthe control P300. It implies that the variations of the hybridP300 have little influence on discrimination; the mean of theSSVEP-B is 0.60 with a range from 0.46 to 0.81, which is asdiscriminative as the P300.

3.3. Efficiency evaluations

To validate the improvement of the combined features, weinvestigate the classification accuracy and the correspondinginformation transfer rate (ITR) in both paradigms. Featuresfrom eight channels (Fz, Cz, Pz, P3, P4, P7, P8 and Oz) are

used for classification. The character recognition accuracy iscalculated with SWLDA for analysis. There are three featuresets used for comparison; it involves the P300 feature set of thecontrol paradigm, the P300 feature set of the hybrid paradigmand the combined feature set (P300 and SSVEP-B) of thehybrid paradigm. The P300 feature set is represented by thefiltered (1–10 Hz) and downsampled (20 Hz) EEG temporalsignals of eight channels during 0∼800 ms after stimulusonset, i.e. it contains 128 feature points. The acquisition ofthe SSVEP-B feature set is the same with that of the P300feature set, except utilizing a different bandpass filter of 14to 16 Hz. Thus the P300 feature sets have 128 feature points,while the combined feature set has 256 points. However, thepredetermined number of feature points is 60, which is thesame for all conditions in the SWLDA training. A 9-fold crossvalidation is adopted to make sure the classification accuracyis robust. EEG data from the 45 characters task is divided into 9segments. 1 segment is used to test the classification accuracysubsequently, while the other 8 segments are used to train theSWLDA classifier. The final accuracy is the mean value of thenine folds. The ITR calculation follows [25].

Comparisons of the character recognition results arepresented in figure 7, by showing the character accuraciesof all subjects as well as ITRs. Consistent with most of theprevious studies, all accuracy curves have a rising trend withan increased number of rounds. For the mean one, accuraciesof P300+SSVEP-B are obviously higher than the others. Forexample, P300+SSVEP-B gets an average of 93.33% at theone round, while it is only 91.85% and 89.81% for P300 in thecontrol and P300 in the hybrid respectively. P300+SSVEP-B reaches above 99% in the three rounds, while it is in thefour rounds for P300 in the hybrid and never for P300 in thecontrol. For individuals, P300+SSVEP-B achieves a higher

7

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 7. Comparison of character accuracies, the corresponding ITR is displayed with the number of rounds used. The solid lines representaccuracy curves while the dotted lines represent ITR curves.

accuracy in most cases. We use two paired t-tests to investigatethe influence of individuals. It shows that the priority ofP300+SSVEP-B is significant in statistics (P300+SSVEP-Bversus P300 in the control: t(59) = 2.900, p-value < 0.001;P300+SSVEP-B versus P300 in the hybrid: t(59) = 3.049,p-value < 0.001). However, there is no significant differencebetween the P300 in the hybrid and the P300 in the control(the P300 in the hybrid versus the P300 in the control: t(59) =1.359, p-value = 0.18). From the view of ITR, P300+SSVEP-B performs better than others in the ITR comparison. Thelargest mean of P300+SSVEP-B reaches 34.2 bit min−1,while it is 32.5 bit min−1 or 30.0 bit min−1 in the other twoconditions. Two paired t-tests demonstrate it is significantlysuperior to others in statistics (P300+SSVEP-B versus P300in the control: t(59) = 2.038, p-value < 0.05; P300+SSVEP-Bversus P300 in the hybrid: t(59) = 2.835, p-value < 0.001).

4. Discussion

4.1. Characteristics of SSVEP-B

SSVEP Blocking is a new EEG feature, which has never beeninvestigated in BCI usage. In this study we prove that thecombination of SSVEP-B and P300 can improve the BCIefficiency. However, the characteristic of SSVEP-B needsmore attention. For example, when the SSVEP-B will beginand end is very important to its usage. Besides lookinginto temporal waves, phase analysis is another powerfultool for investigating this characteristic. Figure 8 shows theEEG signals of channel Oz as well as the correspondingphase presentations. The average result of all subjects isdisplayed in the left sub-graph, the representative individuals

are on the right. From the perspective of an average result,SSVEP oscillation in the target response does not disappearimmediately when coming across the stimulus. It keeps thesame oscillation of 15 Hz as the nontarget for about 100 msbefore fading away, which can be seen clearly in the phaseevolution. Moreover, the dismissing latencies of all subjectsare almost constant, which can be testified by the individualtemporal waves and phase evolutions in the right sub-graphs.However, the recovery time of SSVEP oscillation is quitedifferent from subject to subject. For example, S11 uses about800 ms to recover an SSVEP, while this value is only about600 ms for S12.

As previous studies reported, the steady state visualevoked potential (SSVEP) is a phase locking response tothe repetitive stimulus [26]. Thus inter trail coherence (ITC)analysis may give us a more robust description of an SSVEP.Figure 9 displays the ITC results of all subjects in the hybridexperiment as well as the ERSP results. Compared with theERSP description, the ITC map shows more evident SSVEPand SSVEP-blocking phenomena, especially in subject 2 and7. From the ITC map, we can find a latency of more than1000 ms in S7 and S8, which exceeds the normal range in [7].This phenomenon may be caused by attention attenuation orvisual fatigue for the repetitive SSVEP eliciting and blocking.Since the SSVEP latency varies inversely to the frequency,it indicates that a higher background SSVEP should beconsidered when building a faster speller.

4.2. Influences on ERP

As ERP responses play a significant role in the BCI-speller,any influence on them deserves to be addressed for a

8

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 8. The temporal waves of channel Oz and their phase evolutions are displayed with different responses. The most left sub-graph is anaverage of all subjects. Subjects 1, 9, 11 and 12 are chosen as the representatives.

Figure 9. ERSP and ITC analyses are used in all subjects. The upper sub-graph of each subject is the result of the ERSP, and the bottom oneis the ITC.

comprehensive understanding of this hybrid speller. In order toget confident results, two strict rejection criteria were appliedto select clean data for ERP analysis [27]. The first rejectioncriterion was that EEG would not be used for analysis if itwas contaminated by the previous or following ERPs. Tothis end, EEG data of brain responses were extracted whenthe three preceding and the three following stimuli werenontarget, i.e. the selected EEG had no interference with ERPsbetween −1401 to 1868 ms. The second rejection criterionwas that EEG would not be selected for analysis if it wascontaminated by eye movements. To this end, we discarded theEEG data whose records at Fp1 and Fp2 contained amplitudedifferences exceeding 100 μV. For all conditions, this left uswith averages of 132 target trials and 471 nontarget trials per

participant per condition. Subsequently, the selected target andnontarget trials were averaged per participant, per conditionand per electrode. Finally, we subtracted the averaged cleannontarget epochs from the averaged clean target epochs foreach participant, each condition and each electrode. Furtheranalyses were performed regarding this difference ERP (orendogenous ERP) [27].

Figure 10 displays the ERP differences between thecontrol and the hybrid. The temporal waves and topographiesaverage over subjects. In the temporal wave presentation,channels of Fz, Cz, Pz, P7, P8 and Oz are selected foranalysis. The significant differences (p < 0.05) between thetwo conditions are shaded by gray blocks. Obviously, mostgray blocks appear at 0–300 ms and 450–700 ms, which

9

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 10. ERP comparison between the control speller and the hybrid speller is addressed. Mean waves of Fz, Cz, Pz, P7, P8 and Oz areshown from −200 to 1000 ms after stimuli. The gray blocks present significant statistical differences (p < 0.05) between the control and thehybrid in the corresponding periods. Five ERP components, N1, P2, N2, P3, N5 and P6, are signed in the sub-graph of Cz and the globalresponses at the peak times are interpreted by the topography at the bottom.

indicates the significant variations of ERP of the hybrid arein these periods. The signed ERP components in Cz havea detailed interpretation by topography at the bottom. Forthe first four components, the topography presentations aregenerally similar in spatial trends. For instance, both P3components reach the highest value in the center and stepdown in the peripheral area. However, the distributions of N5and P6 are obviously different from condition to condition.The hybrid condition has stronger N5 and P6 in the frontal andcentral area.

The comparison of the ERP components is shown infigure 11 from the views of amplitude and latency of Cz.Mean and standard errors of each component are displayedin two conditions. T-tests are resorted to reveal the significantdifferences between the two conditions. It is evident that thelatencies of control are significantly shorter than that of hybridin statistics for the first 3 components (N1: t(11) = −4.260,p-value < 0.01; P2: t(11) = −5.946, p-value < 0.01; N2:t(11) = −4.106, p-value < 0.01), while longer for the lasttwo components (N5: t(11) = 4.506, p-value < 0.01; P6:t(11) = 3.399, p-value < 0.01). And the latencies of P3are similar in both conditions. As for amplitude, hybrid N5and P6 are significantly stronger than that of the control (N5:

t(11) = 5.356, p-value < 0.01; P6: t(11) = –2.278, p-value <

0.05),while there is no significant difference between theseconditions for other components.

Compared with the conventional experiment, the firstthree components in the hybrid postpone their latencies withalmost the same delays. Thus, the overall delays result fromthe delayed N1 component. N1 components are sensitive tophysical stimulus factors and index early sensory processingwithin the extra striate visual cortex and it also respondto manipulations of selective attention [28]. It also reflectsthe operation of a discrimination process within the focusof attention [29]. So the delayed N1 indicates the flickerbackground in the proposed speller attracts attention laterin the information processing stream. However, the delayedinformation process does not disturb the subjects’ recognition,which could be reflected by the undelayed and unreduced P3components. Furthermore, the significant differences of thelate components beyond 500 ms indicate a higher complicatedrecognition is activated in the hybrid speller. Since N5 andP6 are less dependent on the physical characteristics ofexternal stimuli, they are more promising in the use of gaze-independent BCI.

Previous studies reveal that early components relatedwith visual stimuli contribute most to the classification in

10

J. Neural Eng. 10 (2013) 026001 M Xu et al

Figure 11. The values of latency and amplitude of each ERP component are displayed. Mean and standard errors average over participantsfor each condition. Condition pairs that significantly differ from each other are indicated by an asterisk (p < 0.05) or two asterisks (p < 0.01).

Figure 12. Feature weights are presented in different conditions, P300 in control, P300 in hybrid and P300+SSVEP-B. For each condition,16 time points plus 8 channels are displayed with weights in classification. The weight of each point in the color map is in the range from 0to 12, which represents the number of selections for building classifiers. If the point is selected by all subjects to build a classifier, then itsweight is 12. And if no subject selects it, then the weight is 0. The curves at the left (time weight distribution) or bottom (channel weightdistribution) are the sums of the weights for the corresponding time or channels. In building classifiers, all trials are used to train SWLDAand the predetermined number of feature points is 60.

overt attention; the speller efficiency will be damaged whenvisual responses are depressed in covert attention [30]. Sowe investigate how features are used by the classifier in theproposed hybrid paradigm. Figure 12 shows the feature weightdistributions for all conditions. In the case of P300 in thecontrol, the most important features are located at 200 msafter stimuli, which does not accord with previous studies thatdemonstrate the high contribution of early components; latecomponents after 300 ms show less importance than earlycomponents in this case. To P300 in the hybrid, the weightdistribution changes with the decreased weights of the earlycomponents and the increased weights of the late components.The distribution in the range from 150 to 550 ms is almosteven. It indicates that the late components improve theircontributions in classification with no harm to the accuracy.From P300+SSVEP-B, we could find a further increase ofweights of late components. The most important features are

located at 300, 400 and 500 ms; the early components furtherreduce their weights. As late components of ERP are moreindependent of the physical stimuli, these findings may suggesta potential choice in covert attention.

4.3. Performance improvements

From section 3.3, we could see the introduction of SSVEP-Bis helpful to improve BCI performance. The enhancement ofaccuracy and ITR is significant but inferior to improving ther2 values (nearly 50% increase). It may indicate the SWLDA,used popularly in the P300 speller, cannot take full advantageof SSVEP-B. Thus, seeking a more felicitous classificationalgorithm would be significant in improving the hybrid BCIperformance. Furthermore, ERP and SSVEP-B are in twodifferent feature spaces whose combination would be anotherimportant issue in achieving high efficiency. Last but not least,

11

J. Neural Eng. 10 (2013) 026001 M Xu et al

there is little experience on how to extract SSVEP-B and itsadvantage is limited by the raw feature extraction method usedin this study. Therefore, a method of a more advanced signalprocessing could have a better detection of SSVEP-B.

4.4. Potential advantages of P300+SSVEP-B

The current P300 speller has three popular presentationmethods, the single character paradigm (SCP), the row columnparadigm (RCP) and the checkerboard paradigm (CBP) [31].Since the SSVEP-B and P300 response are elicited and timelocked to the same target stimulus, this hybrid strategy canbe naturally expanded to RCP and CBP, which will bring alogarithmical increase of ITR according to the matrix size.However, many factors will affect the performance, e.g., theinterval between neighboring targets and the visual anglebetween neighboring characters. If the interval is too shortbetween two neighboring target stimuli, the SSVEP depressedby the first target may not recover to the normal level before thesecond target. Therefore, the interval should be long enoughfor SSVEP recovery to elicit reliable SSVEP blocking features.In addition, Fuchs et al found significant SSVEP suppressionhappens when competing stimuli were closer than 4.5◦ of thevisual angle [32]. Recently Ng et al also proved that when theproximity of competing stimuli lie within 5 visual degrees,the SSVEP based BCI performs poorly [33]. These findingssuggest the SSVEP response would be influenced by neighborflick if too close to the point of the fixation. Thus stimulidistant from each other may provide more robust SSVEP-B.Moreover, Farwell et al [18], Meinicke et al [34] and Sellerset al [35] showed that the ISI could also influence the procedureof P300. Farwell et al reported that a longer ISI could increasethe P300 recognition, which was inconsistent to the other twofindings. Taking into account the situation discussed above,it is worthwhile to test these factors and determine the bestparameters when expanding the current hybrid paradigm toRC or CB paradigm.

BCI illiteracy is a common and hardly resolvedphenomenon for BCI users [36]. Some people cannot presentidentifiable EEG features in some paradigms, such as P300,SSVEP and so on. However, individuals who have a badperformance in eliciting P300 potentials may do well inproducing SSVEP and vice versa. As proved in this paper,a second feature channel could offset the bad performance ofthe first feature channel when containing useful informationof the same task. Previous studies of the pure hybrid BCI oftenseparate the two feature channels into different mental tasks.The two parallel BCI channels could substantially enhanceITRs when users perform well in both tasks. But a goodperformance in one task could not help the other task withbad performances, which may limit the overall efficiency.In our study, two different mental tasks are first combinedfor executing the same operation. Thus the feature channelsof the proposed hybrid BCI could help each other in targetrecognition, thereby reducing the number of BCI illiterates ofeither the P300-speller or the SSVEP-based BCI.

Another potential advantage of P300+SSVEP-B is toconstruct a parallel BCI speller based on several independent

P300+SSVEP-B spellers. Theoretically, SSVEP-B features inthis hybrid paradigm can be elicited in any specified SSVEPfrequency band, which can serve as a switch to identify aspecified sub-speller in a multi-speller system. Suppose thatseveral hybrid BCI spellers following this paradigm withdifferent switch SSVEP-B frequencies are presented to aBCI user at the same time, the system can identify whichspeller contains the user’s intent by recognizing the SSVEPand SSVEP-B frequency bands, and simultaneously selectingthe exact command from the matrix of the detected spellerwith the P300 and SSVEP-B features together. This approachwill allow a very complex command presentation, which maycover all commands of the standard keyboard with the samepresentation rate, and may highly enhance the informationtransfer rate.

5. Conclusion

This study introduces a novel EEG feature, SSVEP blocking,into BCI research, which transforms frequency specifiedSSVEP to a time-locked EEG response. A very simpleparadigm is investigated to prove that SSVEP-B features willbe stably elicited simultaneously with P300 potentials. Thetwo types of features can be combined consistently to select thesame target. The results indicate that the combination of P300with SSVEP-B improves target discrimination greatly; theproposed hybrid paradigm is superior to the control paradigmin spelling performance. This hybrid paradigm provides greatpotential for expanding the design of the BCI speller.

Acknowledgments

This paper was supported by the National Natural ScienceFoundation of China (numbers 81222021, 61172008,81171423, 30970875, 90920015), the National KeyTechnology R&D Program of the Ministry of Science andTechnology of China (number 2012BAI34B02) and theProgram for New Century Excellent Talents in University ofthe Ministry of Education of China (number NCET-10-0618).

References

[1] McFarland D J and Wolpaw J R 2011 Brain–computerinterfaces for communication and control Commun. ACM54 60–66

[2] Vaughan T M et al 2003 Brain–computer interface technology:a review of the second international meeting IEEE Trans.Neural Syst. Rehabil. Eng. 11 94–109

[3] Allison B Z, Wolpaw E W and Wolpaw J R 2007Brain–computer inter-face systems: progress and prospectsExpert Rev. Med. Devices 4 463–74

[4] Vaughan T M, McFarland D J, Schalk G, Sarnacki W A,Krusienski D J, Sellers E W and Wolpaw J R 2006 TheWadsworth BCI research and development program: athome with BCI IEEE Trans. Neural Syst. Rehabil. Eng.14 229–33

[5] Nijboer F et al 2008 A P300-based brain–computer interfacefor people with amyotrophic lateral sclerosisClin. Neurophysiol. 119 1909–16

12

J. Neural Eng. 10 (2013) 026001 M Xu et al

[6] Sellers E W, Vaughan T M and Wolpaw J R 2010 Abrain–computer interface for long-term independent homeuse Amyotroph. Lateral Scler. 11 449–55

[7] Vialatte F-B, Maurice M, Dauwels J and Cichocki A 2010Steady-state visually evoked potentials: focus on essentialparadigms and future perspectives Prog. Neurobiol.90 418–38

[8] Regan D 1989 Human Brain Electrophysiology: EvokedPotentials and Evoked Magnetic Fields in Science andMedicine (New York: Elsevier)

[9] Volosyak I 2011 SSVEP-based Bremen-BCIinterface—boosting information transfer rates J. NeuralEng. 8 036020

[10] Muller-Putz G R et al 2011 Tools for brain–computerinteraction: a general concept for a hybrid BCIFront. Neuroinform. 5 30

[11] Pfurtscheller G, Allison B Z, Brunner C, Bauernfeind G,Solis-Escalante T, Scherer R, Zander T O, Mueller-Putz G,Neuper C and Birbaumer N 2010 The hybrid BCIFront. Neurosci. 4 30

[12] Allison B Z, Brunner C, Kaiser V, Muller-Putz G R, Neuper Cand Pfurtscheller G 2010 Toward a hybrid brain–computerinterface based on imagined movement and visual attentionJ. Neural Eng. 7 26007

[13] Brunner C, Allison B Z, Krusienski D J, Kaiser V,Muller-Putz G R, Pfurtscheller G and Neuper C 2010Improved signal processing approaches in an offlinesimulation of a hybrid brain–computer interfaceJ. Neurosci. Methods 188 165–73

[14] Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner Pand Muller-Putz G R 2010 Self-paced operation of anSSVEP-based orthosis with and without an imagery-based‘brain switch’: a feasibility study towards a hybrid BCIIEEE Trans. Neural Syst. Rehabil. Eng. 18 409–14

[15] Horki P, Solis-Escalante T, Neuper C and Muller-Putz G 2011Combined motor imagery and SSVEP based BCI control ofa 2 DoF artificial upper limb Med. Biol. Eng. Comput.49 567–77

[16] Long J, Li Y, Yu T and Gu Z 2012 Target selection with hybridfeature for BCI-based 2D cursor control IEEE Trans.Biomed. Eng. 59 132–40

[17] Panicker R C, Puthusserypady S and Sun Y 2011 Anasynchronous P300 BCI with SSVEP-based control statedetection IEEE Trans. Biomed. Eng. 58 1781–8

[18] Farwell L A and Donchin E 1988 Talking off the top of yourhead: toward a mental prosthesis utilizing event-relatedbrain potentials Electroencephalogr. Clin. Neurophysiol.70 510–23

[19] Delorme A and Makeig S 2004 EEGLAB: an open sourcetoolbox for analysis of single-trial EEG dynamicsJ. Neurosci. Methods 134 9–21

[20] Tallon-Baudry C, Bertrand O, Delpuech C and Pernier J 1996Stimulus specificity of phase-locked and non-phase-locked40 Hz visual responses in human J. Neurosci. 16 4240–9

[21] Draper N and Smith H 1981 Applied Regression Analysis(New York: Wiley) pp 307–12

[22] Krusienski D J, Sellers E W, Cabestaing F, Bayoudh S,McFarland D J, Vaughan T M and Wolpaw J R 2006 Acomparison of classification techniques for the P300 spellerJ. Neural Eng. 3 299–305

[23] Krusienski D J, Sellers E W, McFarland D J, Vaughan T Mand Wolpaw J R 2008 Toward enhanced P300 spellerperformance J. Neurosci. Methods 167 15–21

[24] Wolpaw J R, McFarland D J, Neat G W and Forneris C A1991 An EEG-based brain–computer interface forcursor control Electroencephalogr. Clin. Neurophysiol.78 252–9

[25] Wolpaw J R, Birbaumer N, Heetderks W J, McFarland D J,Peckham P H, Schalk G, Donchin E, Quatrano L A,Robinson C J and Vaughan T M 2000 Brain–computerinterface technology: a review of the first internationalmeeting IEEE Trans. Rehabil. Eng. 8 164–73

[26] Jian D, George S and Ramesh S 2006 Attentional modulationof SSVEP power depends on the network tagged by theFlicker frequency Cereb. Cortex 16 1016–29

[27] Thurlings M E, Brouwer A M, Van Erp J B, Blankertz Band Werkhoven P J 2012 Does bimodal stimuluspresentation increase ERP components usable in BCIs?J. Neural Eng. 9 045005

[28] Olofsson J K, Nordin S, Sequeira H and Polich J 2008Affective picture processing: an integrative review of ERPfindings Biol. Psychol. 77 247–65

[29] Vogel E K and Luck S J 2000 The visual N1 component as anindex of a discrimination process Psychophysiology37 190–203

[30] Brunner P, Joshi S, Briskin S, Wolpaw J R, Bischof Hand Schalk G 2010 Does the ‘P300’ speller depend on eyegaze? J. Neural Eng. 7 056013

[31] Townsend G, LaPallo B K, Boulay C B, Krusienski D J,Frye G E, Hauser C K, Schwartz N E, Vaughan T M,Wolpaw J R and Sellers E W 2010 A novel P300-basedbrain–computer interface stimulus presentation paradigm:moving beyond rows and columns Clin. Neurophysiol.121 1109–20

[32] Fuchs S, Andersen S K, Gruber T and Muller M M 2008Attentional bias of competitive interactions in neuronalnetworks of early visual processing in the human brainNeuroimage 41 1086–101

[33] Ng K B, Bradley A P and Cunnington R 2012 Stimulusspecificity of a steady-state visual-evokedpotential-based brain–computer interface J. Neural Eng.9 036008

[34] Meinicke P, Kaper M, Hoppe F, Heumann M and Ritter H2002 Improving transfer rates in brain–computer interface:a case study Neural Inf. Process. Syst. 1107–14

[35] Sellers E W, Krusienski D J, McFarland D J, Vaughan T Mand Wolpaw J R 2006 A P300 event-related potentialbrain–computer interface (BCI): the effects of matrix sizeand inter stimulus interval on performance Biol. Psychol.73 242–52

[36] Vidaurre C and Blankertz B 2010 Towards a cure for BCIilliteracy Brain Topogr. 23 194–8

13