A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [Silverstein, Steven]On: 14 May 2010Access details: Access Details: [subscription number 922037403]Publisher Psychology PressInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Clinical and Experimental NeuropsychologyPublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713657736

A comparative study of the MATRICS and IntegNeuro cognitiveassessment batteriesSteven M. Silverstein ab; Judith Jaeger c; Anne-Marie Donovan-Lepore d; Sandra M. Wilkniss e; AdamSavitz f; Igor Malinovsky a; Danielle Hawthorne a; Shane Raines c; Sarah Carson a; Stephanie Marcello a;Stephen R. Zukin c; Stephen Furlong c;Gersham Dent c

a University of Medicine and Dentistry of New Jersey-University Behavioral HealthCare, Piscataway,NJ, USA b University of Medicine and Dentistry of New Jersey-Robert Wood Johnson Medical School,Piscataway, NJ, USA c AstraZeneca, Wilmington, DE, USA d Zucker-Hillside Hospital, Queens, NY,USA e Thresholds, Inc., Chicago, IL, USA f New York Presbyterian Hospital-Weill Medical College ofCornell University-Payne Whitney Westchester, White Plains, NY, USA

First published on: 07 May 2010

To cite this Article Silverstein, Steven M. , Jaeger, Judith , Donovan-Lepore, Anne-Marie , Wilkniss, Sandra M. , Savitz,Adam , Malinovsky, Igor , Hawthorne, Danielle , Raines, Shane , Carson, Sarah , Marcello, Stephanie , Zukin, Stephen R., Furlong, Stephen andDent, Gersham(2010) 'A comparative study of the MATRICS and IntegNeuro cognitive assessmentbatteries', Journal of Clinical and Experimental Neuropsychology,, First published on: 07 May 2010 (iFirst)To link to this Article: DOI: 10.1080/13803391003596496URL: http://dx.doi.org/10.1080/13803391003596496

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

http://www.informaworld.com/smpp/title~content=t713657736

http://dx.doi.org/10.1080/13803391003596496

http://www.informaworld.com/terms-and-conditions-of-access.pdf

JOURNAL OF CLINICAL AND EXPERIMENTAL NEUROPSYCHOLOGY2010, iFirst, 1–16

© 2010 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business

http://www.psypress.com/jcen DOI: 10.1080/13803391003596496

NCEN A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries

Integ-MCCB Comparison Steven M. Silverstein,1,2 Judith Jaeger,3 Anne-Marie Donovan-Lepore,4 Sandra M. Wilkniss,5 Adam Savitz,6 Igor Malinovsky,1 Danielle Hawthorne,1 Shane Raines,3 Sarah Carson,1 Stephanie Marcello,1 Stephen R. Zukin,3 Stephen Furlong,3 and Gersham Dent3

1University of Medicine and Dentistry of New Jersey–University Behavioral HealthCare, Piscataway, NJ, USA2University of Medicine and Dentistry of New Jersey–Robert Wood Johnson Medical School, Piscataway, NJ, USA3AstraZeneca, Wilmington, DE4Zucker-Hillside Hospital, Queens, NY, USA5Thresholds, Inc., Chicago, IL, USA6New York Presbyterian Hospital–Weill Medical College of Cornell University–Payne Whitney Westchester, White Plains, NY, USA

Cognitive impairment is prevalent in schizophrenia and is related to poorer functional and treatment outcomes.Cognitive assessment is therefore now a routine component of clinical trials of new treatments for schizophrenia. Thecurrent gold-standard for cognitive assessment in clinical trials for schizophrenia is the MATRICS (Measurement andTreatment Research to Improve Cognition in Schizophrenia) Consensus Cognitive Battery (MCCB), which wasdeveloped based on expert consensus and incorporates paper-and-pencil tests (and one computerized measure) with anestablished history in the field of neuropsychology. Recently, however, interest has increased in using computerizedbatteries for clinical trials. In this study, we tested 155 people with schizophrenia and 75 healthy control participants onboth the MCCB and IntegNeuro, a touch-screen-based computerized battery with previously demonstrated high levels ofreliability and validity, to determine comparability between test scores. In addition, we assessed test–retest reliability andpractice effects over a one-month interval for both batteries and determined correlations between cognitive test scores andscores on functional outcome measures. High levels of agreement were observed between total battery composite scores(r > .80) and, in a canonical correlation analysis, between all critical single test scores from each battery (rc > .90). Thebatteries demonstrated essentially equivalent sensitivity in discriminating between patients and controls and equivalentlevels of test–retest reliability and practice effects. Correlations between cognitive test scores and functional outcome mea-sures were equivalent between the two batteries and low in nearly all cases. The number of missing data points was greaterwith IntegNeuro, highlighting the requirements for test administrator involvement even with computerized batteries.

Keywords: Schizophrenia; Cognition; Assessment; Computers; Psychometric properties.

This work was supported by a grant to the first author from AstraZeneca. The following authors conceived and designed the studydescribed in this paper: Steven M. Silverstein, Judith Jaeger, Stephen Zukin, Stephen Furlong, and Gersham Dent. The following authorsfacilitated data management and data sharing for this study: Igor Malinovsky, Danielle Hawthorne, and Sarah Carson. The followingauthors served as site principal investigators and coordinated staff training and data collection at their sites: Sandra M. Wilkniss, AdamSavitz, and Anne-Marie Donovan-Lepore. Stephanie Marcello assisted with research assistant training, participant recruitment, and datacollection. Shane Raines analyzed the data for this study. S. M. Silverstein served as a paid consultant for the Brain Resource Company onseveral occasions prior to the initiation of this study. He currently receives funding from the National Institute of Mental Health (NIMH)and AstraZeneca. A. Savitz receives funding from the Stanley Medical Research Institute. S. M. Wilkniss receives funding from NIMH. J.Jaeger, S. Raines, S. R. Zukin, S. Furlong, and G. Dent are employees of AstraZeneca and report no conflicts of interest. A.-M. Donovan-Lepore, S. Marcello, I. Malinovsky, D. Hawthorne, and S. Carson report no biomedical financial interests or conflicts of interest.

Address correspondence to Steven M. Silverstein, University of Medicine and Dentistry of New Jersey, 151 Centennial Avenue,Piscataway, NJ, 08854, USA (E-mail: [email protected]).

Downloaded By: [Silverstein, Steven] At: 16:48 14 May 2010

mailto:[email protected]

http://www.psypress.com/jcen

2 SILVERSTEIN ET AL.

Cognitive assessment is important in schizophrenia,given the number of impairments associated with the dis-order (e.g., attention, memory, learning, problem solv-ing, etc.; Green et al., 2008) and the associations betweenthese findings and poorer functional outcome (Green,1996). This is now a priority area for the NationalInstitute of Mental Health (NIMH). For example, twofoci of the recent Measurement and Treatment Researchto Improve Cognition in Schizophrenia (MATRICS)initiative (Green et al., 2004; Nuechterlein et al., 2008)were to identify relevant neurocognitive treatment tar-gets and develop a test battery that assesses them. TheMATRICS project built upon the earlier development ofstandardized batteries for the assessment of neurocogni-tive functioning in schizophrenia, including the RepeatableBattery for the Neuropsychological Assessment of Schizo-phrenia (RBANS; Gold, Queern, Iannone, & Buchanan,1999; Hobart, Goldberg, Bartko, & Gold, 1999) and theBrief Assessment of Cognition in Schizophrenia (BACS;Keefe, 1999; Keefe et al., 2004). However, it differed frompast test battery development efforts in that it was based onthe input of several stakeholder groups, including academicresearchers, people with schizophrenia, the NIMH, theFood and Drug Administration (FDA), and the pharma-ceutical industry. The resulting MATRICS Cognitive Con-sensus Battery (MCCB) is a paper-and-pencil battery withthe exception of the computerized Identical Pairs Continu-ous Performance Test (CPT-IP). Data on reliability andvalidity of the battery became available in early 2008(Nuechterlein et al., 2008). A useful feature of the MCCB isthat its scores reflect a conorming process on a large (312)and diverse sample (Kern et al., 2008).

A general trend in the assessment field involves the devel-opment of computerized neurocognitive test batteries(Baker et al., 1985; Devivo, Rothland, Price, & Fein, 1999;Elwood, 2001; Kane & Kay, 1992; Morris, Evenden,Sahakian, & Robbins, 1987), and interest has increased inusing computerized tests for clinical trials of treatments forschizophrenia (e.g., O’Halloran et al., 2008). Recently, atouch-screen-based, computerized cognitive assessmentbattery, IntegNeuro (Brain Resource, Ltd), was developedas part of the standardized methodology used with theBrain Resource International Database (BRID; Gordon,Cooper, Rennie, Hermens, & Williams, 2005; http://www.brainnet.net/). It was developed with input from aconsortium of scientists involved in establishing the data-base. The goals behind creation of IntegNeuro, and a com-panion psychophysiological battery called LabNeuro, wereto establish standardized, computer-based cognitive andpsychophysiological assessment batteries (Paul et al., 2005)and to pool data from the world-wide use of these tools intoa single international database (BRID). This database cur-rently contains complete data from over 6,000 people, fromage 6–100 years, and includes various diagnostic groups inaddition to healthy participants, including schizophrenia,attention-deficit/hyperactivity disorder (ADHD), Alzhe-imer’s disease, depression, posttraumatic stress disorder(PTSD), and mild cognitive impairment (Gordon et al.,2005). IntegNeuro and LabNeuro are currently being usedin clinical trials and in clinical settings throughout theworld, with all data collected being included in the BRID.

IntegNeuro consists of an automated stimulus presentationprotocol and response recording involving touch-screenand voice-recording software. The tests were based on exist-ing ones known to be sensitive to brain dysfunction. Fea-tures of the battery include standardized instructions usingboth auditory explanations and visual examples, practicetrials prior to test trials, and largely automated scoring pro-cedures. It is also independent from mouse or keyboardfamiliarity. In addition, the battery includes both languageand nonlanguage paradigms. Test–retest reliability of Integ-Neuro, as assessed in healthy controls, has been shown tobe acceptable for all measures in the battery (Williams et al.,2005). Similarly, there is consistency across internationaltesting sites (Paul et al., 2007). Norms have been establishedfor IntegNeuro within the first cohort of 1,000 stringentlyscreened nonpatient participants (Clark et al., 2006). More-over, IntegNeuro has demonstrated a factor structure simi-lar to the MCCB, and prediction of functional outcome, ina sample of first-episode schizophrenia patients (Williams,Whitford, Flynn, Wong, Liddell, et al. 2008).

The primary aim of this study was to measure the levelof agreement between IntegNeuro and the MCCB in alarge sample of schizophrenia patients and clinicallyhealthy participants. A secondary aim was to measure,and compare, test–retest reliability of IntegNeuro andMCCB in schizophrenia patients and control participants,as well as practice effects. The proposed study was the firstassessment of test–retest reliability of IntegNeuro inschizophrenia and a replication of the Williams et al.(2005) reliability study in healthy participants. In additionto these analyses, relationships between test battery scoresand scores on two standardized tests of functional capa-city, the Social Functioning Scale (SFS; Birchwood,Smith, Cochrane, Wetton, & Copestake, 1990) and theUniversity of California, San Diego (UCSD) Performance-Based Skills Assessment (UPSA-2; Patterson, Goldman,McKibbin, Hughs, & Jeste, 2001) were determined.

METHOD

Performance sites

Data were collected at five sites: two community mentalhealth centers that are part of the University of Medicineand Dentistry of New Jersey’s (UMDNJ’s) UniversityBehavioral Health Care (UBHC) system; the New YorkPresbyterian Hospital–Weill Medical College (WMC) ofCornell University–Westchester Division (White Plains,NY); Zucker-Hillside Hospital (Queens, NY), andThresholds, Inc (Chicago, IL).

Final recruitment totals for the study are reported inTable 1. Note that when participants dropped out afterVisit 3 they were not replaced, and thus the sample sizefor the test–retest component of the study is smaller thanthat for the initial IntegNeuro–MCCB comparison.

Participants

Participants were 155 people with schizophrenia (102 or65% male) and 75 healthy (i.e., without a diagnosable


http://www.brainnet.net

http://www.brainnet.net

INTEG-MCCB COMPARISON 3

major Axis I psychiatric condition) controls (28 or 37%male). The control group was matched to the patientgroup based on group means for age and maternal edu-cation. All patients were outpatients or partial hospitalpatients. The schizophrenia group was 37% African-American, 0.6% Asian, 3.2 Hispanic, 2.5% PacificIslander or Other, and 48.4% White (with 8.3% undeter-mined). The control group was 34.7% African-American,2.7% Asian, 7% Hispanic, 1.3% Pacific Islander orOther, and 53.3% White. Inclusion criteria were identicalto those in the MCCB validation study (Nuechterleinet al., 2008; with minor deviations noted and explainedbelow) to facilitate interpretation of results betweenstudies. A goal of the MCCB validation study was to iso-late change in cognitive function apart from change insymptoms and other clinical features, and this was ourgoal as well. Therefore, relatively asymptomatic patientson stable treatment regimens were studied.

Inclusion criteria were as follows:

1. Between the ages of 18 and 55 years.2. Clinically stable, not hospitalized, and in the residual

(nonacute) phase of their illness for at least 8 weeks.3. Maintained on current antipsychotic and other con-

comitant psychotropic medications for a specifiedperiod of time sufficient to minimize potentialcomplications of assessment of cognitive status (i.e.,at least 6 weeks) and on current dose for at least 2weeks. Any dose changes during study participationcould not exceed ±25% of dose at the beginning ofthe period.

4. No more than a “moderate” severity rating on thehallucinations and delusions items of the Positiveand Negative Syndrome Scale (PANSS; Kay, Opler,& Fiszbein, 1987). Note that the PANSS rather thanthe Brief Psychiatric Rating Scale (BPRS; Ventura,Nuechterlein, Subotnick, Gutkind, & Gilbert, 2000),which was used in the MCCB validation study, wasused for this study.

5. No more than a “moderate” severity rating on posit-ive formal thought disorder (e.g., PANSS Concep-tual Disorganization item score).

6. No more than a “moderate” severity rating on anynegative symptom item (e.g., PANSS Items N1–N7).

7. A minimal level of extrapyramidal symptoms—forexample, Simpson–Angus Scale (Simpson & Angus,1970) total score (excluding items for leg drop andhead drop) less than 5. Note that the MCCB valida-tion study cutoff was 6, but that used the full 10-itemscale whereas we used 8 items.

8. A minimal level of depressive symptoms, as reflectedin a Calgary Depression Scale (Addington, Addington,& Schissel, 1990) total score less than 10.

Exclusion criteria were as follows:

1. Inability to give informed consent.2. Diagnosis of schizoaffective disorder.3. Substance dependence in the past 6 months.4. Positive result on a laboratory toxicology screen

performed during the study.5. Blood alcohol content > 0.05% during any study

visit.6. Diagnosis of dementia or other neurodegenerative

disorder.7. Diagnosis of other neurologic disorder (e.g., epilepsy,

Tourette’s syndrome).8. Significant head injury or other brain injury leading

to cognitive impairment.9. Mental retardation (premorbid IQ < 70).10. First-degree relative with schizophrenia (for controls).11. Presence of a psychotic, mood, or substance abuse

disorder (for controls).12. Electroconvulsive therapy (ECT) treatments within

the past 8 weeks.

Participants took part in 5 research sessions. Demo-graphic, diagnostic, symptom, and functional assess-ment data were collected during Session 1. Half thesample received IntegNeuro during Sessions 2 and 4 andthe MCCB during Sessions 3 and 5, and the other half ofthe sample received the MCCB during Sessions 2 and 4and IntegNeuro during Sessions 3 and 5. Sessions 2 and4 were held one month apart, as were Sessions 3 and 5.Sessions 2 and 3 took place within one week of eachother, as did Sessions 4 and 5. During all testing sessions,participants provided a urine sample for a toxicologyscreen prior to completing any study measures in orderto assure that a homogenous sample was studied withoutthe potential effect of drugs of abuse. Participants forwhom laboratory data came back as positive for anillegal substance were excluded from further participa-tion in the study. In such cases, a new participant wasenrolled to replace the excluded participant, unlessexclusion occurred after Session 3. Participants alsocompleted a breathalyzer examination during all testingvisits to rule out alcohol use. Any participant with ablood alcohol content (BAC) greater or equal to 0.05%was not permitted to continue with the study.

Cognitive assessment batteries

The specific tests that comprise each of the batteries arelisted in Table 2. Tests in the same row measure similarconstructs. Each domain measured by the MCCB has atleast one comparable test in IntegNeuro, with the excep-tion of visual learning. However, the MCCB visuallearning measure is a memory test, and verbal and non-verbal (visual) memory scores are highly correlated inschizophrenia (Calev, Korin, Kugelmass, & Lerer, 1987),

TABLE 1 Number of participants enrolled

Patients Controls

N Male Female Male Female

Total enrolled 250 119 53 29 49Completed Visit 3 230 103 52 28 47Completed all 5 visits 214 93 50 26 45



suggesting that this domain can be adequately assessedvia a single memory test. Nevertheless, we examined theexecutive maze score (which includes a visual learningcomponent—see Appendix), as a visual learning ana-logue to the MCCB visual learning test. IntegNeuromeasures motor speed and language function, domainsnot assessed by the MCCB battery.

IntegNeuro

IntegNeuro tests tap the following domains of cog-nitive function: motor speed, attention/vigilance,working memory, verbal learning, visual learning,speed of processing, language, reasoning and problemsolving, and social cognition. The measures in each ofthe above domains are described in the Appendix.Scoring of responses was conducted using an auto-mated software program for most tests and by handscoring for .wav files. Hand scoring was required forthe two language tests and the verbal memory test.Trained research assistants conducted the hand scor-ing of the wav. files. IntegNeuro typically takes 60–70min to complete.

MCCB

The MCCB taps the following domains: attention/vigilance, working memory, verbal learning, visual learn-ing, speed of processing, reasoning and problem solving,and social cognition. The measures in each of the abovedomains are described in the Appendix. Hand scoringwas required for all tests except the computerized verbalworking memory test. The MCCB typically takes 60–70min to complete.

Diagnostic and symptom ratings

Diagnostic and symptom ratings were completed duringthe baseline session. Participants with schizophreniawere interviewed using the Structured Clinical Interviewfor DSM–IV Diagnosis (SCID-IV; DSM–IV = Diagnosticand Statistical Manual of Mental Disorders–FourthEdition), Patient Version 2.0 (First, Spitzer, Gibbon, &Williams, 1995). The Overview, Mood Disorder, Psy-chotic Disorder, and Substance Abuse Disorder moduleswere administered. Control participants were interviewedwith the nonpatient version of the SCID (SCID-NP).Current symptoms were assessed in schizophreniapatients using the Positive and Negative Syndrome Scale(PANSS; Kay et al., 1987). Depression was assessedusing an instrument specifically developed to assessdepression in schizophrenia, the Calgary DepressionScale (Addington et al., 1990).

Functional assessment

All patient participants completed the UPSA-2 (Pattersonet al., 2001). The UPSA measures a person’s ability to fol-low instructions and complete everyday tasks. Patientsalso completed the SFS (Birchwood et al., 1990). The SFSmeasures seven areas of functioning that are crucial formaintaining individuals with schizophrenia in the com-munity: employment, social withdrawal, prosocial activi-ties, recreation, interpersonal functioning, perceivedindependence competence, and perceived independenceperformance. A total score is calculated by adding all thesubscale scores. All patients were also rated on the Multi-dimensional Scale of Independent Functioning (MSIF).

TABLE 2 Test indices by domain for each battery

Domain MATRICS IntegNeuro™

Motor speed Motor TappingAttention/vigilance Continuous Performance Test–Identical Pairs Continuous Performance TestWorking memory, verbal University of Maryland Letter–Number Span Letter–Number SpanWorking memory, nonverbal WMS–III Spatial Span Digit SpanVerbal learning Hopkins Verbal Learning Test, Revised Span of Visual MemoryVisual learninga Brief Visuospatial Memory Test, Revised Verbal Learning and RecallSpeed of processing Category Fluencyb Word Generationb

Trail Making A Switching of Attentionc

BACS Symbol Coding Choice Reaction TimeLanguage processing Verbal Interference

Spot the Real WordReasoning and problem solving NAB Mazes Executive MazeSocial cognition MSCEIT Managing Emotions Emotion Recognition Task

Note. MATRICS = Measurement and Treatment Research to Improve Cognition in Schizophrenia. BACS = Brief Assessment ofCognition in Schizophrenia. NAB = Neuropsychological Assessment Battery. MSCEIT = Mayer–Salovey–Caruso EmotionalIntelligence Test. WMS–III = Wechsler Memory Scale–Third Edition.aAlthough IntegNeuro™ does not include a specific test for visual learning, the Executive Maze test was used as a surrogate. bDifferentcategories used for word generation across the two batteries. IntegNeuro used the category of “foods” while the MATRICS ConsensusCognitive Battery (MCCB) used the category of “animals.” cAs Switching of Attention test is very similar to Trail Making A and Btests, Switching of Attention test is listed as a speed of processing measure. The Switching of Attention test was listed as attention/vigilance measure in protocol.



Side effects

The Simpson–Angus Scale (Simpson & Angus, 1970),Abnormal Involuntary Movement Scale (AIMS; Guy,1976), and the Barnes Akathesia Scale (Barnes, 2003)were used to assess motor side effects. On the Simpson–Angus Scale, all items except head and leg drop wereadministered. The Hillside Adverse Events scale was alsoadministered to further rule out severe side effects thatcould affect test performance.

Breathalyzer testing

To assess blood alcohol content, we used the AlcoHawk™ABI Professional model. This is a U.S. Department ofTransportation National Highway Traffic SafetyAdministration (DOT/NHTSA) approved professional-grade alcohol breath analyzer that utilizes a state-of-the-artsemiconductor oxide sensor to test breath alcoholcontent quickly and accurately.

Urine toxicology screen

The OnTrak TesTcard 9, by Varian, Inc., was used toscreen for drug use. The TesTcard 9 screens urine sam-ples for the nine most commonly abused drugs—cocaine,tetrahydrocannibol (THC)/cannabis, barbiturates, ben-zodiazepines, PCP, methamphetamine, tricyclic antide-pressants, morphine, and amphetamines. A positivefinding in any drug category except tricyclic antidepres-sants was an exclusion criterion for this study.

RESULTS

Missing data across all sessions

The frequency of missing data was higher for Integ-Neuro than for the MCCB. For IntegNeuro, the totalamount of calculable missing data across all tests was3.91%, and the percentage of participants whose datasetscontained missing data was 32.31%. In contrast, for theMCCB, the total amount of calculable missing dataacross all tests was 0.89%, and the percentage of patientswhose datasets contained missing data was 11.30%. ForIntegNeuro, the main reason for missing data was non-completion of practice tests, which occurred in 2.74% ofall cases, leading to lack of data from nonadministrationof the actual associated test. This accounted for 59.09%of missing data points. Overall, the primary reasons formissing data during testing sessions were incorrectresponses or incorrect technique used during responding(associated with 29.09% of all missing data), palm of thehand or other body part resting on the touch screen(24.24%), lack of response (20.91%), poor recordingquality or absent files (12.42%), and premature respond-ing (10.30%). For the MCCB, the main reason for miss-ing data was incorrect test administration, whichaccounted for 65% of all missing MCCB data. In allcases except one, data loss from incorrect administration

occurred during the Brief Visual Memory Test–Revised.In 15% of MCCB missing data cases, the data were lostafter testing, and in 20% of cases the cause of missingdata was unknown.

Data transformation

All IntegNeuro scores requiring transformation andtheir respective transformations are listed in Table 3. Asrecommended in the MCCB manual, the followingMATRICS scores were transformed using the naturallogarithm function: Trail Making Test Part A comple-tion time, Hopkins Verbal Learning Test, Revised(HVLT–R) total recall score, and NAB Mazes score.

Correlational analyses between test and cognitive domain scores (Pearson r and canonical)

The correlational analyses described below are orga-nized in terms of the seven a priori cognitive factorsassociated with the MCCB.

Attention/vigilance

Pearson product–moment correlation. The correlationmatrix that was created included the IntegNeuro VisualWorking Memory scores of false positive errors, falsenegative errors, reaction time, and d ′, and the MCCBCPT-IP 2-digit, 3-digit, 4-digit, and total d ′ scores.Correlations between MCCB CPT-IP total d ′ and Integ-Neuro attention/vigilance measures ranged from .25 to.55 (p < .001 for all values). The highest correlationbetween attention/vigilance measures (r = .55) wasobserved between the IntegNeuro d ′ and MCCB CPT-IPtotal d ′ indices.

Working memory

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Letter–Number Spantotal trials correct, Span of Visual Memory total score,Forward Digit Span total, and Reverse Digit Span totalscores, and the MCCB Letter–Number Span andWechsler Memory Scale–Third Edition (WMS–III) spa-tial span scores. Correlations between MCCB Letter–Number Span score and IntegNeuro working memorymeasures ranged from .42 to .74 (p < .001 for all values).The highest correlation (r = .74, N = 210) was observedwith the IntegNeuro Letter–Number Span total trialscorrect score. Correlations between MCCB WMS–IIIspatial span score and IntegNeuro working memorymeasures ranged from .41 to .56 (p < .001 for all values).Correlations of .56 were observed for IntegNeuro mea-sures of Span of Visual Memory total and Reverse DigitSpan total score.

Verbal learning

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Verbal Learning andRecall immediate recall and delayed recall scores and the



MCCB HVLT–R scores. Correlations between MCCBHVLT–R total score and IntegNeuro verbal learningmeasures ranged from .47 to .64 (p < .001 for all values).The highest correlation with HVLT–R total score (r = .64,N = 225) was observed for IntegNeuro Verbal Learningand Recall Trials 1–4 total score.

Visual learning

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Executive Maze scoresof total completion time, total errors, total overruns, andtotal trials, and the MCCB Brief Visuospatial MemoryTest–Revised (BVMT–R) individual and total scores.Correlations between MCCB HVLT–R total score andIntegNeuro visual learning measures ranged from .26 to.64 (p < .001 for all values). Correlations with HVLT–Rtotal score of .64 were observed for IntegNeuro Execu-tive Maze measures of total completion time and totalerrors.

Speed of processing

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Average Choice Reac-tion Time, FAS individual and total scores, Switching ofAttention completion time scores, and word generationscore, and the MCCB Category Fluency, Trail MakingTest (TMT) Part A, and BACS symbol coding scores.

Correlations between MCCB Category Fluency scoreand IntegNeuro speed of processing measures rangedfrom .35 to .72 (p < .001 for all values). The highest cor-relation with Category Fluency score (r = .72, N = 226)was observed for IntegNeuro Word Generation totalscore. Correlations between MCCB TMT Part A scoreand IntegNeuro speed of processing measures rangedfrom .35 to .62 (p < .001 for all values). The highest cor-relation with TMT Part A score (r = .62, N = 222) wasobserved for IntegNeuro Switching of Attention Part 1completion time. Correlations between MCCB BACSsymbol coding score and IntegNeuro speed of processingmeasures ranged from .45 to .74 (p < .001 for all values).The highest correlation with BACS symbol coding score(r = .74, N = 217) was observed for IntegNeuro Switchingof Attention Part 2 completion time.

Reasoning and problem solving

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Executive Maze scoresof total completion time, total errors, total overruns, andtotal trials, and the MCCB NAB Mazes score. Correla-tions between MCCB NAB Mazes score and IntegNeuroreasoning and problem-solving measures ranged from.15 to .55 (p < .05 for all values). A correlation withNAB Maze score of .55 was observed for the IntegNeuromeasure of total completion time.

TABLE 3 Transformations used for IntegNeuro™ and MCCB scores

Battery Measure Transformation

IntegNeuro™ Visual Working Memory reaction time 1/xForward Digit Span total score √xReverse Digit Span total score √xLNS total trials correct sin−1 (x/24)Span of Visual Memory total score sin−1 (x/16)Verbal Learning and Recall Immediate Recall Trial 1 score sin−1 (x/12)Verbal Learning and Recall Immediate Recall Trial 2 score sin−1 (x/12)Verbal Learning and Recall Immediate Recall Trial 3 score sin−1 (x/12)Verbal Learning and Recall Immediate Recall Trial 4 score sin−1 (x/12)Verbal Learning and Recall Immediate Recall Trials 1–4 total score sin−1 (x/48)Verbal Learning and Recall Immediate Recall Trial 5 score sin−1(x/12)Verbal Learning and Recall Delayed Recall Trial 6 score sin−1(x/12)Verbal Learning and Recall Delayed Recall Trial 7 score sin−1(x/12)Executive Maze total completion time ln (1/x)Executive Maze total errors ln (1/x)Executive Maze total overruns ln (1/x)Average Choice Reaction Time 1/xSwitching of Attention Part 1 completion time 1/xSwitching of Attention Part 2 completion time 1/xEmotion Identification percentage correct sin−1 (x/100)Emotion Identification reaction time 1/xEmotion Identification total percentage correct sin−1 (x/100)

MCCB HVLT–R Learning total score ln (x)Trail Making Test Part A score ln (1/x)NAB Mazes score ln (x)

Note. MCCB = MATRICS Consensus Cognitive Battery. MATRICS = Measurement and Treatment Research to ImproveCognition in Schizophrenia. LNS = Letter–Number Span. HVLT–R = Hopkins Verbal Learning Test, Revised. NAB =Neuropsychological Assessment Battery.



Social cognition

Pearson product–moment correlation. The correlationmatrix included the IntegNeuro Emotion Identificationpercentage correct and reaction time scores and theMCCB Mayer-Salovey-Caruso Emotional IntelligenceTest (MSCEIT) managing emotions (ME) score. Corre-lations between MCCB MSCEIT ME score and Integ-Neuro social cognition measures ranged from .06 to .40.A correlation of .40 (p < .001, N = 227) was observedbetween the MSCEIT ME score and IntegNeuroEmotion Identification total percentage correct.

Domain and composite scores

Before calculation of domain and composite scores, eachcritical IntegNeuro measure was corrected for the partic-ipant’s age and gender. This age and gender correctionwas conducted for each variable using formulas for theage- and gender-specific mean trend and age-correctedstandard deviations as generated by Brain Resourcefrom their large normative database. Age- and gender-corrected IntegNeuro values were then calculated bysubtracting the age and gender specific trend and divid-ing by the age-adjusted standard deviation.

Domain scores with multiple measures included in thecalculation were calculated as the mean of its compo-nents. The composite score was calculated as the meanof its domain score components. Two composite scoreswere calculated: one composite score with all domainsincluding the visual learning domain score and one com-posite score excluding the visual learning domain scoreto prevent any possible bias by representing ExecutiveMaze performance twice. MCCB domain and compositescores were calculated as age- and gender-correctedt scores as described in the MCCB manual.

Pearson product–moment correlation. Domain scoresand overall composite scores for each battery were com-pared using correlation analyses. All domain and com-posite scores were age and gender corrected. Thecorrelation between the overall composite score for eachbattery was r = .85 (N = 184); the correlation betweencomposite scores when the visual domain score wasexcluded from the calculation of the IntegNeuro compositescore was .84. The correlations between comparabledomain and composite scores for the two batteries arelisted in Table 4. The lowest interbattery correlation wasobserved between the MCCB and IntegNeuro socialcognition domain scores, which was r = .36 (N = 227). Cor-relations between other domains ranged from .54 to .78.

Battery-wise comparisons

Finally, the set of critical IntegNeuro variables wascorrelated with the set of MCCB variables. IntegNeurovariables included in the analysis were: Visual WorkingMemory d′, Letter–Number Span trials correct, Span ofVisual Memory total score, Verbal Learning and RecallImmediate Recall Trials 1–4 total score, FAS averageletter score, word generation total score, average choice

reaction time, Switching of Attention (Parts 1 and 2) com-pletion times, Executive Maze time to completion, andEmotion Identification total percentage correct. MCCBvariables included in the analysis were: CPT-IP total d′score, Letter–Number Span score, WMS–III spatial spanscore, HVLT–R total score, BVMT total score, categoryfluency score, TMT Form A time to completion, BACSsymbol-coding score, NAB Maze total score, andMSCEIT ME score. The canonical correlation analysisindicated a single function with r = .91 and r2 = .84. Therewere five significant pairs of canonical variables. The firstcanonical correlation coefficient was .91 with the firstIntegNeuro canonical variable explaining 39% of theMCCB variance. All other IntegNeuro canonical varia-bles explained less than 3% of the overall MCCB variance;the contributions of each were considered negligible.

Degree of deviation of IntegNeuro and MATRICS test scores from control norms

The effect sizes (Cohen’s d) of the difference betweenschizophrenia patients and healthy controls on each cogni-tive domain score, for each battery, are listed in Table 5.The effect sizes are similar across the two batteriesexcept for the verbal learning and reasoning and prob-lem-solving domains. For these two domains, the Integ-Neuro measures were more sensitive to differencesbetween schizophrenia patients and healthy controls.While the IntegNeuro composite score also appears tobe more sensitive to detecting differences between partic-ipant groups, the effect sizes from the two batteries werenot significantly different.

Relationship between cognitive test scores and functional assessment measures

SFS

For IntegNeuro, correlations between total batteryscore and SFS subscale scores ranged between .05 and.18. The correlation between the IntegNeuro total

TABLE 4 Relationship between IntegNeuro™ and MCCB scores

Cognitive domains N r

Speed of processing 186 .78Attention/processing 192 .55Working memory 190 .70Verbal learning 225 .59Visual learning 212 .62Reasoning and problem solving 225 .54Social cognition 227 .36Composite score 184 .85

Note. Domain and composite scores. MCCB = MATRICSConsensus Cognitive Battery. MATRICS = Measurement andTreatment Research to Improve Cognition in Schizophrenia.N = Number of participants. r = Pearson product–momentcorrelation coefficient.



battery composite score and SFS total score was r = .09(p = .34, N = 122). For the MCCB, correlations betweentotal battery score and SFS subscale scores rangedbetween .012 and .15. The correlation between theMCCB total battery composite score and SFS total scorewas r = .05 (p = .61, N = 131).

UPSA-2

For IntegNeuro, correlations between total batteryscore and UPSA-2 subscale scores ranged from .05 to

.35. The correlation between the IntegNeuro total bat-tery composite score and UPSA-2 total score was r = .33(p < .001, N = 115). For the MCCB, correlationsbetween total battery score and UPSA-2 subscale scoresranged from .13 to .37. The correlation between theMCCB total battery composite score and UPSA2 totalscore was r = .42 (p < .001, N = 121). Data on the MSIFwere not analyzed for this study, and will be presented ina forthcoming paper.

Test–retest reliability

Test–retest reliability over a 1-month interval wascalculated for the critical indices for both batteries.These values are listed in Tables 6 (for IntegNeuro) and 7(for the MCCB). In general, test–retest reliability washigh for all MCCB scores and adequate to high forIntegNeuro test scores except for CPT, Choice ReactionTime (RT), Visual Span, and Verbal Learning andRecall recall scores. The reliability values for these scoreswere significantly lower than those in a previously pub-lished report (Williams et al., 2005). While the lower reli-ability coefficients for these four scores compared totheir MCCB counterparts may reflect a real differencebetween the test batteries, the values may reflect rangerestriction. Specifically, these measures had large amountsof missing data due to participants failing the practicetest prior to attempting the real test (i.e., failed practicetests for these measures accounted for the majority ofmissing IntegNeuro data). In contrast, for the MCCB,there are no criteria that must be passed prior to

TABLE 5 Effect size of the difference between the two groups

on the cognitive domain score for each battery

Cognitive domain

Effect size

MCCB IntegNeuro™

Attention/vigilance 1.39 1.06Working memory 1.05 1.08Verbal learning 0.96 1.58Visual learning 1.34 1.36Speed of processing 1.65 1.46Reasoning and problem solving 0.81 1.55Social cognition 0.95 0.89Composite score 1.67 1.99

Note. Effect size = Cohen’s d (i.e., difference of means dividedby healthy control SD). MCCB = MATRICS Consensus Cog-nitive Battery. MATRICS = Measurement and TreatmentResearch to Improve Cognition in Schizophrenia.

TABLE 6 IntegNeuro test–retest reliability over a 4-week interval

All participantsHealthy control participants only

Schizophrenia participants only

Cognitive domain N ICC r N ICC r N ICC r

Attention vigilanceVisual Memory d′ 167 .53 .53 65 .40 .46 102 .40 .41

Working memoryLNS total trials correct 179 .70 .72 62 .64 .63 117 .64 .64Span of Visual Memory total score 188 .53 .53 68 .57 .44 120 .44 .44

Verbal learningImmediate Recall Trials 1–4 total score 210 .72 .73 69 .54 .62 141 .54 .56

Visual learningExecutive Maze total errors 212 .81 .85 71 .74 .84 141 .74 .78

Speed of processingWord Generation total score 210 .73 .74 69 .56 .85 141 .56 .57Average Choice Reaction Time 162 .65 .66 58 .55 .69 104 .55 .56

FAS average letter score 210 .77 .79 69 .80 .74 141 .74 .75Switching of Attention Part 1 completion time 202 .65 .67 70 .60 .50 132 .60 .62Switching of Attention Part 2 completion time 200 .74 .78 70 .69 .67 130 .69 .72

Reasoning and problem solvingExecutive Maze time to completion 212 .83 .86 71 .71 .85 141 .71 .77

Social cognitionEmotional Identification total percentage correct 210 .79 .80 71 .75 .76 139 .75 .76

Note. ICC calculated according to Shrout and Fleiss’s ICC (1,1). N = number of participants. ICC = intraclass correlation coefficient.r = Pearson product–moment correlation coefficient. LNS = Letter–Number Span.



completing equivalent tests, and so test data areobtained for all participants who attempt a test. It istherefore possible that the lower reliability seen withthese IntegNeuro tests could be due partly to a restric-tion of range due to the loss of data from the mostimpaired patients. This hypothesis is supported by thelarger amount of missing data from patients thanhealthy controls and the assumption that the majority ofmissing data due to failed practice tests would occur inthe most severely ill patient cases. Raju and Brand (2003)explained how restriction of range affects Pearson prod-uct–moment correlations and how correlations can beadjusted to take this into account. Their correction canbe expressed as:

where x = (variance of control group)/(variance of com-bined groups). Range restriction is apparent—and thecorrection applied—when the ratio for x is at least .05(indicating a smaller variance, i.e., restriction, than thatfor the combined groups). For this data set, there wereno cases where this ratio was larger than .50.

Based on the Raju and Brand correction formulas, wereanalyzed the test–retest reliability data from theMCCB to include only those participants who passed the

corresponding IntegNeuro test practice test and assumeda restriction of range. In addition, to make the CPT val-ues more equivalent, we based the MCCB CPT-IP d ′score on the two-digit condition only, since this was theclosest to the IntegNeuro CPT condition (which was sin-gle digit), and therefore this comparison afforded theconceptually clearest test of the extent of similarity intest–retest reliability. The results of these reliabilityrecalculations are presented in Table 8, where it can beseen that the reliability results are more equivalent thanthose presented in Tables 6 and 7. Using the revisedstatistics, across all participants, the average reliabilitycoefficients (r) are .81 for the MCCB and .74 for Integ-Neuro. For schizophrenia participants only, the valuesare .67 for the MCCB and .68 for IntegNeuro, and forcontrols only the values are .79 for the MCCB and .76for IntegNeuro.

Practice effects

In addition to determining reliability across two admin-istrations, we determined whether scores changed signifi-cantly across the two administrations (i.e., practiceeffects). The significance values and effect sizes associatedwith these analyses are reported in Table 9. Significantperformance differences were observed for 7 of the 10MCCB tests and 4 of the 11 IntegNeuro tests. The averageeffect size for IntegNeuro tests was 0.15 (range = 0.03 to0.34), and for the MCCB it was 0.13 (range = 0 to 0.26).

R x

R

(correlation corrected for range restriction) (1/ )

(

= √

× ooriginal correlation value),

TABLE 7 MCCB test–retest reliability over a 4-week interval

All participantsHealthy control participants only

Schizophrenia participants only

Cognitive domain N ICC r N ICC r N ICC r

Attention vigilanceCPT-IP total d′ 208 .84 .86 69 .76 .87 139 .76 .78

Working memoryLetter–Number Span total score 215 .82 .83 70 .79 .80 145 .79 .80WMS–III Spatial Span score 215 .77 .79 70 .72 .79 145 .72 .74

Verbal learningHVLT–R learning total score 216 .76 .76 71 .78 .67 145 .67 .66

Visual learningBVMT–R learning total score 200 .80 .81 71 .69 .79 129 .69 .7

Speed of processingBACS symbol coding score 216 .91 .93 71 .86 .88 145 .86 .87Category Fluency score 215 .74 .75 71 .68 .67 144 .68 .69Trail Making Test Part A score 216 .76 .79 71 .69 .67 145 .69 .72

Reasoning/problem solvingNAB Mazes score 213 .79 .79 71 .68 .78 142 .78 .78

Social cognitionMSCEIT managing emotions score 216 .81 .81 71 .76 .83 145 .76 .77

Note. ICC calculated according to Shrout and Fleiss’s ICC (1,1). N = number of participants. ICC = intraclass correlation coefficient.r = Pearson product–moment correlation coefficient. MCCB = MATRICS Consensus Cognitive Battery. MATRICS = Measurementand Treatment Research to Improve Cognition in Schizophrenia. CPT-IP = Identical Pairs Continuous Performance Test. WMS–III= Wechsler Memory Scale–Third Edition. HVLT–R = Hopkins Verbal Learning Test, Revised. BMVT–R = Brief Visuospatial Mem-ory Test–Revised. BACS = Brief Assessment of Cognition in Schizophrenia. NAB = Neuropsychological Assessment Battery.MSCEIT = Mayer–Salovey–Caruso Emotional Intelligence Test.



DISCUSSION

The specific aim of this research study was to determinethe strength of relationships between scores from theIntegNeuro and MATRICS cognitive batteries. This wastested by having 155 schizophrenia and 75 control par-ticipants complete both batteries, with administrationorder randomized across participants and groups. Inaddition, a month after taking each battery, participantscompleted each battery a second time, to aid in compar-ing the test–retest reliability and practice effect charac-teristics of each battery in the same sample.

Results of the correlational analyses indicated that thetwo test batteries are comparable. The canonical correlation

between the sets of critical test indices for each batterywas high (.91) and accounted for 84% of the variance inthe test battery scores. Moreover, the Pearson correla-tion between the single composite total score for eachbattery was also high (.81).

Analysis of individual cognitive domains indicatedthat while in some domains the two batteries were com-parable, in other domains this was not the case. Thestrongest relationships between MCCB and IntegNeurodomains were observed for working memory (where thehighest correlation between test indices was .73) andspeed of processing (where the highest correlation was.71). The lowest correlations were in the domains ofattention/vigilance, where the correlations ranged from

TABLE 8 Test–retest reliability values after correcting for range restriction

MCCB All Controls Schiz. IntegNeuro All Controls Schiz. All (N) Schiz (n)

Category Fluency .75 .67 .69 Category Fluency .76 .85 .72 210 141Trails A .74 .69 .66 Switching of Attention Part 1 .68 .52 .74 203 133BACS Symbol .93 .88 .87 Choice RT .66 .85 .56 163 105

Verbal Interference .78 .72 .73CPT-IP .85a .87a .77a CPT .67 .79 .56 162 99

.65b .57b .60b

Verbal LNS .83 .80 .80 Verbal LNS .72 .68 .64 178 117WMS–III Visual Spatial Span .77 .79 .71 Span of Visual Memory .56 .65 .50 187 120HVLT–R .78 .78 .66 Verbal Recall and Learning .73 .79 .69 211 142BMVT–R .80 .79 .69 Executive Maze errors .85 .84 .79 197 126NAB-Mazes .81 .78 .76 Executive Maze time .86 .85 .77 211 140MSCEIT–Managing Emotions .81 .83 .77 Emotion Identification .82 .87 .76 211 140

Note. Pearson correlation method. All = all participants. Schiz. = participants with schizophrenia. MCCB = MATRICS ConsensusCognitive Battery. MATRICS = Measurement and Treatment Research to Improve Cognition in Schizophrenia. BACS = BriefAssessment of Cognition in Schizophrenia. CPT-IP = Identical Pairs Continuous Performance Test. LNS = Letter–Number Span.WMS–III = Wechsler Memory Scale–Third Edition. HVLT–R = Hopkins Verbal Learning Test, Revised. BMVT–R = Brief Visu-ospatial Memory Test–Revised. NAB = Neuropsychological Assessment Battery. MSCEIT = Mayer–Salovey–Caruso EmotionalIntelligence Test. RT = reaction time.aValues calculated using total d′, averaged across 2-, 3-, and 4-digit conditions. bValues calculated using the 2-digit condition only.

TABLE 9 Practice effects

MCCB IntegNeuro

Category Fluency (Animals) t = 2.04, p = .04, d = 0.13 Category Fluency t = 0.98, p = .10, d = 0.14Animals Fluency (Letter) t = 1.71, p = .09, d = 0.10

Trails A t = 4.20, p < .001, d = 0.26 Switching of Attention (Trails A) t = 1.58, p = .12, d = 0.12BACS Symbol Coding t = 3.03, p = .003, d = 0.13 Choice RT t = 0.55, p = .58, d = 0.05CPT-IP total d′ t = 3.38, p < .001, d = 0.19 Visual Working Memory (1-back) d′ t = 1.55, p = .13, d = 0.16Verbal LNS t = 2.08, p = .04, d = 0.10 Verbal LNS t = 2.65, p = .009, d = 0.20WMS–III Spatial Span t = 3.00, p = .003, d = 0.19 Spatial Span t = −0.71, p = .48, d = 0.07HVLT–R t = 0.05, p = .96, d = 0.00 Verbal Learning and Recall t = −2.70, p = .008, d = 0.22BVMT–R t = 1.80, p = .07, d = 0.12 Executive Maze (errors) t = 4.65, p < .001, d = 0.28NAB Mazes t = 2.75, p = .007, d = 0.15 Executive Maze (time) t = 5.68, p < .001, d = 0.34MSCEIT–Managing Emotions t = 0.97, p = .34, d = 0.06 Emotion Identification t = −0.59, p = .55, d = 0.03

Note. MCCB = MATRICS Consensus Cognitive Battery. MATRICS = Measurement and Treatment Research to Improve Cognitionin Schizophrenia. BACS = Brief Assessment of Cognition in Schizophrenia. CPT-IP = Identical Pairs Continuous Performance Test.LNS = Letter–Number Span. WMS–III = Wechsler Memory Scale–Third Edition. HVLT–R = Hopkins Verbal Learning Test,Revised. BMVT–R = Brief Visuospatial Memory Test–Revised. NAB = Neuropsychological Assessment Battery. MSCEIT = Mayer–Salovey–Caruso Emotional Intelligence Test. RT = reaction time.



.20 to .55, and social cognition, where the correlationbetween total score on each measure was .42. This varia-bility in correlations can be accounted for by the natureof the tests in each battery. For example, in the socialcognition domain, the MCCB test is a verbally mediatedproblem-solving task that involves anticipating the effec-tiveness of different emotion-based behaviors in an inter-personal situation. In contrast, the IntegNeuro socialcognition measure involves recognizing affect whilelooking at pictures of faces and is equivalent to the ori-ginal Penn emotion identification task. Because socialcognition is a multidimensional construct (Green et al.,2008), it would not be expected that the correlationbetween these two tests would be high. For the attention/vigilance factor, the lowest correlations were observedwith the IntegNeuro false-positive error variable. How-ever, because most participants make few false-positiveerrors, this variable, although important, is not a goodproxy for attentional ability. The most relevant Integ-Neuro variable from this domain is d ′, which reflectssensitivity to the target, and is calculated using signaldetection analyses. The correlation between d ′ and theMCCB CPT-IP total d ′ score was .55, which indicates agreater degree of comparability.

The correlations observed in this study are within thesame range as those from past studies determining com-parability between neurocognitive test batteries. Forexample, in Keefe et al.’s (2004) validation study of theBACS, correlations between BACS measures/factorsand scores from other neuropsychological tests of simi-lar cognitive functions were .61 for verbal memory, from.45 to .59 for executive functioning, .74 for verbal flu-ency, and, for composite scores, .76 for patients and .90for controls. Velligan et al. (2004) obtained a correlationof .72 between total scores on the Brief CognitiveAssessment (BCA) and a battery composed of similarmeasures. A study of the CogTest battery reported cor-relations between individual CogTest measures andanalogous neuropsychological tests to range from .30 to.70 (Sharma & Bilder, 2004). A validation study for theRBANS (Hobart et al., 1999) reported correlationsbetween .20 and .75 between individual RBANS testsand comparable neuropsychological tests, and correla-tions between .52 and .75 between RBANS total scoresand Wechsler Adult Intelligence Scale (WAIS) andWMS composite scores. In the IntegNeuro validationstudy (Paul et al., 2005), all correlations between indi-vidual IntegNeuro measures and paper-and-pencil ver-sions of similar tests exceeded .53. A recent studycompared paper-and-pencil versions of the tests thatmake up the Clinical Antipsychotic Trials of Interven-tion Effectiveness study (CATIE; Keefe et al., 2007) andMATRICS batteries to computerized versions of thesesame tests—the Computerized Multiphasic InteractiveNeurocognitive DualDisplay System (CMINDS; O’Hal-loran et al., 2008). This study found that the CMINDStests produced equivalent scores and were characterizedby equivalent levels of test–retest reliability (but seeKern, Green, Nuechterlein, & Keefe, 2009). Intraclasscorrelations between comparable MCCB and CMINDStests ranged from .64 to .95 during the first testing on

each measure, and test–retest reliability of CMINDStests ranged from .56 to .94. In short, the observed cor-relations between IntegNeuro and the MCCB indicatethat IntegNeuro scores parallel MCCB scores to anequal or greater extent than other batteries’ scores corre-lated with the tests against which these batteries werevalidated in published studies. This is importantbecause, to date, there are no agreed-upon criteria fordetermining whether test batteries are equivalent or not.Because we believe test battery comparability to be on adimensional scale, as opposed to a dichotomous varia-ble, we have chosen not to put forth such criteria in thispaper, but to present the data and our interpretation ofit, recognizing that it might be possible to evaluate theresults differently.

IntegNeuro demonstrated adequate test–retest relia-bility in this study. Intraclass correlations (ICC) forcritical single test scores ranged from .40 to .80 for con-trols and from .40 to .74 for the schizophrenia group.Pearson r values ranged from .46 to .84 for controls andfrom .41 to .78 for the schizophrenia group. For MCCB,the ICC values ranged from .69 to .79 for controls, andfrom .66 to .86 for the schizophrenia group. The latter issimilar to the range of .58 to .85 reported for schizophre-nia patients in the MCCB normative study (Nuechterleinet al., 2008). MCCB Pearson r values ranged from .67 to.87 for controls and from .66 to .87 for the schizophreniagroup. Overall, the ICC values for MCCB tests wereapproximately .09 units higher for controls and .14 unitshigher for patients than those for IntegNeuro tests.Pearson r values were .11 units higher for controls and.12 units higher for the patient sample than those forIntegNeuro tests. When adjusted for range restriction inthe IntegNeuro data due to patient participants’ missingdata points, however, the average differences betweenbatteries, in Pearson r units, were .03 for controls infavor of the MCCB and .01 for patients in favor ofIntegNeuro.

Neither battery demonstrated strong correlations withthe functional outcome measures the SFS or UPSA-2—although correlations between battery composite scores andUPSA-2 composite scores were statistically significant—consistent with the findings from the MCCB normativestudy (Kern et al., 2008). This may reflect the more com-plex and multidetermined phenomenon of real-worldfunctioning compared to specific neurocognitive testperformance and is consistent with the recent NIMH ini-tiative to include functional outcome measures as copri-mary outcome variables in clinical trials. It is notable,however, that the correlations we observed in this studybetween cognitive test scores and functional assessmentmeasures are substantially lower than those reported inother studies (e.g., Bowie, Reichenberg, Patterson,Heaton, & Harvey, 2006; Pietrzak et al., 2009).

Practice effects were statistically significant in manycases, although effect sizes of these score improvementswere small in all cases. The practice effects in this studywere over 50% lower than those reported among healthycontrols and first-episode schizophrenia patients in arecent study (Goldberg et al., 2007), but the latter studyused a different test battery, although there was some



overlap in terms of subtests. While it would seem, intui-tively, that the presence of practice effects would workagainst obtaining significant treatment effects, it isimportant to note the following: (a) Practice effects are aubiquitous problem with cognitive assessment measures;(b) these can be reduced in clinical trials by including atraining phase/administration prior to baseline testing;(c) practice effects do not preclude reliable subsequentand additive change in response to treatment whenceiling effects are not an issue; and (d) when practiceeffects are known, these differences can be used as the“null” hypothesis (rather than no change) against whichtreatment effects are measured. The data presented inthis study can be useful for future studies using thesebatteries by providing such a baseline against whichtreatment-related change can be compared.

An advantage of both batteries is that, even when rela-tively asymptomatic patients are assessed, the differencebetween patient and healthy control groups is large (i.e.,average Cohen’s ds for MCCB and IntegNeuro scores,including composite scores, were 1.23 and 1.37, respec-tively; see Table 5). This is important because the largerthe difference between groups, the greater the range ofchange scores that is possible (i.e., large between-groupdifferences increases sensitivity to change). In the case oftwo domains, verbal learning and reasoning and prob-lem solving, the sensitivity of IntegNeuro was greaterthan 0.5 d units (SD) than that of the MCCB, suggestingthat where these variables are the focus of interestregarding treatment effects, IntegNeuro would be thepreferred assessment battery.

In summary, this study demonstrated the comparabil-ity of the IntegNeuro and MCCB batteries in terms ofequivalence between cognitive domain and total batteryscores, test–retest reliability, practice effects, sensitivityto cognitive deficits in schizophrenia, and correlationswith functional outcome measures. These data suggestthat IntegNeuro is a valid alternative to the MCCB fora cognitive assessment battery for use in clinical trials.Because there are several advantages to the use of astandardized, computerized assessment battery overpaper-and-pencil measures—in terms of standardiza-tion and automatization of test instructions and scor-ing—use of IntegNeuro may have the additionaladvantage of being less resource intensive. An import-ant caveat, however, is that test administrators for com-puterized batteries such as IntegNeuro must be vigilantto aspects of participant behavior that invalidateresponses and lead to missing data (e.g., having sleevesdrag on the touch screen; other response errors due tolack of familiarity with computers, etc). In this study,the frequent occurrence of such errors outweighed thenormal rationale for using computer-administered tests,that being the negative effects of test administrationerrors on paper-and-pencil tests. This echoes findingsfrom the CATIE study, in which the three cognitiveassessment measures with the most missing data were allcomputerized tests (Keefe & Harvey, 2008). It appearsthat, in this study’s use of IntegNeuro, the missing dataproblem reflected primarily automatic skipping of testsfor which the practice test had failed. These problems

have been addressed via the development by BrainResource, Ltd. of: (a) a web-based checklist that eachnew test adminstrator completes to document compre-hension of testing requirements; and (b) a new softwareoption that allows the administrator the options ofrepeating the practise test, and running the test onwhich a participant has failed practice items (BrainResource 2009). The point remains, however, that onlywhen computerized test administrators ensure that dataintegrity is not compromised by participant and testadministration factors will computerized testing trulyconfer its potential advantages.

Original manuscript received 29 September 2009Revised manuscript accepted 4 January 2010

First published online day month year

REFERENCES

Addington, D., Addington, J., & Schissel, B. (1990). A depres-sion rating scale for schizophrenics. Schizophrenia Research,3, 247–251.

Army Individual Test Battery. (1944). Manual of directions andscoring. Washington, DC: War Department, Adjutant Gen-eral’s Office.

Baddeley, A. D., Emslie, H., & Nimmo-Smith, I. (1993). TheSpot-the-Word test: A robust estimate of verbal intelligencebased on lexical decision. British Journal of Clinical Psychology,32, 55–65.

Baker, E. L., Letz, R. E., Fidler, A. T., Shalat, S., Plantamura,D., & Lyndon, M. A. (1985). Computer-based neurobehav-ioral evaluation system for occupational and environmentalepidemiology: Methodology and validation studies. Neu-robehavioral Toxicology and Teratology, 7, 369–377.

Barnes, T. R. E. (2003). The Barnes Akathisia Rating Scalerevisited. Journal of Psychopharmacology, 17, 365–370.

Benedict, R. H. B. (1997). Brief Visuospatial Memory Test–Revised. Odessa, FL: Psychological Assessment Resources.

Birchwood, M., Smith, J., Cochrane, R., Wetton, S., &Copestake, S. (1990). The Social Functioning Scale: Thedevelopment and validation of a new scale of social adjust-ment for use in family intervention programs with schizo-phrenic patients. British Journal of Psychiatry, 157, 853–859.

Bowie, C. R., Reichenberg, A., Patterson, T. L., Heaton, R. K.,& Harvey, P. D. (2006). Determinants of real-world func-tional performance in schizophrenia subjects: Correlationswith cognition, functional capacity, and symptoms. AmericanJournal of Psychiatry, 163, 418–425.

Brain Resource, Ltd. (2009). Brain Resource IntegNeuro Assess-ment Manual, v1. Ultimo, Australia, Brain Resource Ltd.

Brandt, J., & Benedict, R. H. B. (2001). The Hopkins VerbalLearning Test–Revised. Odessa, FL: Psychological Assess-ment Resources.

Calev, A., Korin, Y., Kugelmass, S., & Lerer, B. (1987). Per-formance of chronic schizophrenics on matched word anddesign recall tasks. Biological Psychiatry, 22, 699–709.

Clark, C. R., Paul, R. H., Williams, L. M., Arns, M.,Fallahpour, K., Handmer, C., et al. (2006). Development ofageing across the lifespan using common neuropsychologi-cal indices on an automated test battery. Archives of ClinicalNeuropsychology, 21, 449–467.

Cornblatt, B. A., Risch, N. J., Faris, G., Friedman, D., &Erlenmeyer-Kimling L. (1988). The Continuous Perform-ance Test, Identical Pairs Version (CPT-IP): I. New findingsabout sustained attention in normal families. PsychiatryResearch, 26, 223–238.

Devivo, K., Rothland, J., Price, L., & Fein, G. (1999). Compu-terized assessment of arithmetic computation skills withMicroCog. Journal of the International NeuropsychologicalSociety, 3, 199–200.



Elwood, R. W. (2001). MicroCog: Assessment of cognitivefunctioning. Neuropsychology Review, 11, 89–100.

First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W.(1995). Structured Clinical Interview for DSM IV Axis IDisorders–Patient Edition (SCID-I/P Version 2.0). NewYork: New York State Psychiatric Institute, BiometricsResearch Department.

Gold, J. M., Carpenter, C., Randolph, C., Goldberg, T. E., &Weinberger, D. R. (1997). Auditory working memory andWisconsin Card Sorting Test performance in schizophrenia.Archives of General Psychiatry, 54, 159–165.

Gold, J. M., Queern, C., Iannone, V. N., & Buchanan, R. W.(1999). Repeatable battery for the assessment of neuropsy-chological status as a screening test in schizophrenia: I.Sensitivity, reliability, and validity. American Journal ofPsychiatry, 156, 1944–1950.

Goldberg, T. E., Goldman, R. S., Burdick, K. E., Malhotra, A.K., Lencz, T., Patel, R. C., et al. (2007). Cognitive improve-ment after treatment with second-generation antipsychoticmedications in first-episode schizophrenia: Is it a practiceeffect? Archives of General Psychiatry, 64, 1115–1122.

Golden, C. J. (1978). Stroop color and word test: A manual forclinical and experimental uses. Wood Dale, IL: Stoelting.

Gordon, E., Cooper, N., Rennie, C., Hermens, D., & Williams,L. M. (2005). Integrative neuroscience: The role of a stand-ardized database. Clinical EEG and Neuroscience, 36, 64–75.

Green, M. F. (1996). What are the functional consequences ofcognitive deficits in schizophrenia? American Journal ofPsychiatry, 153, 321–330.

Green, M. F. (1998). Schizophrenia from a neurocognitive per-spective. Boston: Allyn & Bacon.

Green, M. F., Nuechterlein, K. H., Gold, J. M., Barch, D. M.,Cohen, J., Essock, S., et al. (2004). Approaching a consen-sus cognitive battery for clinical trials in schizophrenia: TheNIMH-MATRICS conference to select cognitive domainsand test criteria. Biological Psychiatry, 56, 301–307.

Green, M. F., Penn, D. L., Bentall, R., Carpenter, W. T., Gaebel,W., Gur, R. C., et al. (2008). Social Cognition in Schizophre-nia: An NIMH Workshop on Definitions, Assessment, andResearch Opportunities. Schizophrenia Bulletin, 34, 1211–1220.

Gur, R. C., Sara, R., Hagendoorn, M., Marom, O., Hughett, P.,Macy, L., et al. (2002b). A method for obtaining 3-dimen-sional facial expressions and its standardisation for use inneurocognitive studies. Journal of Neuroscience Methods,115, 137–143.

Guy, W. (1976). ECDEU assessment manual for psychopharma-cology (Rev. ed.). Washington, DC: U.S. Department ofHealth, Education, and Welfare.

Hobart, M. P., Goldberg, R., Bartko, J. J., & Gold, J. M.(1999). Repeatable battery for the assessment of neuropsy-chological status as a screening test in schizophrenia: II.Convergent/discriminant validity and diagnostic group com-parisons. American Journal of Psychiatry, 156, 1951–1957.

Kane, R. L., & Kay, G. G. (1992). Computerized assessment inneuropsychology: A review of tests and test batteries. Neu-ropsychology Review, 3, 1–117.

Kay, S. R., Opler, L. A., & Fiszbein, A. (1987). The positiveand negative syndrome scale (PANSS) for schizophrenia.Schizophrenia Bulletin, 13, 261–276.

Keefe, R. S. E. (1999). Brief Assessment of Cognition in Schizo-phrenia. Durham, NC: Duke University Medical Center.

Keefe, R. S., Bilder, R. M., Davis, S. M., Harvey, P. D.,Palmer, B. W., Gold, J. M., et al. (2007). Neurocognitiveeffects of antipsychotic medications in patients with chronicschizophrenia in the CATIE Trial. Archives of General Psy-chiatry, 64, 633–647.

Keefe, R. S., Goldberg, T. E., Harvey, P. D., Gold, J. M., Poe,M. P., & Coughenour, L. (2004). The Brief Assessmentof Cognition in Schizophrenia: Reliability, sensitivity, andcomparison with a standard neurocognitive battery. Schizo-phrenia Research, 68, 283–297.

Keefe, R. S., & Harvey, P. D. (2008). Implementation consider-ations for multisite clinical trials with cognitive neurosciencetasks. Schizophrenia Bulletin, 34, 656–663.

Kern, R. S., Green, M. F., Nuechterlein, K. H., & Keefe, R. S.(2009). CMINDS does not have identical tests to the CATIEand MATRICS batteries. Commentary on O’Halloran et al.Schizophrenia Research, 107, 327–329.

Kern, R. S., Nuechterlein, K. H., Green, M. F., Baade, L. E.,Fenton, W. S., Gold, J. M., et al. (2008). The MATRICS Con-sensus Cognitive Battery: Part 2. Co-norming and standardiza-tion. American Journal of Psychiatry, 165, 214–220.

Mayer, J. D., Salovey, P., & Caruso, D. R. (2002). Mayer–Salovey–Caruso Emotional Intelligence Test. Toronto,Canada: MHS Publishers.

Morris, R. G., Evenden, J. L., Sahakian, B. J., & Robbins, T.W. (1987). Computer-aided assessment of dementia: Com-parative studies of neuropsychological deficits in Alzheimer-type dementia and Parkinson’s disease. In S. M. Stahl, S. D.Iversen, & E. C. Goodman (Eds.), Cognitive Neurochemis-try (pp. 21–36). Oxford, UK: Oxford University Press.

Nuechterlein, K. H., Green, M. F., Kern, R. S., Baade, L. E.,Barch, D. M., Cohen, J. D., et al. (2008). The MATRICS Con-sensus Cognitive Battery: Part 1. Test selection, reliability, andvalidity. American Journal of Psychiatry, 165, 203–213.

O’Halloran, J. P., Kemp, A. S., Gooch, K. N., Harvey, P. D.,Palmer, B. W., Reist, C., et al. (2008). Psychometric com-parison of computerized and standard administration of theneurocognitive assessment instruments selected by theCATIE and MATRICS consortia among patients withschizophrenia. Schizophrenia Research, 106, 33–41.

Patterson, T. L., Goldman, S., McKibbin, C. L., Hughs, T., &Jeste, D. V. (2001). UCSD Performance-Based SkillsAssessment: Development of a new measure of everydayfunctioning for severely mentally ill adults. SchizophreniaBulletin, 27, 235–245.

Paul, R. H., Gunstad, J., Cooper, N., Williams, L. M., Clark, C.R., Cohen, R. A., et al. (2007). Cross cultural assessment ofneuropsychological performance and electrical brain functionmeasures: Additional validation of an international braindatabase. International Journal of Neuroscience, 117, 549–568.

Paul, R. H., Lawrence, J., Williams, L. M., Richard, C. C.,Cooper, N., & Gordon, E. (2005). Preliminary validity of“Integneuro”: A new computerized battery of neurocog-nitive tests. International Journal of Neuroscience, 115,1549–1567.

Pietrzak, R. H., Olver, J., Norman, T., Piskulic, D., Maruff, P., &Snyder, P. J. (2009). A comparison of the CogState Schizo-phrenia Battery and the Measurement and TreatmentResearch to Improve Cognition in Schizophrenia (MATRICS)Battery in assessing cognitive impairment in chronic schizo-phrenia. Journal of Clinical and Experimental Neuropsychology,14, 1–12.

Raju, N. S., & Brand, P. A. (2003). Determining the significanceof correlations corrected for unreliability and range restric-tion. Applied Psychological Measurement, 27, 52–71.

Reitan, R. (1958). Validity of the Trail Making Test as an indi-cator of organic brain damage. Perceptual Motor Skills, 8,271–276.

Sharma, T., & Bilder, R. (2004). Standardisation and crossvalidation study of Cogtest—an automated neurocognitivebattery for use in clinical trials. European Neuropsychophar-macology, 14, S386.

Simpson, G. M., & Angus, J. W. S. (1970). A rating scale forextrapyramidal side effects. Acta Psychiatrica Scandinavica,212, 11–19.

Spreen, O., & Strauss, E. (1998). A compendium of neuropsycho-logical tests: Administration, norms and commentary (2nded.). New York: Oxford University Press.

Velligan, D. I., DiCocco, M., Bow-Thomas, C. C., Cadle, C.,Glahn, D. C., Miller, A. L., et al. (2004). A brief cognitiveassessment for use with schizophrenia patients in com-munity clinics. Schizophrenia Research, 71, 273–283.

Ventura, J., Nuechterlein, K. H., Subotnik, K. L., Gutkind, D., &Gilbert, E. A. (2000). Symptom dimensions in recent-onsetschizophrenia and mania: A principal components analysisof the 24-item Brief Psychiatric Rating Scale. PsychiatryResearch, 97, 129–135.



Walsh, K. W. (1985). Understanding brain damage: A primer ofneuropsychological evaluation. Edinburgh, UK: ChurchillLivingstone.

Wechsler, D. (1997). Wechsler Memory Scale (3rd ed.). SanAntonio, TX: The Psychological Corporation.

White, T., & Stern, R. A. (2003). Neuropsychological AssessmentBattery. Lutz, FL: Psychological Assessment Resources.

Williams, L. M., Mathersul, D., Palmer, D. M., Gur, R. C.,Gur, R. E., & Gordon, E. (2009). Explicit identification andimplicit recognition of facial emotions: I. Age effects inmales and females across 10 decades. Journal of Clinical andExperimental Neuropsychology, 31, 257–277.

Williams, L. M., Simms, E., Clark, C. R., Paul, R. H., Rowe,D., & Gordon, E. (2005). The test–retest reliability of astandardized neurocognitive and neurophysiological testbattery: “Neuromarker.” International Journal of Neuro-science, 115, 1605–1630.

Williams, L. M., Whitford, T. J., Flynn, G., Wong, W.,Liddell, B. J., Silverstein, S. M., et al. (2008) General andsocial cognition in first-episode schizophrenia: Identifica-tion of separable factors and predication of functional out-come using the IntegNeuro test battery. SchizophreniaResearch, 99, 182–191.

APPENDIX

Description of the tests in each battery

IntegNeuro

Sensorimotor domain

1. Simple Motor Tapping Task: Participants wererequired to tap a circle on the touch screen with theirindex finger, as fast as possible for 60 s. The depend-ent variable was total number of taps with the domi-nant hand.

Attention/vigilance domain

1. Continuous Performance Task: Letters (B, C, D, or G)were presented to the participant on the computerscreen (for 200 ms), separated by an interval of 2.5 s. Ifthe same letter appeared twice in a row, the participantwas asked to press buttons with the index finger of eachhand. Speed and accuracy of response were equallystressed in the task instructions. There were 125 stimulipresented in total, 85 being nontarget letters and 20being target letters (i.e., repetitions of the previous let-ter). The dependent variables were the number of errorsof omission and false positives, and d′.

Working memory domain

1. Digit Span: Participants were presented with a seriesof digits on the touch screen (e.g. 4, 2, 7, etc.; 500-mspresentation), separated by a 1-s interval. The partic-ipant was then immediately asked to enter the digitson a numeric keypad on the touch screen. In the firstpart of the test, participants were required to recallthe digits in forward order (Digits Forwards); in thesecond part, they were required to recall them inreverse order (Digits Backwards). In each part, the

number of digits in each sequence was graduallyincreased from 3 to 9, with two sequences at eachlevel. The dependent measure for each part was themaximum number of digits the participant recalledwithout error.

2. Span of Visual Memory: This test is a computerizedadaptation of the Spatial Span test from theWechsler Memory Scale (Wechsler, 1997). Partici-pants were presented with squares arranged in a ran-dom pattern on the computer screen. The squareswere highlighted in a sequential order on each trial.Participants were required to repeat the order inwhich the squares were highlighted by touching thesquares with their forefinger. Both forward andreverse trials were conducted. The total correct wasthe dependent variable.

3. Letter–Number Span–Verbal (Gold, Carpenter,Randolph, Goldberg, & Weinberger, 1997). This isan orally administered test in which the respondentis instructed to mentally reorder and repeat stings ofletters (alphabetical) and numbers (ascending). Thetrials gradually increase in difficulty. The primaryscore is the total number of trials repeated and reor-dered correctly. All trials were administered, andthere was no discontinuation rule used.

Verbal learning domain

1. Verbal Recall and Learning: The participants wereread a list of 12 words, which they were asked tomemorize. The list contained 12 concrete wordsfrom the English language. Words were closelymatched on concreteness, number of letters, and fre-quency. The list was presented orally four times (andreceived by the participant using headphones). Oneach of the four trials, the participant was requiredto recall as many words as possible by speakingdirectly into the attached microphone. The partici-pant was then presented with a list of distractorwords and was asked to recall them after presenta-tion. Immediately following this, the participant wasasked to recall the 12 words from the original list(short-delay recall trial). A long-delayed recall trialwas completed approximately 20 min later after anumber of intervening tasks. A recognition trial wasthen completed after the delayed trial. The depend-ent variables were the number of words correctlyrecalled across the four learning trials, the immediaterecall trial, and the delayed recall trial and the totalnumber of correctly identified words on the recogni-tion trial.

Visual learning domain

1. Executive Maze Task: This task was a computerizedadaptation of the Austin Maze (Walsh, 1985). Theparticipant was presented with a grid (8 × 8 matrix) ofcircles on the computer screen. The object of the taskwas to identify the hidden path through the grid, fromthe beginning point at the bottom of the grid to theend point at the top. The participant was able to navi-



gate around the grid by pressing arrow keys (up,down, left, right). A total of 24 consecutive correctmoves were required to complete the maze. The partic-ipant was presented with one tone (and a red cross atthe bottom of the screen) if they made an incorrectmove and a different tone (and a green tick at the bot-tom of the screen) if they made a correct move. Thepurpose of the task was therefore to assess howquickly the participant learned the route through themaze and their ability to remember that route. Onlyone maze was presented across trials, and the testended when the participant completed the maze twicewithout error or after 10 min had elapsed. The depend-ent variable was the total maze time. It should benoted that while this measure is identified as a measureof executive function, the requirement to retain themaze in memory for two successive trials introduces anadded visual memory component to the task, andtherefore scores on this measure were also examined inrelationship to the MCCB visual learning domain.

Speed of processing domain

1. Letter Fluency: Participants were required to generate,by speech, words that began with the letters F, A, andS. 60 s were allowed for each letter, and proper nounswere not allowed. Responses were recorded via themicrophone and hand scored. Intrusive or persevera-tive responses were not included in the total numbercorrect. The total number of correct words generatedacross the three trials was the dependent measure.

2. Category Fluency: Animal Naming (Spreen &Strauss, 1998). This is an oral test in which arespondent names as many animals as he or she canin 1 min. The primary score is the total number ofanimals named in 60 s. No points are awarded forperseverations and/or intrusions.

3. Choice Reaction Time Task: Participants were requiredto attend to the computer screen as one of four targetcircles was illuminated in a pseudorandom sequenceover a series of trials. For each trial, the participantwas required to place their index finger in preparationon a start circle displayed on the touch screen. On eachtrial, the participant then had to touch the illuminatedcircle as quickly as possible following presentation. Atotal of 20 trials were administered with a randomdelay between trials of 2–4 s. The dependent variablewas the mean reaction time across trials.

4. Switching of Attention Task: This test is a compu-terized adaptation of the Trail Making Test (ArmyIndividual Test Battery, 1944; Reitan, 1958). Itconsists of two parts. In the first part, the partici-pant was presented with a pattern of 25 numbersin circles and was asked to touch them in ascend-ing numerical sequence (i.e., 1, 2, 3 . . . ). As eachnumber is touched in correct order, a line is drawnautomatically to connect it to the precedingnumber in the sequence. This allowed the partici-pant to visualize the path touched. This task testspsychomotor speed and the basic ability to holdattention on a simple task. The second part of the

test is described below. The dependent variablewas time to completion.

Language domain

1. Verbal Interference: This task taps the ability toinhibit automatic and irrelevant responses and hassimilarities to the Stroop task (Golden, 1978). Theparticipant was presented with colored words pre-sented serially, one at a time. Each word was drawnfrom the following set of lower case words: red, yel-low, green, and blue. The color of each word isdrawn from the following set of colors: red, yellow,green, and blue. Below each colored word is aresponse pad with the four possible words displayedin black and in fixed format. The test has two parts.In Part 1, the participant is required to identify thename of each word as quickly as possible after it ispresented on the screen. In Part 2, the participant isrequired to name the color of each word as quicklyas possible. Each part lasts for 1 min. Responses aremade on the screen by touching the appropriateword on the response pad. The dependent variable ineach part was the number of words correctly identi-fied. This test is a computerized version of the verbalinterference test from the Brief Assessment of Cogni-tion in Schizophrenia (BACS).

2. Spot-the-Word Task: This task is a computerizedadaptation of the Spot the Real Word Test (Baddeley,Emslie, & Nimmo-Smith, 1993). On each trial of thistask, participants were presented with two words onthe touch screen. One of the two words was a validword in the English language (“true” target word),and the second was a nonword foil. Participantswere required to identify, by touching the screen,which of the two words was the true target. The totalcorrect score was the dependent measure.

Reasoning and problem-solving domain

1. Executive Maze Task: This is a computerized task thatassesses constructs equivalent to those assessed by theAustin Maze (Walsh, 1985). The participant was pre-sented with a grid (8 × 8 matrix) of circles on the com-puter screen. The object of the task was to identify thehidden path through the grid, from the beginningpoint at the bottom of the grid to the end point at thetop. The participant was able to navigate around thegrid by pressing arrow keys (up, down, left, right). Atotal of 24 consecutive correct moves were required tocomplete the maze. The participant was presented withone tone (and a red cross at the bottom of the screen) ifthey made an incorrect move and a different tone (anda green tick at the bottom of the screen) if they made acorrect move. The purpose of the task was therefore toassess how quickly the participant learned the routethrough the maze and their ability to remember thatroute. Only one maze was presented across trials, andthe test ended when the participant completed themaze twice without error or after 10 min had elapsed.The dependent variable was the total maze time. It



should be noted that while this measure is identified asa measure of executive function, the requirement toretain the maze in memory for two successive trialsintroduces an added memory component to the task,and therefore this measure taps more cognitivedomains than pure executive function.

Social cognition domain

1. Emotional Identification and Delayed Recall: This isa test of emotional identification based on the onedeveloped at the University of Pennsylvania (Gur,Sara, Hagendoorn, Marom, Hughett, et al., 2002),but modified to centralize eye placement (Williams,Mathersul, Palmer, Gur, Gur, et al., 2009). It hasbeen adapted to a Microsoft Windows environmentby BRC. Participants are presented with a series offaces with different emotional expressions (fear, dis-gust, happy, neutral). They are required to press onthe prompts on the bars on the screen that correctlyidentify the emotional valence presented by the face.The dependent variable is the total correct.

MCCB

Attention/vigilance domain

1. Continuous Performance Test–Identical Pairs(Cornblatt, Risch, Faris, Friedman, & Erlenmeyer-Kimling, 1988). In this computer-administered measureof sustained attention the respondent is instructed toclick a mouse button in response to any two consecut-ive matching numbers that briefly flash on the screen.The there are three sets of stimuli: 2-digit numbers, 3-digit numbers, and 4-digit numbers. The primary scoreis mean d′ value across 2-, 3-, and 4-digit conditions,where d′ is an index of signal/noise discrimination thatis computed by the CPT-IP computer program.

Working memory domain

1. Wechsler Memory Scale–Third Edition: Spatial Span–nonverbal (Wechsler, 1997). The respondent isinstructed to use a board with 10 irregularly spacedblocks to “tap” cubes in the same or reversed sequenceas the administrator. The primary score is the sum ofraw scores on Forward and Backward conditions.

2. Letter Number Span–verbal (Gold et al., 1997). Thisis an orally administered test in which the respond-ent is instructed to mentally reorder and repeatstings of letters (alphabetical) and numbers (ascend-ing). The trials gradually increase in difficulty. Theprimary score is the total number of trials repeatedand reordered correctly (unlike its IntegNeurocounterpart, there is a discontinuation rule).

Verbal learning domain

1. Hopkins Verbal Learning Test–Revised (Brandt &Benedict, 2001). This is an orally administered test in

which a list of 12 words from three taxonomic cate-gories is presented. The respondent is instructed torecall as many words as possible after each of threelearning trials. The primary score is the total numberof words recalled correctly over three learning trials.

Visual learning domain

1. Brief Visuospatial Memory Test–Revised (Benedict,1997). In this paper-and-pencil test the respondent isinstructed to reproduce (draw) six geometric figuresthat are presented to him or her for 10 s. The repro-duction is scored on the basis of accuracy in place-ment of each figure, as well as the accuracy ofdrawing. There are three learning trials, and the prim-ary score is the total recall sum for all learning trials.

Speed of processing domain

1. Brief Assessment of Cognition in Schizophrenia:Symbol Coding (Keefe, 1999). This is a timed paperand pencil test in which respondent uses a key tomatch digits to nonsense symbols. The primary scoreis the total number correct.

2. Category Fluency: Animal Naming (Spreen &Strauss, 1998). This is an oral test in which arespondent names as many animals as he or she canin 1 min. The primary score is the total number ofanimals named in 60 s. No points are awarded forperseverations and/or intrusions.

3. Trail Making Test: Part A (Army Individual TestBattery, 1944; Reitan, 1958). This is a timed paper-and-pencil test in which the respondent draws a lineto connect consecutively numbered circles placesirregularly in a sheet of paper. The participant isinstructed to connect the numbers without lifting thepencil. The primary score is “time to completion.”

Reasoning and problem-solving domain

1. Neuropsychological Assessment Battery: Mazes(White & Stern, 2003). This timed paper-and-penciltest includes seven mazes that the respondent isinstructed to navigate from beginning to end withoutlifting a pencil or crossing the lines. The complexityof mazes is increased with each trial, and points areawarded on the basis of “time to completion” (i.e.,the shorter the completion time, the higher is theamount of points earned).

Social cognition domain

1. Mayer–Salovey–Caruso Emotional Intelligence Test:Managing Emotions (Mayer, Salovey, & Caruso,2002). This task measures the respondent’s ability tomake decisions, informed by his or her emotionalstate. In the task, the person rates the predictedeffectiveness of various alternative behaviors ininterpersonal situations where emotions are acti-vated and must be regulated.


A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries

Documents

Transcript of A comparative study of the MATRICS and IntegNeuro cognitive assessment batteries