Psychometric properties of the MacArthur Competence Assessment Tool-Criminal Adjudication
Transcript of Psychometric properties of the MacArthur Competence Assessment Tool-Criminal Adjudication
Psychological Assessment1998, Vol. 10, No. 4, 435-443
Copyrighl 1998 by the American Psychological AMOCiadon, Inc.1040-3590/98/J3.00
Psychometric Properties of the MacArthur CompetenceAssessment Tool-Criminal Adjudication
Randy K. Otto and Norman G. PoythressUniversity of South Florida
Robert A. NicholsonUniversity of Tulsa
John F. EdensUniversity of South Florida
John Monahan, Richard J. Bonnie,Steven K. Hoge, and Marlene Eisenberg
University of Virginia
This article describes the development of a new clinical instrument for use in assessments of adult
criminal defendants' competence to proceed to adjudication, the MacArthur Competence Assessment
Tool-Criminal Adjudication (MacCAT-CA). The MacCAT-CA was derived from a more comprehen-
sive research instrument (MacArthur Structured Assessment of Competencies of Criminal Defendants;
Hoge, Bonnie, Poythress, Monahan, & Eisenberg, 1997) on the basis of considerations efface validity
for use in legal contexts, psychometric analyses, and advice from mental health experts who reviewed
an earlier prototype. This article presents the results from an National Institute of Mental Health-
sponsored validation study that investigated the psychometric properties of the MacCAT-CA.
At least 25,000 criminal defendants are referred annually for
evaluation of their competence to participate in legal proceed-
ings (Steadman & Hartstone, 1983). Although a number of
measures have been specifically designed to assess defendants'
capacities in this area (e.g., Competence Screening Test [CST],
Lipsitt, Lelos, & McGarry, 1971; Georgia Court Competency
Test [GCCT], Wildman et al., 1980; CADCOMP, Barnard et
al., 1991; Competency Assessment Instrument [CAI], Labora-
tory of Community Psychiatry, 1974; Interdisciplinary Fitness
Interview [IFI], Golding, Roesch, & Schreiber, 1984; Fitness
to Stand Trial Interview, Menzies, Webster, Roesch, Jensen, &
Randy K. Otto, Norman G. Poythress, and John F. Edens, Department
of Mental Health Law and Policy, University of South Florida; Robert
A. Nicholson, Department of Psychology, University of Tulsa; John
Monahan and Richard J. Bonnie, School of Law, University of Virginia;
Steven K. Hoge, School of Law and School of Medicine, University of
Virginia; Marlene Eisenberg, Institute of Law, Psychiatry, and PublicPolicy, University of Virginia.
John F. Edens is now at the Department of Psychology, Sam Houston
State University.
This research was supported by the MacArthur Foundation Research
Network on Mental Health and the Law, and by Grant RO1MH54517-
O1A1 from the National Institute of Mental Health. We are grateful to
Tom Feucht-Haviar for his assistance on a pilot test of this measure.
Norman G. Poythress, John Monahan, Richard J. Bonnie, and Steven
K. Hoge are authors of the MacArthur Competence Assessment Tool-
Criminal Adjudication (MacCAT-CA) published by Psychological As-
sessment Resources. Randy K. Otto, Norman G. Poythress, Robert A.
Nicholson, John F. Edens, John Monahan, Richard J. Bonnie, and Steven
K. Hoge are authors of the MacCAT-CA manual, also published byPsychological Assessment Resources.
Correspondence concerning this article should be addressed to Randy
K. Otto, Department of Mental Health Law and Policy, Florida Mental
Health Institute, University of South Florida, 13301 Bruce B. DownsBoulevard, Tampa, Florida 33612-3899. Electronic mail may be sent to
Eaves, 1984), each exhibits various limitations (Grisso, 1991;
Melton, Petrila, Poythress, & Slobogin, 1997; Nicholson, 1992).
Briefly, some measures were designed to serve primarily as
screening instruments (e.g., CST and GCCT) and appear mainly
to be collections of items having various legal content but little
or no underlying conceptual structure. Others (e.g., CAI and IFI)
lack standardized administration and criterion-based scoring,
relying instead on the judgment of the clinician to determine,
on a case-by-case basis, which questions to pose to the defendant
and how to rate the responses received. None is designed to
provide quantitative indexes of discrete competence-related abil-
ities, nor have interpretative norms based on large, national
samples been developed for any of these measures.
Given the pivotal role of a defendant's competence, one of the
major research initiatives of the MacArthur Foundation Research
Network on Mental Health and the Law was the development
of a standardized research instrument for evaluating criminal
defendants' psycholegal abilities related to competence to pro-
ceed to adjudication. The MacArthur Structured Assessment
of Competencies of Criminal Defendants (MacSAC-CD; Hoge,
Bonnie, Poythress, Monahan, & Eisenberg, 1997) was based on
Bonnie's (1992, 1993) comprehensive theory of legal compe-
tence, and it contains measures of discrete competence-related
abilities. Items in the MacSAC-CD were modeled after those
created by Grisso, Appelbaum, Mulvey, and Fletcher (1995) for
the assessment of competence to consent to treatment (see, more
generally, Grisso & Appelbaum, 1998). Each measure in the
MacSAC-CD involves standardized administration and crite-
rion-based scoring, thus reducing much of the discretionary
(and potentially idiosyncratic) scoring that plagues existing
measures.
Hoge et al. (1997) recently described the MacSAC-CD
and presented the primary findings from a field study that in-
volved both incompetent and competent defendants as research
participants. The results of this field study indicated that the
435
436 OTTO ET AL.
MacSAC-CD had satisfactory psychometric properties, con-
struct validity, and potential classification utility for research
applications. Significant differences in the expected direction
between mean scores of hospitalized incompetent (HI), jail
unscreened (JU), and jail treated (JT; i.e., jail inmates who
were receiving mental health services but whose competence
was not raised as an issue) participants were obtained on each
measure. Within a subset of the HI group, the measures were
sensitive to changes in the patients' or defendants' clinical condi-
tion; reassessment of those patients or defendants who had been
deemed clinically restored to competence revealed that scores
on the MacSAC-CD increased, reflecting gains in competence-
related abilities. Although the primary field study used only
male defendants (W = 366), a second study involving 106
female defendants produced similarly encouraging findings con-
cerning the validity of the MacSAC-CD (Poythress, Hoge, et
al., 1998).
Having demonstrated the potential utility of the MacSAC-CD
for the assessment of competence-related abilities, the network's
next priority was to develop a clinically portable measure for
actual use in clinical practice. The field study revealed that
administration time for the MacSAC-CD (approximately 1.5 to
2 hours) exceeded considerably that required by other currently
available measures; further, the need to reduce the length of the
measure was indicated by some duplication in scoring proce-
dures and redundancy across separate measures (see Hoge et
al., 1997). In light of these considerations, and keeping in mind
the importance of face validity for application in the legal con-
text, the MacSAC-CD, an instrument comprising 7 measures
containing 47 items, was streamlined to one comprising 3 mea-
sures containing 22 items. Decisions about item retention were
informed by an examination of (a) the item-total correlations
for each item with its original MacSAC-CD measure, and (b)
the impact on coefficient alpha for the MacSAC-CD measure if
the item was removed.
The resulting 22-item clinical instrument, the MacArthur
Competence Assessment Tool-Criminal Adjudication (Mac-
CAT-CA; Poythress, Nicholson, et al., in press) evaluates three
discrete competence-related abilities: understanding (the ability
to understand general information related to the law and adjudi-
catory proceedings), reasoning (the ability to discern the poten-
tial legal relevance of information, and capacity to reason about
specific choices that confront a defendant in the course of adju-
dication), and appreciation (rational awareness of the meaning
and consequences of the proceedings in one's own case). These
abilities are assessed through three separate measures on the
MacCAT-CA. The content of both the Understanding (eight
items) and Reasoning (eight items) measures refers to a hypo-
thetical legal scenario in which a defendant is charged with
aggravated assault. Questions comprising the Appreciation mea-
sure (six items) refer specifically to the examinee's own case.
Items are scored on a 0-2 scale, with higher scores indicative
of higher levels of capacity. The specific legal content of each
item is summarized in the Appendix. The MacCAT-CA was
reviewed by experienced forensic clinicians for ease of adminis-
tration and was pilot tested to determine administration time,
which ranges from 25 to 55 minutes. Because the MacCAT-CA
was not normed on persons with IQ estimates below 60 (see
below) it is not recommended for use with this population.
From 1996 to 1998, we conducted a multistate study, funded
by the National Institute of Mental Health, to establish the clini-
cal utility of the MacCAT-CA. In this article, we report the
findings regarding the psychometric properties and construct
validity of the instrument. Our clinical protocol, comprising the
MacCAT-CA and other measures relevant to the investigation of
the validity of the MacCAT-CA, was administered to 729 pre-
trial, fekmy defendants in eight states.
The selection of states as potential sites for data collection
was informed by two considerations. First, state systems for
providing forensic evaluation services vary considerably in their
organizational structure. For example, the typology of forensic
service delivery systems proposed by Grisso, Cocozza, Stead-
man, Fisher, and Greer (1994) consisted of five system types
(i.e., traditional inpatient, modified-traditional, community-
based, private practitioner, and mixed evaluation systems) as
well as a ' 'not classified'' category. Second, states differ in the
presence or absence of specific training requirements for foren-
sic examiners and in the availability of state-sponsored training
programs designed to meet those training requirements (Farkas,
DeLeon, & Newman, 1997). Both of these variables-—organiza-
tional structure for forensic service delivery and the existence
of state-sponsored forensic training—could influence the
thresholds for determining incompetence that are applied across
states. Therefore, gathering data from each system type and
from states with and without forensic examiner training was
necessary to inform judgments about the generalizability of
findings on the basis of the MacCAT-CA. To provide representa-
tive national data, we selected individual states that represented
each of the six categories identified by Grisso et al. (1994). In
addition, where possible, within types of service delivery sys-
tems, states with and without forensic examiner training were
selected.
Method
Eligibility Criteria ami Recruitment Procedures
Data were collected- from three groups of felony defendants: (a)
defendants admitted to forensic psychiatric units after being adjudicated
incompetent to proceed (HI); (b) defendants in jail who were receiving
treatment for mental health problems but who were presumed competent
(JT); and (c) randomly selected jail inmates who were presumed com-
petent (JU). Participants were recruited from six states representing
categories in the Grisso et al. (1994) typology: Washington (tradi-
tional); Louisiana (private practitioner); Oklahoma (community-
based); South Carolina (modified-traditional); Wisconsin (mixed); and
Utah (not classified). According to the survey by Farkas et al. (1997),
none of these six states reported the availability of extensive, state-
sponsored forensic evaluator training programs. Therefore, two addi-
tional states (Michigan [modified-traditional] and Alabama [commu-
nity-based]) that provide systematic training for forensic examiners also
were included.
Both male and female felony defendants were eligible for participation
in the study; participants were paid $10 for their time. Potential partici-
pants had to be English-speaking, between the ages of 18 and 65, without
a diagnosis suggesting organicity, and they had to have a prorated Wechs-
ler Adult Intelligence Scale-Revised (WAIS-R; Wechsler, 1981) Full
Scale IQ of 60 or greater. For the HI sample, participants were recruited
within 14 days of hospital admission to minimize the effects of treatment
on their performance on the research measures. All but 4(1%) partici-
pants were assessed within 10 days of admission.
Potential HI participants were identified as they entered the forensic
hospital for competence restoration treatment; essentially all eligible
MAcCAT-CA 437
participants were approached by research assistants. Jail inmate partici-
pants were recruited from lists of defendants provided by cooperating
public defender offices in each state. Jail inmates receiving mental health
services (potential JT participants) were identified by the research assis-
tant with the assistance of jail mental health staff and randomly selected
for invitation to participate in the study. The remaining inmates were
randomly selected for participation as members of the JU group.
Potential participants were approached by a research assistant and
informed about the study using informed consent procedures approved
by a university institutional review board. Those expressing an interest
in participating were screened by the research assistant with respect to
their competence to consent to the research project and their cognitive
functioning (see below). After participants completed the research pro-
tocol, $10 was placed in their institutional accounts. Refusal rates for
the HI, JT, and JU samples were 22%, 9%, and 9%, respectively. Whereas
the 22% refusal rate is higher than desirable, it is lower than is often
found in studies of acutely mentally ill people. For example, in studies
of competence to consent in civil contexts (consent to voluntary hospital-
ization; consent to treatment), Appelbaum, Mirkin, and Bateman (1981)
and Grisso, Appelbaum, and Hill-Fbtouhi (1997) reported refusal rates
of 33% and 43%, respectively.
Measures
Measures gathered on each participant consisted of historical and
demographic information, estimated IQ, measures of current adjustment
and psychopathology, the MacCAT-CA, and a treating clinician's global
estimate of competence to proceed (HI participants only). Measures
were administered in the order listed below.
Estimate of cognitive-intellectual functioning. To estimate partici-
pants' cognitive abilities and screen out potential participants with mark-
edly limited abilities, we administered to all potential participants the
Information and Picture Completion subtests of the WAIS-R. Perfor-
mance on these two scales is highly correlated with WAIS-R Rill Scale
IQ scores (r = .86 and r = .92, respectively; Kaufman, Ishikuma, &
Kaufman-Packer, 1991). Because cognitive functioning can be impaired
by underlying mental disorder, this score is best considered as an index
of current functioning rather than a measure of baseline intellectual
capacity. Thirty-six potential HI participants were excluded because they
obtained WAIS-R Rill Scale IQ estimates of below 60, whereas 8 poten-
tial JT participants and 3 potential JU participants were excluded on
these grounds.
Background interview data. Background information on the partici-
pants was gathered through a brief structured interview. Information
obtained included history of psychiatric treatment, arrest history, educa-
tional level, and highest socioeconomic status achieved.
Brief Psychiatric Rating Scale-Anchored (SPSS). Scored after the
administration of a 15-minute interview, the BPRS (Overall & Gorham,
1962) provides a reliable and valid estimate of the presence and severity
of psychopathology. We used the anchored version (Woerner, Man-
nuzza, & Kane, 1988) in which the severity of 18 symptoms of mental
disorder (e.g., depression and orientation) are rated on a 7-point, an-
chored Likert scale. Ratings on the 18 items are summed to provide a
global measure of current psychopathology, and four subscale scores
that provide indices of psychoticism, depression, emotional withdrawal,
and hostility can be calculated (Overall & Porterfield, 1963).
MMPI-2 Psychoticism scale. The 25-item Psychoticism scale was
derived from the MMPI-2 item pool. Based on a five-factor model of
personality especially relevant for pathological populations (Harkness &
McNulty, 1994), the Psychoticism scale measures the gross degree of
correspondence between an individual's internal models of reality and
the external social and physical world (Harkness, McNulty, & Ben-
Porath, 1995). Estimates of internal consistency (coefficient alpha) rang-
ing from .70 to .84 have been reported across diverse nonclinical and
clinical samples. The scale also yields appropriate mean differences
across these samples, as well as expected patterns of correlations with
Table 1
Demographics by Group
Variable
AgeMSD
SexMale (%)Female (%)
EthnicityWhite (%)Non-White (%)
Education
MSD
SESM
SD
JU
31.339.18
93.96.1
35.065.0
11.261.98
56.1010.96
JT
32.469.35
88.811.2
50.2
49.8
11.352.51
56.2512.45
HI
36.0110.66
89.410.6
49.850.2
11.44
2.68
57.9413.13
Note. JU = unscreened jail inmates; JT = jail inmates receiving mentalhealth services; HI = hospitalized incompetent defendants; SES = socio-economic status.
measures of theoretically relevant and divergent constructs. This measure
was added to the research protocol after data collection already had
been initiated; nevertheless, data were available for 89% (n = 647) of
the participants.
MacCAT-CA. The MacCAT-CA, the clinical instrument derived
from the MacSAC-CD research instrument, was administered to all
participants.
Chan data. Relevant background information on the participants
was gathered, in part, by way of institutional chart review. A review of
the hospital chart or jail records provided information on participants'
age, race, admission date, and current psychiatric status including diag-
nosis, treatment status, and medications.
Clinical judgment regarding competence. After HI participants
completed the research protocol, a hospital mental health professional
familiar with the participant's mental state and adjustment offered a
global judgment of the participant's competence to proceed using an
anchored 6-point Likert scale (1 = clearly incompetent, 6 = clearly
competent}.
Results
Demographic Data
Table 1 provides descriptive information about the hospital-
ized and jailed defendant samples recruited for this study (for
ffl, n = 283; for JT, n = 249; for JU, n = 197).' Participants
were predominantly male (90%), and gender was distributed
similarly across the HI, JT, and JU samples, x2(2, N = 729)
= 3.90, p = .14. Although there was a balance of White and
non-White participants in the study, they were somewhat dispro-
portionately distributed across the three samples, X2(2, N =
729) = 12.99, p < .002. HI participants as a group were signifi-
cantly older than participants in the JU sample, f(724) = 25.01,p < .001, and the JT sample, f(724) = 16.40, p < .001.
The mean years of education reported by participants was
11.36 (SD = 2.45), and level of education did not vary signifi-
1 Different degrees of freedom for the analyses that follow reflect
missing data on these variables for a small number of participants.
438 OTTO ET AL.
Table 2
Mental Status Characteristics by Group
JU JT HI
Variable M SD M SD M SD Univariate F df
Estimated WAIS-R Full Scale IQ 85.23 11.99 85.49 14.61 83.21 14.05 2.17 2,726
MMPI-2 Psychoticism 63.32 14.13 70.48 18.40 74.68 19.64 20.93* 6,646BPRS total 29.16 6.99 35.95 9.07 38.80 10.04 68.26* 2,726BPRS Psychoticism 3.41 1.16 4.% 2.61 7.55 3.72 131.96* 2,722BPRS Depression 8.70 3.96 10.76 4.55 8.20 3.99 24.48* 2,722BPRS Hostility 5.00 2.26 6.15 2.69 6.74 3.09 23.53* 2,722BPRS Emotional Withdrawal 3.91 1.50 4.62 2.33 5.40 2.95 22.27* 2,722
Note. JU = unscreened jail inmates; JT = jail inmates receiving mental health services; HI = hospitalized
incompetent defendants; WAIS-R = Wechsler Adult Intelligence Scale-Revised; MMPI-2 = MinnesotaMultiphasic Personality Inventory-2; BPRS = Brief Psychiatric Rating Scale-Anchored.
"p < .001.
cantly across groups. Using Hollingshead and Redlich's (1958)
two-factor index of social position, participants identified the
highest level of employment ever held. The mean scores (see
Table 1) equate with Social Class 4 (e.g., clerical and sales
workers, technicians). The group means did not differ signifi-
cantly in terms of highest socioeconomic status attained by
participants.
The participants in the various samples reported similar
amounts and types of involvement with the criminal justice
system. The mean number of estimated misdemeanor and felony
arrests for the total sample was 5.24 (SD = 9.52) and 1.91 (SD
= 2.79), respectively.2 There were no significant differences
between the groups in terms of their estimated number of felony
arrests, F(2,721) = .35, p = .71, or misdemeanor arrests, F(2,
714) = 1.11, p = .33.
Clinical Measures
Group characteristics on measures of psychopathology and
cognitive functioning are summarized in Table 2. The three
samples did not differ in terms of their estimated current cogni-
tive-intellectual functioning. There was a significant group ef-
fect as measured by the MMPI-2 Psychoticism scale, with the
HI sample demonstrating greater levels of psychopathology than
both the JT, r(646) = 2.55, p = .011, and JU, t(646) = 6.46,
p < .001, samples. Similarly, participants in the HI sample
obtained higher scores on the BPRS total score than the JT and
JU samples, rs(726) = 3.65 and 11.58, respectively, as well as
higher scores on the BPRS Psychoticism, fs(722) = 10.49 and
15.74, Hostility, is(722) = 2.61 and 6.85, and Emotional With-
drawal, B(722) = 3.69 and 6.61, subscales, respectively (all
ps < .01). Similar to findings in the MacSAC-CD field study
(Hoge et al., 1997), HI participants were less depressed than
JT participants, f(722) = 6.75, p < .001, but did not differfrom JU participants, r(722) = 1.30, p = .19.
MacCAT-CA Performance and Comparisons
Internal consistency. Cronbach's alpha and the mean and
range of interitem correlations for each of the MacCAT-CA
measures are provided in Table 3. These values serve as indices
of scale reliability and item homogeneity, respectively (Fiske,
1971). The alphas ranged from .81 (Reasoning) to .85 (Under-
standing) to .88 (Appreciation), indicating good internal consis-
tency for these measures. These estimates are comparable to
those obtained using the MacSAC-CD research measure (Hoge
et al., 1997). The mean interitem correlations were .36, .42, and
.54 for Reasoning, Understanding, and Appreciation, respec-
tively, indicating appropriate homogeneity of item content for
all three measures. Although the Appreciation measure has the
fewest items, its relatively greater item homogeneity enabled it
to produce the highest estimate of scale reliability.
Interrater reliability. Tb evaluate interrater reliability, we
drew a sample of 48 protocols from the database (2 HI, 2 JT,
and 2 JU from each state). Scoring assigned by the original
research assistant who completed the protocol was removed,
and 42 protocols were mailed to each research assistant for
rescoring.3 One protocol was not returned; hence, the interrater
reliability analysis was based on 47 total cases scored by each
of the eight research assistants. Three analyses of variance
(ANCTVAs) were conducted on the total scores from the Under-
standing, Reasoning, and Appreciation measures, respectively,
to compute an intraclass correlation for each measure. In each
of the ANOVks, rater was treated as a random variable (see
Shrout & Fleiss, 1979, Model 2). As several authors have em-
phasized, the intraclass correlation is superior to the traditional
product-moment correlation as an index of reliability (e.g., see
Bartko & Carpenter, 1976). Interscorer reliability for the three
measures as estimated by this procedure ranged from very good
to excellent, with intraclass R = .75 for Appreciation, .85 for
Reasoning, and .90 for Understanding (see Table 4). Although
explicit scoring guidelines are provided for each item on each
measure, the scoring criteria are more rigorous for the Under-
standing items, and less so for the Appreciation measure (where
a judgment about the plausibility of a defendant's "reasons" is
required). The pattern of intraclass correlations for the three
2 One participant who reported 1,000 misdemeanor arrests was de-
leted from this analysis.3 Research assistants did not rescore the six protocols they originally
completed that were subsequently mailed to the remaining research assis-
tants. Therefore, only 42 new protocols were mailed to them.
MACCAT-CA 439
Table 3
Internal Consistency ofMacArthur Competence Assessment
Tool— Criminal Adjudication Measures
Interitem correlation
Measure M Range
UnderstandingReasoningAppreciation
.85
.81
.88
.42
.36
.54
.31 -.53
.21 -.53
.4S-.62
measures is consistent with the degree of rigor in the scoring
guidelines.
Correlations Between MacCAT-CA Scales and Clinical
Measures
Support for the construct validity of the MacCAT-CA is found
in the pattern of correlations between MacCAT-CA measures
and select clinical variables. Ideally, measures of competence-
related abilities should correlate positively with estimated IQ,
but they should correlate negatively with measures of psychopa-
thology, particularly with measures of psychoticism. As revealed
in Table 5, this is the pattern of relationships obtained in the
present study, providing evidence of convergent validity for the
MacCAT-CA.
Consistent with the findings of Hoge et al. (1997) with the
research instrument (MacSAC-CD), the MacCAT-CA measures
correlate negatively with disturbed thinking as measured by the
Table 4
Interrater Reliability ofMacArthur Competence Assessment
Tool-Criminal Adjudication Measures
Measure/Item no. ICC
Understanding1234567g
Reasoning9
10111213141516
Appreciation171819202122
.90
.82
.88
.80
.88
.82
.92
.90
.86
.85
.93
.75
.87
.34
.90
.74
.72
.45
.75
.73
.66
.57
.70
.53
.42
Table 5
Correlations Between MacArthur Competence Assessment
Tool-Criminal Adjudication and Clinical Measures
Variable Understanding Reasoning Appreciation
Estimated WAIS-RFull Scale IQ
MMPI-2 PsychoticismBPRS TotalBPRS Psychoticism
BPRS DepressionBPRS HostilityBPRS Emotional
WithdrawalClinician Ratings of
Competence'
.41-.33-.23-.40
.18-.01
-.34
.36
.34-.31-.29-.48
.18-.10
-.27
.42
.14-.21
-.36-.52
.18-.28
-.19
.49
Note. WAIS-R = Wechsler Adult Intelligence Scale-Revised; MMPI-2 = Minnesota Multiphasic Personality Inventory-2; BPRS = BriefPsychiatric Rating Scale-Anchored, p < .001 for all correlations exceptHostility-Understanding correlation (p = .71) and Hostility-Reasoningcorrelation (p = .007)." Correlations for clinician ratings are based only on data from the 283
hospitalized incompetent participants.
BPRS Psychoticism subscale and MMPI-2 Psychoticism scale,
and with psychopathology generally as measured by the BPRS
total score. Present cognitive ability, as measured by perfor-
mance on select subtests of the WMS-R, correlated positively
with performance on the MacCAT-CA. Similar to results ob-
tained with the research version of the MacCAT-CA (the Mac-
SAC-CD; Hoge et al., 1997), presumed competent jail defen-
dants showed higher levels of depression as measured by the
BPRS than hospitalized incompetent defendants.
Global ratings of the HI participants' competence to proceed
were offered by forensic clinicians who were knowledgeable
about their clinical conditions. As expected, clinicians' ratings
of competence were moderately correlated with performance on
the MacCAT-CA (Understanding, r = .36; Reasoning, r = .42;
Appreciation, r = .49) presumably due, in part, to the restriction
in range (i.e., only hospitalized incompetent defendants received
such clinician ratings). These results provide evidence of con-
current validity for the MacCAT-CA.
Sample differences. Table 6 provides the mean and the stan-
dard deviation of each sample for each of the three MacCAT-CA
psycholegal ability measures: Understanding, Reasoning, and
Appreciation. These means differed significantly in a multivari-
ate analysis of covariance (MANCOV\) in which type of pre-
trial forensic evaluation service delivery system and mandated
training requirements were entered as covariates, multivariate
F(18, 2,160) = 9.79, p < .001." For each measure, planned
orthogonal comparisons revealed that the HI sample scored sig-
nificantly lower (i.e., were more impaired regarding their cam-
Note. ICC = intraclass correlation.
' Although these two variables are important to assess the generaliz-
ability of MacCAT-CA scores, they are potential confounds that might
obscure within-state group differences (HI vs. JT and HI vs. JU) across
states with differing organizational structures and training requirements.
It should be noted, however, that not including these covariates in this
analysis resulted in essentially similar results, with defendants in the HI
group obtaining significantly lower scores than those in the JT and JU
groups across all three MacCAT-CA measures.
440 OTTO ET AL.
Tible 6
MacArthur Competence Assessment Tool-Criminal Adjudication Descriptive Statistics by Group and
Multivariate Analysis of Covariance (MANCOVA) Results
JU(n = 197)
Measure
UnderstandingReasoning
Appreciation
M
12.5013.27
11.44
SD
3.082.641.01
IT(n = 249)
M
12.56
12.9011.02
SD
3.252.901.63
HI(n = 283)
M
9.119.337.89
SD
4.194.314.01
HI(R)(n = 90)
M
10.7811.149.87
SD
3.41
3.753.00
HI(C)(n = 170)
M
8.14
8.236.58
SD
4.174.234.08
Univariate F'
(2, 720)
79.1295.92
121.07
Univariate F"
(3,696)
73.3588.27
129.08
Note. Clinician ratings of competence were unavailable for 23 hospitalized incompetent (M) defendants, who were not included in the secondMANCOVA. JU = unscreened jaU inmates, JT = jail inmates receiving mental health services; HI(C) = clinically affirmed incompetent ffl defendants;ffl(R) = residual HI defendants. All F values significant at p < .001.• Univariate F for MANCOVA comparing JU, JT, and HI samples. " Univariate F for MANCOVA comparing JU, JT, HI(R), and HI(C) samples.
petence-related abilities) than the JU sample [Understanding,
((720) = 11.37; Reasoning, ((720) = 12.72; Appreciation,
((720) = 13.74, allps < .001]. The planned comparisons con-
trasting the means from the HI and JT samples provide a more
stringent test of group discrimination. As expected, and lending
support for the claim that the MacCAT-CA can discriminate
between competent and incompetent mentally ill defendants, the
HI group mean was significantly lower than the JT group mean
for each MacCAT-CA measure [Understanding, ((720) = 10.04;
Reasoning, ((720) = 10.66; Appreciation, /(720) = 12.84, all
ps < .001].
This MANOTvA was repeated after separating the HI sample
into two groups on the basis of clinicians' ratings of compe-
tence. The HI(C) sample comprised those HI participants who
were clinically confirmed as incompetent on the basis of inde-
pendent ratings provided by a treatment-team clinician at the
time that the research protocol was administered. The HI(R)
sample comprised the residual HI participants whose clinician
ratings reflected a clinical impression that the individuals were
at least marginally competent at the time of protocol administra-
tion. A significant multivariate effect was obtained, F(18,
2,088) = 9.82, p < .001, for this analysis. Planned comparisons
indicated that the HI(C) sample obtained lower mean scores
on each of the MacCAT-CA measures when compared to the
HI(R) group, rs(696) = 6.05, 6.98, and 9.93 for Understanding,
Reasoning, and Appreciation, respectively; all ps < .001. Fur-
• thermore, despite obtaining higher scores than the HI(C) sam-
ple, the HI(R) sample obtained lower mean scores on each of
the MacCAT-CA measures when compared with both the JU
sample, (s(696) = 5.19, 5.53, and 4.70 for Understanding, Rea-
soning, and Appreciation, respectively; all ps < .001, and the
JT sample, rs(696) - 3.88, 3.66, and 3.62 for Understanding,
Reasoning, and Appreciation, respectively; all ps < .001, sug-
gesting greater impairment in competence-related abilities (see
Table 6).
We note that we did not include race as a covariate in these
analyses, although race does have a statistically significant bivar-
iate correlation with Understanding (—.15) and Reasoning
(—.09).' Partial correlations, controlling for cognitive function-
ing (i.e., prorated IQ scores), indicated nonsignificant correla-
tions between race and MacCAT-CA performance. Thus, we
attribute the difference in MacCAT-CA scores between White
and non-White participants to differences in current cognitive
functioning.
The preceding analyses documented the statistical signifi-
cance of the differences between groups of participants on the
MacCAT-CA measures. Table 7 provides information regarding
the magnitude of those between-group differences. The measure
of effect size used to estimate group differences was Cohen's
(1977) d, which is defined as follows:
d = -
where Xi and X2 are the means of the two groups being com-
pared, and s is the pooled within-group standard deviation. Thus,
Cohen's d expresses the difference between means relative to
within-group variation. As the first two columns of Table 7 show,
comparisons involving all hospitalized incompetent defendants
yielded effect sizes of approximately 1 SD across the three
MacCAT-CA measures. Further, as can be seen in the third and
fourth columns of the table, substantially larger effect sizes were
obtained from comparisons involving the clinically affirmed
incompetent defendants. In every case, the between-group dif-
ferences exceeded 1 SD, ranging from almost 1.25 to more than
1.5 SDs. In contrast, more moderate effects were observed in
comparisons involving incompetent defendants who were not
clinically affirmed as such. The latter effect sizes clustered
around one half of a standard deviation.
Discussion
We have described the derivation of a new clinical tool for
use in the assessment of criminal defendants' adjudicative com-
petence, the MacCAT-CA. We also have presented research
findings concerning its psychometric properties and potential
clinical utility in evaluating psycholegal abilities relevant tocompetence determinations. These results suggest that the Mac-
CAT-CA measures of Understanding, Reasoning, and Apprecia-
tion have good interrater reliability (intraclass Rs ranging from
.75 to .90) and strong internal consistency (as > .80, mean
5 It is noted, however, that race only accounted for a small percentageof the variance associated with the Understanding (2%) and Reasoning(1%) scores.
MACCAT-CA 441
Table 7
Effect Sizes (Cohen's d)for Comparisons Between Samples of Jailed Defendants
and Hospitalized Incompetent Defendants
Comparison
Measure JU-ffl JT-HI JU-HI(C) JT-HI(C) JU-Hl(R) JT-HI(R)
UnderstandingReasoningAppreciation
.901.061.13
.91
.961.00
1.201.451.69
1.211.331.54
.54
.70
.84
.54
.56
.55
Note. JU = unscreened jail inmates; JT = jail inmates receiving mental health services; HI = hospitalizedincompetent defendants; HI(C) = clinically affirmed incompetent HI defendants; HI(R) = residual HIdefendants.
interitem rs ranging from .36 to .54), as well as evidence of
construct validity (expected patterns of correlation with mea-
sures of cognitive ability, psychopathology, and clinical judg-
ments of degree of impaired competence).
The psychometric properties of the MacCAT-CA scales ob-
served in this study compare favorably with those reported for
other measures of adjudicative competence, such as the 22-item
CST (Lipsitt et al., 1971) and the 21-item GCCT-Mississippi
State Hospital revision (GCCT-MSH; Nicholson, Briggs, &
Robertson, 1988). Although the individual MacCAT-CA scales
contain fewer items, they produced estimates of internal consis-
tency reliability that are similar to those reported for the other
competency measures. In addition, differences between compe-
tent and incompetent defendants on the MacCAT-CA scales are
comparable to those obtained with the CST and GCCT-MSH.
Reanalysis of data reported in a quantitative review comparing
competent and incompetent defendants (Nicholson & Kugler,
1991) yielded a mean effect size of 1.05 SDs across six studies
(n = 429) involving the CST and a mean effect size of 1.20
SDs across four studies (n = 539) involving the GCCT. In the
present study, effect sizes for the MacCAT-CA scales ranged
from almost 1 SD for comparisons with all incompetent defen-
dants to more than 1.5 SDs for comparisons with clinically
affirmed incompetent defendants. Interesting to note, the Under-
standing scale, which covers information similar to that assessed
by other knowledge-based competency measures (e.g., the char-
acteristics of criminal prosecution and defense, the nature of
criminal charges, the consequences of conviction), tended to
yield smaller effects than either the Reasoning or Appreciation
scales, which tap domains of competence-related abilities not
represented on other measures.
The MacCAT-CA will be available to the field after develop-
ment of clinical interpretive norms, currently in progress. Norms
will be derived using this eight-state sample of 729 defendants,
providing data about the distribution of MacCAT-CA perfor-
mance among competent and incompetent criminal defendants.
These norms will serve as guideposts to inform individualized
judgments regarding the competence of defendants, more spe-
cifically, defendants' abilities to understand and reason about
legally relevant information, and to appreciate the significance
of such information in the context of their own legal situation.
Although the MacCAT-CA is useful in evaluating criminal
defendants whose competence has been questioned, it is not
offered as a measure of legal competence per se. That is, we
do not anticipate the derivation of a single optimal cutoff score
for the empirical classification of a defendant as competent
or incompetent. Other psycholegal abilities are relevant to the
ultimate determination of competence (e.g., memory for specific
events related to one's own case; coherency of speech) and a
court may consider other nonpsychological factors (e.g., legal
complexity of the case) in making the dichotomous judgment
about "competence." As a result, no single measure, no matter
how nuanced, can encompass all of the •potentially relevant fac-
tors. Additionally, the MacCAT-CA includes no scales designed
to assess the examinee's test-taking set. Some defendants may
be motivated to appear to lack capacity when in fact, they do
not. As is the case with any forensic evaluation, the examiner
must consider the possibility of exaggerated and fabricated
deficits or psychopathology. When such deception is suspected,
it is necessary to use other measures designed to assess the
defendant's test-taking set.
Nevertheless, the MacCAT-CA, through its measurement of
understanding, reasoning, and appreciation capacities, offers a
more systematic and differentiated analysis of the psycholegal
abilities relevant to adjudicative competence than do existing
measures. Grounded in a comprehensive theory of adjudicative
competence, the MacCAT-CA offers the additional features of
standardized administration and criterion-related scoring that
are missing from other interview-based measures. Used as part
of a comprehensive evaluation that includes a relevant history
(e.g., psychiatric history), current mental-status evaluation, and
inquiry about case-specific memories, the MacCAT-CA should
enhance the thoroughness and quality of clinical evaluations of
adjudicative competence.
References
Appelbaum, P. S., Mirkin, S. A., & Bateman. A. L. (1981). Empirical
assessment of competency to consent to psychiatric hospitalization.
American Journal of Psychiatry, I3S, 1170-1176.
Barnard, G., Thompson, J. W., Freeman, W. C., Robbins, L., Gies, D.,
& HanKns, G. C. (1991). Competency to stand trial: Description
and initial evaluation of a new computer-assisted assessment tool
(CADCOMP). Bulletin of the American Academy of Psychiatry and
Law, 19, 367-381.
Bartko, J. J., & Carpenter, W. T. (1976). On the methods and theory of
reliability. Journal of Nervous and Mental Disease, 163, 307-317.
Bonnie, R. J. (1992). The competence of criminal defendants: A theoret-
ical reformulation. Behavioral Sciences and the Law, W, 291-316.
Bonnie, R. J. (1993). The competence of criminal defendants: Beyond
Dusky and Drope. University of Miami Law Review, 47, 291-316.
442 OTTO ET AL.
Cohen, J. (1977). Statistical power analysis for the behavioral sciences(Rev. ed.). New York: Academic Press.
Farkas, G. M., DeLeon, P. H., & Newman, R. (1997). Sanity examinercertification: An evolving national agenda. Professional Psychology:Research and Practice, 28, 73-76.
Fiske, D. W. (1971). Measuring the concepts of personality. Chicago:Aldine.
Golding, S. L., Roesch, R., & Schreiber, J. (1984). Assessment andconceptualization of competency to stand trial: Preliminary data onthe Interdisciplinary Fitness Interview. Law and Human Behavior, 9,321-334.
Grisso, T. (1991). Clinical assessment for legal decision making: Re-search recommendations. In S. A. Shah & B. D. Sales (Eds.), Lawand mental health: Major developments and research needs. (DHHSPublication No. ADM 91-1875, pp. 49-80). Rockville, MD: NationalInstitute of Mental Health.
Grisso, T., & Appelbaum, P. S. (1998). Assessing competence to consentto treatment. New York: Oxford University Press.
Grisso, X, Appelbaum, P. S., & HUl-Fotouhi, C. (1997). The MacCAT-TA clinical tool to assess patients' capacities to make treatment deci-sions. Psychiatric Services, 48, 1415-1419.
Grisso, X, Appelbaum, P. S., Mulvey, E. P., & Fletcher, K. (1995).The MacArthur treatment competence study. II: Measures of abilities
related to competence to£onsent to treatment. Law and Human Behav-ior, 19, 127-148.
Grisso, X, Cocozza, J., Steadman, H., Fisher, W., & Greer, A. (1994).The organization of pretrial forensic evaluations services: A nationalprofile. Law and Human Behavior, 18, 377-394.
Harkness, A. R., & McNulty, J. L. (1994). The personality psychopa-thology five (PSY-5): Issues from the pages of a diagnostic manualinstead of a dictionary. In S. Strack & M. Lorr (Eds.), Differentiatingnormal and abnormal personality. New %rk: Springer.
Harkness, A. R., McNulty, J. L., & Ben-Porath, Y. S. (1995). The per-sonality psychopathology five (PSY-5): Constructs and MMPI-2scales. Psychological Assessment, 7, 104-114.
Hoge, S. K., Bonnie, R. J., Poythress, N. G., Monahan, J., & Eisenberg,M. (1997). The MacArthur Adjudicative Competence Study: Devel-opment and validation of a research instrument. Law and HumanBehavior, 21, 141-179.
Hollingshead, A. R., & Redlich, F. C. (1958). Social class and mentalillness. New York: Guilford Press.
Kaufman, A. S., Ishikuma, X, & Kaufman-Packer, J. L. (1991). Amaz-ingly short forms of the WMS-R. Journal of Psychoeducational As-sessment, 9, 4-15.
Laboratory of Community Psychiatry (1974). Competency to stand trial
and mental illness. (DHEW Publication No. ADM 74-103). Rock-ville, MD: Department of Health, Education, & Welfare.
Lipsitt, P., Lelos, D., & McGarry, A. L. (1971). Competency for trial:A screening instrument. American Journal of Psychiatry, 128, 105-109.
Melton, G. B., Petrila, J., Poythress, N., & Slobogin, C. (1997). Psycho-logical evaluations for the courts: A handbook for mental healthprofessionals and lawyers (2nd ed.). New Tfork: Guilford Press.
Menzies, R. J., Webster, C. D., Roesch, R., Jensen, F., & Eaves, D.(1984). The Fitness Interview Test: A semi-structured instrument forassessing competency to stand trial. Medicine and Law, 3, 151-162.
Nicholson, R. (1992, August). Defining and assessing criminal compe-tencies. Paper presented at the 100th Annual Convention of the Ameri-can Psychological Association, Washington, DC.
Nicholson, R. A., Briggs, S. R., & Robertson, H. C. (1988). Instrumentsfor assessing competency to stand trial: How do they work? Profes-sional Psychology: Research and Practice, 19, 383-394.
Nicholson, R. A., & Kugler, K. E. (1991). Competent and incompetentcriminal defendants: A quantitative review of comparative research.Psychological Bulletin, 109, 355-370.
Overall, J., & Gorham, D. (1962). The Brief Psychiatric Rating Scale.Psychological Reports, 10, 799-812.
Overall, J., & Porterfield, J. (1963). Power vector method of factoranalysis. Psychometrika, 28, 415-422.
Poythress, N. G., Hoge, S. K.. Bonnie, R. J., Monahan, J., Eisenberg,M., & Feucht-Haviar, T. (1998). The competence related abilities ofwomen criminal defendants. Journal of the American Academy ofPsychiatry and Law, 26, 215-222.
Poythress, N. G., Nicholson, R., Otto, R. K., Edens, J. F., Bonnie, R. J.,Monahan, J., & Hoge, S. K. (in press). Manual for the MacArthurCompetence Assessment Tool- Criminal Adjudication (MacCAT-CA).Odessa, FL: Psychological Assessment Resources.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses inassessing rater reliability. Psychological Bulletin, 86, 420-428.
Steadman, H. J., & Hartstone, E. (1983). Defendants incompetent tostand trial. In J. Monahan & H. J. Steadman (Eds.), Mentally disor-dered offenders: Perspectives from law and social science. New M>rk:Plenum.
Wechsler, D. (1981). The Wechsler Adult Intelligence Scale-Revised:Manual. San Antonio, TX: Psychological Corporation.
Wildman, R., Batchelor, E., Thompson, L., Nelson, F., Moore, J., Pat-terson, M., & DeLaosa, M. (1980). The Georgia Court CompetencyTest: An attempt to develop a rapid quantitative measure for fitnessfor trial. Unpublished manuscript, Forensic Services Division, CentralState Hospital, Milledgeville, Georgia.
Woerner, M. G., Mannuzza, S., & Kane, J. M. (1988). Anchoring theBPRS: An aid to improved reliability. Psychopharmacology Bulletin,24, 112-118.
MAcCAT-CA 443
Appendix
Legal Content of the MacCAT-CA Measures
A. Legal Content of the Understanding Measure1. Understanding the roles of defense attorney and prosecutor.2. Understanding both the act and mental elements of a serious
offense.3. Understanding the elements of a less serious offense.4. Understanding the role of a jury.5. Understanding the responsibilities of a judge at a jury trial.6. Understanding sentencing as a function of the severity of the
offense.7. Understanding the process of a guilty plea.8. Understanding the rights waived in pleading guilty.
B. Legal Content of the Reasoning Measure9. Reasoning about evidence suggesting self-defense.
10. Reasoning about evidence related to criminal intent.11. Reasoning about evidence of provocation.12. Reasoning about motivation for one's behavior.13. Reasoning about the potential impact of alcohol on one's
behavior.14. Capacity to identify information that might inform the decision
to plead guilty versus plead not guilty.
15. Capacity to identify both potential costs and potential benefits ofa legal decision (e.g., pleading guilty).
16. Capacity to compare one legal option (e.g., accepting a pleabargain) with another legal option (e.g., going to trial) in termsof advantages and disadvantages.
C. Legal Content of the Appreciation Measure17. Plausibility of defendant's beliefs about the likelihood of being
treated fairly by the legal system.18. Plausibility of defendant's beliefs about likelihood of being
helped by his/her lawyer.19. Plausibility of defendant's beliefs about whether to disclose case
information to his/her attorney.20. Plausibility of defendant's beliefs about likelihood of being found
guilty.21. Plausibility of defendant's beliefs about likelihood of being pun-
ished if found guilty.22. Plausibility of defendant's beliefs about whether to accept a plea
bargain.
Received April 23, 1998Revision received June 21, 1998
Accepted July 30, 1998 •
750 First Street, NE - Washington, DC 20002-424220̂ ?J36- S5
American Psychological Associat750 First Street, NEWashington, EC Z0002-424Z
75IM=ir5t Street, NE - Washington, DC 20002-4;
l ^ ^ « ^ a a l ^ ' ^ o d a i t : ^ W ^ d a : - > a s . ""™
n Psychological Associate ISO First Street, KB
shington, DC 20002-4242
nstruetlons to Publisher
of bonOi, mongagBi. or other HtuitUn of ttw p
eolation mgsi Be puDKOaa: It rnt