ORIGINAL ARTICLE

Predictors of Use of Evidence-Based Practices for Childrenand Adolescents in Usual Care

Charmaine K. Higa-McMillan • Brad J. Nakamura •

Ashley Morris • David S. Jackson • Lesley Slavin

� Springer Science+Business Media New York 2014

Abstract Practice data from 74 therapists providing public

mental health services to 519 youth ages 5–19 were exam-

ined. Multilevel modeling suggested child and therapist

characteristics predicted use of practices derived from the

evidence-base (PDEB) and use of practices with minimal

evidence support (PMES). Longer episode length predicted

greater receipt of PDEB; older youth, males, and youth in

out-of-home levels of care were more likely to receive

PMES; and youth receiving an evidence-based treatment

program were less likely to receive PMES. Professional

specialty and theoretical orientation significantly predicted

PDEB whereas therapist characteristics did not predict

PMES. Implementation implications are discussed.

Keywords Usual care � Evidence-based practices �Therapists � Implementation � Dissemination � Youth

Introduction

A growing body of research is dedicated to examining

effective ways to disseminate and implement programs

with empirical support (e.g., Aarons et al. 2011; Damsch-

roder et al. 2009; Proctor et al. 2009). This body of

research holds promise to improve mental health services

for children and adolescents. A complimentary approach to

understanding how to effectively implement new programs

is the study of practices already in place in usual care

settings (Garland et al. 2010a, b). The examination of usual

care practices can provide information about the overlap

and discrepancies between programs with demonstrated

effectiveness and what is already occurring in usual care. It

may also be helpful to identify the predictors of evidence-

based practice (EBP) usage in usual care settings so that we

can draw on these predictors when developing implemen-

tation programs.

In one of the first studies to date to compare the prac-

tices of usual care clinicians to the evidence-based litera-

ture, Garland and colleagues (2010a) found that for youth

referred for disruptive behavior problems, clinicians pro-

vided an array of treatment strategies, however, all strate-

gies were delivered at low intensity. Further, while some

strategies that were consistent with the evidence base for

disruptive behavior problems (e.g., psychoeducation,

problem-solving, positive reinforcement) were delivered

frequently, there were also a number of approaches derived

from the evidence-base that were rarely observed (e.g.,

role-play, homework). Consistent with these findings, two

recent studies on children receiving public mental health

services for traumatic stress and anxiety reported similar

findings (Borntrager et al. 2013a; Higa-McMillan et al.

2011, respectively).Both studies found that exposure, the

most commonly occurring practice among evidence-based

treatment protocols for youth with anxiety and traumatic

stress, was reported as being used infrequently. Borntrager

et al. (2013a) reported that youth receiving out-of-home

services with a diagnosis of post-traumatic stress disorder

Parts of this research study were presented in May 2013 at the Seattle

Implementation Research Conference in Seattle, WA.

C. K. Higa-McMillan (&) � A. Morris

Department of Psychology, University of Hawaii at Hilo, 200 W.

Kawili St., Hilo, HI 96720, USA

e-mail: higac@hawaii.edu

B. J. Nakamura

University of Hawaii at Manoa, Honolulu, HI, USA

D. S. Jackson � L. Slavin

Child and Adolescent Mental Health Division, Hawaii

Department of Health, Honolulu, HI, USA

123

Adm Policy Ment Health

DOI 10.1007/s10488-014-0578-9

(PTSD) were less likely than youth without a diagnosis of

PTSD to receive practices derived from the evidence-base

(PDEB). Both studies found that as youth age increased,

clinicians reported greater use of PDEB. On the other hand,

Higa-McMillan et al. (2011) found that youth with a

comorbid diagnosis of a disruptive behavior disorder in

addition to primary anxiety or who were receiving an

evidence-based service that is targeted to reduce DBDs

(i.e., Multisystemic Therapy, Multidimensional Treatment

Foster Care, or Functional Family Therapy) were less

likely to receive PDEB for anxiety.

Equally as important to understanding the practices that

usual care practitioners use (and do not use) are the prac-

titioner characteristics that predict use of EBPs. In a fol-

low-up study of Garland et al.’s (2010a) observational data

of practices in usual care, Brookman-Frazee et al. (2010)

examined child, family and therapist characteristics that

predicted greater use of PDEB. While they found several

child and family characteristics predicted greater use of

PDEB, the only therapist characteristics that predicted

greater use of PDEB were theoretical orientation and

clinical experience. Specifically, they found that therapists

with cognitive-behavioral or behavioral orientations pro-

vided more PDEB than therapists who self-identified as

eclectic or with other orientations and therapists with fewer

months in practice provided more PDEB than those with

more months in practice. Other therapist variables such as

license status, position, and training discipline (i.e., mar-

riage and family therapy, psychology, social work) did not

predict usage patterns in usual care.

In addition to therapist background characteristics, other

therapist- and organizational-level characteristics such as

therapist knowledge of EBPs, therapist attitudes towards

EBPs and organizational support for EBPs may also be

important predictors of EBP implementation in usual care

settings. Indeed, a number of theories in Dissemination and

Implementation Science have identified multiple factors

contributing to the successful implementation of a new

innovation. In particular, Damschroder and colleagues

(2009) proposed the Consolidated Framework for Imple-

mentation Research (CFIR), a comprehensive analysis of

the common factors among existing implementation theo-

ries. The CFIR includes five major domains: the inter-

vention itself, the inner setting, the outer setting, the

process by which implementation is accomplished, and the

individuals involved. With regard to characteristics of

individuals, the CFIR identifies five characteristics com-

mon across the field including knowledge and beliefs about

the intervention, self-efficacy, individual stage of change,

individual identification with organization, and personal

attributes. Most research to date has focused on knowledge

and beliefs about the intervention with less research

examining the latter four variables highlighted in the CFIR.

For example, in a study of 29 school mental health

providers, Stephan and colleagues (2012) examined self-

reported knowledge (none to significant) and frequency of

use (never to frequently) of PDEB for attention deficit-

hyperactivity disorder, disruptive behavior disorders,

depression, and anxiety and compared these to interviewer-

rated quality of treatment for youth depression. They found

that self-rated knowledge scores were significantly corre-

lated with self-reported frequency of use of these practices.

They also found that whereas knowledge of common

practices derived from the evidence base were significantly

correlated with interviewer-rated quality of depression

treatment, frequency of use of these practices was not

significantly correlated with interviewer-rated quality of

treatment for depression.

In another study of 197 children’s mental health care

providers, Jensen-Doss et al. (2009) found that positive

attitudes towards EBPs significantly predicted higher self-

reported usage of such practices and that positive provider

attitudes towards EBPs were predicted by higher ratings of

perceived colleague support, higher ratings of perceived

agency support, higher ratings of the quality of prior EBP

training, and lower ratings of institutional barriers. Simi-

larly, Nelson and Steele (2007) found in a survey of 214

providers across 15 states that positive and negative atti-

tudes mediated the relationship between how open the

clinician perceived their environment to be towards EBPs

and EBP use by the clinician. In other words, training in

EBPs and working in a setting that is open to EBPs may

impact attitudes towards EBPs, thereby increasing EBP

usage. Although these studies provide empirical support for

the theory that positive attitudes towards EBPs predict

greater use of EBPs in practice, they did not examine actual

use of EBPs with youth receiving mental health services

from their provider samples and they did not account for

child characteristics that may likely play a role in the

selection of practices by providers in usual care.

The current investigation attempted to replicate and

extend this body of work by examining predictors of

practices reported by usual care providers in a large

statewide publicly-funded mental health system for chil-

dren and adolescents. Using multilevel modeling (MLM),

this study examined both child- and provider-level vari-

ables that predicted greater use of PDEB as well as prac-

tices minimally found in evidence-based protocols. In

addition to examining child and service characteristics

(age, gender, diagnosis, functional impairment, service

type, and length of service) and provider background

characteristics (years trained, license status, professional

discipline, theoretical orientation), we also examined

characteristics of providers hypothesized to influence

greater use of EBPs including provider knowledge of and

attitudes towards EBPs.

Adm Policy Ment Health

123

Methods

Therapist Participants

Therapist participants were invited to participate at a state

sponsored workshop in EBPs. Of the 397 practitioners who

attended the trainings, 240 (63.3 %) completed one or

more questionnaires from the pre-training survey battery

(see Nakamura et al. 2011a, b). Sixty-three of these prac-

titioners provided mental health services in the schools and

thus their practice data was not available for analysis. Of

the remaining 177 practitioners who were affiliated with

the state of Hawaii’s Child and Adolescent Mental Health

Division (CAMHD), 74 (41.8 %) had at least one practice

measure in the Child and Adolescent Mental Health

Management Information System (CAMHMIS) for a youth

receiving mental health services through CAMHD. Par-

ticipants ranged in age from 25 to 61 (M = 42.4,

SD = 10.8), 64.9 % were female (n = 48), and the pri-

mary ethnicities reported were: White (n = 34; 45.9 %),

Asian (n = 11; 14.9 %), and Hawaiian or Pacific Islander

(n = 5; 6.8 %). Participants reported an average of

6.6 years of clinical training (SD = 5.3), 7.5 years of

clinical experience (SD = 5.5), and 31.1 % (n = 23)

reported holding a state license to practice. Therapist par-

ticipants reported being trained in a variety of different

professional disciplines: marriage and family therapy

(n = 35; 47.3 %), psychology or psychiatry (n = 21,

28.4 %), counseling (n = 12; 16.2 %), social work (n = 5;

6.8 %), and other (n = 1; 1.4 %). When asked about the-

oretical orientation, 86.5 % of participants (n = 64) iden-

tified as Eclectic or selected more than one theoretical

orientation that they identified with and 13.5 % (n = 10)

selected Cognitive-Behavioral Therapy as their primary

theoretical orientation. On average, participants reported

having an active caseload of 7.9 (SD = 4.8) and received

approximately 1.8 h of supervision per week (SD = 1.1).

Youth Participants

Youth were selected for participation if they received

mental health treatment from a contracted provider agency

of the CAMHD between May 2006 and May 2008, had at

least 30 days of available treatment data (mean episode

length = 6.9 months; SD = 5.7 months) and were receiv-

ing treatment for anxiety, attentional, depressive, disruptive

behavior, and/or trauma-related problems. Because youth

were nested within therapists, youth were included only if

their therapist participated in the study. This resulted in 519

youth (64.5 % boys) ages 5 to 19 (M = 14.1; SD = 2.9).

More than half reported a multiracial background

(n = 332; 64.5 %) with other non-mixed races reported at

10.8 % White, 10.0 % Native Hawaiian/Pacific Islander,

5.0 % Asian, and 2.1 % Black/African American. Three-

quarters of youth carried comorbid diagnoses (n = 389;

75.0 %) with primary diagnoses including disruptive

behavior disorders (n = 241; 46.4 %), depressive disorders

(n = 112; 21.6 %), attention-deficit/hyperactivity disorder

(n = 84, 16.2 %), and anxiety or traumatic stress disorders

(n = 82, 15.8 %). Just over half of the youth were

receiving services in in-home settings (n = 300, 57.8 %)

and 22.4 % (n = 116) were receiving services through a

packaged evidenced-based program (i.e., Functional Fam-

ily Therapy, Multisystemic Therapy or Multidimensional

Treatment Foster Care).

Measurement

Child and Adolescent Functional Assessment Scale

(CAFAS; Hodges 1998)

The CAFAS is a 200-item scale that measures a youth’s

level of functional impairment. Based on their experience

with the youth, case managers in the CAMHD review

behavioral descriptions ordered by level of impairment

within eight domains of functioning. The subscales of

school role performance, home role performance, com-

munity role performance, behavior toward others, mood/

emotions, mood/self-harmful behavior, substance use, and

thinking are calculated by scoring the highest level of

impairment (i.e., severe = 30, moderate = 20, mild = 10,

no/minimal = 0) that describes the youth within the

respective domain of items. A total score is calculated by

summing across the eight subscales (i.e., CAFAS Total-8).

The CAFAS has been found to have acceptable internal

consistency across items, inter-rater reliability, stability

across time, and concurrent validity (Hodges and Gust

1995; Hodges and Wong 1996). Of note for the current

study, case managers within the CAMHD are certified for

administration annually. For the purposes of this study, the

CAFAS Total-8 score obtained closest to the treatment

episode start was used to estimate level of functional

impairment at treatment start.

Evidence-Based Practice Attitude Scale (EBPAS; Aarons

2004; Aarons et al. 2010)

The EBPAS is a 15-item well-established measure of cli-

nician attitudes towards EBPs. Participants respond on a

five-point Likert-scale (0 = ‘‘not at all’’ to 4 ‘‘to a very

great extent’’) the extent to which they agree to a particular

statement. Higher mean scores indicate more favorable

attitudes. The EBPAS has demonstrated a four factor

structure and good internal consistency (Aarons 2004;

Aarons et al. 2010). Given that all four scales were highly

correlated in this sample and in order to reduce

Adm Policy Ment Health

123

multicollinearity, we only examined the EBPAS total score

in the MLM. Cronbach’s alpha for the EBPAS total in this

sample was .82, which is slightly higher than that reported

in previous studies (a = .77 in Aarons 2004; a = .76 in

Aarons et al. 2010). The EBPAS total mean in this sample

was 2.93 (SD = .48), which is also slightly higher than that

reported in previous studies (M = 2.30, SD = .45, Aarons

2004; M = 2.33, SD = .45, Aarons et al. 2010).

Knowledge of Evidence Based Services Questionnaire

(KEBSQ; Stumpf et al. 2009)

The KEBSQ is a 40-item measure assessing awareness

knowledge of various evidence-based and non-evidence-

based techniques for youth with anxious/avoidant (A),

depressed/withdrawn (D), disruptive behavior (B), and

attention/hyperactivity (H) problems. Respondents are

asked to circle all problem areas for which a particular type

of practice element is drawn from an evidence-based pro-

tocol. Each individual item is then scored on a scale from

zero to four, with correctly endorsed and omitted responses

per problem area each receiving one point each. As an

example for the present study, exposure has been classified

as being derived from an evidence-based protocol for

anxious/avoidant problems (Chorpita and Daleiden 2007).

In this case, a respondent would get one point for circling

A, one point for not circling D, one point for not circling B,

and one point for not circling H, for a grand total of four

points. In order to differentiate a no-response (e.g., the

participant refused to answer the question) from actively

choosing to indicate that a particular technique is not from

an evidence-based protocol for any of the four problem

areas, participants have the option of circling the letter N

(None) for each item. Total possible scores on the KEBSQ

can range from zero to 160. The KEBSQ has demonstrated

adequate test–retest reliability in a sample of graduate level

and community clinicians (r = .56) and the ability to dis-

criminate between these two populations. Given that the

surveys were administered in 2008 and 2009, the KEBSQ

was scored using definitions of the evidence-base found in

the 2007 CAMHD Biennial Report (Chorpita and Daleiden

2007). Consistent with Stumpf et al.’s (2009) original

study, a technique was considered as derived from an

evidence-based protocol for a particular problem area if

that technique was utilized in 10 % or more of all treatment

protocols receiving Best (Level 1) or Good (Level 2)

Support for that specific problem area. For this study the

KEBSQ total score, which can range from 0 to 160 (40

items ranging from 0 to 4 points per item), was examined.

The KEBSQ total mean score for this sample was 91.3

(SD = 8.8), which was about five points lower than that

reported by Stumpf et al. (2009).

Monthly Treatment and Progress Summary (MTPS;

CAMHD 2008)

The MTPS is a therapist report form designed to measure

treatment targets, clinical progress, and intervention practice

elements. For the purposes of this study, only the practice

elements (PEs; e.g., activity scheduling, catharsis) portion on

the MTPS were examined. On a monthly basis, therapists

identify upto 63 predefined PEs they used in their treatment

with youth clients. The MTPS PEs have demonstrated

acceptable one-month stability estimates (average kap-

pas = .65; Daleiden et al. 2004) and structural validity with

support for three factors corresponding to behavior man-

agement interventions, self-control practices, and family

interventions (Orimoto et al. 2012). Further, in a study

comparing the MTPS PEs and audio recordings from therapy

sessions, 12 of the 63 predefined PEs were determined to be

100 % valid according to clinician-coder agreement utilizing

extensiveness and experiential scales (the other PEs were not

available for coding; Borntrager et al. 2013a, b). As the

MTPS includes PEs that are found in EBPs as well as PEs

that are not included in EBPs, two scores were derived from

the MTPS. Using the CAMHD 2007 Biennial Report

(Chorpita and Daleiden 2007), PDEB were defined as prac-

tices occurring in 30 % or more of level 2 or better treatment

protocols for the problem areas examined in this study (i.e.,

anxiety, attentional problems, depression, disruptive behav-

ior, and traumatic stress). In the CAMHD 2007 Biennial

Report, level 1 treatments are similar to ‘‘well-established’’

treatments as defined by the American Psychological Asso-

ciation Division 12 Task Force for Promotion and Dissemi-

nation of Psychological Procedures (1995) and level 2

treatments are similar to ‘‘probably efficacious’’ treatments

(see Chorpita and Daleiden 2007; Chorpita et al. 2011). This

resulted in 24 PEs (see Table 1) which were summed and

then divided by the total number PEs the youth received over

the course of their entire treatment episode (i.e., PDEB =

Sum of PEs from evidence-based protocols/sum of total

PEs). In order to examine whether therapist characteristics

predicted use of practices with minimal evidence-support

(PMES), four treatment outcome research experts were

consulted to review the remaining list of PEs on the MTPS.

These experts determined that common factors (e.g., sup-

portive listening) and practices that are drawn from evidence-

based protocols for problem areas not included in the study

(e.g., motivational interviewing for substance abuse) should

not included in the PMES score. The remaining PEs were

found to occur in 8 % or less of level 1 or level 2 protocols

for the problem areas in this study. This resulted in ten PEs

(see Table 1) which were summed and then divided by the

total number of PEs the youth received over their treatment

episode (i.e., PMES = Sum of PEs from minimally sup-

ported protocols/sum of total PEs).

Adm Policy Ment Health

123

Therapist Background Questionnaire (TBQ)

The TBQ was developed for this study and assesses basic

demographic information (age, gender, ethnicity/race, ethnic

identity), training and experience information (degrees earned,

state license, professional specialty, theoretical orientation,

years of clinical training, years of clinical experience), and

work setting information (agency name/type, position, clinical

setting, current caseload, hours of supervision per week).

Procedure

The University of Hawai‘i Institutional Review Board

approved this study. Half-day voluntary continuing educa-

tion trainings in EBPs for youth internalizing and external-

izing concerns were held between May 2008 and July 2009

across the state of Hawaii’s four counties. Questionnaires

were administered to attendees prior to workshop partici-

pation. If a practitioner attended more than one training

workshop, his questionnaire from the first training he atten-

ded was utilized for analyses. Prior to any data collection, all

participants underwent standardized Institutional Review

Board-approved notice of privacy and consent procedures.

Youth clients and their legal guardian(s) provided written

informed consent after receiving a complete description of

CAMHD’s notice of privacy and disclosure procedures. The

CAFAS was administered as close to the start of treatment

as possible in conjunction with a diagnostic assessment by a

staff or contracted CAMHD clinician (e.g., staff psycholo-

gist, psychiatrist, community psychologist).

Data Preparation

MTPS scores (PDEB and PMES) as well as predictor vari-

ables from the EBPAS, KEBSQ, and therapist and child

demographics were examined for both statistical outliers and

distribution normality. Standardized z-scores were calcu-

lated for these variables and responses in excess of 3.29

(p \ .001, two-tailed test) were considered outliers (Ta-

bachnick and Fidell 2007). Using this convention, a small

number of outliers were identified for years of training

(n = 1; 1.3 %), years of full-time clinical experience

(n = 2; 2.6 %), typical number of active cases (n = 1;

1.3 %), and episode length (n = 10; 1.9 %).Based on outlier

analysis, skewness, and distribution shape, inverse (reci-

procal) transformations were completed for years training,

years experience, active cases, and episode length. However,

transformations failed to reduce outliers and distributions

remained significantly skewed. As such, scores were chan-

ged for variables with more than one outlier by taking the

next most extreme score and adding one unit (Fields 2009;

Tabachnick and Fidell 2007). For instance, one therapist

identified as having 33 years of training beyond their

undergraduate degree. The next most extreme score was

23 years of training. Thus one unit (1 year) was added to the

second most extreme score (i.e., 23 ? 1 = 24) and this

number replaced the most extreme outlier (i.e., 33).

Missing data were handled using the missing value

analysis module of SPSS 19.0 (SPSS, 2010) and missing

Table 1 MTPS practice use by therapist (N = 74) and youth

(N = 519)

MTPS practices Type of

practice

Therapist

percent use

(N = 74)

Youth

percent

received

(N = 519)

Activity scheduling PDEB 89.2 69.8

Behavioral contracting PDEB 33.8 7.4

Cognitive PDEB 93.2 82.1

Commands PDEB 85.1 79.6

Communication skills PDEB 97.3 91.2

Exposure PDEB 27.0 9.9

Goal setting PDEB 43.2 11.1

Ignoring or DRO PDEB 52.7 24.7

Maintenance/relapse Prevention PDEB 66.2 33.7

Modeling PDEB 87.8 57.8

Parent/teacher monitoring PDEB 82.4 60.1

Parent/teacher praise PDEB 87.8 73.2

Personal safety skills PDEB 16.2 4.3

Problem solving PDEB 97.3 85.8

Psychoeducation child PDEB 79.7 61.9

Psychoeducation parent PDEB 85.1 66.7

Relaxation PDEB 74.3 44.2

Response cost PDEB 51.4 27.0

Self-monitoring PDEB 82.4 51.0

Self-reward/self-praise PDEB 78.4 45.9

Social skills training PDEB 90.5 72.0

Tangible rewards PDEB 86.5 68.3

Therapist praise/rewards PDEB 91.9 76.8

Time out PDEB 67.6 42.4

Catharsis PMES 29.7 13.2

EMDR or tapping PMES 9.5 1.4

Free association PMES 41.9 13.8

Hypnosis PMES 9.5 1.6

Interpretation PMES 40.5 26.1

Line of sight supervision PMES 70.3 44.4

Mentoring PMES 73.0 40.3

Milieu therapy PMES 54.1 31.3

Thought field therapy PMES 10.8 2.9

Twelve step program PMES 35.1 18.5

MTPS monthly treatment and progress summary, PDEB practice

derived from the evidence base, DRO differential reinforcement of

other behavior, PMES practices with minimal evidence support,

EMDR eye movement desensitization and reprocessing

Adm Policy Ment Health

123

data patterns were examined using Little’s missing com-

pletely at random (MCAR) test. Data was missing from the

following variables: years trained (1.9 %), KEBSQ

(3.3 %), EBPAS (3.3 %), and CAFAS (8.1 %) and Little’s

MCAR was significant (v2 = 257.46, df = 52, p \ .001),

suggesting that data was not MCAR. As such, data were

imputed (fully conditional) and all results are reported for

the pooled dataset (average of 20 imputed datasets) except

in instances where multiple imputation procedures did not

yield pooled estimates; for those instances, results from

analyses with non-imputed data are reported.

Data Analytic Strategy

Given the nested nature of the data in this study, MLM using

SPSS version 19 was employed. Youth were defined as the

Level 1 unit of analysis and therapists were defined as the Level

2 unit. Three models were created for each of the two dependent

variables (i.e., PDEB and PMES): (a) null models (i.e., no

predictors models), (b) level 1 models (i.e., child characteris-

tics) and (c) level 2 models (i.e., therapist characteristics).

In order to reduce multicollinearity, highly correlated

predictor variables were examined and one of the pair was

selected for inclusion in final models. In particular, years of

training was selected over years of clinical experience

(r = .59) because the latter variable was more skewed. Also,

supervision hours per week, number of active cases, and

receiving a packaged EBP program were all highly correlated.

Given that we were interested in predicting use of PDEB, the

latter of the three variables was retained in final analyses.

Thus, the final predictor variables in the level 1 models were

(a) child age, (b) child gender, (c) primary diagnosis (anxiety,

trauma, depression, or attention compared to disruptive

behavior), (d) CAFAS Total, (e) receiving services through an

evidence-based program (yes, no), (f) level of care (in-home,

out-of-home), and (g) length of treatment episode. The final

predictor variables in the level 2 models were (a) years trained,

(b) licensed (yes/no), (c) professional specialty (counseling,

marriage and family, other, or psychiatry and psychology

compared to social work), (d) primary theoretical orientation

(Eclectic, CBT), (e) KEBSQ Total, and (f) EBPAS Total.

Alpha was set a priori at .05.

Results

Null Models (No Predictors)

In order to determine if a multilevel could be examined, no

predictors (i.e., null) models were examined. For both

PDEB and PMES, intercepts varied significantly across

therapists (Wald Z Intercept = 3.686, p \ .001 and 4.983,

p \ .001, respectively) and residual parameters varied

significantly within therapists (Wald Z Residual = 14.333,

p \ .001 and 14.845, p \ .001, respectively). Intraclass

correlation coefficients (ICCs) provide estimates of vari-

ance for the dependent variables. The ICCs for PDEB

(.416) and PMES (.553) suggest that 41.6 % of the vari-

ance in PDEB scores and 55.3 % of the variance in PMES

scores is accounted for by youth, service, and therapist

characteristics. Thus, we rejected the null hypothesis and

developed multilevel models to explain the variability in

intercepts between and within therapists.

Level 1 Models (Youth and Service Characteristics)

Results of the level 1 models predicting PDEB and PMES

are presented in Table 2 (significant results are in bold

font). Episode length was the only significant predictor of

PDEB with longer episode lengths predicting greater use of

PDEB (t = 2.874, p = .004). On the other hand, several

youth and service characteristics predicted PMES. Older

youth and males were more likely to receive PMES

(t = 2.769, p = .006 and t = -3.322, p = .001, respec-

tively). Further, youth in out-of-home levels of care were

more likely to receive PMES (t = 6.865, p \ .001)

whereas youth receiving services from an evidence-based

program (i.e., Functional Family Therapy, Multisystemic

Therapy or Multidimensional Treatment Foster Care) were

less likely to receive PMES (t = -2.977, p = .003).

Level 2 Models (Therapist Characteristics)

Results of the level 2 models predicting PDEB and PMES

are presented in Table 3 (significant results are in bold

font). Significant youth and service characteristics from

model 1 were included in model 2 (i.e., episode length for

PDEB and youth age, youth gender, service type, and level

of care for PMES). These characteristics remained signif-

icant in the level 2 models. Therapists who reported having

a Psychology or Psychiatry specialty used significantly

more PDEB than therapists with a Social Work specialty

(t = 2.180, p = .029). No other differences among the

professional specialties were noted. In addition, therapists

who identified with a Behavioral or Cognitive-Behavioral

theoretical orientation used significantly more PDEB than

therapists who identified with an Eclectic theoretical ori-

entation (t = -2.599, p = .009). No significant differ-

ences among therapists were noted for PMES.

Discussion

This research builds on and extends the work to date

examining youth and therapist characteristics that predict

use of PDEB in usual care. Regarding youth and service

Adm Policy Ment Health

123

level characteristics, we found that youth who had longer

episodes of care were more likely to receive PDEB. This

may simply indicate that there was greater opportunity for

use of any practice relative to youth who had shorter

lengths of stay. Related to this, youth in out-of-home levels

of care were more likely to receive PMES. Given that

therapists have greater contact with youth in out-of-home

care, it may be that therapists simply had more contact with

Table 2 Level 1 models predicting PDEB and PMES

Predictor PDEB PMES

b SE t p b SE t p

Youth agea 0.000 0.001 -0.004 0.997 0.002 0.001 2.769 0.006

Youth gendera -0.003 0.009 -0.353 0.724 -0.013 0.004 -3.322 0.001

Primary diagnosisb

Anxiety -0.006 0.017 -0.362 0.718 0.006 0.008 0.844 0.399

ADHD 0.008 0.011 0.693 0.489 0.001 0.006 0.104 0.917

Depression 0.007 0.010 0.684 0.494 -0.009 0.005 -1.900 0.057

Trauma 0.011 0.013 0.817 0.414 0.001 0.006 0.104 0.917

CAFAS total 0.000 0.000 -0.119 0.906 0.000 0.000 1.341 0.180

Service typec 0.000 0.018 0.017 0.987 -0.023 0.008 -2.977 0.003

Level of cared -0.029 0.016 -1.793 0.073 0.047 0.007 6.865 0.000

Episode length 0.000 0.000 2.874 0.004 0.000 0.000 -1.556 0.120

Statistically significant values (p \ 0.05) are given in bold

PDEB practices derived from the evidence base, PMES practices with minimal evidence support, ADHD attention-deficit/hyperactivity disorder

PDEB ICC = .425; PMES ICC = .382a 0=Male; 1=Femaleb Compared to disruptive behavior, 0=non-evidence-based service; 1=evidence-based service (Functional Family Therapy; Multisystemic

Therapy, or Mulidimensional Treatment Foster Care)c 0=In-home level of cared 1=Out-of-home level of care

Table 3 Level 2 models

predicting PDEB and PMES

Statistically significant values

(p \ 0.05) are given in bold

PDEB practices derived from

the evidence base, PMES

practices with minimal evidence

supporta 0=Male; 1=Femaleb 0=Non-evidence-based

service; 1=Evidence-based

service (Functional Family

Therapy; Multisystemic

Therapy, or Mulidimensional

Treatment Foster Care)c 0=In-home level of care;

1=Out-of-home level of cared 0=Not licensed; 1=Licensede Compared to social workf Compared to cognitive-

behavioral

Predictor PDEB PMES

b SE t p b SE t p

Level 1 predictors

Youth age – – – – 0.002 0.001 3.073 0.002

Youth gendera – – – – -0.013 0.004 -3.327 0.001

Service typeb – – – – -0.018 0.009 -2.086 0.037

Level of carec – – – – 0.050 0.008 6.612 0.000

Episode length 0.000 0.000 3.166 0.002 – – – –

Level 2 predictors

Years trained 0.000 0.002 0.002 0.938 0.000 0.001 0.282 0.778

Licensedd -0.031 0.022 -1.411 0.158 0.009 0.010 0.885 0.376

Professional specialtye

Counseling -0.006 0.033 -0.172 0.864 0.011 0.016 0.695 0.487

Marriage and family -0.020 0.027 -0.743 0.457 0.009 0.013 0.687 0.492

Psychology and psychiatry 0.083 0.038 2.180 0.029 -0.001 0.018 -0.048 0.962

Other 0.040 0.037 1.076 0.282 -0.006 0.018 -0.365 0.715

Theoretical orientation

Eclecticf -0.073 0.028 -2.599 0.009 0.013 0.014 0.928 0.353

KEBSQ total 0.000 0.001 0.186 0.852 0.000 0.000 -0.335 0.738

EBPAS total -0.005 0.019 -0.259 0.796 -0.003 0.009 -0.371 0.711

Adm Policy Ment Health

123

youth and thus use more practices in general. We also

found that males were more likely to receive PMES. It is

not clear why this pattern emerged. Additional research is

needed to determine whether there truly are gender dis-

crepancies in practice use. If future research confirms this

finding it will be important to address this gender

inequality. On the positive side, we found that youth

receiving services from an evidence-based program (i.e.,

Functional Family Therapy, Multisystemic Therapy or

Multidimensional Treatment Foster Care) were less likely

to receive PMES, suggesting that these programs are not

including practices that are inconsistent with the evidence-

base. Denneny and Mueller (2012) found similar results in

a study of Multisystemic Therapy in the Hawaii CAMHD

system (2013), suggesting robustness of these findings.

Although we found that older youth were more likely to

receive PMES, Brookman-Frazee et al. (2010) found that

older youth received a higher Child EBP score. However, this

may be an artifact of the different methods employed by the

two studies. For instance, the Brookman-Frazee et al. study

examined youth ages 4–13 whereas the current study inclu-

ded older adolescents (age range = 5–19; mean = 14.1).

The Brookman-Frazee et al. study also examined disruptive

behavior only whereas we included other common diagnoses

in childhood. Further, while the Brookman-Frazee et al.

study employed observational methods to assess practices

and the current study relied on self-reported practices, it is

unclear whether (or how) this methodological artifact may

have resulted in the difference in findings.

Consistent with Brookman-Frazee and colleagues’ study

(2010), we found very few therapist characteristics that

predicted use of PDEB and no characteristics that predicted

PMES. Therapists who identified with a Behavioral or

Cognitive-Behavioral theoretical orientation used signifi-

cantly more PDEB than therapists who identified with an

Eclectic theoretical orientation. It is possible that because

many of the PDEB are consistent with cognitive-behavioral

theory, therapists with a CBT orientation are more likely to

use practices consistent with their orientation. An alterna-

tive but related explanation is that therapists with a CBT

orientation have an epistemology that values science-based

treatments and as such, are more likely to use PDEB with

their youth clients. We also found that therapists who

reported a Psychology or Psychiatry specialty used sig-

nificantly more PDEB than therapists with a Social Work

specialty. This is somewhat concerning give that a large

portion of the mental health workforce is made up of Social

Workers, with Psychologists and Psychiatrists increasingly

playing a larger role in mental health administration and

supervision. Historically, Social Work has focused less on

psychotherapy and the provision of mental health services

and more on connecting individuals in need of services to

those services. Thus there has been less of an emphasis on

training of EBPs in schools of Social Work in the past,

however, there are signs of positive changes in this area

(Barth et al. 2012).

Contrary to expectations, knowledge of EBPs and atti-

tudes towards EBPs did not predict greater use of PDEB

nor did they predict less use of PMES. These findings may

relate to ceiling effects for the attitude measure and

instrumentation issues for the knowledge measure. Con-

cerning EBP attitudes in the current study, it is noteworthy

that the mean total score for the EBPAS was 2.93

(SD = .48), suggesting very positive attitudes towards

EBPs. Indeed, the overall attitude score in the current

sample was greater than those indicated in Aarons’ (2004)

original and Aarons et al. (2010) follow up studies. This

skew in the data, combined with a small standard deviation,

may have resulted in less overall variance to draw from

when examining the relationship between attitudes and

PDEB and PMES scores. This skew may be a result of the

context from which the sample was drawn. As noted ear-

lier, participants were providers who voluntarily partici-

pated in trainings on youth EBPs in Hawaii, a state that has

experienced large system reform with an emphasis on

increased attention towards EBP implementation (Nakam-

ura et al. 2011a, b; Nakamura et al. 2012).

Concerning EBP knowledge, the KEBSQ assesses

knowledge at the awareness (i.e., is a specific technique

part of a larger EBP protocol) rather than the how-to or

implementation (i.e., what are the specific steps associated

with implementing a certain technique) level, and the

relationship between these two types of knowledge are in

need of further investigation (Nakamura et al. 2011a, b).In

addition to the concerns mentioned above, the finding that

attitudes and knowledge did not predict PDEB or PMES

scores speak to the importance of investigating other

individual level, as well as inner and outer context factors

such as organization variables important for implementing

youth EBPs in usual care (e.g., Damschroder et al. 2009).

For instance, in a study of therapists from communities

receiving funding from the Comprehensive Community

Mental Health Services for Children and their Families

Program, Aarons et al. (2009) reported that private agen-

cies (compared to public agencies), organizational support

for EBPs, and attitudes towards EBPs all predicted greater

use of EBPs. When they examined the complex relation-

ships among these variables, they found that the model that

fit their data best was one where the relationship between

EBP use and agency type was mediated by organizational

support for EBPs. Further, while agency type and organi-

zational support predicted more favorable attitudes towards

EBPs, in the final model, attitudes towards EBPs did not

significantly predict greater use of EBPs though there was a

small effect in the hypothesized direction. Aarons and

colleagues (2009) reconciled these contradictory findings

Adm Policy Ment Health

123

by hypothesizing that in the absence of organizational

support of EBPs, attitudes towards EBPs are likely to

predict use of EBPs but when organizations do support

EBPs, individual provider attitudes towards EBPs are not

as important.

Limitations

Although the results of the current study are promising with

regard to continued investigation of predictors of youth ther-

apist practices in usual care, a few limitations in addition to

those mentioned above are noteworthy. First, as mentioned

earlier the current therapists were drawn from a larger sample

of providers that voluntarily signed up to participate in train-

ings on youth EBPs. Along these lines, some caution should be

exercised when interpreting these results and readers are urged

to remember that the provider sample may not be representa-

tive of the broad spectrum of usual care therapists. A second

notable concern surrounds the self-reported nature of the

MTPS, the instrument underlying calculation of the PDEB and

PMES outcome variables. As mentioned above, within the

CAMHD reporting infrastructure, therapists self-report on the

practices they utilized with a youth for any given reporting

month. As such, readers are encouraged to remember the self-

report aspect of the outcome variables of PDEB and PMES, as

well as the potential flaws that typically accompany such

methodology. A third noteworthy caveat concerns instru-

mentation specificity for the assessed constructs of attitudes

and knowledge in the current study. In addition to the concerns

already mentioned above, the level of specificity between the

constructs of attitudes, knowledge, and self-reported PDEB

and PMES practice patterns varied widely. For example,

although the construct of EBP attitude was assessed with a

psychometrically tested instrument, the EBPAS queries about

EBP attitudes in a general sense (as opposed to attitudes about

the 24 specific techniques that make up the PDEB score or the

12 techniques that make up the PMES score). Building upon

this example, the knowledge measure asked about a set of 40

techniques, some of which contributed to the PDEB scores and

some of which contributed to PMES scores. Building upon this

work, forthcoming studies examining attitudes towards,

knowledge of, and self-reported usage patterns of EBPs may

benefit from assessing all three constructs at the same level of

specificity. Indeed, research is emerging that suggests that

therapists’ attitudes towards EBPs can vary as a function of the

type of EBP for which they are queried (Reding et al. 2007).

Conclusion

Given that findings in this study largely replicate previous

findings in different samples, in different contexts, and with

different methodologies, one potential conclusion is that

these findings are relatively robust and that as a field we

should be turning our attention and efforts to other levels as

suggested by theories of implementation science such as

organizational factors, leadership, and the process by which

implementation is accomplished (e.g., Aarons and Som-

merfeld 2012; Damschroder et al. 2009; Proctor et al. 2009).

However, care should be taken to not abandon research on

individual factors just yet, as work in this area will most

likely gain benefit from continued scientific advances related

to the broader EBP dissemination and implementation

movement. For instance, it could be argued that our field has

not yet agreed upon a standardized definition of, or metric

system for EBP, the very construct upon which voluminous

amounts of research are beginning to orbit. It is noteworthy

that large governing bodies playing important roles in dis-

semination and implementation (e.g., American Psycholog-

ical Association, Society of Clinical Child and Adolescent

Psychology, Substance Abuse and Mental Health Services

Administration, American Academy of Pediatrics) all have

slightly varying definitions of this construct. Concerning an

EBP metric system, some groups define EBP at the level of

brand-named treatment manuals, whereas others may utilize

technique commonalities (practice elements) across those

protocols or even broad-based treatment approaches (e.g.,

Cognitive Behavior Therapy). Further complicating these

issues, EBP also varies as a function of youth characteristics

including diagnoses and age, and also changes over time as

the treatment outcome literature continues to mature. As a

result, instrumentation efforts for EBP and related individual

level constructs such as knowledge and attitudes can be

deceivingly complex. Owing to these and other complicating

issues, noise or error variance has the real potential to nullify

important research findings in studies such as this one and

partially account for discrepant findings between various

investigative groups. In the end then, despite our findings

that only a few youth and provider level factors predicted

practice patterns for therapists, it may still be fruitful to

continue with research in this area. Borrowing from the area

of treatment outcome research, the perennial question of

‘‘What works for whom, how and under what circum-

stances?’’ as applied to therapists and EBP implementation

may best be approached through efforts that continue to be

multilevel in nature, that also allow for penetrating media-

tional and mode rational models.

References

Aarons, G. A. (2004). Mental health provider attitudes toward adoption

of evidence-based practice: The evidence-based practice attitude

scale (EBPAS). Mental health services research, 6(2), 61–74.

Aarons, G. A., Glisson, C., Hoagwood, K., Kelleher, K., Landsverk,

J., & Cafri, G. (2010). Psychometric properties and US National

Adm Policy Ment Health

123

norms of the evidence-based practice attitude scale (EBPAS).

Psychological Assessment, 22(2), 356.

Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a

conceptual model of evidence-based practice implementation

in public service sectors. Administration and Policy in Mental

Health and Mental Health Services Research, 38(1), 4–23.

Aarons, G. A., & Sommerfeld, D. H. (2012). Leadership, innovation

climate, and attitudes toward evidence-based practice during a

statewide implementation. Journal of the American Academy of

Child and Adolescent Psychiatry, 51(4), 423–431.

Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009).

Evidence-based practice implementation: The impact of public

versus private sector organization type on organizational

support, provider attitudes, and adoption of evidence-based

practice. Implementation Science, 4(1), 83.

American Psychological Association Task Force on Promotion and

Dissemination of Psychological Procedures, Division of Clinical

Psychology. (1995). Training in and dissemination of empiri-

cally-validated psychological treatments: Report and recommen-

dations. The Clinical Psychologist, 48, 3–23.

Barth, R. P., Lee, B. R., Lindsey, M. A., Collins, K. S., Strieder, F.,

Chorpita, B. F., et al. (2012). Evidence-based practice at a

crossroads the timely emergence of common elements and common

factors. Research on Social Work Practice, 22(1), 108–119.

Borntrager, C., Chorpita, B. F., Higa-McMillan, C. K., Daleiden, E.

L., & Starace, N. (2013a). Usual care for trauma-exposed youth:

Are clinician-reported therapy techniques evidence-based? Chil-

dren and Youth Services Review, 35(1), 133–141.

Borntrager, C. F., Chorpita, B. F., Orimoto, T., Love, A., & Mueller,

C. W. (2013b). Validity of clinician’s self-reported practice

elements on the monthly treatment and progress summary. The

Journal of Behavioral Health Services and Research. doi:10.

1007/s11414-013-9363-x.

Brookman-Frazee, L., Haine, R., Baker-Ericzen, M., Zoffness, R., &

Garland, A. (2010). Factors associated with use of evidence-

based practice strategies in usual care youth psychotherapy.

Administration and Policy in Mental Health, 37, 254–269.

Child and Adolescent Mental Health Division. (2008). Instructions

and codebook for provider monthly summaries. Honolulu:

Hawaii Department of Health, Child and Adolescent Mental

Health Division.

Chorpita, B.F. & Daleiden, E. L. (2007). 2007 Biennial report:

Effective psychosocial interventions for youth with behavioral

and emotional needs. Child and Adolescent Mental Health

Division, Hawaii Department of Health.

Chorpita, B. F., Daleiden, E. L., Ebesutani, C., Young, J., Becker, K.

D., Nakamura, B. J., et al. (2011). Evidence-based treatments

for children and adolescents: An updated review of indicators

of efficacy and effectiveness. Clinical Psychology, 18(2),

154–172.

Daleiden, E., Lee, J., & Tolman, R. (2004). Child and Adolescent

Mental Health Division: 2004 annual report. Honolulu: Child

and Adolescent Mental Health Division.

Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander,

J. A., & Lowery, J. C. (2009). Fostering implementation of

health services research findings into practice: A consolidated

framework for advancing implementation science. Implement

Science, 4(1), 50.

Denneny, D. & Mueller, C. (2012). Do empirically supported

packages or their practices predict superior therapy outcomes

for youth with conduct disorders. Poster presented at the Forty-

Sixth Annual Convention of the Association for Behavioral and

Cognitive Therapies, National Harbor, MD.

Field, A. (2009). Discovering statistics using SPSS. CA: Sage

publications.

Garland, A., Brookman-Frazee, L., Hurlburt, M., Accurso, E.,

Zoffness, R., Haine-Schlagel, R., et al. (2010a). Mental

health care for children with disruptive behavior problems: A

view inside therapists’ offices. Psychiatric Services, 61(8),

788–795.

Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M.,

& Accurso, E. C. (2010b). Methodological challenges of

characterizing usual care psychotherapeutic practice. Adminis-

tration and Policy in Mental Health and Mental Health Services

Research, 37(3), 208–220.

Higa-McMillan, C. K., Jackson, D., & Daleiden, E. L. (2011).

The Venn Diagram of usual care and evidence-based care:

Identifying non-overlapping areas to fine-tune dissemination

of evidence-based practices for child anxiety. In C. K.

Higa-McMillan (Chair), Comparing Usual Care and Evidence-

Based Care for Children and Adolescents: Advancing Dissem-

ination in the twentyfirst Century. Symposium presented at the

annual meeting of the Association of Behavioral and Cognitive

Therapies, Toronto.

Hodges, K. (1998). Child and adolescent functional assessment scale

(CAFAS). Ann Arbor: Functional Assessment Systems.

Hodges, K., & Gust, J. (1995). Measures of impairment for children

and adolescents. Journal of Mental Health Administration, 22,

403–413.

Hodges, K., & Wong, M. M. (1996). Psychometric characteristics of a

multidimensional measure to assess impairment: The child and

adolescent functional assessment scale (CAFAS). Journal of

Child and Family Studies, 5, 445–467.

Jensen-Doss, A., Hawley, K. M., Lopez, M., & Osterberg, L. D.

(2009). Using evidence-based treatments: The experiences of

youth providers working under a mandate. Professional Psy-

chology, 40(4), 417.

Nakamura, B. J., Chorpita, B. F., Hirsch, M., Daleiden, E., Slavin, L.,

Amundson, M. J., et al. (2011a). Large-scale implementation of

evidence-based treatments for children 10 years later: hawaii’s

evidence-based services initiative in children’s mental health.

Clinical Psychology, 18(1), 24–35.

Nakamura, B. J., Higa-McMillan, C. K., & Chorpita, B. F. (2012).

Sustaining Hawaii’s evidence-based service system in children’s

mental health. In D. H. Barlow & K. McHugh (Eds.), The

dissemination of evidence-based psychological treatments. New

York: Oxford University Press.

Nakamura, B., Higa-McMillan, C., Okamura, C., & Shimabukuro,

S. (2011b). Knowledge of and attitudes towards evidence-

based practices in community child mental health practi-

tioners. Administration and Policy in Mental Health, 38,

287–300.

Nelson, T. D., & Steele, R. G. (2007). Predictors of practitioner self-

reported use of evidence-based practices: Practitioner training,

clinical setting, and attitudes toward research. Administration

and Policy in Mental Health and Mental Health Services

Research, 34(4), 319–330.

Orimoto, T. E., Higa-McMillan, C. K., Mueller, C. W., & Daleiden,

E. L. (2012). Assessment of therapy practices in community

treatment for children and adolescents. Psychiatric Services,

63(4), 343–350.

Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson,

C., & Mittman, B. (2009). Implementation research in

mental health services: An emerging science with conceptual,

methodological, and training challenges. Administration and

Policy in Mental Health and Mental Health Services Research,

36(1), 24–34.

Reding, M. E. J., Chorpita, B. F., Lau, A. S., & Innes-Gomberg, D.

(2007). Providers’ attitudes toward evidence-based practices: Is

it just about providers, or do the practices matter, too?

Adm Policy Ment Health

123

Administration and Policy in Mental Health and Mental Health

Services Research, 34(4), 411–419.

Stephan, S., Westin, A., Lever, N., Medoff, D., Youngstrom, E., &

Weist, M. (2012). Do school-based clinicians’ knowledge and

use of common elements correlate with better treatment

quality? School Mental Health, 4(3), 170–180.

Stumpf, R. E., Higa-McMillan, C. K., & Chorpita, B. F. (2009).

Implementation of evidence-based services for youth assessing

provider knowledge. Behavior Modification, 33(1), 48–65.

Tabachnick, B. G., & Fidell, L. S. (2007). Using Multivariate

Statistics (6th ed.). Needham Heights: Allyn & Bacon.

Adm Policy Ment Health

123