http://heb.sagepub.com/Behavior

Health Education &

http://heb.sagepub.com/content/37/6/815The online version of this article can be found at:

 DOI: 10.1177/1090198110366002

2010 37: 815 originally published online 4 November 2010Health Educ BehavLindley

Shelly-Ann K. Bowen, Ruth P. Saunders, Donna L. Richter, Jim Hussey, Keith Elder and LisaRütten Framework−−Interventions: Introducing the Rogers

Assessing Levels of Adaptation During Implementation of Evidence-Based  

Published by:

http://www.sagepublications.com

On behalf of: 

  Society for Public Health Education

can be found at:Health Education & BehaviorAdditional services and information for    

  http://heb.sagepub.com/cgi/alertsEmail Alerts:

 

http://heb.sagepub.com/subscriptionsSubscriptions:  

http://www.sagepub.com/journalsReprints.navReprints:  

http://www.sagepub.com/journalsPermissions.navPermissions:  

http://heb.sagepub.com/content/37/6/815.refs.htmlCitations:  

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

815

Assessing Levels of Adaptation During Implementation of Evidence-Based Interventions:

Introducing the Rogers–Rütten Framework

Shelly-Ann K. Bowen, PhDRuth P. Saunders, PhDDonna L. Richter, EdD

Jim Hussey, PhDKeith Elder, PhD

Lisa Lindley, DrPH

Most HIV-prevention funding agencies require the use of evidence-based behavioral interventions, tested and proven to be effective through outcome evaluation. Adaptation of programs during implementation is com-mon and may be influenced by many factors, including agency mission, time constraints, and funding streams. There are few theoretical frameworks to understand how these organizational and program-related factors influ-ence the level of adaptation. This study used constructs from both Rogers’s diffusion theory and Rütten’s framework for policy analysis to create a conceptual framework that identifies determinants hypothesized to affect the level of adaptation. Preliminary measures of these constructs were also developed. This framework and its measures assess organizational and program-related factors associated with adaptation and could serve as a model to assess implementation and adaptation in fields outside of HIV prevention.

Keywords: HIV/AIDS; community-based organizations; evidence-based interventions; implementation; adaptation

The estimated number of new Human Immunodeficiency Virus (HIV) infections for the United States in 2006 was 56,300 (Hall et al., 2008). Incidence estimates continue to demonstrate a disproportionate HIV infection incidence rate among Blacks (83.7/100,000) and Hispanics (29.3/100,000) compared with Whites (11.5/100,000; Centers for Disease

Shelly-Ann K. Bowen, Bureau of Community Health and Chronic Disease Prevention, South Carolina Department of Health and Environmental Control, Columbia. Ruth P. Saunders, Health Promotion, Education, Behavior, Arnold School of Public Health, University of South Carolina, Columbia. Donna L. Richter, Institute for HIV Prevention Leadership, Arnold School of Public Health, University of South Carolina, Columbia. Jim Hussey, Department of Biostatistics and Epidemiology, Arnold School of Public Health, University of South Carolina, Columbia. Keith Elder, Health Sciences, University of Alabama Birmingham. Lisa Lindley, Depart-ment of Global and Community Health, College of Health and Human Services, George Mason University, Fairfax, Virginia.

Address correspondence to Shelly-Ann Bowen, 1800 St. Julians Place, Columbia, SC 29201; phone: 803-545-4488; e-mail: bowensk@dhec.sc.gov.

Health Education & Behavior, Vol. 37(6): 815-830 (December 2010)DOI: 10.1177/1090198110366002© 2010 by SOPHE

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

816 Health Education & Behavior (December 2010)

Control and Prevention [CDC], 2007; Hall et al., 2008). With an HIV vaccine still in its development stages, prevention programs are the best approach to stemming the HIV epidemic. There is strong evidence that behavioral prevention interventions designed to reduce HIV risk behaviors are powerful and effective tools in curbing the HIV epidemic (Collins, Harshbarger, Sawyer, & Hamdallah, 2006; Jemmott & Jemmott, 2000; Saleh-Onoya et al., 2008; Stanton et al., 1996).

Behavioral prevention interventions are theory driven and provide information, educa-tion, counseling, and risk reduction to reduce HIV transmissions and risk in vulnerable populations (Foster & Niederhausen, 2002). These interventions are often implemented by community based organizations (CBOs; Mantell & Divittis, 1990), but it is widely recognized that the impact of evidence-based HIV prevention programs is diminished when translating evidence-based HIV prevention programs from research to practice settings because there are a number of challenges to their successful implementation (Gandelman, Desantis, & Rietmeijer, 2006; Kelly, Heckman, et al., 2000; Lee, Altschul, & Mowbray, 2008; Solomon, 2002).

The difficulty of transitioning HIV prevention research into HIV prevention practice may relate to determinants particular to the CBO (Blankenship, Bray, & Merson, 2000; Sumartojo, 2000). Effective implementation of science-based interventions requires organizational support, adequate staffing, and sufficient resources (CDC HIV/AIDS Prevention Research Synthesis Project, 1991; Sogolow et al., 2000).

To ensure implementation fidelity, the CDC funds CBOs implementing effective evidence-based interventions (EBIs) that require careful training of CBO personnel, close monitoring of the fidelity of procedures, and ongoing monitoring of effectiveness (Sogolow et al., 2000). This article introduces the Rogers-Rütten conceptual framework as a model for guiding research on dissemination, adoption, and implementation of EBIs in CBOs. The framework uses key organizational and program determinants that are hypothesized to influence the level of adaptation of EBIs during program implementation. Preliminary measures for key constructs in the framework, determinants, and level of adaptation are presented.

DIFFUSION THEORY

Considerable research suggests that practitioners face challenges when evidence-based innovations are translated into the real world (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Kilbourne et al., 2004; Lee et al., 2008). The diffusion of innovations theory (Rogers, 1995) provides key tenets for understanding factors that influence imple-mentation of innovations such as HIV prevention programs. Important considerations for applications of this theory in organizational settings include characteristics of the innova-tion (i.e., HIV prevention program) and the organization, as well as an understanding of the organizational assimilation process that occurs through a series of nonlinear “stages” that broadly include adoption, implementation, and routinization (Greenhalgh et al., 2004). The main characteristics of innovations likely to be implemented (Rogers, 1995) include relative advantage, that is, innovations that have clear, unambiguous advantage in either effectiveness or cost-effectiveness are more likely to be implemented (Greenhalgh et al., 2004; Miller, 2001); compatibility, that is, innovations that are compatible with organiza-tional or professional values, norms, and ways of working are more likely to be implemented (Greenhalgh et al., 2004; Rogers, 1995); and complexity, that is, innovations at the orga-nizational level that are perceived by adopters as simple to understand, with few response barriers, are more likely to be implemented (Greenhalgh et al., 2004; Miller, 2001).

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 817

Reinvention, or adaptation, is another tenet of the diffusion theory and pertains to changes made to an innovation as it is implemented. Changes to innovations are part of the imple-mentation process (Rogers, 1995). Many federal agencies and private organizations that promote prevention curricula as being effective advise CBO program managers to implement the interventions precisely as specified in curriculum guides to ensure positive results (McK-leroy et al., 2006). However, studies (Berman & McLauglin, 1976; Eveland, 1977; Tappe, Galer-Unti, & Bailey, 1995) show that adaptation to curricula occur more often than empiri-cal studies report and is commonly done because of contextual factors, including agency mission, politics, time constraints, and funding streams (Collins et al., 2006).

Ultimately, the effectiveness of prevention interventions depend on effectively translat-ing the original research trial intervention into community-based practice while maintaining the desired level of fidelity (DiClemente, 2000; Kelly et al., 2000). Few or no guidelines are in place to facilitate adaptation of effective programs while ensuring that they remain effective (Collins et al., 2006). A better understanding of factors influencing adaptation may facilitate the development of such guidelines.

In this study, we augment Rogers’s (1995, 2003) theory to more fully address elements related to policy development and implementation in organizational settings. The second theoretical perspective presented here is the model of policy determinants by Rütten et al. (2003a), which complements Rogers’s theory of diffusion by addressing the external determinants of program implementation.

MODEL OF POLICY DETERMINANTS

The theory of policy determinants offers salient constructs to the development of the conceptual framework on dissemination, adoption, and implementation of EBIs. Constructs include opportunities, goals, resources, and obligations. These factors influence imple-mentation of new policies in complex organizational systems. The theory posits that policy determinants, such as tangible goals of the policy, sufficient resources, or organizational capacity, are crucial for implementation of a policy and ultimately of behavior change at the population level. This model offers a way to understand implementation of interven-tions in a variety of settings and with a variety of health issues (Rütten et al., 2003a).

Opportunities refer to environmental changes such as those resulting from new federal funding streams and mandates for HIV prevention; for example, when the CDC makes new funding available for CBOs working with a particular population that is considered to be in dire need of an intervention. Specifically for this study, the opportunity is CDC’s mandate that CBOs implement EBIs in order to continue receipt of funding. Goals take into account concreteness of the objectives and the substantive attributes of the innova-tion (Rütten et al., 2003b). For example, the CBO will consider how simple, relevant, and applicable the goals of the EBIs are before they decide to implement them. Resources pertain to the organizational capacity (e.g., skilled staff and space) and funds in the orga-nization (Rütten et al., 2003b). If the CBO has all the necessary resources, then it is more likely to implement the intervention without changes.

A specific set of obligations determines the context in which a mandate is implemented. These CBO obligations, mainly to the funder and the community, can be defined as the personal and professional duties of the CBO’s director as well as the historical relationship the CBO has with its service community. That is, when an agency receives funding to implement an EBI, both the director and agency staff are in contractual agreement with the funder to implement the EBI.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

818 Health Education & Behavior (December 2010)

The diffusion theory stage of interest in this study is implementation. Implementation is the stage that modification or adaptation to the intervention is likely to occur (Rogers, 1995, 2003). Using the combined Rogers–Rütten framework, we examine the outcome “level of adaptation” operationalized as the amount of change made to the core elements of the inter-vention. The determinants, shown in Figure 1 and defined in Table 1, are purported by the authors as factors that influence the implementation process. The authors hypothesize that the determinants are associated with level of adaptation of the program being implemented.

METHOD

Rogers–Rütten Conceptual Framework

As shown in Figure 1, the Rogers–Rütten conceptual framework includes the following components: relative advantage of using the EBI in the organization compared to using “home grown” interventions, compatibility of the EBI to the priority population that the organization has served, complexity of the goals of the EBI, obligations (service and funding) of the CBO, and resources of the CBO that could influence the implementation process.

This framework combines program characteristic constructs from Rogers’s (1995) diffusion theory—relative advantage, compatibility, and complexity—and organizational constructs from Rütten et al.’s (2003a) theory—opportunities, goals, resources, and obligation. The construct “complexity” from Rogers (1995) and the construct “goals” from Rütten et al. (2003a) are similar and have been combined to create the single construct “complexity of goals.”

The precipitating “opportunity” in this study is the CDC mandate that CBOs must implement EBIs to receive funding using the rationale that these EBIs have been proven effective in research settings (see Figure 1). This opportunity also constitutes a change in policy for the CBOs. Presumably, CBOs receiving funding to implement EBIs have adopted the new approach of only implementing science-based interventions.

Preliminary Measures of Constructs in the Conceptual Framework

Conceptual definitions for each of the independent variable’s relative advantage, compatibility, goals, funding and service obligations, and resources are presented below.

Opportunities

Relative Advantage

Compatibility

Complexity of Goals

Obligation

Resources

Level ofadaptation

DETERMINANTS IMPACT

Figure 1. Rogers–Rütten study framework.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

819

Tabl

e 1.

P

roxy

Var

iabl

es f

or I

mpl

emen

tati

on

Com

mon

Cor

e A

reas

Ele

men

ts N

eede

d to

Mai

ntai

n F

idel

ity

for

Eac

h In

terv

enti

on

Hea

lthy

Rel

atio

nshi

ps3M

VS

ISTA

Str

eet S

mar

tV

oice

s/V

oces

Gro

up

char

acte

rist

ics

1. A

ll p

arti

cipa

nts

are

of

the

sam

e ge

nder

.2.

All

par

tici

pant

s ar

e of

th

e sa

me

sexu

al

orie

ntat

ion.

3. B

etw

een

6 an

d 10

pa

rtic

ipan

ts a

re p

rese

nt

at e

ach

sess

ion.

1. B

etw

een

6 an

d 12

pa

rtic

ipan

ts a

re p

rese

nt a

t ea

ch s

essi

on.

1. B

etw

een

10 a

nd

12 p

arti

cipa

nts

are

pres

ent a

t eac

h se

ssio

n.

1. B

etw

een

6 an

d 10

pa

rtic

ipan

ts a

re

pres

ent a

t eac

h se

ssio

n.

1. B

etw

een

4 an

d 8

part

icip

ants

are

pre

sent

at

eac

h se

ssio

n.

Cor

e to

pics

1. S

tres

s an

d co

ping

ski

lls

are

disc

usse

d in

the

cont

ext o

f di

sclo

sing

H

IV s

tatu

s to

sex

pa

rtne

rs.

2. P

erso

nal f

eedb

ack

repo

rts

(PF

Rs)

are

use

d to

mot

ivat

e th

e pa

rtic

ipan

ts.

1. T

he d

ual i

dent

ity

cult

ure

of B

lack

MS

M is

di

scus

sed

in th

e fi

rst

sess

ion.

2. T

he d

iffe

renc

e am

ong

asse

rtiv

e, a

ggre

ssiv

e, a

nd

nona

sser

tive

beh

avio

rs is

ta

ught

.3.

Par

tici

pant

s ar

e ta

ught

co

ping

ski

lls

1. A

ll th

e se

ssio

ns

begi

n w

ith

read

ing

a po

em.

2. T

here

is a

di

scus

sion

abo

ut

havi

ng p

ride

in

ones

elf.

3. T

he d

iffe

renc

e am

ong

asse

rtiv

e,

aggr

essi

ve, a

nd

nona

sser

tive

beha

vior

s is

taug

ht.

1 P

arti

cipa

nts

are

taug

ht h

ow to

use

pe

er s

uppo

rt to

id

enti

fy tr

igge

rs to

un

safe

beh

avio

r.2.

The

dif

fere

nce

amon

g as

sert

ive,

ag

gres

sive

, and

no

nass

erti

ve

beha

vior

s is

taug

ht.

1. T

here

is a

dis

cuss

ion

abou

t hav

ing

prid

e in

on

esel

f.2.

The

dif

fere

nce

amon

g as

sert

ive,

agg

ress

ive,

and

no

nass

erti

ve b

ehav

iors

is

taug

ht.

3. T

here

is a

dis

cuss

ion

abou

t how

to n

egot

iate

sa

fer

sex

beha

vior

s.

Cor

e m

etho

ds1.

Dec

isio

n-m

akin

g sk

ills

ar

ound

HIV

sta

tus

disc

losu

re a

re ta

ught

.2.

The

sug

gest

ed te

achi

ng

met

hods

out

line

d in

the

curr

icul

um a

re u

sed

to

teac

h be

havi

oral

ski

lls.

1. T

he s

ugge

sted

teac

hing

m

etho

ds o

utli

ned

in th

e cu

rric

ulum

are

use

d to

te

ach

beha

vior

al s

kill

s.2.

Par

tici

pant

s ar

e ta

ught

ho

w to

neg

otia

te s

afer

sex

be

havi

ors.

1. T

he s

ugge

sted

te

achi

ng m

etho

ds

outl

ined

in th

e cu

rric

ulum

are

us

ed to

teac

h be

havi

oral

sk

ills

.

1. T

he s

ugge

sted

te

achi

ng m

etho

ds

outl

ined

in th

e cu

rric

ulum

are

use

d to

teac

h be

havi

oral

sk

ills

.

1. T

he s

ugge

sted

teac

hing

m

etho

ds o

utli

ned

in th

e cu

rric

ulum

are

use

d to

te

ach

beha

vior

al s

kill

s.2.

Mat

eria

ls a

re u

sed

wit

hout

cha

nges

.

(con

tinu

ed)

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

820

Tabl

e 1.

(c

onti

nued

)

Com

mon

Cor

e A

reas

Ele

men

ts N

eede

d to

Mai

ntai

n F

idel

ity

for

Eac

h In

terv

enti

on

Hea

lthy

Rel

atio

nshi

ps3M

VS

ISTA

Str

eet S

mar

tV

oice

s/V

oces

3. P

arti

cipa

nts

are

taug

ht

how

to n

egot

iate

saf

er

sex

beha

vior

s.

2. P

arti

cipa

nts

are

taug

ht h

ow to

ne

goti

ate

safe

r se

x be

havi

ors.

Cor

e m

ater

ials

1. C

urri

culu

m m

ater

ials

ar

e us

ed w

itho

ut

chan

ges.

1. I

nfor

mat

iona

l too

ls (

such

as

han

dout

s) a

re p

rovi

ded

in th

e se

ssio

n.

1. I

nfor

mat

iona

l to

ols

(suc

h as

ha

ndou

ts)

are

prov

ided

in th

e se

ssio

n.

1. I

nfor

mat

iona

l too

ls

(suc

h as

han

dout

s)

are

prov

ided

in th

e se

ssio

n.

1. I

nfor

mat

iona

l too

ls (

such

as

han

dout

s) a

re

prov

ided

in th

e se

ssio

n.

NO

TE

: 3M

V =

Man

y M

en, M

any

Voi

ces;

SIS

TA =

Sis

ters

Inf

orm

ing

Sis

ters

abo

ut T

opic

s on

AID

S; M

SM

= m

en w

ho h

ave

sex

wit

h m

en.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 821

The independent variables were measured using a 5-point Likert-type scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree).

An index to assess the dependent variable, level of adaptation during implementation, was also developed, and it used a 4-point response format (1 = never, 2 = some of the time, 3 = most of the time, and 4 = all the time). Index development procedures are described below.

Dependent Measure Development

Level of Adaptation. Adaptation was assessed for five group-level interventions (GLIs)—Healthy Relationships; Many Men, Many Voices (3MV); Sisters Informing Sisters about Topics on AIDS (SISTA); Street Smart; and Voices/Voces. Level of adaptation for each EBI was assessed using a proxy measure of adherence to the four core components com-mon to all programs: group characteristics, core topics, core methods for behavior change, and core materials. These elements of the curriculum are thought to be responsible for program effectiveness in facilitating behavior change and they represent the intent, theory, and internal logic of each intervention, as well as the interventions’ recommended activi-ties and delivery methods (Gandelman et al., 2006). Table 1 presents the index items used to assess the adaptation of each of the GLIs assessed in this study.

Across the five GLIs, the number of required elements within each of the four core components (group characteristics, core topics, core methods for behavior change, and core materials) varied by intervention. Therefore, in the first step to operationalize the items for the implementation or adaptation index, each published curriculum was used to identify all elements that comprised implementation fidelity for each of the GLIs. Second, the ele-ments for each intervention were classified into the four core components (group charac-teristics, core topics, core methods, and core materials). Third, index items were developed for each intervention based on the number of required elements in each of the four core components. Thus, the number of items to assess each core component varied across pro-grams based on features unique to each of the GLIs. Fourth, the mean score was used as an indicator of implementation fidelity for each of the four core components; higher scores indicated greater fidelity. Thus, even though the number of items within a category may vary from program to program (e.g., different numbers and types of topics are covered), these scores are comparable across the different EBIs. Finally, a single score to reflect adaptation was created by taking the average of each mean across the four core components (Table 1), with a higher score indicating greater fidelity or lower adaptation.

Group Characteristics. These were defined as the required number of intervention participants from the target population for whom the intervention is designed. For example, Healthy Relationships is targeted toward groups of 5 to 12 HIV-positive heterosexual men and women and HIV-positive men who have sex with men, 3MV targets groups of 6 to 12 men of color who have sex with men, SISTA target groups of 10 to 12 African American heterosexual women, Street Smart targets groups of 6 to 10 runaway youth, and Voices/Voces targets groups of 4 to 8 men and women of color.

Each EBI curriculum specifies the core topics, core methods, and core materials. Core topics are the key issues that are to be discussed at each session, core methods are the techniques needed to be used in teaching the topics, and core materials are the tangible items needed to teach the lesson effectively. Using Healthy Relationships as an example, core topics of the curriculum would include a session in which stress is defined and coping skills are reinforced in three life areas: (a) disclosing to family and friends, (b) disclosing to sex partners, and (c) building healthier and safer relationships. Core methods of delivery

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

822 Health Education & Behavior (December 2010)

of these topics would include using modeling, role-play, and feedback to teach and practice skills related to coping with stress, teaching decision-making skills around the issue of disclosure of HIV status, and providing participants with personal feedback reports to motivate change of risky behaviors and continuance of protective behaviors. Core materials would be the resources required, such as using popular movie clips, to set up scenarios around disclosure and risk reduction to stimulate discussions and role-play handouts or an overhead projector to implement the session.

Independent Measures

Relative Advantage. This two-item scale developed for the study measured the degree to which the selected EBI is perceived as better than the program it is replacing. The items were reverse coded, summed, and the mean of the sum of scores was used for the analyses, with a higher score indicating that the EBI was perceived as advantageous compared to prior CBO practices.

Compatibility. This three-item scale created for the study assessed how well goals of the chosen EBI fit with the CBO’s goals, mission, values, beliefs, and experiences. A higher mean score indicated higher levels of compatibility.

Complexity of Goals. The original eight-item scale used in Rütten’s health policy impact study demonstrated an internal consistency reliability of α = .54 (Rütten et al., 2003b). For the present study, four items from the original scale were adapted and used to assess the perceived complexity or simplicity of the broader goals of the program. A higher mean score indicated lower levels of complexity of goals.

Obligations. For the purpose of the study, five items from the original seven-item obli-gation scale were adapted and used to assess the two types of obligations: funding and service. The original scale developed by Rütten demonstrated an internal consistency reli-ability of α = .66 (Rutten et al., 2003b). The two-item funding obligation scale in the present study measured the CBO’s level of need to use the EBI as driven by their desire to receive or retain CDC funding. The three-item service obligation scale measured the CBO’s need to use the EBI based on their service community’s need for the program. The mean score of the responses was used for analysis, with higher scores indicating higher levels of fund-ing and service obligation.

Resources. Seven items of the original 10-item scale developed by Rütten were used for the present study and were designed to measure the capacity of the organization, including having trained personnel available to implement the program, sufficient finances to carry out the program, organizational support to implement the program, and outside support for program implementation. The original 10-item scale had an internal consistency reliability of α = .79 (Rütten et al., 2003b). The 7 items used in the present model were then summed to provide sum of scores. The mean of the scores was used for analyses, with higher scores indicating higher levels of resources.

Design Characteristics

The study used a cross-sectional design and obtained information using an online questionnaire service. Through the service, the URL to the questionnaire was individually emailed to each of the participants. Eligibility criteria for this study included HIV

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 823

prevention program managers who had completed the CDC/ASPH Institute for HIV Prevention Leadership (IHPL; Richter et al., 2006; Richter et al., 2007) since 2000, who worked for their original CBO at the time the questionnaire was administered, and whose CBOs were implementing one of the five GLIs. IHPL is an intensive capacity-building program designed to enhance the HIV prevention capacity of CBOs that serve popula-tions and/or communities affected by or at risk for HIV/AIDS, especially minority communities.

Analysis

Data were analyzed using Statistical Analysis System (SAS v.9.0). Initial data analysis included descriptive statistics for normality indicating that the data demonstrated a normal distribution. Frequency procedures and means were also performed for the demographic variables. Calculations for means along with ranges and standard deviations were performed on the dependent and independent variables from the questionnaire.

The content validity was ascertained using expert assessment of instrument items and their applicability with constructs from the Rogers–Rütten framework. Bivariate Pearson correlational analysis was used to establish construct validity by assessing the relationship between the independent (determinants) and dependent (implementation or adaptation) subscales. Cronbach’s alpha was used to assess the internal consistency of each independent variable. For all analyses, the significance was set at p <.05 unless otherwise stated.

RESULTS

Response Rates and Sample Characteristics

Of the 220 HIV program managers approached for the study, 99 participants (45% response rate) completed the questionnaire, of which 63 (29% useable response rate) were used for analysis; 36 questionnaires had only partial responses on key items.

The respondents for each organization consisted of executive directors (19%), program managers (42%), and other senior staff members (39%). The reporting CBOs were implementing one of five types of programs: Healthy Relationships (16%), 3MV (14%), SISTA (40%), Street Smart (5%), and Voices/Voces (25%).

Total operating expenditures were categorized as either less than $299,999 (61%) or more than $300,000 (39%) for HIV prevention programs. Fifty-nine percent of the respond-ing CBOs employed 29.9 or less paid full-time equivalents (FTEs) and 41% employed more than 30 FTEs. For HIV prevention, 48% of the CBOs employed 3.9 or less FTEs whereas the other 52% employed 4.0 to 6.0 FTEs. Most CBOs (63%) have been operating for 10 years or less and serve a population size of 100,000 or more (66%), and the racial and ethnic population most frequently targeted for intervention among the reporting CBOs was African American (68%).

Determinant Scale Reliabilities and Correlations

The description of the determinant scale, including internal consistency reliabilities, means, and scale items for each of the determinants, is presented in Table 2. Internal consistency reliabilities for the determinant scales demonstrated Cronbach’s alpha values ranging from .63 to .88. Means ranged from 3.1 to 4.2, indicating that perceptions of EBI

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

824 Health Education & Behavior (December 2010)

Table 2. Description of Determinants scales (Independent Variables)

VariableNo. of Items

Internal Reliability (α) Ma Instrument Items

Relative advantage

2 .63 3.1 To what extent do you agree with the following?1. This EBI takes a long time to implement

compared to interventions that are not EBIs.2. Non-EBIs are less costly than EBIs.

Compatibility 3 .77 4.2 To what extent do you agree with the following?1. This EBI is compatible with the goals of my

CBO.2. This EBI fits well with the goals of my CBO.3. The scheduling of sessions in this EBI is

compatible with the work schedule of employees in my CBO.

Goals 4 .88 4.1 To what extent do you agree with the following?1. The goals of the EBI are clear.2. The EBI is easy to implement.3. The organization of the curriculum is easy to

follow.4. The format of the curriculum is easy to follow.

Funding obligation

2 .76 3.3 To what extent do you agree with the following?

1. Our funding agency required us to implement the EBI.

2. Our CBO will lose funding if we do not use this EBI.

Service obligation

3 .77 4.0 To what extent do you agree with the following?1. We used this EBI because of its scientific

merit.2. The current HIV trends in our community

indicate the need to use this EBI.3. Providing this EBI is part of our CBO’s

mission.Resources 7 .84 4.2 To what extent do you agree with the following?

1. The staff has the skills to implement this intervention.

2. There is enough space at my CBO to conduct the intervention sessions.

3. There are enough personnel at my CBO to conduct the intervention sessions.

4. Our CBO fully supports the intervention.5. My CBO’s stakeholders support the

intervention.6. My CBO’s collaborators support the

intervention.7. Our CBO has the supplies needed to

implement this EBI.

NOTE: EBI = evidence-based intervention; CBO = community-based organization.a. Range of scores: 1= strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 825

attributes ranged from neutral (i.e., 3 = neither agree or disagree) to somewhat positive (i.e., 4 = strongly agree).

Correlations among the determinant scale scores are presented in Table 3. Compatibility was found to have significant positive correlations with relative advantage (r = .25, p < .05), goals (r = .40, p < .05), service obligation (r = .49, p < .001), and resources (r = .41, p < .05). Thus, perceptions of compatibility are positively correlated with perceptions that the new program is perceived as advantageous compared to the former programs, that new program goals were perceived as relatively easy to understand, that the new program is consistent with service obligations, and that the organization has the needed resources to carry out the new program. Similarly, goals were found to be significantly positively cor-related with service obligations (r = .34, p < .05) and resources (r = .48, p < .05), indicating that perceptions of clear EBI program goals are associated with perceptions that the new EBI program is consistent with service obligations and having sufficient organizational resources to carry out the new program. Finally, there was a significant positive correlation between service obligations and resources (r = .51, p < .05), indicating a positive associa-tion between perceptions that the new EBI program is consistent with service obligations and having sufficient organizational resources to carry out the new program.

No significant correlations were found between funding obligations and any other variables.

Level of Adaptation Index and Determinant Scales Correlations

In Table 3 are the correlations between level of adaptation and the determinants. Relative advantage was the only determinant that was found to be significantly correlated with adaptation during program implementation (r = .29, p < .05). Thus, perception that the EBI is advantageous compared to the intervention previously used is positively associated with implementing the EBI with lower levels of adaptation.

DISCUSSION

Research on the topic of adaptation of EBIs in HIV prevention is becoming prominent in the literature (Chillag et al., 2002; Collins et al., 2006; Galbraith, 2004; Kelly et al., 2000). The purpose of this article was twofold: (a) to introduce the Rogers–Rütten model

Table 3. Adaptation Scales Correlations

Relative Funding Service Level of Advantage Compatibility Goals Obligation Obligation Adaptation (n = 60) (n = 60) (n = 60) (n = 60) (n = 60) (n = 63)

Relative advantage – .29*Compatibility .25* – .09Goals .19 .40* – .11Funding obligation .006 .03 .06 – .20Service obligation .14 .49** .34* -.01 – .20Resources .19 .41* .48* .16 .51** .18

**p < .001. *p < .05.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

826 Health Education & Behavior (December 2010)

as a framework to explore factors that affect program implementation as indicated by level of adaptation of the EBIs and (b) to report preliminary measures for the framework’s constructs, relative advantage, compatibility, goals, resources, and obligations.

A key discussion point in this study is the low response rate. The sampling universe consisted of 220 HIV program managers who were implementing EBI(s) in their CBOs’ target population. The researchers had no prior knowledge of which CBOs among the total study population were implementing GLIs. Only later at analysis through the survey’s response tree did we gain a better understanding. One of the early questions asked the respondent if the CBO was implementing a GLI; if the response was no, then the respon-dent completed demographics only. During analysis, we found that only 99 participants responded that they were implementing a GLI. Of these 99 responses, only 63 were use-able as 36 had incomplete responses on key items.

To increase the response rate, a modified Dillman’s (2000) Internet and Interactive Voice Response Approach was used to ensure a maximum response rate. Two weeks prior to the date the survey was going to be launched, a prenotification email was sent to the participants notifying them of the upcoming survey. A second letter was sent on the day the survey was launched containing the uniform resource locator (URL) to the survey. After the survey was launched, email reminders were sent each week for 3 months to participants who had not completed the survey. Data collection was extended for an extra 2 weeks, during which time personalized emails were sent to nonrespondents.

To the best of our knowledge, this is the first study to examine the development of a framework for identifying and defining factors that influence adaptation of programs being implemented in community settings (or conversely, fidelity of implementation). This article also describes how these constructs, including relative advantage, compatibility, goals, resources, and obligations, and program adaptation are measured. There are no scales or instruments that have been validated in this field to which we can compare the results of the Rogers-Rütten framework used in this study. Most studies have investigated factors influencing fidelity or adaptation of implementation qualitatively. This framework can serve as a tool that can guide researchers as they give increased real-world thought to intervention design. For example, by engaging with practitioners, researchers will better understand the impact that agency policies, mission statements, and funding streams have on EBI implementation (Jenkins & Carey, 2005).

This study also presents preliminary measures to assess organizational factors that influence adaptation during implementation. All subscales demonstrated acceptable to very good internal consistency reliabilities (Devillis, 2003), with Cronbach’s alpha values ranging from α = .63 to .88. Most scales were moderately correlated, which was expected; however, the correlations were not high, which provides support for using each as an independent measure.

Furthermore, this study presents a technique to assess implementation fidelity and adaptation that involved a method of assessing the core components across interventions. This approach makes it possible to use a single score to directly compare the implemen-tation of different interventions. We believe this approach to assessing implementation overcomes a common limitation in these types of studies (e.g., organizations implement-ing different programs) and could enable the field to continue to move forward in under-standing the implementation and adaptation process.

In general, findings of this study are timely and consistent with previous recommenda-tions for organizational assessment before implementing EBIs (Gandelman et al., 2006; Peterson & Randall, 2006). Specifically, the study found an association between perceptions that an EBI is advantageous compared to interventions that were previously implemented.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 827

Thus EBIs that are considered advantageous to previously used programs are implemented with higher fidelity (or less adaptation). These findings are similar to previous authors who have reported that CBOs are more likely to implement EBIs that are closely related to the community they seek to affect (Collins et al., 2006; Lyles, Crepaz, Herbst, & Kay, 2006).

This study provides preliminary support for the validity of the Rogers–Rütten conceptual model. These initial findings take us one step further into the building of the empirical evidence associated with implementation of EBIs with less adaptation. As a result, it begins to fill the need for empirical, data-driven evidence (Gandelman et al., 2006) showing preliminary support for the validity of the Rogers–Rütten model.

LIMITATIONS

There are some limitations to our study that should be considered. The primary limita-tion to this study is the low response rate. Therefore, there may be a selection bias in that CBOs who were less likely to be implementing the GLI with less fidelity may have chosen not to respond to the survey. Furthermore, although IHPL seeks to maintain an up-to-date alumni database, many of the nonrespondents were participants of the very early years of IHPL whose addresses may have changed. Some also were no longer program managers and/or employees of the CBO that they were representing at the time of the intervention. Also, participants in this study were IHPL participants who are taught organizational management and prevention program implementation skills at the Institute, and it is pos-sible that they tended to provide socially desirable responses on the survey.

In addition, this study used a cross-sectional research design; therefore, study conclu-sions are limited to correlational relationships, not causation, and the scales used in the study require further validation with a larger sample size that would make it possible to conduct more sophisticated analyses. Another limitation is that of recall bias for the primary questionnaire respondent. It is likely that some HIV prevention program managers who responded to the questionnaire had not implemented the EBI recently. This study was conducted in 2007, and the compendium of HIV Prevention Interventions with Evidence of Effectiveness was first published in 1999. With the exception of 3MV, which was published in 2006, all the GLIs have been established interventions since 1999. Ideally, data collection for this type of study should take place close to the time at which CBOs first started using the EBIs.

IMPLICATIONS FOR PRACTICE

In spite of limitations, we feel this study contributes to the investigation of factors influencing implementation of EBIs in community settings. Given the vast differences in community-based organizational structures, identifying common significant factors that influence level of adaptation during implementation of an EBI is a daunting task. The complexity and variation in services, funding streams, and other factors have made it extremely difficult in the past to identify and assess the factors that affect implementation. This study presents first step empirical data on factors affecting level of adaptation of HIV prevention programs implemented by CBOs as an early approach to exploring these complex factors. The findings build on qualitative studies that have illustrated factors at the organizational level that act as barriers to delivery of HIV prevention services by CBOs (Chillag et al., 2002; Miller, 2001).

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

828 Health Education & Behavior (December 2010)

The development of the Rogers-Rütten framework was a first step in attempting to understand the various program and organizational factors that affect implementation or adaptation in service organization settings. It is hoped that these findings will help research-ers developing behavioral interventions to consider the realities that local community organizations face as they serve their target populations. These include the pervasive challenge of accommodating resource constraints, the importance of providing culturally appropriate programs, and meeting requirements from funders. The world in which CBOs operate is complex and dynamic, and very different compared to the controlled and well-funded context in which the interventions were created. Therefore, some adaptation may be appropriate. Adapting a compatible intervention can reduce the costs of designing a new program, increase community ownership of the program, and potentially build the capacity of local community members. Furthermore, adaptation while maintaining pro-gram integrity is an opportunity to bridge the science and practice as practitioners and researchers gain information about the unique needs of the population being served and how these needs affect and change interventions. Adaptation also calls practitioners to consider the “theory” in the programs they deliver so that they are well implemented and properly evaluated (Lee et al., 2008).

References

Berman, P., & McLauglin, M. (1976). Implementation of educational innovation. Education Forum, 40, 345-370.

Blankenship, K. M., Bray, S. J., & Merson, M. H. (2000). Structural interventions in public health. AIDS, 14, S11-S21.

Centers for Disease Control and Prevention. (2007). Racial/ethnic disparities in diagnoses of HIV/AIDS—33 States, 2001-2005. MMWR Morbidity and Mortality Weekly Report, 56, 189-193.

CDC HIV/AIDS Prevention Research Synthesis Project. (1991). Compendium of HIV prevention interventions with evidence of effectiveness. Atlanta, GA: Centers for Disease Control and Prevention.

Chillag, K., Bartholow, K., Cordeiro, J., Swanson, S., Patterson, J., Stebbins, S., . . . Sy, F. (2002). Factors affecting the delivery of HIV/AIDS prevention programs by community-based organizations. AIDS Education and Prevention, 14(Suppl. A), 27-37.

Collins, C., Harshbarger, C., Sawyer, R., & Hamdallah, M. (2006). The diffusion of effective behavioral interventions project: Development, implementation, and lessons learned. AIDS Education and Prevention, 18(4 Suppl. A), 5-20.

Devillis, R. F. (2003). Scale development: Theory and applications. London, UK: Sage.DiClemente, R. J. (2000). Future directions for HIV prevention research. In J. L. Peterson &

R. J. DiClemente (Eds.), Handbook of HIV prevention (pp. 83-102). New York, NY: Kluwer Academic/Plenum.

Dillman, D. A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York, NY: John Wiley.

Eveland, J. D. (1977). The innovative process in public organizations [Mimeo]. Ann Arbor: University of Michigan, Department of Journalism.

Foster, S., & Niederhausen, P. (2002). Federal HIV/AIDS spending: A budget chartbook, fiscal year 2002. Retrieved from http://www.kff.org/hivaids/1633-index.cfm

Galbraith, J. (2004). An examination of the re-invention process of a health promotion program: The changes and evolution of “FOCUS ON KIDS” HIV prevention program (Doctoral dis-sertation, University of Maryland, College Park).

Gandelman, A. A., Desantis, L. M., & Rietmeijer, C. A. (2006). Assessing community needs and agency capacity—An integral part of implementing effective evidence-based interventions. AIDS Education and Prevention, 18(4 Suppl. A), 32-43.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

Bowen et al. / Rogers-Rütten Framework 829

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innova-tions in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581-629.

Hall, H. I., Song, R., Rhodes, P., Prejean, J., An, Q., Lee, L. M., . . . Janssen, R. S.; HIV Incidence Surveillance Group. (2008). Estimation of HIV incidence in the United States. Journal of the American Medical Association, 300, 520-529.

Jemmott, J. B., 3rd, & Jemmott, L. S. (2000). HIV risk reduction behavioral interventions with heterosexual adolescents. AIDS, 14(Suppl. 2), S40-S52.

Jenkins, R. A., & Carey, J. W. (2005). Decision making for HIV prevention planning: Organizational considerations and influencing factors. AIDS and Behavior, 9(2 Suppl.), S1-S8.

Kelly, J. A., Heckman, T. G., Stevenson, L. Y., Williams, P. N., Ertl, T., Hays, R. B., . . . Neumann, M. S. (2000). Transfer of research-based HIV prevention interventions to community service providers: Fidelity and adaptation. AIDS Education and Prevention, 12(5 Suppl.), 87-98.

Kelly, J. A., Somlai, A., DiFranceisco, W. J., Otto-Salaj, L. L., McAuliffe, T. L., Hackl, K. L., . . . Rompa, D. (2000). Bridging the gap between the science and service of HIV prevention: Transferring effective research-based HIV prevention interventions to community AIDS service providers. American Journal of Public Health, 90, 1082-1088.

Kilbourne, A., Schulberg, H., Post, E., Rollman, B., Belnap, B., & Pincus, H. (2004). Translating evidence-based depression management services to community-based primary care services. Milbank Quarterly, 82, 631-659.

Lee, S., Altschul, I., & Mowbray, C. (2008). Using planned adaptation to implement evidence-based programs with new populations [Original paper]. American Journal of Community Psychology, 41, 290-303.

Lyles, C. M., Crepaz, N., Herbst, J. H., & Kay, L. S. (2006). Evidence-based HIV behavioral prevention from the perspective of the CDC’s HIV/AIDS Prevention Research Synthesis Team. AIDS Education and Prevention, 18(4 Suppl. A), 21-31.

Mantell, J., & Divittis, A. (1990). Evaluating AIDS prevention: Contributions of multiple disciplines. San Francisco, CA: American Evaluation Association.

McKleroy, V. S., Galbraith, J. S., Cummings, B., Jones, P., Harshbarger, C., Collins, C., . . . Carey, J. W.; ADAPT Team. (2006). Adapting evidence-based behavioral interventions for new settings and target populations. AIDS Education and Prevention, 18(4 Suppl. A), 59-73.

Miller, R. L. (2001). Innovation in HIV prevention: Organizational and intervention characteristics affecting program adoption. American Journal of Community Psychology, 29, 621-647.

Peterson, A. S., & Randall, L. M. (2006). Utilizing multilevel partnerships to build the capacity of community-based organizations to implement effective HIV prevention interventions in Michigan. AIDS Education and Prevention, 18(4 Suppl. A), 83-95.

Richter, D. L., Dauner, K. N., Lindley, L. L., Reininger, B. M., Oglesby, W. H., Prince, M. S., . . . Potts, L. H. (2007). Evaluation results of the CDC/ASPH Institute for HIV Prevention Leader-ship: A capacity-building educational program for HIV prevention program managers. Journal of Public Health Management and Practice, Suppl., S64-S71.

Richter, D. L., Potts, L. H., Prince, M. S., Dauner, K. N., Reininger, B. M., Thompson-Robinson, M., . . . Jones, R. (2006). Development of a curriculum to enhance community-based organizations’ capacity for effective HIV prevention programming and management. AIDS Education and Prevention, 18(4), 362-374.

Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York, NY: Free Press.Rütten, A., Luschen, G., von Lengerke, T., Abel, T., Kannas, L., Rodriguez Diaz, J. A., . . . van

der Zee, J. (2003a). Determinants of health policy impact: A theoretical framework for policy analysis. Sozial- und Praventivmedizin, 48, 293-300.

Rütten, A., Luschen, G., von Lengerke, T., Abel, T., Kannas, L., Rodriguez Diaz, J. A., . . . van der Zee, J. (2003b). Determinants of health policy impact: Comparative results of a European poli-cymaker study. Sozial- und Praventivmedizin, 48, 379-391.

Saleh-Onoya, D., Braxton, N., Sifunda, S., Reddy, P., Ruiter, R., van den Borne, B., . . . Wingood, G. M. (2008). SISTA South Africa: The adaptation of an efficacious HIV prevention trial conducted

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from

830 Health Education & Behavior (December 2010)

with African-American women for isiXhosa-speaking South African women. SAHARA J: Journal of Social Aspects of HIV/AIDS Research Alliance/SAHARA, Human Sciences Research Council, 5, 186-191.

Sogolow, E. D., Kay, L. S., Doll, L. S., Neumann, M. S., Mezoff, J. S., Eke, A. N., . . . Anderson, J. R. (2000). Strengthening HIV prevention: Application of a research-to-practice framework. AIDS Education and Prevention, 12(5 Suppl.), 21-32.

Solomon, J. (2002, January). Fidelity vs. flexibility in the replication of teen pregnancy STD/HIV/AIDS prevention programs. Paper circulated for the National Institute of Child Health and Human Developments “The Science of Intervention Replication” Workshop, Bethesda, MD.

Stanton, B. F., Li, X., Ricardo, I., Galbraith, J., Feigelman, S., & Kaljee, L. (1996). A randomized, controlled effectiveness trial of an AIDS prevention program for low-income African-American youths. Archives of Pediatrics & Adolescent Medicine, 150, 363-372.

Sumartojo, E. (2000). Structural factors in HIV prevention: Concepts, examples, and implications for research. AIDS, 14(Suppl. 1), S3-S10.

Tappe, M., Galer-Unti, R., & Bailey, K. (1995). Long-term implementation of the teenage health modules by trained teachers: A case study. Journal of School Health, 65, 411-415.

at GEORGE MASON UNIV on April 16, 2013heb.sagepub.comDownloaded from All in-text references underlined in blue are linked to publications on ResearchGate, letting you access and read them immediately.