Implementation of School-Wide Positive Behavioral Interventions and Supports (PBIS) in Elementary...

EDUCATION AND TREATMENT OF CHILDREN Vol. 31, No. 1 2008

Implementation of School-Wide PositiveBehavioral Interventions and Supports (PBIS)in Elementary Schools: Observations from a

Randomized Trial

Catherine P. Bradshaw

Center for the Prevention of Youth ViolenceJohns Hopkins Bloomberg School of Public Health

Wendy M. Reinke

Louis D. Brown

Katherine B. Bevans

Johns Hopkins Bloomberg School of Public Health

Philip J. Leaf

Center for the Prevention of Youth ViolenceJohns Hopkins Bloomberg School of Public Health

Abstract

High fidelity implementation of school-wide PBIS has been linked with

improvements in student and staff behavior, but less is known about which

aspects of the model may be present in schools prior to training, and whether

some features of PBIS are implemented faster than others. The present

study examines the progression of school-wide PBIS implementation using

School-wide Evaluation Tool data collected across three years from 21 schools

randomly assigned to receive training in PBIS and 16 schobls not trained in

PBIS. Trained schools evidenced significantly higher levels of implementation

fidelity. Non-trained schools showed some increases, but lagged behind

trained schools on all subscales except Systems for Responding to Violations.

Findings suggest that program trainers and behavior support coaches should

concentrate initial efforts on strategies for Defining and Teaching Expectations,

whereas less time may be needed for developing Systems for Responding to

Violations. Recommendations for high fidelity implementation of school-widePBIS are provided.

ncreased legislative pressure on schools to prevent disruptive

and violent behavior has likely contributed to the popularity of

Correspondence to Catherine PR Bradshaw, Center for the Prevention of Youth Violence,Johns Hopkins Bloomberg School of Public Health, Johns HopkinsUniversity, Baltimore,MD 21205; e-mail: [email protected].

SPages1-26

BRADWSHAW et al.

preventive interventions,, such as Positive Behavioral Interventionsand Supports (PBIS). This whole-school preventive interventionstrategy is currently implemented in over 5,000 schools nationwide(Sugai & Homer, 2006) and aims. to prevent disruptive behavior andenhance the school's organizational climate by creating and sustaininga comprehensive system of behavioral support. Previous researchhas indicated that implementation of school-wide PBIS is associatedwith a reduction in office discipline referrals (Taylor-Greene et al.,1997) and suspensions (Homer, SugAi, Todd, & Lewis-Palmer, 2005),and improvements in academic performance (Nelson, Martella, &Marchand-Martella, 2002).

Because PBIS procedures are rooted in basic and commonly un-derstood behavioral, social learning, and organizational principles,it is likely that some schools.implement aspects of school-wide PBISbefore receiving formal training in the procedures. To date, there hasbeen no research systematically documenting the features of school-wide PBIS that are present in schools prior to training, or describingthe progression of school-wide PBIS implementation once training hasoccurred. Identifying core components of school-wide PBIS that areevident in the absence of formal "training and determining the com-ponents of school-wide PBIS which are implemented with the great-est ease may facilitate the training process. This information may alsohelp behavior support coaches, district liaisons, and other PBIS train-ers concentrate their support efforts on aspects of the model which aremost challenging to implement.

Positive Behavioral Interventions and Supports

Whole-school PBIS is the application of a continuum of posi-tive behavior supports, initially developed for children with specialeducation needs, to the entire school environment. The PBIS modelis three-tiered, which includes a layer of "primary supports" imple-mented school-wide (referred to from here on as school-wide PBIS),a "secondary supports" level of targeted interventions for groups§ ofchildren with greater needs, and a "tertiary supports" layer of moreintensive supports for individual students (for a review, see Sugai,Homer, & Gresham, 2002). This proactive, preventive approach con-trasts with a reactive approach where student behavior problems areaddressed only after they have occurred. The school-wide PBIS modelutilizes universal positive preventive support strategies that providesystematic training of expected social behaviors and reinforcement ofthose behaviors to all students in the school. Approximately 80% to90% of students are projected to respond successfully to the School-wide component of PBIS. A second group of students (about 5-10%).who engage in problem behaviors beyond acceptable levels, even in

2

IMPLEMENTATION OF SCHOOL-WIDE PBIS 3

the presence of school-wide PBIS, are provided secondary supportsthat include efficient group-based interventions with increased struc-ture and contingent feedback (e.g., Behavior Education Program,check-in/check-out procedures). A third and even smaller number ofstudents (1-5%) who enter schools with significant skills deficits thatdo not respond to school-wide or secondary interventions, will needmore intensive individualized interventions in order to succeed inschool (e.g., function-based behavioral supports). The main objectiveof the three-tiered PBIS model is to establish a positive school climatefor all students and create systems to foster durable changes in thebehavior of higher-needs students (see Homer et al. (2005) and Sugai'and Homer (2002; 2006) for a full description of the implementationof school-wide PBIS).The present paper focuses on the school-wide primary supports

level of this three-tiered model, which targets all students in an ef-fort to prevent behavior problems before they occur. The school-widePBIS model is multi-component and requires that schools have strongadministrative leadership and commitment to the model; have buy-in from school staff; have adequate personnel, time, and resources tosupport the model; and have access to an information system to trackand manage their student discipline data. An important aspect of PBISis the on-going evaluation of program implementation fidelity.

Measuring fideliht of school-wide PBIS. The developers of PBIScreated the School-wide Evaluation Tool (SET; Sugai, Lewis-Palmer,Todd, & Homer, 2001) as a measure of the degree to which schools areimplementing the key features of school-wide PBIS (Homer, Todd,Lewis-Palmer, Sugai, & Boland, 2004). The SET is typically completedannually by a trained external observer who assesses the followingseven key features of school-wide PBIS: Expectations Defined (three tofive positive sch6ol-wide behavioral expectations are defined); Behav-

iorial Expectations Taught (these expectations are taught to all childrenin the school); System for Rewarding Behavioral Expectations (rewardsare provided for meeting the behavioral expectations); System for Re-sponding to Behavioral Violations (a consistently implemented contin-uum of consequences for problem behavior is in place); Monitoringand Evaluation (behavior patterns are monitored and the informationis used for ongoing decision-making); Management (an administratoractively supports and is involved in the PBIS efforf, and a comprehen-sive school-wide behavior support team is formed); and District-LevelSupport (the school district provides support to the school'in the formof functional policies, staff training, and data collection opportuni-ties).

The instrument yields seven subscale scores (ranging from 0%

I

BRADWSHAW et al.

to 100%), with higher scores indicating greater program fidelity. Anoverall summary score can be computed by averaging all seven keyfeatures (referred to as the Overall SET score), which also ranges 0-100%. The developers of PBIS posited that the intended benefits ofthe program occur when school-wide PBIS is implemented with atleast 80% fidelity (Homer et al., 2004; Sugai et al., 2001). Furthermore,schools are expected to continue to experience improvements or stabi-lize once this level of implementation has been reached. Although theSET is widely used to track implementation fidelity of school-widePBIS procedures, there has been little published literature that sys-tematically utilizes the SET to document the adoption of the key fea-tures of school-wide FBIS (see Homer et al., 2005 and Nelson, 1996 fora brief discussion of this issue).

Overview of the Current Study

The present study examines the progression of school-wide PBISimplementation using data from 37 elementary schools that volun-teered to participate in a group randomized trial of PBIS. Of theseschools, 21 received training in PBIS (referred to as "trained schools")and 16 did not receive training and refrained from implementing themodel for the duration of the five-year study (referred to as "non-trained schools"). The SET data from these two groups were ana-lyzed to determine the impact of training in PBIS on school-wide PBISimplementation fidelity. We also documented which componentsof school-wide PBIS were implemented in trained and non-trainedschools across the three-year study. Lastly, we provided recommen-dations for behavior support coaches and school personnel workingto support high fidelity implementation of school-wide PBIS.

Hypotheses. Our primary hypotheses focused on the impact oftraining in PBIS on the implementation of the core features 'of themodel. Consistent with prior research indicating that training in PBISleads to sustained changes in the schools' internal discipline practicesand systems (Nersesian, Todd, Lehmann, & Watson, 2000; Taylor-Greene & Kartub, 2000), we anticipated that training in PBIS wouldbe associated with a significant increase in all SET subscale scores. Wealso predicted that formal training in PBIS would hasten a paradigmshift in behavior management approaches, from a traditionally pu-nitive to a positive preventive approach. Our secondary hypothesesfocused on the progression of PBIS implementation. Given that manyof the key concepts of school-wide PBIS are accessible and easily un-derstood by school staff, it was hypothesized that at baseline both thetrained and non-trained schools would display some components ofschool-wide PBIS. We also expected non-trained schools to continue

4

I

IMP,LEMENTATION OF SCHOOL-WIDE PBIS

to implement some components of school-wide PBIS, but at a lowerlevel than the trained schools across the three-year trial. Specifically,we expected that non-trained schools would display the traditionalfocus on punitive or reactive responses to behavioral disruptions(Strain, 2004), and thus would continually receive lower scores thantrained schools on Defining Behavioral Expectations and Teaching Behav-ioral Expectations. However, due to the increased legislative pressureon schools to monitor student infractions and increase student perfor-mance, we predicted that both trained and non-trained schools wouldshow increases in Monitoring and Evaluation across the three years ofthe study.

Method

Participants

Data for the present study come from a large-scale randomizedtrial of PBIS conducted in the state of Maryland. A total of 37 public el-ementary schools from five suburban and rural school districts volun-teered to participate in the trial. Baseline demographic characteristicsof the participating schools (e.g., student enrollment, student mobilityrate, school-suspension rate, and performance on standardized tests)are provided in Table 1. The Committee on Human Subjects Researchat the Johns Hopkins Bloomberg School of Public Health providedapproval for this study.

Procedure

Design. The study followed a group randomized trial design(Murray, 1998). Schools within district were first matched on selectdemographic characteristics (e.g., urbamicity, percentage of studentswith free/reduced meals status, percentage of minority students) andthen randomly assigned to either the intervention or control condi-tion using a random number table. Two districts had equal numbersof matched schools assigned to intervention and control condition.In the remaining three districts, the matched schools were randomly

S assigned to condition at a ratio of approximately two trained schoolsto one comparison school. Trained schools were over-sampled basedon the power calculations focused on school-level outcomes (e.g., of-fice discipline referrals) among the trained schools. The 16 schoolsrandomized to the non-trained condition agreed not to implementschool-wide PBIS for the duration of the five-year study. A MANOVAwas conducted in SPSS 13.0 and indicated that there were no signifi-cant differences in baseline school-level characteristics (see Table 1) byintervention condition, Wilks' A =.674, F (9, 19) = 1.022, p = .46.

5

6

Trained Schools (n = 21)

SchoolCharacteristics

SchoolEnrollment

Student toTeacher Ratio

% Free/ReducedMeals

% SpecialEducationStudents

% CaucasianStudents

% StudentMobili4y

% SuspensionRate

AcademicPerformanceMath (%)t

Academic

Mean SD Range

471.76 132.78 233-701

18.48 4.33 13.19-30.83

42.93 19.22 15.4-71.2

Non-Trained Schools (n - 16)

Mean SD Range

505.50 188.57 227-983

18.61 4.69 12.52-29.54

36.25 20.93 5.2-62.5

13.24 4.27 6.7-21.9 15.08 6.66 8.8- 34.7

53.81 33.16

25.88 8.24

7.73 7.43

47.20 22.37

2.43-99.61

10.7-43.6

0- 32.52

10.7-98.2

67.51

20.51

5.06

46.96

28.99 7.44-100

7.19 10.2-36.1

4.73 0-17.18

19.05 6.6-69.8

50.66 19.32 13.2-83.8 52.94 16.43 26.7-78.5PerformanceReading (%)t

Note. -Percentage of 5th grade students who scored in the "proficent" or "advanced"range on the state's standardized test. The overall MANOVA on the school-levelcharacteristics indicated no significant difference between schools trained in PBIS andthe non-trained schools at baseline, Wilks' A = .674, F (9,19) 7 1.022, p =.46.

BRADWSHAW et al.

Table IDescriptives for School-level Characteristics

for Trained and Non-Trained Schools at Baseline

IMPLEMENTATION OF SCHOOL-WIDE PBIS

Training in school-wide PBIS. The 21 schools randomly assignedto the PBIS training condition formed internal PBIS teams comprisedof five to six members (e.g., staff, teachers, administrators). These in-ternal PBIS teams attended a two-day summer training led and sup-ported by the developers of PBIS (e.g., Drs. George Sugai, Rob Homer)and their collaborators (e.g., Drs. Teri Lewis-Palmer, Terrance Scott).Ongoing support for the schools is provided through local behaviorsupport (PBIS) coaches, a regional PBIS coordinator, and the Mary-land State PBIS Leadership Team. To ensure consistently high levels ofimplementation fidelity, PBIS schools attended annual two-day sum-mer "booster" training events. Additional supports and training wereprovided to the schools' behavior support (PBIS) coaches four timesa year.

Measure

The School-Wide Evaluation Tool (SET; Sugai et al., 2001) wasused to assess school-wide PBIS implementation fidelity. A previouslydescribed, the SET consists of 29 items organized into the followingseven subscales that represent the seven key features of school-.widePBIS: Expectations Defined, Behavioral Expectations Taught, System for

Rewarding Behavioral Expectations, System for Responding to BehavioralViolations, Monitoring and Evaluation, Management, and District-LevelSupport. Each item of the SET is scored on a three-point scale with 0 =not implemented, 1= partial implementation, and 2 =full implementation.Each subscale is represented by a percentage (from, 0% to 100%), withhigher scores indicating greater program fidelity. The seven subscalescores were averaged to yield an Overall SET score.

Administration.of the SET. During a SET assessment, the trainedobserver determined the degree to which a school had each of themodel's seven critical features in place,by reviewing written materi-als and established discipline procedures (e.g., discipline handbook,school improvement goals, behavioral incident summaries) and not-ing visual displays of thethree to five expected behaviors posted in 10specified locations throughout the school (e.g., hall, classrooms, caf-eteria, library). SET assessors conducted brief interviews about schoolprocedures, policies, and standards for positive behavior and rule in-fractions with administrators (30 minutes each), students (at least twostudents, one boy and one girl) per class per grade level (1-3 minuteseach), and school staff (using a random number table, at least 10 staffmembers were interviewed for approximately 2-5 minutes each). TheSET was conducted on a single day during the month of May andlasted approximately five hours.

Prior research on the psychometric properties of the SET. Homer et al.(2004) found the SET to have strong psychometric properties. Specifi-

7

BRADWSHAW et al.

cally, their research indicated that the SET was internally consistent(Cronbach's alpha = .96) and trained administers had high interob-server reliability (range = 98.4% - 100% across 17 schools). The test-re-test reliability was also high, and averaged 97.3% for the Overall SETscore across 14 to 20 days (Homer et al., 2004).

Training of SET assessors. A total of 15 SET assessors were hiredby the project, with approximately 10 assessors conducting SETs forthe project each year. Each assessor conducted between 1 and 10 SETson an annual basis, with a modal number of 4 SETs conducted per as-sessor at baseline and 6 thereafter. The SET assessors were primarilymaster and doctorate level professionals (e.g., teachers, special edu-cators, school counselors, educational trainers) who were currentlyworking or had recently retired from full-time work in an education-al setting. They were hired by the project and kept unaware of theschools' intervention status during all phases of the study. An initialtwo-day didactic group SET training session was conducted by Dr.Teri Lewis-Palmer (a developer of the SET) on interpretation of items,strategies for administering the SET, and the mechanics and scoring ofthe SET. Two additional days of one-on-one training and shadowingoccurred in a PBIS elementary school (which was not included in thetrial) by Dr. Teri Lewis-Palmer and a local lead trainer (Ms. Susan Bar-rett). All assessors obtained at least 85% interobserver agreement withthe trainer prior conducting a 'SET in a project school. The averageinterobserver agreement was 91.62% for the initial SET training. Onan annual basis, a one-day group refresher training session was led bythe local lead trainer for all previously trained SET assessors, duringwhich core aspects of the administration and scoring of the SET werereviewed. All assessors again obtained at least an 85% interobserveragreement level with the lead SET trainer in a non-project school. TheSET's detailed scoring guide and documentation form were rescoredby project staff to ensure accuracy. The interrater reliability of the SETscoring guide was high (r = .99, p < .0001), and all scoring discrepan-cies were resolved through consultation with the local lead trainer.As noted above, SETs were conducted at ýchools in both conditions atbaseline (May jrior to randomization and PBIS training), and annu-ally in the spring (May) thereafter.

Preliminary Analyses

The Cronbach's alphas for the 29-item SETs conducted on thecurrent sample of 37 schools were .85 at baseline, .92 after the firstyear of the trial, and .90 after two years of the trial. A MANOVA in-dicated there were no significant differences in baseline (prior to ran-dom assignment) SET scores, Wilks' A = .795, F (8, 20) = .645, p = .73.

8


Furthermore, a series of ANOVAs indicated there were no significantdifferences in Overall SET scores based on SET assessor at any of thethree time points [baseline: F (9,27) =.89, p=-.5 4 ; Year 1: F(9, 27) 1.71,p = .14; Year 2: F(9, 27) = 1.58, p = .17].

Overview of the Analyses

To test our primary hypotheses regarding the impact of trainingin PBIS on implementation fidelity, we conducted repeated measuresGLM analyses in SPSS 13.0 to determine the effect of training in PBISprocedures on SET scores. Because the number of schools and possi-bly the amount of district-level support for PBIS varied by school dis-trict, all multivariate and univariate analyses included school districtas a covariate. We also conducted a cross-sectional MANOVA on theSET scores two and three years after the training to determine at whattime point the schools in the two conditions differed significantly intheir levels of positive behavior supports. To examine our secondaryhypotheses regarding the progression of PBIS implementation, we.used a descriptive approach to quantify the percentincrease in levelsof positive behavior supports observed among schools assigned to thetrained and non-trained conditions. We conducted a repeated mea-sures GLM analysis on the non-trained schools to determine whetherthere was a significant increase in the level of positive behavior sup-ports. Lastly, we conducted an item-level analysis to further pinpointwhich specific aspects of PBIS were implemented by schools in eachcondition.

Results

Impact of PBIS Training on School-Wide PBIS Implementation Fidelity

To examine our primary hypothesis regarding the impact oftraining on implementation of positive behavior supports, we 'con-ducted repeated measures GLM analyses on the SET scores fromthe three years of the study, and results are reported in Figure 1. Theanalyses revealed a significant intervention effect (i.e., interactionbetween intervention condition and time) on the Overall SET scoreand all SET subscales, except System for Responding to Behavioral Viola-tions. This means that over the three years of the study, trained schoolsoutperformed non-trained schools in program fidelity on all but onesubscale. Inspection of the data reported in Figure 1 suggests a con-sistent pattern whereby the trained schools increased sharply in ratesof school-wide PBIS implementation within the first year, but leveledoff slightly thereafter.

To further explore whether there were significant difference inthe mean SET subscale scores by intervention condition at the end

I

.9

10 BRADWSHAW et al.

Figure 1. Adjusted, Mean SET Scores (Controlling for School District) forTrained and Non-trained Schools at Baseline, and after One and Two Yearsof Training.

(1) Overall SET

Legend: -n-'Trained -e-Non-Trained

(2) Define Expectations

(3) Teach Expectations

(4) Reward System

Inn./ .

80%

60%

4U%

20%

0%

Baseline Year 1 Year 2

100%

80%-

60%--

40%

20%

0% .Baseline Year 1 Year 2

1nno!.11)0%

80%

60%

4U'* ... ......

20%

0% 1Baseline Year 1 Year 2

inn- .!

80%

60%

40%

20%

0%-Baseline Year I Year 2

(1) Overall SET: Wilks' A =.458, F (2,26) = 15.38, p < .001, partial q2 = .54(2) Define Expectations: Wilks' A .620, F (2,26) = 7.95, p = .002, partial T12 .38(3) Teach Expectations: Wilks' A =.680, F (2,26) = 6.11, p =.007, partial 112= .32(4 Reward System: Wilks' A = .762, F (2,26) 4.05, p = .030, partial 112 =.24

S' *A''" " " 4........... ".9•-'-


(5) Respond toBehavioral Violations

Legend: -m-Trained -o--Non-Trained

(6) Monitoring andEvaluation

(7) Management

(8) District Support

100%.

80%-

60%"

40-y

20%]

0% sBaseline Year I Year 2

0no

80%

60%

40%

20%

0%

100%

80%

60%

40%

20%

0%

100%]

60%

60%-

a--

0�

Baseline Year 1 Year 2

.f ir

Baseline Year I Year 2

40% F r-

*�0� I.

Baseline Year I Year 2

(5) -Respond to Behavioral Violations: Wilks' A - .988, F (2, 26) = .157, p = .855(6) Monitoring System: Wilks' A =.656, F (2,26) - 6.82,p .004, partia -r2=.34(7) Management: Wilks' A .610, F (2,26) - 8.30,p .002,partial,12 .39(8) District Support: Wilks' A = .894, F (2, 26) = 1.54, p = .233

f

I

11

BRADWSHAW et al.

of study Years 2 and 3, we conducted cross-sectional MANOVAs onthe SET scores after one and twoyears of participation in the study.Controlling for school district effects, the analy'ses revealed that thetrained schools scored significantly higher on Overall SET scores thannon-trained schools'at both time points, Year 1: Wilks' A = .474, F (8,20) = 2.769, p = .03, -2 = .526; Year 2: Wilks' A =.131, F (8,20) = 16.58, p<.001, -2 = .869.

Taken together, the cross-sectional MANOVAs and repeatedmeasures GLM analyses indicated that schools trained in PBIS imple-mented the school-wide components of the model with significantlygreater fidelity than non-trained schools. Despite the overall effect,trained and non-trained schools did not differ in their implementationof System for Responding to Behavior Violations. Schools in both condi-tions evidenced significant improvements in this domain across thethree years of the study, Wilks' A =.61, F (2, 26) = 8.356, p < .01,112 = .39(i.e., main effect of time). ,

Progression of School-Wide PBIS Implementation

Our secondary hypotheses focused on the core elements ofschool-wide PBIS which were present in the schools at baseline, andon the progression of PBIS implementation for schools in both condi-tions across the three study years.

Components of School-Wide PBIS present at baseline. Schools in bothconditions evidenced some aspects of school-wide FBIS at baseline.Specifically, Systems for Responding to Violations tended to be the mostprevalent key feature among the schools at baseline, with a mean scoreof 67.84 (SD = 17.01) across all schools. In contrast, Defining BehavioralExpectations (M = 29.73, SD = 29.95) and Teaching Behavioral Expectations(M = 21.08, SD = 21.45) tended to be the least prevalent at baseline.

Trends in implementation across time. To examine the progressionof implementation that occurred across both trained and non-trainedschools, we determined the percentage of schools in each conditionwhich met the 80% implementation fidelity threshold on the OverallSET score at each time point (see Table 2). Specifically, 66.67% (14 outof 21) of the trained schools achieved the 80% implementation level onthe Overall SET score by the end of the first year. By the end of Year2, 85.71% (18 of the 21) of the trained schools met the 80% implemen-tation level on the Overall SET score. All the trained schools whichhad met the 80% implementation fidelity level by the end of Year 1maintained high levels of implementation fidelity through Year 2. Thenon-trained schools also implemented some aspects of school-widePBIS. Although most non-trained schools consistently received Over-all SET scores between 30% and 50%, 1 of the 16 non-trained schools

12


scored above 80% at the end of Year 1, and 3 of the non-trained schoolsreached 80% by the end of Year 2.

To determine if there was a significant increase in the level ofschool-wide PBIS over time in the non-trained schools, we conducteda second set of repeated measures GLM analyses for the Overall SETand subscale scores for the non-trained schools only (n = 16). Control-ling for school district, the non-trained schools showed a significantincrease in their Overall SET scores, Wilks' A = .54, F (2, 10) = 4.33, p< .05, partial 12 = .46, and on the System for Responding to BehavioralViolations subscale, Wilks' A =.19, F (2, 10) = 20.86, p < .01, partial 12

= .81. There was also a marginally significant increase in the DistrictSupport subscale, Wilks' A =.59, F (2, 10) = 3.51, p =.07, partial .12 =.41.Although the gains on these subscales were significant, it is importantto note that the magnitude of these changes (i.e., the 112 effect size esti-mates) was not as great as those experienced by the schools formallytrained in PBIS.

To further assess the progression of school-wide PBIS in bothtrained and non-trained schools, we examined the individual itemswhich comprised the SET subscales. Table 2 reports the percentageof schools in each condition that received the maximum score foreach item across the three years of the study. Inspection of the per-cent achieved data indicates there was a large increase in the number oftrained schools reaching the maximum scores on SET items within ayear of receiving training. Compared to the non-trained schools, thetrained schools showed greater increases in the number of schoolsperforming at the maximal level across nearly all subscales. The onlyexception to this finding was for the System for Responding to BehavioralViolations subscale, whereby both the trained and non-trained schoolsexhibited relatively high performance on subscale items at baselineand an increasing trend for all items on this subscale over time. AtYear two, 15 (71.43%) of the trained schools and 12 (75.00%) of thenon-trained schools scored 80% or higher on this subscale. Consistentwith the repeated measures GLM analyses, the relatively high levelsof implementation fidelity for the System for,Responding to BehavioralViolations subscale evidenced by both trained and non-trained'schoolswere sustained through Year 2 of the study.

Discussion

The current study aimed to determine the impact of train-ing in school-wide PBIS on the implementation of positive behaviorsupports. We also explored implementation fidelity among schoolstrained in PBIS compared to non-trained schools to illustrate the pro-gression of school-wide PBIS adoption and sustainability over time.

I

13

BRADWSHAW et al.

N

'.3C)

>1

-4

CuC)

C)

C)U)CuCO

N.4CuC)

-4

vo

"oE-4

LZJ

C-C

0

r. i

4 .C

8uU)

."-,,tJ

tU)

C)

C,)

00U

to

C)CuCuC-F.;.Cu0

N

U)

00

C')

C)CuCU.4

1-4

EiC)

*0C)Cu

COC)

0*.0Cu

r,qI ZN

Pom c

CDLO

N Nl

Col'C '.

0'. 0'.-4 -4

'.o 'd '.o1"'. 1C ZN

'0 0ZN 0

-4 � 06

C)

SU)

C)

o .0

C) 0Ob

U) 0 ML

C) 0

U) U) )

r4 cli 00

0

Cum

CD . Co - 0',kO 1

00

C)

C4.

%~. 0C, rl

"NI:

I'Dco

C6'IV~

A

00

t7%

%0 0alL't Ni

to

0'.

NZri

Nl m'

- 0)"a00

cu C

10- rl. 0 a

0 ~cu I

U) C2 0 CXJý4 cli cli

14

C)

q0l C

N 0

Cu 0li C 0


LQ

N cq

o DN N mC w

-4

N

CC

C)

-4

CCC)

'-C

C)

C)U)CC

N

CCC)

-4

CCC)

C)

CC

0U

C)

CCI-

-44

rýLn

1s Ln

N 0

.l i M

0'.

NN

-4 r

CD Q

0

-.0 -.

a) U

*0

6.9 U)

C) -0CD

SC)U)

LA*0CC

C)

0Dq

00 c

00

G% 0

In,

LAN-4 �

'.0

LAN

-� etS-4 '.0

NCD~

eCD

00lcLo

--

o

.4.

-IV

-4 00.

040

00'.CC)

'.0

N'.0

0 N-4. .�C

CDq00D

CD CD*0 q0m 6 0 6 c

aR t -, :tt *- C -0

Q) 0

.2 C

to. (nC cn U .0

15

g 00,4 '0 '.0 0 6

cl I ,Lq

BRADWSHAW et al.

00

Rn06

in

-4 en

'-.4 in

Cin

in [Nen

rn

0

0

U

"an

I,

I.;

0z

'-.4

Lo0)

1-4 en

Ln

co

[N

coco

" "OO

[Nin N

en-4 '0 -

cu

*.0 0 .

Ca . ~ 0 ~ 00

00 cu .c

to .2 E Fi

r.4--1--C

caL Cw cc

ca 0 Ccsl e ~ ~ I e

16

CD

co

R06

C NLQ

CDC4

CD CDIQ q

CD N q

in[N ccoe

CD

co

NIZ?

CD

C,

C)

ini[eonc

mn

00CD

m CD

U,Mr.

0

*0

0~Cl.

0

'i


C)iU),~C,

0M

0)Lto

0) C-1 to'-4 ýo

co M) 00CCr I 001v t

(U0)

-4

(U0)

0)

0)

(U

I- 0) CD '

Nl 0 N 0. N

U) co

a\ C4-4 -4oU

00�

C') -4�4 \0

N

-4c�)

InN1

to '-4co)

00

eN 0'-4

C')

U) -4-4 N

cq In

j!

tz ( U

2 U)

0 cJ 0)U) v0 a)

0 0

-000 U

r ~ I CD

c)

ri>4

C'g~) C )

0 0-I:v Pgn

CD CR LQ

"00 ,~ %0

0

U)

0t0

.0•

U

U) to

ON 0 ý -4 0U

G\

C') m cq a l C'o'C 0'

0)

0

> (a

2 .0)

o0 .

20o

to to)

04 , -

5 Ei(a .i0E

v~ i

17

0 , 0 i m 06

rq eq . 6 q 0\ U) N 0

-4I..(U0)

0).20)U)(U

U)

,* 0 0t 0

N N' 00 in0 )N Dc"O NW to 0 0 " \ 00 Uq)G

'-.4ON

U)00

.0U)

0)

(U4..

,,,

0UJ

0)

0

(U

0

0).20)

0..'4)

(UI-.0)0)

-4

0

I

BRADWSHAW et al.

, -4' N ci

In- 0

Ln CO n

CO

'0

to*00

.i

C-

0,

z.

'-41.. 0-)0 -0 'o'd

�4- ,-4IN

COcO?mN1

ceO

C)

C)

Cf)

C-4 '

o'* g'0 ~C) I 0 p Ctq L

CD 1ON1 m~

07%

CA ~

0 0O

00 0-)

1O Cf) -

- R IN i- '

-O M 'I0-

o 0

0 cU)W -5 U

.. 0 (A tu0 . 0)(n c

U) co

0 0

%0 rz c6 00

t

"0U)

.!4

1ý 0 " 0DCDq

106U-n

0. 0 ,tr) 0C)

0 D C'i 00 6 Ov"- %D 10 *)

to

100

CO

C-4 C)

\0

cr% N

Ln

Cf)

toiCN a%

C-4

I10

CO 00

N- C>

"I 00

Cý N LO

w -

0

-0 0ra0

;a 0~- 0 0

.0 0

U10.0 C4< 0-

'-4 N C

18

C.4

-4C..00>..

0

0Cu

m10

'-4

0060D

NDtCA. m

rl

00

-0

C)-0

Uu

C)O

QcJo

F3.

U) 0

0.

00

00

tot

0 Q

0ý

0060n,

0U

0

0

0

0n


In the following sections, we discuss some of the major findings of thepresent study and conclude with recommendations for training andmaintaining high fidelity implementation of school-wide PBIS.

Impact of PBIS Training on School-Wide Implementation Fidelity

The repeated measures and cross-sectional multivariate analysesindicated a significant impact of PBIS training on school-wide PBISimplementation fidelity. Most of the growth in fidelity occurred withinthe first year of implementation. Specifically, the PBIS schools shoývedsubstantial increases in the Overall SET scores and across the majorityof SET subscales. The increases showed by trained schools were im-mediate and consistent across the two years following training. Thesharp increases between baseline and Year I SET scores suggests thatthe training and support provided to the schools in this sample wassufficient to promote high implementation fidelity in a relatively shortperiod of time. Although the developers of PBIS have theorized thatit takes three to five years to implement the model (Sugai & Homer,March 2006), these data suggest that most schools were able to imple-ment the program with high fidelity in just one to two years. How-ever, it may take the full three to five years for these organizationalchanges to translate into changes in student behavior, such as reducedbehavior problems and increased academic performance. An impor-tant next step in this research is to examine the association betweenvariations in implementation'fidelity and changes in student and staffbehavior. Specifically, we intend to use multilevel procedures (e.g.,HLM) to model the impact of PBIS training and implementation fidel-ity on individual behavior change.

Baseline Levels of School-Wide PBIS

As predicted, schools displayed some components of school-wide PBIS, even prior to formal training in the model. In particular,all schools displayed fairly high levels of Systems for Responding to Be-havioral Violations, indicating that they applied a continuum of conse-quences for problematic student behavior. In contrast, sch ools at base-line tended to score low on System for Teaching Behavioral Expectationsand System for Defining Behavioral Expectations. The greater emphasison reactive and punitive responses to behavioral violations, coupledwith low levels of positive behavior support, is consistent with tradi-tional models of behavior management (Strain, 2004). These findingssuggest that a primary objective of FBIS training should be to encour-age a shift in educational approaches from punishment-based strate-gies to teaching and reinforcing prosocial behaviors.

I

19

BRADWSHAW et al..

Components of School-Wide PBIS Present in Non-Trained Schools

It was also hypothesized that the non-trained schools wouldcontinually implement some components of school-wide PBIS, butthat growth in two coie features (Defining Behavioral Expectations andTeaching Behavioral Expectations) would be more pronounced in trainedschools relative to non-trained schools. The results indicate that thetrained schools experienced considerable increases in implementationin the Defining Behavioral Expectations, Teaching Behavioral Expectations,and Rewarding Behavioral Expectations subscales (see Table 2), whereasthe non-trained schools showed only minimal increases in DefiningBehavioral Expectations and Teaching Behavioral Expectations subscales,and evidenced a slightly increasing trend in the Rewarding BehavioralExpectations subscale (See' Figure 1). The small increase in SET scoresamong the non-trained schools may be due in part to the explicit na-ture of the questions on the SET. For example, asking the principaland school staff whether a "school has agreed upon 3 to 5 positivelystated school rules that are publicly displayed in 8 to 10 locations" mayhave prompted school personnel to consider doing so. The questionsregarding a behavioral rewards system also may have led to increasedawareness of the importance of using rewards. However, while thenumber of non-trained schools reporting to utilize a reward systemincreased, fewer non-trained schools reported linking the reward sys-tems with posting or teaching school-wide behavioral expectations.This suggests that the non-trained schools were missing several keycomponents to effective school-wide PBIS. Thus, it isý recommendedthat school personnel teach behavioral expectations to students in asimilar way as they teach academics (see Darch & Kameenui, 2004).Furthermore, school staff should devise and implement a system thatconsistently provides rewards to students who demonstrate those be-havioral expectations.

It is important to note that the SET was not developed to dis-guise school-wide PBIS procedures, but to assess and guide imple-mentation of school-wide PBIS. School personnel can, to a certainextent, self-train on positive behavior support strategies either bydeducing information from the SET interview process or through in-formation available on the Internet. However, the current study sug-gests that explicit training in the development and application of thekey school-wide PBIS systems, as well as assistance from a behaviorsupport coach in the shift toward positive b6havior management ap-proaches, are likely to result in more substantial gains in school-widePBIS implementation.

With regard to the types of changes observed in the non-trainedschools, these schools did exhibit a significant increase on the System

20

IMPLEMENTATION OF SCHOOL-WIDE FBIS

for Responding to Behavioral Violations subscale across the three years ,of the study. Based on the percent change data reported in Table"2, itappears that these improvements were primarily due to an increase inthe number of schools using a formal crisis plan for managing danger-ous situations, having it available, and training staff on procedures touse during a crisis. While these activities.are important for effectiveimplementation of school-wide PBIS, they are not unique to school-wide PBIS and should occur in all schools. In fact, under No Child LeftBehind (NCLB) schools are required to establish a school safety plan,which includes discipline policies, security procedures, preventionactivities, student codes of conduct, and a crisis management plan forresponding to violent or traumatic incidents on school grounds. Thus,the questions on this particular subscale may no longer effectively dif-ferentiate schools based on school-wide PBIS fidelity, but may be use-ful in identifying schools in compliance with federal requirements.

Similarly, non-trained schools tended to show gains in the Moni-toring and Evaluation subscale, which assesses the schools' processfor monitoring and reporting discipline referrals. Attention towardaccountability and documentation has increased dramatically sinceNCLB was signed into law in January 2002. UnderNCLB, states mustreport school safety statistics to the public on a school-by-school ba-sis. Thýrefore, it was not surprising to see an increase in the collec-tion and use of discipline data within all schools. Although the non-trained schools shoWed some increases on this subscale, they contin-ued to have significantly lower implementation levels than the trainedschools. Further, the increases among non-trained schools were mostspecific to having an office discipline referral form (item 1) and a sys-tem for collecting and summarizing discipline data (item 2). Althoughnon-trained schools tended to increase in Monitoring and Ev;aluationoverall, they did not appear to utilize the collected information formaking decisions about implementation of programs or other sup-ports. Trainers working with new PBIS teams may consider allocat-ing less training time to the construction of a monitoring system, andmore time on how to effectively use the data to monitor effectivenessand create meaningful change with positive outcomes for students.

It is interesting that schools in both conditions reported a sig-nificant increase in District Support across the three years of the study(i.e., main effect of time), Wilks' A =.52, F (2, 26)-1 12.13, p < .01, partial

112 = .48; however, this effect did not vary by intervention status. Thereare only two items on this subscale (i.e., a budget for positive sup-ports, and a district contact for positive supports). While the growthin District Support tended to be greater for the trained schools, it ispossible that the schools in the comparison condition perceived that

I

21,

2BRADWSHAW et al.

some of the support provided through their participation in the re-search project connoted increased district support. Although the uni-versity researchers were iesponsible for the overall management ofthe data collection for the project, ,the school districts were collabora-tors on the trial and participated in some site visits and were keptabreast of communications to and from the schools. Consequently,schools in both conditions may have perceived greater connection tothe school district as a result of their participation in the research trial.Furthermore, the non-trained schools received a modest honorarium($1000) from the researchers in appreciation for their participation inthe project; these schools may have used this money to support otherstudent programs.

District and State-Level Support of PBIS

It is important to note that the 37 schools were clustered withinfive large school districts. Each district had a different number of par-ticipating schools and a slightly different PBIS support infrastructure(e.g., district-level coordination, financing); therefore, we statisticallyadjusted for school district in our analyses. Due to 'the small numberof districts participating in the trial, we lack sufficient.power (Murray,1998) to draw meaningful conclusions about specific aspects of thedistricts' support that may have influenced implementation fidelity orsustainability. Findings from state-wide evaluations of PBIS suggestthat having a district-level budget to support PBIS and a coordina-tor who has dedicated time to provide training, support, and on-sitetechnical assistance are critical for high fidelity implementation andsustainability of school-wide PBIS (Barrett, Bradshaw, & Lewis-Palm-er, in press). Future'research with a larger number of diverse schooldistricts should examine aspects of district support (e.g., coaching,coordination, training, technical assistance, financing) which are criti-cal to implementing and sustaining high fidelity implementation ofschool-wide PBIS.

Although a unique feature of this study was its ability to com-pare schools with and without formal PBIS training, both groups ofschools were, located within a state actively engaged in large-scaleimplementation of PBIS (See Barrett et al., in press for a summary ofthe Maryland PBIS state-wide initiative). This is both a strength andpotential limitation of the current study. Specifically, the large supportand training network may have contributed to the speed with whichthe trained schools reached high implementation fidelity; however,the extensive state-wide network may have led to an increased aware-ness and familiarity with school-wide PBIS procedures among thenon-trained schools. Therefore, the extent to which the findings for

2-2

RMVPLEMENTATION OF SCHOOL-WIDE PBIS

the trained schools will generalize to schools in other contexts wherethe training and support network may be less extensive is unclear.Furthermore, all schools volunteered to participate in the study withthe hope of receiving training in PBIS. Thus, this sample may includeschools with influential staff members who weie more knowledge-able about the school-wide PBIS model than is typical. Similarly, theseschools may have been more open to the model's emphasis on positivebehavior supports. Further research should be conducted in schoolsthat are less knowledgeable and accepting of school-wide PBIS, suchas those that are mandated to implement the model. The current sam-ple only included elementary schools. Additional research is neededon middle schools and particularly on high schools, as it is likely thatthe school-wide PBIS model may require ldnger to reach the 80% fi-delity threshold due to uni4ue implementation challenges in highschools (Sugai, Flannery & Bohanon-Edmonson, 2004).

Conclusion and Recommendations

The findings of the present study suggest several'important rec-ommendations for school personnel attempting to implement school-wide PBIS in new or recently trained schools. One recommendationis to conduct a baseline SET and use this information to determinewhich, if any, aspects of school-wide PBIS are already present in theschool prior to formal training. This baseline data could then be usedby administrators, PBIS behavior support coaches, and PBIS trainers toidentify the strengths and weaknesses specific to the school. Trainingscould then be tailored to target the needs of the school. For example,the trainer might provide a brief review of school-wide componentscurrently in place, but spend more time focusing on the componentswhich are less familiar and currently underutilized, thereby allowingthe training to be targeted and efficient.

In addition to conducting a baseline SET, schools could conductSET evaluations on a regular basis (e.g., monthly, quarterly, bi-annu-ally) to monitor progress of the school-wide components. This wouldallow for a more accurate determination of the time at which schoolsfirst implement specific components of school-wide PBIS, and howwell each component is maintained over time. Additionally, havingmore frequent checks on fidelity coupled with the provision of feed-back to school personnel, would likely decrease the amount of timeit takes schools to reach the threshold for effective implementation.After completing the SET, evaluators should provide school personnela summary of the SET data, on both the item- and subscale-level. Thisongoing feedback of current performance would inform additionaltraining or changes in current practices, leading to swifter positive

23

•4BRADWSHAW et al.

outcomes for the students. Furthermore, school personnel would beable to review their progress over time and celebrate their successes.

As noted above, the schools in the current sample tended to havethe highest baseline scores on the System for Responding to BehavioralViolations and the Monitoring and Evaluation subscales of the SET. Thissuggests that less training time may need to be spent reviewing theseareas. In contrast, Defining Behavioral Expectations, Teaching BehavioralExpectations, and System for Rewarding Behavioral Expectations tendedto be relatively low at baseline. There was, however, considerablegrowth after training in PBIS on Defining Behavioral Expectations andSystem for Rewarding Behavioral Expectations, but less so for the Teachingof Behavioral Expectations. These data and anecdotal responses from thePBIS coaches working with the trained schools suggest that schoolsfind it challenging to develop the lessons for teaching the expecta-tions and review the expectations infrequently. We recommend that asignificant portion of the initial training in PBIS focus on strategies forteaching the behavioral expectations, including the development ofwritten lesson plans: Furthermore, school staff should be reminded tobriefly review the lessons on a daily basis for the first week of school,on monthly basis thereafter, and after all vacations and breaks.

The current study aimed to enhance understanding of the man-nernin which school-wide PBIS is adopted and maintained over time.Because of the accessibility of school-wide PBIS and the familiaritywith underlying theoretical constructs, some administrators may betemped to implement the program without formal training or sup-port. However, the findings of the current study suggest that schoolswithout formal training in PBIS are most likely to implement the com-ponents of the model which are consistent with traditional behavioraldiscipline approaches (e.g., Responding to Behavioral Violations), butare less likely to establish and teach positive behavioral expectations,and reward children for meeting those expectations. In contrast, mosttrained schools appear to be able to implement these and the otherkey components of school-wide PBIS within a year or two of initialtraining. These findings should aid trainers and PBIS behavior sup-port coaches in directing their training and support efforts for PBISschool teams during the early stages of implementation and increasesustainability.

Acknowledgements

Support for this project comes from the Centers forDisease Control andPrevention (R49/CCR318627 and 1U49CE 000728) and the NationalInstitute of Mental Health (1R01MH67948-1A, T32 MH19545-11, T32MH18834). The authors would like to acknowledge the MarylandPBIS State Leadership Team and Dr. Susan Keys for their assistanceand support of this project.

24


References

Barrett, S., Bradshaw, C. P., & Lewis-Palmer, T. (in press). Marylandstate-wide PBIS initiative: Systems, evaluation, and next steps.Journal of Positive Behavior Interventions.

Darch, C., & Kameenui, E. (2004). Instructional classroom management:A proactive approach to behavior management. (2 nd Edition). Up-per Saddle River, NJ: Pearson-Merrill Prentice Hall.

Homer, R. H., Sugai, G., Todd, A. W., & Lewis-Palmer, T. (2005).School-wide positive behavior support. In L. Bambara & L.Kern (Eds.), Individualized supports for students with problembehaviors: Designing positive behavior plans (pp. 359-390). NewYork: Guilford Press.

Homer, R., Todd, A., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland,J., (2004). The School-wide Evaluation Tool (SET): A researchinstrument for assessing school-wide positive behavior sup-ports. Journal of Positive Behavior Interventions, 6, 3-12.

Murray, D. M. (1998). Design and analysis of group-randomized trials.New York: Oxford Press.

Nelson, J. R. (1996). Designing schools to meet the needs of studentswho exhibit disruptive behavior. Journal of Emotional and Be-havioral Disorders, 4, 147-161.

Nelson, J. R., Martella, R. M., & Marchand-Martella, N. (2002). Maxi-mizing student learning: The effects of a comprehensive,school-based program for preventing problem behaviors.Journal of Emotional and Behavioral Disorders, 10, 136-48.

Nersesian, M., Todd, A., Lehmann, J., & Watson, J., (2000). School-wide behavior support through district-level system change.Journal of Positive Behavioral Interventions, 2, 244-246.

Strain, P. (2004). A not so good job with "Good Job": A response toKohn 2001. Journal of Positive Behavior Interventions, 6, 55-59.

Sugai, G., Flannery, B., & Bohanon-Edmonson, H. (2004). School-widepositive behavior support in high schools: What Will it take?http://www.pbis.org/files/PBSMonographComplete.pdf. Ac-cessed December 15, 2005.

Sugai, G., & Homer, R. (2002). The evolution of discipline practices:School-wide positive behavior supports. Child and Family Be-havior Therapy, 24, 23-50.

Sugai, G., & Homer, R., (March 2006): Adoption, implementation,durability, & expansion of SW-PBS. Paper presented at theThird Annual International Association for Positive Behavior

I

25

26 BRADWSHAW et al.

Supports Convention. Reno, NV. http://Www.pbis.org/past-conferencepresentations.htm Accessed April 7, 2006.

Sugai, G., & Homer, R. (2006). A promising approach for expandingand sustaining school-wide positive behavior support. SchoolPsychology Review, 35, 245-259.

Sugai, G., Homer, R., & Gresham, F. (2002). Behaviorally effectiveschool environments. In M. Shinn, H. Walker, & G. Stoner(Eds.) Interventions for academic and behavior problems II: Pre-vention and remedial approaches. (pp. 315-350). Bethesda, MD:NASP.

Sugai, G.,,Lewis-Palmer, T., Todd, A., & Homer, R.H. (2001). School-wide evaluation tool. Eugene: University of Oregon.

Taylor-Greene, S., Brown, D., Nelson, L., Longton, J., Gassman, T., Co-hen, J., et al. (1997). School-wide behavioral support: Startingthe year off right. Journal of Behavioral Education, 7(1), 99-112.

Taylor-Greene, S., & Kartub, D. T. (2000). Durable implementation ofschool-wide behavior support: The high five program. Journalof Positive Behavior Interventions, 2, 233-235.

COPYRIGHT INFORMATION

TITLE: Implementation of School-Wide Positive BehavioralInterventions and Supports (PBIS) in Elementary Schools:Observations from a Randomized Trial

SOURCE: Educ Treat Child 31 no1 F 2008

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited.

Implementation of School-Wide Positive Behavioral Interventions and Supports (PBIS) in Elementary...

Documents

Transcript of Implementation of School-Wide Positive Behavioral Interventions and Supports (PBIS) in Elementary...