Program evaluation: Relationships as key to student development

NEW DIRECTIONS FOR YOUTH DEVELOPMENT, NO. 120, WINTER 2008 © GIL G. NOAMPublished online in Wiley InterScience (www.interscience.wiley.com) • DOI: 10.1002/yd.290 151

This article describes general steps in RALLY eval-uations as well as the implementation and selectedoutcomes of a school that implemented RALLY dur-ing the 2007–2008 school year.

8Program evaluation: Relationships askey to student development

Tina Malti, Sarah E. O. Schwartz, Cindy H. Liu, Gil G. Noam

IN THIS ARTICLE, we describe the general purposes of the evalua-tion component of RALLY programming. Then, through a pilotstudy in an urban middle school, we investigate how the RALLYProgram services affect students’ resiliency, learning potential, andrisks and the implementation of RALLY services and program sat-isfaction among students, practitioners, and teachers. This in-formation will contribute to a deeper understanding of how investments in RALLY can facilitate development and resilience instudents, promote academic success, and reduce mental healthrisks. With these goals, we also aim to contribute more broadly to a way to evaluate the conditions necessary to allow students tothrive socially, emotionally, and academically.

RALLY evaluationsEvaluation is an important program component of RALLY andhelps to track the progress of the RALLY Program. In general,

152 YOUTH DEVELOPMENT MEETS MENTAL HEALTH

new directions for youth development • DOI: 10.1002/yd

scientific evidence that a preventive program is effective is a pre-requisite for success.1 Reaching the program goals set out in thesecond article in this volume requires first and foremost an under-standing of the needs of the population and the specific individualsit serves. The purposes of the RALLY evaluations are to improveprogram design by helping to identify the needs and resources ofthe students, school, and community; provide early detection forstudents in urgent need of services by utilizing a holistic assessment(see the seventh article) and continually assess the program’s effec-tiveness in meeting these needs (see the previous article as well);highlight the strengths and weaknesses of the program so that suc-cess can be replicated and areas of improvement can be addressed;and offer evidence of program effectiveness for current and poten-tial funders. We share this work in order to provide ideas to allthose who are implementing evaluation on a tight budget and wantto connect student assessment to program evaluation.

RALLY has developed a six-phase plan to help to implementevaluations effectively in a school:

Step 1: Conduct a program needs assessment. This phase begins longbefore the start of the program and is necessary to help under-stand the youth population whom the program aims to serve, aswell as the existing school and community resources.

Step 2: Create an evaluation plan including a holistic assessment proce-dure and related systems. The plan has systems for collecting holis-tic diagnostic assessments, tracking, and analyzing data.

Step 3: Administer individual student needs assessments to gather base-line data. Unlike the program needs assessment, this step takesplace at the beginning of the school year to assess the individualneeds of the students served.

Step 4: Gather participant feedback and program quality at checkpointsthroughout the year. This step consists of administering surveys,interviews, and observations assessing the quality of program ser-vices and participant satisfaction throughout the year.

Step 5: Administer posttests and analyze data to assess program impactand progress toward program goals. Data from the posttest admin-

153PROGRAM EVALUATION


istered at the end of the year are compared to baseline data fromthe beginning of the year.

Step 6: Compare results with a control group. Ideally, the outcomes of agroup that has benefited from RALLY services are compared toa group that has not received these services. This approach iscalled quasi-experimental in that it shares rigor with the designused in experiments (randomizing those who receive an inter-vention and those who do not). Given the fact that we are work-ing with natural settings and cannot easily assign students toclassrooms or schools by the luck of the draw, however, the quasi-experimental design is the most realistic to use. What we can doin the future is randomly assign schools and interventions towhole classrooms, but that is different from assigning individualstudents randomly to classrooms.

RALLY evaluations do not focus exclusively on students’ aca-demic outcomes. Rather, our developmental and relational per-spective requires a holistic view of development and resiliencies,risks, and supportive relationships (see the previous article andarticle two). We also made recommendations for a holistic mea-surement rationale in the previous article.2

RALLY evaluations are usually conducted by researchers, butit is also possible for these evaluations to be conducted by inter-nal staff such as teachers, practitioners, or the student supportteam within a school if adequate training is provided. Thisoption is important to mention because many schools may nothave sufficient funding to allow researchers to implement a for-mal evaluation of RALLY programming.

An empirical pilot studyThe pilot study investigates implementation quality and studentchanges in resiliency, learning interest, and risks in classes whereRALLY was implemented. It aims to provide evidence that the RALLY Program can enhance resiliency and development,improve academic functioning and learning potential, and decrease



risk behavior. In sum, we investigate the following evaluation questions:

Student outcomes: How were students’ resiliency, development,learning potential, and risks affected by the RALLY Program?

Program implementation and satisfaction: How were various programcomponents implemented? How satisfied were program partic-ipants and providers with the services implemented?

Design

Our evaluation took place within a RALLY Program implementedin a U.S. urban middle school during the 2007–2008 school year.The design is based on mixed qualitative and quantitative, multi-informant evaluation methods for an integrated approach, includingdata from holistic pre- and postassessments administered to studentsat the start and end of the school year; focus groups and observationalprocess data during the program year; written surveys with practi-tioners, students, and teachers; and interviews with program andschool leaders at the end of the school year. This ingegrated approachis well suited to investigating strategies of success in implementation,program satisfaction, and student outcomes.

A researcher not involved in any part of the intervention conductedthe evaluation. In this study, since we piloted several interventionsin the implementation of this program, we did not include a controlgroup. Although the RALLY Program has conducted evaluationsfrom the beginning, we wanted to integrate the holistic assessmentcomponent with the evaluation component, and for that reason weconducted a new pilot, which we are reporting here. This integrationappeared useful, because the assessments are simple yet describewell the students’ issues and needs (see the previous article). Thismakes the evaluation data useful for students and staff.

Participants

Ninety-two students in grades 7 and 8 in the 2007–2008 school yearparticipated in the RALLY Program that was implemented in the



urban middle school examined here. There were forty-four girls (48percent) and forty-eight boys (52 percent) with a mean age of 13.7years (SD = 0.78). The demographics of the student population atthe school are as follows: Hispanic (65 percent), black (27 percent),white (5 percent), Asian (2 percent), and unspecified (1 percent). Theaverage test scores of students in grades 6 through 8 at this schoolare significantly below the state average. Approximately 83 percentof all students are eligible for the free lunch program. The studentsfrom our sample came from diverse neighborhoods, and many ofthem live in neighborhoods with primarily Latino and black fami-lies. Many of the students live in housing projects situated withinthe poorer areas of the neighborhood where an increase in crimehas been observed and documented by several newspaper articles.However, several community organizations established in the neigh-borhood provide activities before and after school.

The RALLY team consisted of six classroom teachers, six prac-titioners, the school program director, and the clinical coordinatorof RALLY, as well as two professional researchers and several stu-dent research assistants. Written informed parental consent wasobtained for both the assessments and participation in the differ-ent intervention components.

Measures

Due to space constraints, we give only a brief overview of the measures:

• Student interviews (preassessment). Interviews with the adoles-cents administered by the practitioners included open-endedquestions on the adolescent’s resiliencies and needs regardingmental health as well as interests for after-school activities (seethe volume appendix).

• Student questionnaires (pre- and postassessment). The measureswe chose represent our holistic assessment procedure as out-lined in the previous article. The measures were indicators of (1) development, (2) resiliencies, (3 symptoms, (4) relationships,and (5) program evaluation (only postassessment).



• Social-cognitive development. The Sociomoral Reflection Mea-sure, Short Form (SRM-SF) is a group-administered paper-and-pencil instrument designed to assess the developmental levels ofcognitive-moral skills.3

• Resiliency. A scale was developed by the authors to assess re-siliency. The items were designed to measure selected basicresiliency factors such as empathy, trust in others, relationshipskills, and emotion regulation skills, as well as academic skills,such as interest in learning and caring about school.

• Symptoms. The Youth Self Report was administered to assessthe behavioral and emotional functioning of adolescents.4

• Supportive relationships. Self-created items measured perceivedsupport by parents, peers, and teachers.

• Program evaluation. Students evaluated various components ofthe RALLY Program such as academic support, program satis-faction, and gains from the program in terms of facilitatingresiliencies and learning potential.

• Focus groups (process assessments). Student focus groups were con-ducted with twenty-six students over the course of the year, includ-ing students from all six homerooms that worked with RALLY.

• Observations (process assessments). RALLY researchersobserved the work of the practitioners over the course of the yearby following them in classrooms and in selected interventions(participant observation).

• Practitioner and teacher questionnaires (postassessments). Thepractitioner questionnaire included questions exploring theirrole, as well as evaluating their perception of their effectivenessin supporting students academically, socially, and emotionally. Inaddition, both the practitioners and teachers rated the quality ofteacher and staff collaborations, the impact of activities onRALLY students, and the overall program.

• Interviews with RALLY director and clinical coordinator andschool director (postassessments). The program director, thecoordinator of clinical and group services, and a school directorwere interviewed about their evaluation of implementation qual-ity and program effects on student outcomes.



Procedure

The preassessments (student questionnaires and interviews) wereconducted at the beginning of the school year and the postassess-ments (student, practitioner and teacher questionnaires, interviewswith program leaders) at the end of the school year. Focus groupsand observations took place during program implementation.

RALLY outcomesEffects on students’ resiliencies, learning potential, and risks.

The first evaluation question focuses on the effects of the RALLYProgram services on students’ resiliencies, academic skills andlearning potentials, and risks. At the beginning of the program year, a holistic assessment was implemented to help the RALLYteam understand adolescents’ social-cognitive development andresiliencies, risks, and relationships (see the second and seventharticles). Eight percent of the students were at the subjective-physical developmental level, which indicates thinking in egocen-tric and impulsive terms. Two-thirds of the students were at thereciprocal-instrumental developmental level, which is described bythinking in individualistic terms. Twenty-five percent of the stu-dents were at the mutual-inclusive developmental level, whichincludes the ability to take others’ perspectives. According to arecent international study, the mutual-inclusive level is prominentamong early adolescents,5 whereas the reciprocal instrumentaldevelopmental level is the most prominent in nine to eleven yearolds. Students also reported on several resiliencies, showingmedium to high levels of learning motivation and caring aboutschool, low trust in others, and medium levels of empathy and rela-tionship skills. A significant number of students showed symptomson the Youth Self Report: 15 percent of the students reported clin-ical levels of internalizing symptoms (affective, anxiety, and somaticproblems), and 16 percent reported clinical levels of externalizingsymptoms (attention deficit hyperactivity disorder, oppositionaldefiant, and conduct problems). Furthermore, an additional 25



percent showed internalizing problems or externalizing pro-blems at a borderline clinical level. These numbers indicate thatour students have elevated risks (see the first article). Regardingsupportive relationships, students reported high family sup-port (M = 2.51, SD = 0.83) but lower relationship quality withteachers (M = 1.74, SD = 0.90) and support by peers (M = 2.24, SD = 0.88). It is also important to note that 30 percent of studentsreported having experienced a negative life event during the pre-vious year such as a death or interpersonal or family problems, thusindicating a high level of trauma in this sample.

As expected, we found various relations between risks andresiliencies as well as academic interests at the beginning of theprogram year. For example, measures of resiliency such as caringabout school and emotion regulation skills were negatively relatedto both internalizing and externalizing behavior (care about school:r (70) = –.36, p < .01 and r (70) = –.41, p < .001, respectively, emo-tion regulation: r (72) = –.33, p < .01, and r (72) = –.44, p < .001).

In other words, the more that students cared about school andfelt they could control their emotions, the fewer symptoms theyhad. This confirms the view that mental health, resiliencies, andacademic outcomes are linked (see the first article), which calls foran integrated strategy to promote students’ growth.

In the following sections, we describe student outcomes at theend of the program year. We first present academic outcomes andthen data on development, resiliency, and risks.

Academic outcomesI learned from my RALLY prevention practitioner to try hard and nevergive up no matter how hard it is. ––RALLY student

Eighty percent of students agreed or sometimes agreed that prac-titioners helped them do their schoolwork, and 93 percent agreedor sometimes agreed that practitioners helped them to think moreabout their life and their future. However, no changes in interestin learning and caring about school were observed from pretest to



posttest, as perceived by students. We have found similar findingsin the past. Normatively, the rates of attachment and caring aboutschool go down in this age group, so if the rates remain stable, thatis potentially an indicator for success. Moreover, 92 percent of stu-dents reported that they agreed with the statement, “RALLY peo-ple help keep me on track.” In focus groups, students explainedhow practitioners “make you do your work,” but that they ask youto do it “in a way like [they are] asking you for a favor, not like ateacher just telling you to do it.” As a result, they said, students aremore likely to do their work if a practitioner asks them. A few stu-dents said that practitioners would meet individually with them togo over their report cards and make a plan for how to improve theirgrades in the following quarter, which they felt was very useful.

The practitioners confirmed this view: all of them agreed thatthey had helped students with their schoolwork and thinking moreabout their life and their future. Practitioners felt less successful infacilitating learning interest, which resonates with the students’perception. Learning is a complex process and frequently needscollaboration with teachers and parents, which may explain thislower evaluation. It is also a reason that afterschool is so importantbecause it can support learning that is more self-chosen. TheRALLY-designated classroom teachers all agreed or strongly agreedthat the RALLY Program specifically helped to improve students’academic performance, as well as increased students’ interest andmotivation to learn.

In sum, the RALLY Program had selected positive effects on aca-demic outcomes such as doing schoolwork and thinking about lifeand the future as perceived by students, practitioners, and teachers.

Changes in resiliency, development, and risks

We compared the self-reported resiliencies from the pretest withthe resiliencies from the posttest (Figure 8.1) using repeated-measure analyses of covariance (ANCOVAS) with developmentallevel as covariate and the corresponding resiliency as within sub-ject factor. (Eighty-six percent of the seventy-nine students who



Figure 8.1. Mean values of resiliency scales in pretest andposttest

participated in the pretest also participated in the posttest. Thislower number was in part due to the fact that some of the studentsmoved away and were not in the program at the end of the year.Others were not in class when the postassessments were adminis-tered. Thus, we cannot entirely exclude the possibility that thedropout students were systematically different from the studentsfor whom pretest and posttest data were available.)

The findings indicate that students reported more empathy afterhaving received RALLY Program services than at the start of theprogram, F(1, 68) = 11.21, p < .01, η2 = .15. Furthermore, there wasan increase in trust by the end of program, F(1, 66) = 4.76, p < .05,η2 = .07. Relationship skills with peers remained at a high levelthroughout the program year. Nevertheless, 82 percent of studentsagreed that RALLY people helped them get along better with oth-ers at school. RALLY staff also successfully facilitated mediationsbetween students. The RALLY clinical director described how



some students who originally felt disconnected from other studentsin the school met each other in the RALLY space and created theirown support for each other. These students eventually began shar-ing with each other even when adults were not present. The stu-dents also showed increased emotion regulation skills, F(1, 68) =34.00, p < .001, η2 = .34.

Overall, 83 percent of the practitioners agreed that RALLY hadpositive impacts on students’ resiliency. The RALLY-designatedclassroom teachers all agreed or strongly agreed that the RALLYProgram helped to increase students’ relationship skills and resiliency.Thus, the program had an impact on adolescents’ social-emotionalcompetencies as perceived by students, practitioners, and teachers.This is an important finding, as most studies indicate that it is hardto get high levels of consistency across different reporter groups.

In regard to students’ development, pre- and posttesting showedno changes in social-cognitive development as indicated by thesociomoral measures. This finding could be related to the fact thatthe postassessment took place during a time when students weresomewhat burned out with testing (having just finished district aca-demic testing); thus, it is possible that the developmental test at thepostassessment underestimated the actual development of the stu-dents. This test requires a great deal of concentration and writingand is sensitive to motivational test-taking factors. It could alsomean that we need to be even more focused on bringing about thischange, which we anticipate to occur based on the relationshipsbetween students and practitioners without sufficient training inthe developmental skills that go into it. However, socioemotionalresiliencies at the postassessment significantly predicted social-cognitive development at the postassessment after controlling forsocial-cognitive development at the preassessment: F(4, 26) = 3.07,p < .05. This finding indicates that socioemotional resiliencies suchas emotion regulation skills increased from the beginning to theend of the program year, after controlling for social-cognitivedevelopment. This is important, as it shows that development islinked to socioemotional resiliencies, the latter being increased bythe program. Furthermore, 83 percent of the practitioners reported



that RALLY had positively influenced students’ development. They also reported that they helped students to better understand others’ points of view. In sum, findings on this student outcomewere mixed. Although there was not an overall increase in devel-opment demonstrated in pre- and posttesting, practitioners per-ceived an increase.

In regard to risks, teachers and practitioners reported decreasesof problem behavior at the end of the program year. Sixty-sevenpercent of the practitioners agreed that RALLY had helped todecrease behavioral problems, and all the teachers agreed orstrongly agreed that the RALLY Program specifically helped toreduce students’ behavior problems. There was a consensus amongthe teachers that the program was most successful in supporting at-risk students. Thus, behavioral problems decreased from the per-spectives of practitioners and teachers but not from the students’self-report. In contrast, we found no decrease in self-reportedsymptoms from pre- to posttest. However, 35 percent of the chil-dren reported to have experienced a negative life event during theprogram year such as a death, family problems, or interpersonalproblems. Furthermore, one of the students in the eighth grade wasshot and killed on the street outside his home during the schoolyear, thus leading to a school atmosphere that presumably aggra-vated the risks evident in some of the children. Thus, students whoare getting better in terms of symptoms are balanced by studentswho join the ranks of more symptoms in reaction to new problems.In a high-risk environment, it might be important to stabilizesymptoms and increase the positive issues such as resilience. That isindeed part of our practice: to focus greatly on the increase ofsocioemotional resilience and relationships to support students asa counterbalance to the many stresses and risks the students are fac-ing. A practitioner works with a student two days a week, some-times quite informally and briefly; the family, the community, andthe school environments shape the students’ day-to-day experienceseven days a week. This is not to say that more progress is needed,but a way of saying that we cannot expect any program to reduceall of a child’s symptoms.



Future studies are needed with larger samples, which will helpto consider subgroups of children and their qualitative change pro-files in risk factors. This would help to detect more specific changesin children with different risk factors and developmental orga-nization, and independent of specific life events and program ser-vices. Also, we need to be able to determine which students reducesymptoms and which students become more aware of their symp-toms, thus increasing the numbers on the test as a response to bet-ter self-awareness and more openness to report them.

A case vignette: Promotion of a student’s empathy and prosociality, by Lindsay Amper, RALLY practitioner

Nobody was really A’s friend, but everybody followed his lead.He was known for a trend that he started in which one studentslaps another student’s neck so hard that it leaves a red spot. Hisintimidating behavior often went unnoticed by adults in school,but so did his low self-esteem. My position as a RALLY practi-tioner allowed me to observe and work with students in a varietyof settings. It was not long before I picked up on the powerfuland negative influence he had on others.

During our initial conversations, A identified himself as beinga bad kid and feeling that he had no skills. Subsequently ourinterventions focused on relationship building and empathy, bol-stering his self-esteem, and helping him to envision and worktoward a future. Sessions were also used as a time for empower-ment: he taught me how to play checkers and to talk using slang.His lessons for me were a great source of pride for him. Discus-sions led to how he could use his power in a positive way. Andthen it happened: A stood up for the girl whom he had onceteased relentlessly and demanded that his classmates leave heralone. Amazingly, they did. An excerpt from a conversation thatfollowed after he stood up for the girl in class illustrates thebeginning of his prosocial leadership:

A: I don’t want you to think that I just did that thing with Susiebecause you were there.



P: (THE PRACTITIONER): Do you think that is what I thought?A: Yeah, cuz like you don’t normally see me do things like that.P: It made me very happy to see that. What made you do it?A: It wasn’t nice or nothing. And like it looked like she was going

to cry. They torture her every day, and that must feel really badto have people making fun of you like that all the time in school.School isn’t where that is supposed to be happening. . . . Yeah,well it just wasn’t right. I wish that other people noticed, though.

P: I think they did, A. Didn’t you hear what they said? Whenthey said something to you, you said, “Hey, man, I’m notlaughing.” And you weren’t. I saw you, and you had a verystraight face. I think that they noticed. I certainly did.

Throughout the rest of the year, A continued to demonstrateacts of kindness in the classroom. Shifting from a position ofaggressive dominance to a position of leadership and empathy,A was able to influence others with his prosocial values and sensibility.

By the end of the school year, A had applied for a summer job,considered running for student council president, and agreed todo a public speaking job for RALLY the following year. Otherstudents no longer feared him; they befriended him. Teachersno longer ignored him; they mentored and nurtured him.

Both practitioner and student are able to benefit from theirrelationship. One of the many lessons that I learned from A was the need to nurture the positives and believe in them evenwhen the students do not believe in themselves.

It is important to note that the changes in A are viewed by usas part and parcel of a developmental shift that we observe inmany students we work with (see also other case examplesthroughout this issue). The RALLY intervention encourages the“belonging world” that grows in the middle school years (withthe increased capacities of empathy, interest in others, and friend-ship). The interventions also discourage aggression and impul-sivity, so prevalent in the actively-engaged and assertive worlds.



Relationships as key to students’ development

In regard to perceived relationship quality, students reported higherpeer support, t(71) = –2.03, p < .05, but lower family support,t(68) = 2.95, p < .01 after the program end compared to programstart. As RALLY aims to promote interpersonal relationships, it ispromising that peer support was perceived as higher than before bythe students. It is not surprising that perceived family support didnot improve overall, as this RALLY site did not target family inter-ventions. However, RALLY did show success in initiatives directedtoward the larger school community. Based on interviews with stu-dents and program leaders, RALLY facilitated community-schoolpartnerships, which had a significant impact on the whole schoolcommunity, leading, in the words of the RALLY director, to “agreater sense of caring for each other, belonging, and contribution.”

Practitioners also reported success in relationship building. Theystrongly agreed that they established strong relationships with stu-dents (M = 3.67, SD = .52, on a four-point scale from 1, stronglydisagree, to 4, strongly agree). On average, most practitionersreported knowing more than half of the students in their classroomwell. Practitioners achieved this objective by working to ensure thatstudents felt that they were noticed, heard, and valued. This wasalso supported by student survey data, with 87 percent reportingthat they missed RALLY staff when they were not around. Evenmore important, 85 percent of students reported that they felt com-fortable talking about their feelings with RALLY practitioners, and97 percent of students agreed with the statement, “RALLY peoplecare how I’m doing.” At an age where life transitions and commu-nity stressors present frequent emotional challenges, having anearly outlet to process feelings can be a strong prevention methodto keep these emotions from being bottled up or erupting.

When asked in interviews about the strengths of the program,both the program director and the clinical coordinator describedthe relationships that practitioners developed with students. Theclinical coordinator emphasized the impact on students of “havingan adult who listens to them and they can go to; whether it’s a bad



day from a love drama or violence, they can talk to somebody aboutthat.” The relationships that RALLY staff built with students madethe RALLY office a safe space for students where they knew theywould be supported and cared for. In agreement with this view, theteachers felt that the overall impact of the program on the studentsand on the school as a whole was positive.

RALLY Program implementation and satisfaction

The second evaluation question focused on the way in which theRALLY Program was implemented and levels of satisfactionamong students, practitioners, and teachers. We briefly describethe implementation of RALLY services, and then elaborate on atti-tudes toward specific interventions as well as overall satisfactionwith the program.

Implementation of RALLY services. RALLY offerings includedclassroom support, informal RALLY groups (such as lunch bunches,tutoring, a student advisory board, and arts groups), formal RALLYgroups (including after-school groups), one-on-one mentoring (orsupport) or counseling with the RALLY clinician, and referrals toexternal agencies. While some aspects of RALLY were directed atthe whole class (such as classroom support and lunch bunches), stu-dents were referred to other services (such as formal RALLY groupsor counseling) based on the assessment findings (see the previousarticle) and conjoint RALLY team discussions.

As part of RALLY’s whole-classroom and prevention strategies,almost all students participated in one or more informal RALLYgroups (such as lunch bunches or tutoring). Thirty percent werereferred to external agencies such as community after-school pro-grams or a day-long workshop on community building and reduc-ing community violence. Another 30 percent received one-on-onesupport from the RALLY clinician, outside therapy, or practi-tioners. In addition, 13 percent of the students participated in formal groups such as a clinical group focusing on loss, an empow-erment group, and a sports-based group. Thus, informal groupreferrals were made for all students, and additional referrals weremade for tier two or tier one students who needed higher levels of



support. There were also students who were referred to specificinterventions but were unable to connect to resources due to vari-ous obstacles, such as transportation, student or family motivation,scheduling, or insurance. This was particularly true for the one-on-one support and counseling (15 percent of RALLY participantswere referred to such services but did not actually attend, in con-trast to the 30 percent who did receive those services). In addition,8 percent of RALLY participants were referred to formal groupactivities but did not actually attend (in contrast to 13 percent ofRALLY participants who did participate in formal groups). Thenumber of unsuccessful referrals for informal groups was muchsmaller, at only 4 percent, possibly since informal groups requiredless paperwork and in many cases occurred during the schoolday,minimizing logistical obstacles. These findings led us to increasethe in-school counseling services.

Evaluation of RALLY services and quality of collaboration. Wefirst elaborate the students’ attitudes toward intervention compo-nents of the program:

• In-class support. While more than half of the students held very pos-itive attitudes toward the in-class support provided by practitioners,a significant percentage also held more neutral attitudes toward it,marking “it’s okay.” This could be due to the fact that some practi-tioners found it most effective to work primarily with a smallernumber of students who needed extra support in class, and as aresult may not have had a significant impact on the other studentsin the class. It is a tribute to the relationships that practitioners heldwith students that only 4 percent of students reported that they didnot like it when practitioners were in class, since most practitioners,as another adult presence in the classroom, at times played a disci-plinary role, checking behavior and redirecting students’ attention.

• Informal groups. Students’ attitudes toward informal groups suchas lunch bunches were quite positive, with 72 percent of studentsreporting that they were “great.” Students appreciated that lunchbunches were a place where you could “express your feelings.”One student stated, “We get to hang out and talk—we get to talk



about feelings.” For many, lunch bunches were a place wherestudents could express themselves and talk about their feelingswithout feeling stigmatized.

• One-on-one sessions with practitioners. Overall, students’ attitudestoward one-on-one meetings with their practitioners were notas high as attitudes toward lunch bunches: 43 percent reportedthat they were great, whereas 55 percent said that they wereokay. This finding may indicate support for RALLY’s strategy ofdevoting more time to whole-group interventions and less timeto one-on-one pull-out interventions. However, all the studentssaid that they were more likely to talk to practitioners than toguidance counselors or teachers if they had a personal problem.This finding is supported by the overall evaluation of the pro-gram, where 85 percent of students said they “feel comfortabletalking about feelings to RALLY people.”

While the evaluation focused on student outcomes and student per-ceptions of the program, practitioners’ perceptions of different inter-vention components were also solicited through surveys as well as focusgroups. According to practitioners, one-on-one support, informal andformal groups, schoolwide activities, and RALLY support space wereevaluated as positive or very positive (see the fifth article). The onlyservice that was evaluated as somewhat mixed was the in-class support:34 percent evaluated this service as negative or neutral, whereas 66 per-cent reported it as positive. The latter finding may be related tothe fact that the efficacy of in-class support strongly depends onpractitioner-teacher interactions and how much the practitioner feelsthat he or she can affect the classroom climate. It also sometimes putsthe practitioner in the role of disciplinarian or at least as someone whoholds high expectations to conform to the classroom expectations andfocused task completion, something students do not necessarily like.

Practitioners were also asked how they perceived the trainingRALLY provided, as well as the strength of collaboration betweenthe people and groups involved in RALLY. Quality of training, col-laboration, and communication between the parties is of key impor-tance to high-quality program implementation. Practitioners



perceived the formal and informal supervision as very useful (Ms =4.50 [formal] and 4.33 [informal supervision], scale ranging from1 = not at all useful to 5 = very useful). The overall support and train-ing provided by the program was also perceived as high (M = 3.83).In contrast, the weekly seminar and the one-day training at thebeginning of the program year were perceived as somewhat less use-ful (M = 2.67 [seminar], M = 3.00 [one-day training]). Focus groupsrevealed that these trainings were perceived as less connected topractitioners’ daily work than, for example, informal supervision.

Practitioners also assessed the quality of the collaboration withthe various people involved in RALLY. The perception of the dif-ferent collaborations was predominantly neutral to positive. Thecollaboration with the homeroom teacher was perceived as pre-dominantly positive (67 percent), whereas the collaboration withother teachers, community organizations, and family was perceivedas somewhat more neutral (60 to 67 percent). This is not surpris-ing, as practitioners worked most closely with homeroom teachers.

The teachers reported favorably on their experience collaborat-ing with the RALLY staff. The three teachers who returned ques-tionnaires agreed or strongly agreed that the RALLY staff formedclose relationships with students and worked effectively with them.However, because only half of the classroom teachers participatedin this survey, we cannot entirely exclude the possibility that onlythe teachers with a positive attitude completed the questionnaires.However, practitioner reports reflect similar positive perceptions,making negative evaluations unlikely. Nevertheless, one teacherrecommended improving communication about ongoing problemsand referrals of students. Furthermore, the teachers evaluated thepractitioners as being respectful of classroom teachers’ agendas,collaborating effectively with teachers, and making teachers feelsupported. One teacher said, “Often teaching is an isolating expe-rience. As the only adult in the room you must be all things to allpeople. A capable practitioner means you have twice the eyes, twicethe ears, and twice the time to listen and guide the children.”Teachers’ positive reporting of interactions with practitioners isnotable, as sharing the classroom with another adult requires an



adjustment. As one teacher stated: “I needed to learn that the prac-titioner had his job to do too. Sometimes he wanted to meet witha student when I wanted to teach a lesson. We had to figure outhow to work around each other, and how to talk about our plans.”

Program satisfaction

Regarding overall program satisfaction, there was a high level ofsatisfaction reported by students, practitioners, and teachers. Whenstudents were asked to complete the sentence, “RALLY is . . . ,” 99percent of students wrote in a positive adjective or experience, and93 percent agreed that they wished they had RALLY every year.Overall, practitioners also held positive attitudes toward RALLY,with 60 percent reporting being satisfied with the RALLY Pro-gram. Eighty percent of the practitioners evaluated the impact ofparticipation in the RALLY Program on professional developmentas positive. The teachers also reported being highly satisfied withthe program as a whole, as did the principal and the administrators.

ConclusionThe RALLY Program services provide social, emotional, and aca-demic benefits to the students it serves. In this pilot study of oneyear in a RALLY Program, student participation in services wasreported as having an impact on resilience and academic skills. Morespecifically, there was an increase in empathy, emotion regulation,and trust in others, as demonstrated in the pre- and posttest com-parison. Emotion regulation skills were related to students’ higherlevel of development at program end after having controlled fordevelopment at the beginning of the program. Furthermore, over80 percent of the students reported that the program helped themwith their social relationships at school and helped them to improvetheir schoolwork. Thus, evidence shows that RALLY promotedresiliencies that facilitate development. Furthermore, practitionersreported feeling successful in providing students with support toenhance their academic success, resiliency, and development, as well



decrease problem behaviors. This view was supported by the teacherreports. However, students’ self-reported social-cognitive develop-ment stayed stable, and the level of risk factors did not decline. Pos-sibly these latter findings indicate that students became more awareof their emotions and more honest in reporting their problems as aresult of program participation and discussions with RALLY staff.But because not all referrals that were thought to be beneficial weresuccessfully completed (as a result of logistical or motivationalobstacles), this finding may also indicate that more effort needs tobe put into a fast and comprehensive referral system that covers asmany referrals as an individual adolescent might need.

Previous evaluations of schools that implemented RALLY alsorevealed evidence that RALLY Program services can improveresiliency, mental health, and academic functioning in adolescents.For example, over the course of one program year in a U.S. urbanpublic school, two-thirds of the students working with RALLYshowed improvement in their school functioning.6 Furthermore,42 percent of the tier one and tier two students (those with highlevels of risk) showed increased grade point averages at the pro-gram’s end. More than three-quarters of these students showeddecreased involvement in antisocial behaviors compared to thepretest. Regarding classroom outcomes, a great majority of studentsshowed improved attitudes toward school, conflict resolution skills,and social skills in the classroom; they also showed a clear decreasein problem behaviors in the classroom when compared to thepretest data.

One of the greatest benefits of RALLY is the increase in sup-portive relationships: positive adults within the school setting whooffer guidance and care to students. These adults also act as facili-tators of developmental growth, such as establishing trust in oth-ers, a cornerstone for the majority of adolescents whose assessmentsindicated they were predominantly at the reciprocal-instrumentallevel in their development. Thus, creating caring and supportiverelationships with practitioners and working with the developmen-tal needs of the participants yielded a key to program success andhelped facilitate students’ resiliency. This is supported by previous



findings that supportive relationships and the creation of a caringschool environment are key to students’ growth.7 Moreover, devel-opmental resiliency can be increased by targeting the overall group,but also by offering one-on-one support or group treatment for stu-dents with different needs.

Despite the mostly positive outcomes, several barriers preventedthe program from reaching its maximum capacity. These includedinsufficiently rapid or incomplete referrals and at times weak per-ceived collaboration and integration of some of the service compo-nents. Ensuring that students reach their maximum potential requiresa high degree of collaboration, integration, and communicationamong project components.

Furthermore, several limitations of this evaluation report need tobe mentioned as well. First and foremost, we included only self-report data in the pre-post test design. Although this informationwas backed up by information after the assessment from practition-ers and teachers, this methodology needs to be supplemented in thefuture by observations and additional objective data. And becausethis was a pilot study, no control group was included, and thereforeit is not possible to draw causal conclusions on the interventioneffects. Future studies with rigorous empirical designs are plannedand will be conducted once funding is secured. Third, many of themeasures were rather limited (there were two or three items perscale), and although reliabilities were predominantly strong, valida-tion studies are certainly needed for a few of the instruments, espe-cially those pertaining to resiliency and relationships.8

Despite these limitations, the RALLY Program model hasdemonstrated a strong buy-in from teachers, administration, guid-ance, and other school staff over more than ten years of services,with a solid network of community partners who are committed tosupporting it. The findings are promising and show that a focus oncaring and supportive relationships and bridging different servicesto students is likely to be helpful to promote students’ resiliencyand development. Furthermore, a fast and efficient referral systembased on holistic assessments is needed to implement program ser-vices appropriate to the adolescent’s development needs and inter-



ests. Finally, collaboration and supervision are of key importanceto ensure program quality.

While we focused here on the evaluation for the RALLY Program,this issue is also about the RALLY approach. Thus, we presented theevaluation method here not only as a way to explore whether we areshowing positive evaluation results, but to describe how school-basedprograms can make themselves evidence based without spending abig part of their budget. Thus, the RALLY approach to evaluationincludes building in assessment and having it be short and focused soit can be done in one classroom session, which helps to avoid a lot ofproblems. The use of preassessments helps to understand the stu-dents so that evaluation and assessments do not need to be done sep-arately. Continuous qualitative and quantitative program qualityevaluation and satisfaction during the program year help to ensurehigh program quality. This helps also to ensure that the data collectedare not only for funders and outside organizations or the evaluationcommunity, but that they are applicable to program management andplanning for the following year. Finally, if possible, it is important toget data from all parties—students, practitioners, teachers, adminis-trators, community partners and families—because this will help tounderstand the entire system in which the student is embedded andhelps him or her to develop.

In summary:

• Evaluation is an important program component of RALLY andhelps to define its gains. RALLY’s six-step plan aims to ensurehigh-quality evaluation for schools that implement the program.

• One RALLY Program implemented at an urban middle schoolwas described to illustrate the holistic assessment approach andevaluation strategies for practitioners, researchers, and othersinterested in RALLY. We analyzed student outcomes and pro-gram implementation quality.

• The findings showed effects on a student’s resiliency and social-emotional learning as well as academic success as indicated bystudent, practitioner, and teacher reports. The practitioners andteachers also reported a decrease in behavioral problems.



• Relationships to practitioners proved to be of key significancefor changes in students’ resiliency and academic outcomes.

• Regarding program implementation, high numbers of studentswere referred to services. However, there was also a group of stu-dents who were not successfully connected to resources due to various obstacles. The quality of services was perceived ashigh, and collaboration among providers was predominantly per-ceived as positive. Supervision was also evaluated as positive overall, although some components need improvement. Program satisfaction, as reported by students, practitioners, teachers, andprogram leaders, was high overall.

• Future studies with mixed quantitative and qualitative researchdesigns are needed to evaluate RALLY in a comprehensive way.

Appendix: RALLY Initial InterviewGil G. Noam, Tina Malti, Lizzie Salzfass, and the RALLY Team(Performed by the practitioner in dialogue with every youth in theclassroom)

1. ID:2. Name:3. Homeroom:4. Birthdate (month, day, year):5. Gender:6. Today’s Date:7. Address:8. Home Phone number:

a. Who is the parent/guardian we can reach at this number?b. What languages does this person speak?c. Does any adult in your home speak English?

9. How do you get to and from school? Please circle your answer(s).

Walk Public transportation School bus Ride

If ride, from who?



10. What is your class schedule/teachers?11. Who makes up your immediate family? How many siblings do

you have, and how old are they?12. Which adults take care of you at home?13. What do the adults in your home do for work? (Please be spe-

cific: for example: mechanic, nurse, teacher, receptionist, takescare of children)Adult(s) in your home (please specify, e.g., mother):Type of Work:

Basic School Questions

1. Favorite subject in school (circle only one):

ELA Math Social Studies Science Gym Health Art

Dance Theater

Why is this your favorite subject?2. Least favorite subject in school (circle one):

ELA Math Social Studies Science Gym Health Art

Dance Theater

Why is this your least favorite subject?3. I usually do my homework (circle one):

Every day A few days a week Sometimes Rarely

4. I have someone to help me with my homework: Yes / No If yes, who?

5. I would like to get more help with homework: Yes / No6. What I typically do after school (please be specific!):7. Which afterschool activity do you like best?8. Is there anything else you’d like to do after school, such as join

organized sports, dance, afterschool program, etc.?9. What I typically do on the weekends (please be specific!):

Day Night

Saturday

Sunday



10. Is there anything else you’d like to do during the weekend?11. How many hours per day do you:

Watch TV ____Play video games ____Go on the Internet ____

12. Do you have a paid job? If so, what do you do?13. Do you have an unpaid job? Do you have chores? If so, what

do you do/what kind of chores do you have?

Guiding Interview Questions

School• How much do you like school, on a scale of 1 to 10 (1 = not at

all, 10 = school is okay)? Why?• How would you describe yourself as a student?• What are you good at in school? What do you want to improve at?• What do you like about school? What do you want to change

about school? Why?

Activities• What music and artists do you enjoy listening to?• What are your favorite sports? Do you like watching or playing

these sports?• What are some hobbies, activities, or games that are your favorites?• What kind of food do you like to eat?

Relationships• If you needed to talk to an adult, who would you talk to? (If

answer is no one, ask, “Which adult do you feel knows you best?) Why?• Tell me about some of your friends. What are they like? What

do you do together?• Who are some people in your family you feel close to? Why do

you feel close to this person?• What sorts of things do you do with your family?

Self and development• What are you the most proud of about yourself? Why are you

proud of this?



• What would you like to get better at? Why?• What do you think are some of your challenges? How do you

want to change this?• Who is someone you admire, and why?• Is there anything else you would like me to know about you?

Notes1. European Health Report. (2005). Public health action for healthier children

and populations. Geneva: WHO Regional Office for Europe.2. Malti, T., Liu, C. H., & Noam, G. G. (in press). Holistic assessment in

school-based, developmental prevention. Journal of Prevention and Interventionin the Community; Noam, G. G., Goldstein, C., & van Dyken, L. (2003).RALLY: Program manual. Cambridge, MA: Harvard University, and Belmont,MA: McLean Hospital.

3. Gibbs, J. C., Basinger, K. S., & Fuller, D. (1992). Moral maturity: Mea-suring the development of sociomoral reasoning. Mahwah, NJ: Erlbaum.

4. Achenbach, T. M., & Edelbrock, C. (1987). Manual for the Youth Self-Reportand Profile. Burlington, VT: University of Vermont Department of Psychiatry.

5. Gibbs, J. C., Basinger, K. S., Grime, R. L., & Snarey, J. R. (2007). Moraljudgment development across cultures: Revisiting Kohlberg’s universalityclaims. Developmental Review, 27, 443–500.

6. Maike, M. M., & Nixon, A. (2007). Tacoma Public Schools: Responsive advocacyfor life and learning in youth: RALLY. Unpublished report, Tacoma Public Schools.

7. Berkowitz, M. W., Sherblom, S., Bier, M., & Battistich, V. (2006). Edu-cating for positive youth development. In M. Killen & J. G. Smetana (Eds.),Handbook of moral development (pp. 683–701). Mahwah, NJ: Erlbaum.

8. Tolan, P. H., & Dodge, K. A. (2005). Children’s mental health as a pri-mary care and concern. A system for comprehensive support and service.American Psychologist, 60, 601–614.

TINA MALTI is a visiting research scientist at McLean Hospital and Har-vard University.

SARAH E. O. SCHWARTZ is a research fellow at the Program in Educa-tion, Afterschool, and Resiliency at McLean Hospital and Harvard University.

CINDY H. LIU is a postdoctoral fellow of clinical psychology at HarvardMedical School–Children’s Hospital of Boston.

GIL G. NOAM is the founding director of the RALLY Program and asso-ciate professor of developmental psychopathology at McLean Hospital andHarvard University.

https://www.researchgate.net/publication/7589799_Children's_Mental_Health_as_a_Primary_Care_and_Concern_A_System_for_Comprehensive_Support_and_Service?el=1_x_8&enrichId=rgreq-d5371077c1d7f0dbdf5d0014b5c2b4ab-XXX&enrichSource=Y292ZXJQYWdlOzIzOTQwMzY4O0FTOjI3MTg1ODIyMjIzNTY0OEAxNDQxODI3NDU5MTYz



Program evaluation: Relationships as key to student development

Documents

Transcript of Program evaluation: Relationships as key to student development