P.E.P. Wellness Triage: Stakeholder Evaluation

Running Head: STAKEHOLDER EVALUATION 1

P.E.P. Wellness Triage: Stakeholder Evaluation

Myrna Davis-Washington

University of the Rockies

\

Running head: STAKEHOLDER EVALUATION2

Abstract

“Stakeholder evaluation” is essential to the effectiveness and

longevity of every successful project in every successful

organization. As the cornerstone of project management, it is an

important strategic planning process that expresses the quality

of any program or project through the viewpoints, values, and

needs of the stakeholders. Successful managers use stakeholder

evaluation to identify the stakeholders in their projects, to win

the support of all those who are affected by their projects, and

to help ensure that their projects succeed where others fail. The

purpose of this paper is to examine stakeholder evaluation for

the “Performance Enhancement Program (P.E.P.) Health and Wellness

Triage,” a board-requested, high school-level health and wellness

project in a low-income school district in Montbello, Colorado,

an area of high urban community violence with predominantly

African American and Mexican American students. Three of the


project’s major stakeholders (Faculty and Staff, Students, and

Project Instructors) are examined and paired with appropriate

needs assessments, progress evaluations, and key indicators. This

paper discusses stakeholder evaluation for this project by

exploring four of its major components: Identification, Needs

Assessments, Progress Evaluation, and Key Outcome Indicators.

P.E.P. Wellness Triage: Stakeholder Evaluation

The quality of any program or project expresses itself

through the viewpoints, values, and needs of its stakeholders. As

the cornerstone of project management, the “stakeholder

evaluation” is an important strategic planning process that is


essential to the effectiveness and longevity of every successful

project in every successful organization. Successful managers

use stakeholder evaluations to identify the stakeholders in their

projects, to win the support of all those who are affected by

their projects, and to help ensure that their projects succeed

where others have failed. (Deubel, 2003; Michigan Department of

education, 2009; MindTools Ltd., 2010)

In this paper, we examine stakeholder evaluation for the

“Performance Enhancement Program (P.E.P.) Health and Wellness

Triage” as the final project in the class, “ORG 6350: Wellness

Program Leadership.” P.E.P. is a board-requested, high school-

level health and wellness project being implemented in a low-

income school district in Montbello, Colorado, an area of high

urban community violence with predominantly African American and

Mexican American students. The district school board, members of

the school board, faculty and staff, students, parents, and the

community will all have a “stake” in this project and will be

involved in every stage of the project, from design and

implementation to decision-making and receipt of services. This


paper will discuss stakeholder evaluation for this project by

exploring four of its major components: Identification, Needs

Assessments, Progress Evaluation, and Key Outcome Indicators.

Literature Review

For the astute and ‘seasoned’ project or program manager,

stakeholder evaluations have been the cornerstone of stakeholder

management for the past thirty years. Hambrick (1979) noted the

importance of needs assessments in setting program priorities,

setting policy agendas, and identifying administrative programs.

Hambrick (1979) suggested that program managers use common

techniques such as problem checklists, interview card games,

organization cards, process cards, the Delphic approach (where

respondents are sent an instrument for getting their initial

judgments on a subject), assistance solitaire and conferencing,

and ranking (which requires respondents to rank problem areas,

their severity, and the type of solution needed.

Ten years later, Rossett (1989) developed a systematic

approach to assessing needs called “gap analysis.” Gap analysis

examined the discrepancy between the way things are (“actual” and


the way they could or should be (“optimals”) by assessing

information in three areas: proficiency, criticality, and

frequency. Rossett (1989) suggested that five tools (i.e.,

interviews, surveys, observations, focus groups, and document

examination) could be used to assess stakeholders’ opinions on

four kinds of information: optimal performance, actual

performance, subject, skills, new system, or technology; and

causes of the problem.

At the Youth Services Bureau of Ottawa-Carleton in Canada,

however, trainers and developers realized something profound.

After using such methods as staff questionnaires, structured

interviews of front-line supervisors about their staff, and a

critical-incident method, they realized that the best method of

assessing needs was simply to ask staff members to indicate the

job tasks in which they would most like to be trained (Darou,

1990). In other words, the most effective method of assessing

needs is simply to “communicate” with people by talking to them.

In practical terms, the complete needs assessment allowed the

agency to develop a credible training plan based on four basic


principles: (1) Do the needs assessment first; (2) Quantify

everything; (3) Use several sources of information; and (4) Base

needs on work tasks (Darou, 1990).

Several studies found that the most effective procedure for

assessing stakeholder needs involved: (1) identifying the

stakeholders; (2) prioritizing the stakeholders; (3)

understanding the stakeholders by taking a closer look at some of

the groups; and (4) analyzing the stakeholder needs

(MindTools,Ltd., 2010; The World Bank Group, 2001).

Additionally, Kelly (2010) suggested that an integrated approach

(integrating planning, assessment, and improvement) will help

improve the quality of teaching, learning, and service to

students by planning, analyzing process performance, developing

improvement solutions, implementing the solutions, documenting

the process, and evaluating and planning for continuous

improvement. Curran and Totten (2010) also stated that

successful strategic planning involved gaining buy-in from key

participants and other stakeholders, conducting an environmental

assessment, analyzing strengths, weaknesses, opportunities and


threats; developing planning and financial assumptions;

establishing the strategic framework, setting strategies, and

establishing performance measures.

To address the component of evaluation, one study (Kitson

and Straus, 2010) reexamined Rossett’s (1989) “gap” analysis, in

which researchers systematically analyzed the discrepancy between

“optimal” outcomes (what should or could be happening in the

project) and “actual” outcomes (what is happening in the project).

Kitson and Straus (2010) also suggested that researchers look

beyond the “gap” to discover “why” gaps in performance exist and

why problems exist in a program. Straus, Tetroe, Graham, and

Zwarenstein (2010) stated that the selection of a strategy for

evaluation depends on whether researchers want to enhance local

knowledge or provide generalized information. Several authors

also stated that if the implementation of a program targets a

behavior for which a strong evidence of benefit exists (i.e., on-

site bullying), measuring the impact of the program in terms of

whether the behavior has occurred, rather than whether a change

in clinical outcomes has occurred, may be appropriate (Straus,


Tetroe, Graham, and Zwarenstein, 2010; Srabstein and Leventhal,

2010).

Srabstein and Leventhal (2010) described “bullying” as a

psychological health hazard and a major health problem that

demands the concerted and coordinated time and attention of

health-care providers, policy-makers and families. These authors

defined bullying as a multifaceted form of mistreatment that is

characterized by repeated exposure of one person to physical

and/or emotional aggression including teasing, name calling,

mockery, threats, harassment, taunting, hazing, social exclusion,

and/or rumors (Srabstein and Levanthal, 2010). This will be

useful in classifying and evaluating student on-site bullying.

Brewer and Clippard (2002) studied burnout and job

satisfaction among student support services personnel by using

the Maslach Burnout Inventory and the Job Satisfaction Scale.

According to these authors, the variables that are related to

burnout fall into three categories: (1) environmental and

organizational factors such as work overload, role conflict, and

the work environment itself, (2) personality factors (i.e.,


idealism, obsession, introversion, extroversion, anxiousness),

and (3) demographic characteristics (females, younger employees,

and single employees have higher burnout rates) (Brewer and

Clippard, 2002). These researchers also indicated that high

burnout has been linked to poor job satisfaction, low

organizational commitment, desire to quit one’s job, and

attrition (Brewer and Clippard, 2002). Findings from their

research disclosed a significant negative relationship between

emotional exhaustion and total job satisfaction, a significant

positive relationship between personal accomplishment and total

job satisfaction, and an overall significant relationship between

the three components of burnout and total job satisfaction: (1)

emotional exhaustion (which has been shown to have a significant

negative relationship with total job satisfaction); (2)

depersonalization (also shown to have a significant negative

relationship with total job satisfaction); and (3) personal

accomplishment (which has been shown to have a significant

positive relationship with total job satisfaction) (Brewer and

Clippard, 2002).


One study (Teufel-Shone, Siyuja, Watahomigie, and Irwin,

2006) suggested that assessment and evaluative interviews be

open-ended and semi-structured. These researchers used a

formative assessment to design a school-based and family-outreach

intervention program they named SPARK 15 (Sports, Play, Action,

and Recreation for Kids) and a youth-oriented fitness assessment

called ‘Fitnessgram.’ The authors also presented intervention

strategies that build upon the importance of family involvement

and community support by implementing family events and by

reporting program outcomes locally, regionally, and nationally

(Teufel-Shone, Siyuja, Watahomigie, and Irwin, 2006). Findings

suggested that supportive programs have consistent adult

leadership, structured activities, and a positive local and

regional image (Teufel-Shone, Siyuja, Watahomigie, and Irwin,

2006).

Finally, one study (Poraj, 2010) pointed out the fact that

most of the Faculty and Staff stakeholders in the P.E.P. Wellness

Triage would be female. This study used The Hope Scale, the

Maslach Burnout Inventory, and three other instruments to classify


teachers into three groups: ‘passive teachers,’ ‘frustrated

teachers,’ and ‘adapted teachers’ (Poraj, 2010). Findings showed

that passive teachers included women who were not distinctive in

any way; that frustrated teachers were characterized by very

unfavorable traits that lend themselves to burnout; and that

adapted teachers were full of energy and enthusiasm for work and

had high job satisfaction (Poraj, 2010). This is strong evidence

for using the Maslach Burnout Inventory to measure Faculty and Staff

burnout in the P.E.P. Wellness Triage. According to this study,

most teachers in the proposed school district for the P.E.P.

Wellness Triage will be either passive of frustrated. By

involving the adapted teachers in the planning and implementation

of the project from the beginning, the evaluation team can gain

their commitment and use it to recruit the other Faculty and

Staff stakeholders who are resistant, unwilling, disinterested,

or too passive to participate in a program of this nature.

Identifying Stakeholders

Generally speaking, “stakeholders” are people who have a

“stake” in the project; anyone who is affected by the project,


who has influence or power over it, or who has an interest in its

successful or unsuccessful conclusion. Possible stakeholders in

the PEP Wellness Triage include the district school board

members, the faculty (teachers) and staff (administrative and

non-teaching) at the district’s two high schools, the students

and their parents at both high schools, the communities

surrounding both schools, the project’s team members, the local

press, the Internet community (since most high schools now use

blended education formats and have websites), the local

government, community businesses, and other interest groups in

the proposed districts. Since all of these stakeholders have the

power to either block of advance the project by their level of

interest and participation, it is important to prioritize and

classify the stakeholders according to the level of power and

interest that they can exert over the project, the project’s team

members, and the other stakeholders in the project (Johnson and

Breckon, 2007).

In this paper, we will focus on the three most influential

and important stakeholders in the P.E.P. Wellness Triage project:


(1) “Faculty and Staff,” (2) “Students,” and (3) “Project

Instructors.” While these three groups of stakeholders are

‘major’ stakeholders in this project, they also provide astute

and diligent project managers with the information needed to

analyze the power structure, politics, and personalities of the

school’s organization. Their ability to produce an effect within

the project, to make things happen, to make a difference in

realizing the project’s outcomes, and to exert power over every

aspect of the project is an essential part of stakeholder

management; one that will balance, control, define, and

accomplish the goals of the project. From a positive

perspective, when these three stakeholders interact harmoniously,

the program will run smoothly and efficiently and the goals of

the district and the project will be achieved and even exceeded.

However, when there is friction between any two or all of these

stakeholders, the program will struggle to achieve its goals and

will most likely be short-lived. (Johnson and Breckon, 2007).

“Faculty” stakeholders are those stakeholders employed in

teaching positions at each of the two school sites. “Staff”


stakeholders are those employed in non-teaching or administrative

positions at the two district high schools. Faculty and Staff

were chosen as stakeholders in this project because they stand to

benefit or lose more than any other group in the project.

Understandably, the constant stress created by the on-site

bullying and criminal activity at both high schools and the

frustration caused by the schools’ low academic statuses (which

Faculty stakeholders feel is a reflection of their skills and

performance) are reflected in a high ‘burnout rate” (i.e., high

absenteeism, high turnover, and health problems) and high job

dissatisfaction. Their ‘stake’ in this project is closely tied to

their job performance, job satisfaction, personal mental and

physical health, and even their personal relationships and home

lives. The success or failure of this project will have a ripple

effect in every aspect of their lives. If the project is

successful in improving grades and reducing on-site bullying,

these stakeholders will be the first to notice, as their careers

and job performance will reflect the program’s success. By the

same token, if the project fails, the Faculty and Staff will


receive the brunt of its failure on personal and professional

levels.

As a group, Faculty (teachers) and Administrative Staff

(principals, assistant principals, and school nurses and

psychologists) are “high power, interested people.” While the names

of these stakeholders are not known at the writing of this paper,

it is safe to say that this is the second largest group of

stakeholders in this project. These stakeholders are the people

with whom the project team members must fully engage and make the

greatest efforts to satisfy. Non-teaching Staff (secretarial

staff, janitorial staff, cafeteria staff, and security staff) are

“low power, interested people” who can often be very helpful with

the details of the project (i.e., scheduling, class locations,

school security, participant safety). Because both are

‘interested’ stakeholders, managers should strive to keep them

adequately informed by talking with them and keeping the lines of

communication open to ensure that no major issues arise on a day-

to-day basis. One of the reasons that this group is such a

powerful group of stakeholders is because both formal and


informal power structures play a role in organizational politics,

and the personalities and egos of these stakeholders usually

interact to dominate both the power and political networks of any

project. One dissatisfied stakeholder in this group can undermine

the success and efficiency of the entire project. On the other

hand, a satisfied Faculty and Staff member will add impetus to

the program and will facilitate the project’s growth and

reception among students, parents, and the other stakeholders

(MindTools Ltd., 2010; Johnson and Breckon, 2007).

Although “Faculty and Staff” may wield a substantial amount

of power and influence over the project, “Students” are by far

the largest and most important group of stakeholders in this

project. It is expressly for this group that this program has

been requested by the board and it is through their acceptance

and participatory efforts that the project will achieve its goals

and ultimately succeed. As a group, Students are usually “high

power, less interested people” in whom the project manager must put

enough work to keep them satisfied, but not so much that they

become bored with the project’s message (MindTools Ltd., 2010).


Currently, there are 1, 864 full-time high school students in the

proposed school district; 978 at School A and 886 at School B.

It would not be realistic to anticipate that all of the students

will attend the P.E.P. Wellness Triage program. However, in order

to qualify for participation in the project, all student

participants must be currently enrolled as full-time students (13

– 21 years old) in grades 9 – 12 at either of the district’s high

schools. In addition, any student who has been involved in any

reported on-site bullying or criminal activity (victim and

perpetrator) and with a grade point average below ‘1.9’ will be

required to participate in the program. Faculty, staff, and

parents may request that a student be required to attend classes,

as well. In addition, breaking the group down by gender and

ethnic origin will give managers and planners insight into how to

vary the program’s classes and seminars.

“Instructors” are employees of the school district who have

been employed for the express purpose of implementing the

program’s scheduled classes. Although the project design calls

for four (4) instructors at the start of the program, that number


is subject to change pending the stakeholder needs assessments.

As a group, Instructors will be involved in all phases of the

project and will serve as integral members of the “data team”

whose function is to generate and analyze data that the planners

require (Johnson and Breckon, 2007). All Project Instructors will

have at least a bachelor’s degree (although a Master’s degree is

preferable, since many of the assessment instruments require that

those administering the instruments have a Master’s) in a health-

related field and will be industry-certified (i.e., ACE, IDEA,

AFAA) as a health and wellness leader. In addition, a background

that includes an understanding of music and dance is preferable,

but not required; as development of spatial intelligence and

musical intelligence will enhance Instructor performance. These

educational requirements are necessary because they will

attenuate and mediate any professional friction that may occur

between instructors and the faculty and staff, and between

instructors and students.

There are many reasons that this group was chosen as

stakeholders. Firstly, they have a vested interest or stake in


the success of this program because the success and growth of

this program means that they will remain employed as long as they

desire to stay with the program. As a group, they are also “high

power, interested people” with whom the project manager must

fully engage and make the greatest efforts to satisfy. As

stakeholders, Instructors will reflect the quality of the

program’s design and will interact directly with every other

stakeholder in the project, with the most frequent

interrelationships occurring between the instructors and the

faculty, staff, students, and parents. In addition, Instructors

will act as liaisons between the project team managers and the

other stakeholders, as well as, between the school board and the

other stakeholders. To ensure the success of the project, the

school district and project manager must ensure that each

Instructor position is filled by a qualified, productive, and

personable person who reflects the vision and goals of the

project. (Johnson and Breckon, 2007; MindTools Ltd., 2010).

Assessing the Needs of Stakeholders


A “Needs Assessment” is a systematic process to acquire an

accurate, thorough picture of the strengths and weaknesses of a

school community that can be used in response to the needs of the

stakeholders to determine priority goals, to develop a plan, and

to allocate funds and resources (USDE, 2010). After identifying

and prioritizing the stakeholders, it is important to develop a

more in-depth understanding of the stakeholders and their impact

on the project by assessing their feelings, needs, values, and

opinions about the project, its team members, and each other. How

do they feel about the project? What are their reactions to the

school board’s request for this project? What time of day are

they most active (morning, noon, evenings)? What financial or

emotional interests (positive or negative) do they have in the

outcome of the project? What expectations do they have in the way

of information (what information do they need from the project

and its team members?), communication (what is the best way of

communicating the information to them?), implementation (how do

they want to receive the information from the project?), variety

(what motivates them?), and quality (what do they think of the


project and its team members, and who generally influences their

opinions?)? An effective strategy that answers all of these

questions and works at nearly every level of project planning and

implementation is to appeal to the stakeholders at the individual

level because, although they all identify with their

organization, their ultimate concern is usually for themselves

(Johnson and Breckon, 2007). In other words, build enthusiasm in

the project by answering every stakeholder question about the

benefits of participating in the PEP Wellness Triage with the

answer to “WIIFM?” (What’s in it for me?).

To keep stakeholders from viewing the needs assessment as a

threat to their personal space and an invasion of their privacy,

and responding with fear, uncertainty, and resistance; the

project manager should strive to maximize the stakeholders’

influence over the project. One of the most effective ways to

‘sell’ stakeholders on ‘buying into’ the project is to involve

them in the planning and decision-making process as early in the

project planning process as possible. Some of the most commonly-

used needs assessments include: (1) a problem checklist that


includes a list of problems or topics which respondents are asked

to rank in order of importance; (2) the interview card game that

is used on people from different levels in a city organization or

school; (3) process cards; cards that represent school functions

and activities are matched with parts of the school that the

respondent sees as being weak; (4) the Delphic approach in which

respondents are sent an instrument for obtaining their initial

judgments on the project; (5) questionnaires and surveys

indicating preferences and opinions; (6) structured interviews

with faculty about the students and other stakeholders; (7)

observations; (8) focus groups; (9) document examination; and

(10) “gap analysis”, a systematic search for details about the

discrepancy between what is happening (“actual”) and what should

be happening (“optimals”). However, although any and all of these

strategies may prove to be effective tools for assessing the

needs of stakeholders, a comparison of the different methods used

for needs assessment, indicates that the simplest and most

effective method of assessing the needs of the stakeholders is

simply to ‘talk’ (person-to-person, by telephone, or e-mail) to


each stakeholder about their feelings, opinions, preferences, and

the areas in which they feel they need the most work and

attention. (Johnson and Breckon, 2007; Hambrick, 1979; Darou,

1990; Allison, 1989; MindTools Ltd., 2010).

After soliciting ideas from team member stakeholders (Site

Coordinators and Project Instructors), the project manager will

integrate the ideas received from team member focus groups and

communications (meetings, meetings, meetings … as many as are

needed) into a tentative plan based upon a general consensus of

the team members and their backgrounds, knowledge- and skills-

bases and present them to the remaining stakeholders in the first

‘open’ discussion forum. This will be a (second) well-advertised,

short (one hour) “brainstorming” session, during which the

project manager and the “data team” will present the project, the

board’s reason for requesting it, and the tentative plans to all

stakeholders in a questionnaire asking for ‘ideas’ about when,

where, and how the P.E.P. Wellness Triage should be implemented.

Since the data team is only looking for ideas at this stage of


the planning process, this questionnaire can be presented in an

open-ended question format.

After collecting the data, analyzing it, and writing a ‘new’

plan based upon a synthesis of the stakeholders’ ideas that are

consistent with the project’s vision, the drafted plan will then

be discussed in a third forum discussion and revised as often and

as much as needed until an acceptable, workable synthesized model

is achieved. Additionally, it is important to have just enough

necessary supporting data to keep the plan realistic, but not so

much that it dulls the data team’s enthusiasm for the project.

(Johnson and Breckon, 2007).

In order to mitigate opposition to the project before it is

expressed, it is important to anticipate and accommodate

opposition to the program by identifying its source and analyzing

the motives of the opponents. At this stage of the project’s

planning process, the project manager must initiate strategies

for neutralizing resistance to the ‘change’ that the project

infers before it is expressed. To identify the forces that resist

the project’s changes, the project manager will conduct a fourth


one-hour focused discussion of the project’s drafted plan and the

most commonly identified areas of resistance: budget (or lack of

money), the staff, and the organizational culture. (Johnson and

Breckon, 2007).

Strategies to offset resistance to the project by powerful

individuals include: (1) anticipating the source of and nature of

the resistance; (2) honestly reassuring the stakeholders; (3)

explaining the need for and the logic of the project and the

logic of the plan (4) asking stakeholders to help design the

program (people who participate in creating the project will be

committed to implementing it); (5) giving key individuals and

powerful stakeholders desirable roles in the design and

implementation of the project; and (6) disbursing personal

advantages early on in the project to co-opt a potential source

of resistance (Johnson and Breckon, 2007). Generally speaking, in

pushing the project, managers should support the one-third of

stakeholders who advocate the program, focus on the one-third who

are open to the program but are uncertain of their feelings, and

address, in a limited way, the concerns of the one third who


resist the project. (Johnson and Breckon, 2007). By utilizing

these strategies, project managers can be assured that if the

“high- or low-power, high-interest” stakeholders are persuaded of

the benefits of the program, the resisters will usually ‘tag’

along. (Johnson and Breckon, 2007; MindTools ltd., 2010).

To address any anticipated resistance within the power

structures of the P.E.P. Wellness Triage, the project manager

will use a revolutionary (top-down) approach followed by planned

evolutionary (bottom-up) change. Since the type of change

proposed by this program is similar to a revolutionary change

than occurs when a new leadership assumes control, the project

manager will confront any anger, denial, or depression during the

“honeymoon period” (the first few weeks or months of the program)

when the stakeholders expect the change and are usually eager to

support the program and its team members. In using the planned

evolutionary change, the project manager will promulgate the

project’s vision and the steps needed to realize it by

articulating the project’s vision and keeping the vision and the


interim steps in front of everyone else. (Johnson and Breckon,

2007).

To assess the needs of the Faculty and Staff, their level of

“burnout,” and their level of job satisfaction with their jobs in

this low-income, urban environment, two reliable and valid

instruments and a demographic form to collect the data will be

used (Brewer and Clippard, 2002). The first instrument, the

Maslach Burnout Inventory (MBI), is a twenty-two-item inventory that

uses a six-point Likert scale to measure the three components of

burnout: emotional exhaustion, depersonalization, and personal

accomplishment) using a six-point Likert scale. The second

instrument is the Job Satisfaction Scale (JSS), a fourteen-item

inventory used to measure the three dimensions of job

satisfaction: intrinsic satisfaction (which measures intrinsic

qualities such as opportunity to help students, challenge derived

from the job, and feelings of professional success),

organizational satisfaction (which measures qualities such as

clarity or guidelines to perform job, recognition by supervisor,

and quality of supervision), and salary and promotion (which


measures satisfaction with salary and benefits and opportunity

for promotion (Brewer and Clippard, 2002). The demographic

variables that will be compared to burnout and job satisfaction

will be: current position, years in current position, school

district, years in school district, marital status, gender,

ethnicity, and age. In addition to looking at the three

components of burnout and their correlation to Faculty and Staff

total job satisfaction, the data team will also be looking to see

whether these variables vary by any of the demographic variables.

(Brewer and Clippard, 2002).

The MBI and the JSS will also be administered as quarterly

stakeholder ‘evaluations’ that will serve as part of formative

evaluations that include the development of a consensus of goals,

an assessment of client reactions to the services that have been

provided, a retrospective review of the processes used in

planning and implementing the project, the adequacy of the

schedule and its content, and any other problematic areas in

program implementation (Johnson and Breckon, 2007). In addition,

the most current set of data will be compared to the previous


quarter’s scores to create a summative evaluation of the

program’s efficiency at achieving one of the project’s three

goals (“decrease attrition”) and to meet the needs as indicated

by the “Faculty and Staff total job satisfaction” score (a key

indicator)..

One of the reasons that these inventories will adequately

assess the needs of the Faculty and Staff is because they are

relatively brief and can be administered at the end of one of the

regularly-scheduled monthly seminars or at the end of a weekly

class or seminar without infringing upon the stakeholders’ time

and energy. In addition to measuring the program’s effect on

Faculty and Staff, the data will indicate where funding should be

spent (i.e., Relaxation Therapy, Art Therapy, Cardio-Therapy),

which programs are more efficient, what improvements should be

made as the program develops, and how close the project has come

in meeting its goals.

In addition to encouraging students to participate in all

open decision-making discussion forums in the planning phase of

the project, the data team will assess Student stakeholder needs


by using a combination “demographic/perceptions form.” Variables

included on this form will include: enrollment, age, gender,

grade level, ethnicity, program services used, type of discipline

referrals, number of (physical/verbal) on-site bullying incidents

involved in during the last quarter as victim, number of

(physical/verbal) on-site bullying incidents involved in during

quarter as perpetrator; school, family, program, and self

satisfaction levels (using a Likert scale of 1 – 5, where 1 =

very dissatisfied, 2 = somewhat dissatisfied, 3 = neutral, 4 =

somewhat satisfied, and 5 = very satisfied), and an open-ended

question soliciting ideas for improvements.

Since all of the project’s outcomes (“Improve student

grades, “Decrease attrition,” and “Reduce on-site bullying and

criminal activity”) are directly related to students, Student

stakeholders’ needs will be assessed using multiple measures such

as the above demographic/ perceptions forms,

questionnaires/surveys, standardized test scores, and GPA letter

grade distributions. In addition, two other instruments will be

used to address problems of on-site bullying and criminal


activity: the Bullying-Behavior Scale (BBS) and the Multidimensional School

Anger Inventory (MSAI) (VINJ, 2007a). The BBS is a reliable

psychometric instrument consisting of six forced-choice items,

three of which refer to being the perpetrator of negative

physical actions (i.e., hit and pushed, picked on, bullied) and

three of which refer to being the perpetrator of negative verbal

actions (i.e., teased, name-calling, laughed at) (VINJ, 2007a).

The MSAI is a 36-item, Likert-type age-appropriate instrument

designed to measure affective, cognitive, and behavioral

components of anger among youths (VINJ, 2007b). Subtest and

scores include: anger experience, hostility, destructive

expression, and positive coping (VINJ, 2007b).

The data generated from the above measures and surveys will

be used to design an effective mentoring program that addresses

the “knowledge-to-action cycle” and identifies the gaps that

exist between Student knowledge and practice (Kitson and Straus,

2010). The multiple measures, the BBS, and the MSAI are logical

and inclusive for most Students in this stakeholder group because

they are relatively short (≤ 15 minutes), easy-to-understand,


easily administered, and manually-graded. When students realize

that the data from these surveys will be used to create programs

that address their academic, social, physical, and emotional

needs, they may be more inclined to cooperate with Faculty and

Staff during school hours and to participate in the project’s

assessment and evaluation process.

Finally, the needs of the Project Instructors will be

assessed from the initial planning phases of the project. As

noted above, Instructors have been communicating with other team

members since their initial employment by the school district.

Initially, Instructor needs will be assessed using questionnaires

and surveys indicating their preferences, opinions, and equipment

needs; focus groups and discussion forums; “gap analysis”

(Rossett, 1989) indicating any problems that stand between what

is happening and the project’s goals (what ‘should’ be

happening); and as many meetings, conversations, phone calls, and

e-mails as are necessary to ensure that all of their needs are

met. In addition, problem ranking questionnaires will be used

throughout the program (Hambric, 1979). A knowledgeable,


personable, satisfied Instructor will act as ‘ambassador’ for the

project and will ultimately convey his or her enthusiasm and

satisfaction with the program to the other stakeholders.

Evaluating Stakeholder Progress

Selection of a strategy for evaluation depends on whether

the goal is to measure the occurrence of a behavioral change or

to enhance ‘local knowledge’ (whether the project is working or

not in the context in which it was implemented). (Straus, Tetroe,

Graham, Swarenstein, et al., 2010). By anticipating the

stakeholders’ goals based upon the assessment of their needs, the

project manager and data team can plan a fitting method of

evaluating the project’s progress through the use of formative

and summative assessments.

Formative assessments are conducted in the early phases of

the project to create a feedback loop that will enable the data

team to make periodic adjustments to improve the progam (Johnson

and Breckon, 2007). Formative assessments include the needs

assessments of each group, the development of a consensus on

goals, assessments of stakeholder reactions to the services that


have been provided by the program, and quarterly retrospective

reviews of the processes used in planning and implementing the

program (Johnson and Breckon, 2007). For instance, each

stakeholder group (Faculty and Staff, Students, and Project

Instructors) can review together whether enough Instructors have

been employed by the project or whether the addition of more

Instructors would improve the quality and implementation of the

program. In addition, an evaluation will be made of the adequacy

of the class schedule, the appropriateness of class content, the

effectiveness of the project advertising, the effectiveness of

seminars, or any other problematic areas in the project’s

implementation from the perspective of all stakeholders. The

results of the formative study will be summarized in writing,

with conclusions and recommendations and distributed to all

stakeholders and the sponsoring organization (the school board).

The bottom line is that by analyzing the data to determine what

might have prevented the problems from occurring, the data team

will gather valuable information that will prove useful in the

future (Johnson and Breckon, 2007).


Summative evaluations will be conducted later in the

program’s implementation stage and will focus on whether the

goals and objectives of the PEP Wellness Triage have been met.

In addition, outcome evaluations will be used to show what

behaviors have been changed (i.e., on-site bullying, attrition,

student academic performance) and impact evaluations will be

conducted to report on the incidences of on-site bullying, crime,

or violence. (Johnson and Breckon, 2007). Questions that will be

asked on these evaluations include: How many Student stakeholders

have improved their GPA during the past quarter? How many

Student stakeholders have improved their scores on standardized

tests? How many Faculty and Staff stakeholders have decreased

their level of “burnout” during the past quarter? How many

Faculty and Staff have increased their total level of job

satisfaction during the past quarter? How many Faculty and Staff

have transferred or quit during the past quarter? How many

Student stakeholders have been involved in verbal/physical

(circle one) on-site bullying incidents as victim/perpetrator

(circle one) during the past quarter? How many Student


stakeholders have been involved in criminal activities (by type)

in the last quarter?

To obtain the above information, the data team will gather

information on average student GPA from the administrative office

or from the school board. Since standardized tests (i.e., CSAP

and/or SAT and ACT) are only administered once or twice a year,

results will be reported during each quarter, but will only be

gathered yearly. In addition, Student scores from quarterly

administrations of the BBS and the MSAI will be used to evaluate

whether the program has had an effect on student levels of

hostility, aggression, and violence.

In addition to answering the above questions on Faculty and

Staff, the data team will also evaluate Faculty and Staff scores

from the MBI and the JSS. Scores from the current quarter and

the previous quarter will be compared and the results will be

made readily accessible to all stakeholders, as well as to the

school board, to use in improving the program, which is the main

reason for conducting a process evaluation (Johnson and Breckon,

2007). However, because summative evaluations require a more


sophisticated research design and more time, effort and,

resources than do formative evaluation studies, the basic set of

evaluation question described above will help determine whether

the program objectives are being or have been accomplished and,

if so, by what percentage of the participants (Johnson and

Breckon, 2007). The idea is to quantify all of the findings from

the data so that they are meaningful to the sponsoring

organization and to the stakeholders. This necessitates the

identification of the key indicators that are directly attributed

to the goals of each group.

Key Indicators

As defined by the school board’s goals for this project, the

outcomes for this project are: (1) Improve Grades, (2) Decrease

Attrition, and (3) Reduce on-site bullying and criminal activity.

As they apply to each group, the outcome for Faculty and Staff

stakeholders is to decrease attrition that is attributable to

their high levels of “burnout; the outcomes for the Student

stakeholders are to improve grades, decrease attrition, and to

reduce on-site bullying; and the outcomes for Instructor


stakeholders are directly related to improving grades and

reducing on-site bullying and criminal activities for the

students and decreasing attrition for Faculty and Staff and for

Students. From this perspective, if Student stakeholders and

Faculty and Staff meet their goals, the Instructors will have

achieved their outcomes, as well. On a personal level,

Instructors will need to remain current with industry

certifications, guidelines, and C.E.C.’s so that they can deliver

quality information and instruction to the other stakeholders.

This will ensure that their growth is commensurate with the

growth of the program.

The key indicators for “Improve Grades” will be “average

student GPA” and “average standardized test scores.” The key

indicators for “Decreasing Attrition” among students will be

“average number of absences during evaluation period” and “number

of students who dropped out of school during the evaluation

period.” The key indicators for decreasing attrition among

Faculty and Staff will be “average number of absences during

evaluation period,” “number of Faculty and Staff quitting and/or


transferring during evaluation period,” “Faculty and Staff

burnout level” (from the MBI) and “Faculty and Staff job

satisfaction score” (from the JSS).

As defined by Srabstein and Levanthal (2010), “bullying” is

characterized by repeated exposure of one person to physical

and/or emotional aggression including teasing, name calling,

mockery, threats, harassment, taunting, hazing, social exclusion,

and/or rumors. These classifications will be useful in

classifying on-site bullying. The key indicators for decreasing

attrition and reducing on-site bullying and criminal activities

will be “number of absences during evaluation period,” “number of

students involved in on-site bullying as victims,” “number of

students involved in on-site bullying as perpetrators,” “number

of students involved in criminal activities (by type) as

perpetrator,” and “number of students involved in criminal

activities (by type) as victims.” Since all of these results have

little meaning unless they are quantified, these numbers will be

compared to the previous period’s numbers to reveal whether the

key indicators have increased or decreased during the evaluation


period. If average GPA and average standardized test scores have

risen, then the program has achieved its goals and the project is

successful. Similarly, if the other key indicators have

decreased, then the project is a success and has been effective

in achieving the project’s goals as defined in the vision and

mission statement of the project.

In the end, it is true that the quality of our project will

express itself through the assessment and evaluation the

viewpoints, values, needs, and progress of its stakeholders. How

else can we determine whether the project has met the needs of

its stakeholders? How can we determine whether our project was

successful in achieving it expressed outcomes? Stakeholder needs

assessments and progress evaluations are, indeed, the cornerstone

of project management. They are important strategic planning

processes that determine the effectiveness and longevity of every

successful project in every successful organization. If a project

is to be successful, then its managers should use stakeholder

needs assessments and progress evaluations to identify the

stakeholders in their projects, to win the support of all those


who have a stake in their projects, and to help ensure that their

projects succeed where others have failed. By examining the

stakeholder needs assessments and progress evaluations, we have

identified all those who have a vested “stake” in the outcome of

our project: members of the district school board, faculty and

staff, students, their parents, the surrounding community,

members of our project team, the local press, and even the world

community (the Internet and the national press). By involving

stakeholders at every stage of the project, from design to

implementation, we have taken a giant step in ensuring that our

project will successfully meet the needs of its stakeholders.

References

Brewer, E. W. & Clippard, L. F. (2002). Burnout and job

satisfaction among Student Support Services


personnel. Human Resource Development Quarterly, 13(2), 169-186.

(ProQuest Document ID: 128929461).

Curran, C., & Totten, M.. (2010). Mission, strategy, and

stakeholders. Nursing Economics, 28(2), 116-8. Retrieved from

ProQuest database. (Document ID: 2026180461).

Darou, W. G.. (1990, March). Training the people who help

troubled kids. Training and Development Journal, 44(3), 54.

Abstract retrieved from ProQuest database. (Document

ID: 817859).

Eaton, D. K., Kann, L., Kinchen, S., Ross, J., et al. (2006).

Youth Risk Behavior Surveillance-United States, 2005. The

Journal of School Health, 76(7), 353-72. Retrieved from ProQuest

database. (Document ID: 1141736811).

Hambrick, Ralph. (1979). Needs assessment: Alternative

approaches. Southern Review of Public Administration, 3(3), 360.

Abstract retrieved from ProQuest database. (Document

ID: 1318539).


Johnson, J. A. & Breckon, D. J. (2007). Managing health education and

promotion programs: Leadership skills of the 21st century (2nd ed.).

Sudbury, MA: Jones and Bartlett Publishers.

Kelly, H. (2010). Integrating planning, assessment, and

improvement in higher education [review of book Integrating

Planning, Assessment, and Improvement in Higher Education ]. Planning for

Higher Education, 38(4), 65-67. Retrieved from ProQuest

database. (Document ID: 2066512741).

Kitson, A., & Straus, S. (2010). The knowledge-to-action cycle:

identifying the gaps. Canadian Medical Association.

Journal, 182(2), E73-7. Retrieved from ProQuest database.

(Document ID: 1976388541).

MindTools Ltd. (2010). Stakeholder analysis: Winning support for your

projects. Retrieved from

http://www.mindtools.com/pages/article/newPPM_07.htm

Michigan Department of Education. (2009). A focus on evaluation: A

stakeholder evaluation handbook. Retrieved from

http://www.michigan.gov/mde/0,1607,7-140-54504_18668_18688-

59315--,00.html#append


Poraj, G.. (2010). Psychological models of female teachers’

functioning in their professional role. International Journal of

Occupational Medicine and Environmental Health, 23(1), 33-46.

Retrieved from ProQuest Health and Medical Complete.

(Document ID: 2028262971).

Rossett, Allison. (1989). Assess for success. Training and

Development Journal, 43(5), 55. Abstract retrieved from

ProQuest database. (Document ID: 817695).

Srabstein, J., & Leventhal, B. (2010). Prevention of bullying-

related morbidity and mortality: a call for public health

policies: World Health Organization. Bulletin of the World Health

Organization, 88(6), 403.

Straus, S., Tetroe, J., Graham, I., Zwarenstein, M., Bhattacharyy

a, O., & Shepperd, S. (2010). Monitoring use of knowledge

and evaluating outcomes. Canadian Medical Association.

Journal, 182(2), E94-8. Retrieved from ProQuest database.

(Document ID: 1976388581).

Teufel-Shone, N. I., Siyuja, T., Watahomigie, H. J., & Irwin,

S. (2006). Community-based participatory research:


Conducting a formative assessment of factors that influence

youth wellness in the Hualapai community. American Journal of

Public Health, 96(9), 1623-8.

The World Bank Group. (2001). Tools: Getting started: Identifying stakeholders.

Retrieved from

http://web.mit.edu/urbanupgrading/upgrading/issues-tools/too

ls/Ident-stakeholders.html

U.S. Department of Education (USDE). (2010). Needs Assessments.

Retrieved from www.dpi.state.nd.us/grants/needs.pdf

Violence Institute of New Jersey (VINJ). (2007a). Searchable

Inventory of Instruments: Assessing Violent Behavior and Related Constructs in

Children and Adolescents: Bullying-Behavior Scale. Retrieved from

http://vinst.umdnj.edu/VAID/TestReport.asp?Code=BBS

Violence Institute of New Jersey (VINJ). (2007b). Searchable

Inventory of Instruments: Assessing Violent Behavior and Related Constructs in

Children and Adolescents: Multidimensional School Anger Inventory.

Retrieved from http://vinst.umdnj.edu/VAID/TestReport.asp?

Code=MSAI

P.E.P. Wellness Triage: Stakeholder Evaluation

Documents

Transcript of P.E.P. Wellness Triage: Stakeholder Evaluation