PADM : Evaluation of Nonprofit Organizations Spring 2015 Instructor Course Location Course...

PADM : Evaluation of Nonprofit Organizations Spring 2015

Instructor

Daniela C. Schröter, Ph.D.

Director of Research

The Evaluation Center

Western Michigan University

Cell phone: 269‐267‐8227

Email: [email protected]

Skype: Daniela.Schroeter

Office hours: By appointment

Course Location

4735 Sangren Hall

Western Michigan University

Main Campus

Kalamazoo, MI

of 11

Course Description (WMU Catalog)

Examination of how nonprofit programs are assessed for need and evaluated as to their

operations and outcomes. The course includes discussion of the role and conduct of

research on the program evaluation process, performance expectations of multiple

stakeholders, as well as the methods of effective evaluation and analysis.

Credit and Course Hours

This 3 credit class meets from 6:00 PM to 9:00 PM on Tuesdays beginning January 13, 2015 and

ending April 7, 2015. No class is scheduled in the weeks of

February 10 due to instructor’s work travel (independent work on assignments is scheduled and one assignment is due)

March 10 due to spring break.

There is a possibility that class may be cancelled for March 3 due to scheduled work travel of the

instructor. Details will be provided well ahead of time. Course content will be available online

through E‐Learning and would be covered during the March 17 meeting. Assignments are due as

scheduled. If the class should be cancelled, group work is strongly encouraged.

Learning Objectives

By the end of this course you should be able to:

Accurately use key evaluation concepts and terminology.

Describe, distinguish among, and critically evaluate the usefulness and validity of selected evaluation models and approaches.

Design solid, mixed‐method evaluations under various conditions and constraints.

Critically evaluate evaluations (metaevaluation).

Throughout the course we will also collectively work on improving our communication and

interpersonal skills, which are vitally important in evaluation, especially:

Conveying constructive criticism in a professional and tactful manner

Facilitating discussion to achieve dialogue among all participants

Writing clearly and concisely for academic and nonacademic audiences

Making succinct and professional oral presentations

Teaching Philosophy

Cognitive science suggests that students learn most effectively when they actively construct the

meaning of material by articulating and applying the information. Consequently, this course

emphasizes active learning. In class, instructor‐led presentations will be integrated with small‐

and large‐group discussions, case studies, role‐play, and real‐world examples of evaluation.

Outside of class, students will need to read material to be applied in class, and complete

of 11

assigned homework with particular attention to writing and critical thinking and reflection.

Moreover, students are responsible for attending and actively participating in class.

Most class sessions will employ the following format:

Instructor’s summary of the main parts of the previous assignments; as appropriate, review and discussion of students’ performance or feedback; and overview of the current class session

Presentation and discussion of the assigned readings and projects

Question and answer session to clear up any areas of confusion and discuss any controversial issues.

In‐class application activities.

Overview of next assignment(s)

Academic Integrity

You are responsible for making yourself aware of and understanding WMU’s policies and

procedures regarding students’ rights and responsibilities:

http://catalog.wmich.edu/content.php?catoid=23&navoid=938

These policies include the code of honor, academic conduct, cheating, fabrication, falsification

and forgery, multiple submission, plagiarism, complicity and computer misuse, among others. If

there is reason to believe you have been involved in academic dishonesty, you will be referred

to the Office of Student Conduct. You will be given the opportunity to review the charge(s). If

you believe you are not responsible, you will have the opportunity for a hearing. You should

consult with your instructor if you are uncertain about an issue of academic honesty prior to the

submission of an assignment or test.

To access the WMU Code of Honor and general academic policies on such issues as diversity,

religious observance, student disabilities, to name but a few, please visit

http://www.wmich.edu/conduct/index.html

Need for Accommodation

To assure compliance with the Americans with Disabilities Act, faculty members at Western

Michigan University need to know how a disability will impact student participation and work in

courses. Any student registered with Disability Services for Students who would like to discuss

accommodations for this class should contact the instructor of record in a timely manner.

Students with documented disabilities who are not registered with DSS should call the office at

(269) 387‐2116 or visit wmich.edu/disabilityservices. Students cannot request academic

accommodations without scheduling an appointment and meeting with a DSS staff member. If a

student does not register with DSS, their academic accommodations/modifications cannot be

executed.

of 11

WMU Resources

WMU’s Libraries and the Writing Center provide important resources for graduate students

which you are encouraged to use as part of this course. You can access the libraries via your

GoWMU account. Many of the recommended readings are freely available through the library.

The writing center can support your writing endeavors and can be located at this site:

http://wmich.edu/casp/writingcenter/

E‐Learning

E‐Learning is accessible through your GoWMU account:

https://gowmu.wmich.edu/cp/home/displaylogin.

Course presentations and supplementary, required readings and tools will be made available

through E‐Learning. Students are encouraged to build discussion groups and use E‐Learning in

other ways that facilitate learning.

Textbooks and Readings

Required Textbook and Supplemental Readings and Tools

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program

evaluation: Alternative approaches and practical guidelines

(4th ed.). Upper Saddle River, NJ: Pearson Education, Inc.

Other required readings can be downloaded from the E‐Learning portal or Websites. Students

are encouraged to share materials and resources as part of the course through the e‐learning

system.

E‐Learning

Stufflebeam’s (2011) metaevaluation checklist and Excel workbook

Rubrics and tools developed during the course

Supplementary materials provided during the course

of 11

Resources on Websites

Better Evaluation: http://betterevaluation.org/

W. K. Kellogg Foundation: http://www.wkkf.org/resource‐directory#pp=10&p=2&q=Evaluation

FSG: http://www.fsg.org/

Collective Impact Forum: http://www.collectiveimpactforum.org/

Stanford Social Innovation Review: http://www.ssireview.org/

USAID Project Starter: http://usaidprojectstarter.org/

WMU Evaluation Center Checklists: http://www.wmich.edu/evalctr/checklists/checklistmenu.htm

The State of Evaluation: http://www.stateofevaluation.org/

Recommended Continued Readings

Alkin, M.C. (Ed., 2004). Evaluation Roots: Tracing Theorist’s Views and Influences. Thousand

Oaks, CA: Sage.

Burkhardt, J.T., Schröter, D.C., Magura, S., Means, S.N., Coryn, C.L.S. (2015). An Overview of

Evidence‐Based Program Registers (EBPRs) for Behavioral Health. Evaluation and

Program Planning, 48, 92‐99.

Chelimsky, E. (1997). The political environment of evaluation and what it means for the develop‐

ment of the field. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21st century:

A handbook (pp. 53‐68). Thousand Oaks, CA: Sage.

Christie, C. A. (2007). Reported influence of evaluation data on decision makers’ actions.

American Journal of Evaluation, 28(1), 8‐25.

Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C., (2011). A systematic review of

theory‐driven evaluation practice from 1990 to 2009. American Journal of Evaluation,

32(3), 199‐226.

Coryn, C. L. S., Schröter, D. C., & Hanssen, C. E. (2009). Adding a time‐series design element to

the Success Case Method to improve methodological rigor: An application for non‐profit

program evaluation. American Journal of Evaluation, 30(1), 80‐92.

Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization.

Review of Educational Research, 56(3), 331‐364.

Davidson, E.J. (2004). Evaluation methodology basics: The nuts and bolts of sound evaluation.

Newbury Park: Sage.

Dewey, J. D., Montrosse, B. E., Schröter, D. C., Sullins, C. D. & Mattox II, J. R. (2008). Evaluator

competencies: What’s taught versus what’s sought. American Journal of Evaluation,

29(3), 268‐287.

Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and

working logic. In D. M. Fournier (Ed.), Reasoning in evaluation: Inferential links and leaps

(pp. 15‐32). New Directions in Evaluation, 68. San Francisco, CA: Jossey‐Bass.

of 11

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation

standards (3rd ed.). Thousand Oaks, CA: Sage.

Matthison, S. (Ed.) (2005). Encyclopedia of Evaluation. Thousand Oaks, CA: Sage.

Mertens, D.M. & Wilson, A.T. (2012). Programevaluation theory and practice: A comprehensive

guide. New York: The Guilford Press.

Miller, R. L., & Campbell, R. (2006). Taking stock of empowerment evaluation: An empirical

review. American Journal of Evaluation, 27(3), 296‐319.

Morra Imas, L.G. & Rist, R. C. (2009). The road to results: Designing and conducting effective

development evaluations. World Bank. (Available for free online!)

Newman, D. L., Scheirer, M. A., Shadish, W. R., & Wye, C. (1995). Guiding principles for

evaluators. In W. R. Shadish, D. L. Newman, M. A. Scheirer, & C. Wye (Eds.), Guiding

principles for evaluators (pp. 19‐26). New Directions in Evaluation, 66. San Francisco, CA:

Jossey‐Bass.

Patton, M. Q. (2010). Developmental Evaluation. Applying Complexity Concepts to Enhance

Innovation and Use. Guilford Press, New York.

Patton M. Q. (1997). Utilization‐Focused Evaluation: The New Century Text. Thousand Oaks, CA:

Sage Publications.

Picciotto, R. (2003). International trends and development evaluation: The need for ideas.


Posavac, E.J. and Carey, R.G. (2003) Program Evaluation: Methods and Case Studies, 6th

Edition. Englewood Cliffs, NJ: Prentice‐Hall.

Shulha, L. M., & Cousins, J. B. (1997). Evaluation use: Theory, research, and practice since 1986.


Scriven, M. (1991). Evaluation thesaurus (4th ed.). Thousand Oaks, CA: Sage.

Scriven, M. (1994a). The final synthesis. Evaluation Practice, 15(3), 367‐382.

Scriven, M. (2007). Key evaluation checklist (KEC). Kalamazoo, MI: Western Michigan University,

The Evaluation Center.

Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for

program evaluators. American Journal of Evaluation, 26(1), 43‐59.

Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 22(2),

183‐209.

Stake, R. E. (2004) Standards‐Based and Responsive Evaluation. Thousand Oaks, CA: Sage

Publications.

Tourmen, C. (2009). Evaluators’ decision making. American Journal of Evaluation, 30(1), 7‐30.

of 11

Course Components

Course components have been developed with a portfolio in mind and include:

1. Critical reflections (20%) 2. Metaevaluation (30%) 3. Evaluation Plan (40%) 4. 360˚ Evaluation (10%)

Assignment policy

Assignments shall be submitted via e‐mail to [email protected] by 5 PM on the day

they are due. For each day an assignment is late, 10% will be deducted from your grade for that

assignment.

Critical Reflections (20%) – Individual Assignment

In preparation for the five class sessions, beginning January 20 and ending February 17, develop

brief (250 – 500 words) critical reflections that are based on the assigned readings and linked to

your professional or academic experiences. You may use the book’s discussion questions and

application exercises to develop ideas for this assignment. Moreover, these reflections can be

used in developing your group evaluation proposal and/or your individual metaevaluation

project. Each of these assignments is worth 4 percent of your total grade. To obtain all points

the assignment should

Consist of 250 – 500 words each

Be accurate (punctuation, grammar, spelling, etc.)

Refer to credible resources and use APA guidelines for citations (see http://www.apastyle.org/)

Be relevant to the course content

Consider work or experience from outside the course

Be delivered by 5pm the day of the class they are assigned for

Metaevaluation (30%) – Individual Assignment

The metaevaluation consists of an evaluation of a report from a nonprofit program evaluation

that you choose. To support your metaevaluation project, you will be provided with a checklist

and workbook developed by Stufflebeam (2011). This tool is based on the Joint Committee on

Standards for Educational Evaluation’s (2011) “Program Evaluation Standards.” These standards

have been adapted throughout the world and are widely used in program evaluation regardless

of disciplinary context and content (i.e., not just educational evaluation).

Criteria, indicators, and standards for this assignment will be developed as part of the class in

form of a rubric. Ultimately, your metaevaluation report is worth 20% of your final grade and

your final metaevaluation presentation is worth 10% of your final grade.

of 11

Evaluation Plan (40%) – Group Assignment

Groups of students are asked to develop an evaluation proposal based on publicly available

materials. Evaluation plans may be developed based on an evaluation RFP or a program of

interest to you. Draft sections of this plan are assigned throughout the course. As a result,

students will have an opportunity to obtain critical feedback from their peers and the instructor

throughout the course. At the end of several class sessions, students will receive 20 – 30

minutes of time to discuss their work, ask questions, and share their progress. Larger sections of

the plan have to be handed in to the instructor at the assigned days for formative feedback. As a

result, you will have an opportunity to improve the plan before its final delivery.

The final product should integrate the key readings of the class. Specifically the evaluation plan

should include an evaluation background and context section, an evaluation design and

methods section, and a management plan. Final products will be graded as follows:

1. Evaluation background and context (10%) 2. Evaluation design and methods (10%) 3. Evaluation management plan (10%) 4. Final presentation (10%)

Evaluation proposals should be:

Feasible under consideration of proposed budgets and time lines

Doable: good proposal are practical and can be done

Clear and concise: Well written and to the point

Well‐structured/organized

Complete: include all key elements discussed in class

360˚ Evaluation Report (10%) – Peer Impact Assignment

The 360˚ Evaluation is intended to provide feedback about each student and your group

experience overall. Each student will write a paper, including

A project description (Introduction): The same for all group members.

A method section, bringing to bear the logic of evaluation. In essence, identify criteria and standards, discuss how you measured performance and provide a synthesis of criteria with facts into evaluative conclusions. The same for all group members. In this section you will identify how you will connect individual students’ performance with a recommended percentage for the assignment.

A findings section, including

An assessment of your own performance (Self‐assessment: ≈ 1‐2 pages). What percentage would you give yourself? Why?

An assessment of each peer within the group (Peer assessment: ≈ ½ ‐1 page per peer), including allocation of percentage points attributed to each student (i.e., 0=no credit to 10=full credit). What percentage would you give each group member? Why?

of 11

An assessment of the group work overall (Synthesis and conclusions: ≈ 1‐2 pages). How would you rate your group’s performance overall? Why?

A concluding statement which may include recommendations for future uses of group assignments of this type.

The page numbers provided in the brackets are rough estimates to serve as guidelines. If you

feel that more or less information is necessary, you are welcome to do so. Please keep in mind

to be concise and to the point.

Grading

Component Percentage Scale

Critical Reflections

5 reflections, 4% each 20%

≥ 95% = A 89% < 95% = BA 83% < 89% = B 77% < 83% = CB 71% < 77% = C 65% < 71% = DC 59% < 65% = D < 59% = E

Metaevaluation

20% Report 10% Presentation

30%

Evaluation Plan

10% Background & Context

10% Evaluation Design

10% Management Plan

10% Presentation

40%

360⁰ Evaluation 10%

of 11

Course Schedule

Date Topics Readings In‐class Activity Assignment due dates

Individual Group

Jan. 13 Course overview and introductions

The state of evaluation in nonprofit organizations

Assessment of student needs

Paper towel exercise

Jan. 20 Introduction to evaluation FSW Book Ch. 1 – 3

Stufflebeam’s metaevaluation checklist

Group building for evaluation plan

Evaluation report for metaevaluation

Critical reflection 1

Jan. 27 Evaluation models and approaches I

FSW Book Ch. 4 – 6

WKKF Logic Model Guide

Logic modeling


Feb. 3 Evaluation models and approaches II

FSW Book Ch. 7 – 9 Mini group presentations to introduce the case for the evaluation plan


Draft logic model for evaluation plan

Feb. 10 NO CLASS: Independent work time

Work on evaluation plan or metaevaluation


Feb. 17 Summary of models and approaches

Practical guidelines I. Background and context


Work on evaluation plan Critical reflection 5

Feb. 24 Practical guidelines II. Framing and managing evaluations

FSW Book Ch. 13 – 14 Interim feedback

Work on evaluation plan

Draft background and context for evaluation plan

of 11

Date Topics Readings In‐class Activity Assignment due dates

Individual Group

March 3

Practical guidelines III. Data interpretation in evaluation

(Instructor might be in Slovenia for project work. If so, students will need to work on their assignments.)


Discussion of interim feedback and means for midterm course improvement


Draft management plan

March 10 Spring Break

March 17 Practical guidelines IV. Evaluation reporting and use

Future trends


Review and critique of rubrics for metaevaluation, evaluation plan, and presentations


Draft metaevaluation

Draft methods section for the evaluation plan

March 24 Peer review and applications Independent reading

Q & A on metaevaluation and 360⁰ report


Draft 360⁰ evaluation report

March 31 Presentation of final evaluation plans

Independent reading

Student group presentations of evaluation plans

Final evaluation plan

April 7 Final presentations of metaevaluations

Independent reading

Individual presentations of metaevaluation and peer feedback

Final metaevaluation

Final 360⁰ evaluation report

FSW = Fitzpatrick, Sanders, & Worthen

PADM : Evaluation of Nonprofit Organizations Spring 2015 Instructor Course Location Course...

Documents

Transcript of PADM : Evaluation of Nonprofit Organizations Spring 2015 Instructor Course Location Course...