Capacity Building in Higher Education: A Small Step Toward an Assessment Habit of Mind

Assessment Office Crawford Hall 230 & 231 2550 Campus Road, Honolulu, HI 96822

(808) 956-4283 (808) 956-6669

manoa.hawaii.edu/assessment

of 25

Presentation at the American Evaluation Association, November 11-14, 2015, Chicago, IL.

Slide 1

Capacity Building in Higher Education: A Small Step Toward an Assessment Habit of Mind

Monica Stitt-BerghUniversity of Hawai‘i at Mānoa

Capacity Building in Higher Education: A Small Step Toward an Assessment Habit of Mind by Monica Stitt-Bergh, University of Hawai’i at Mānoa Abstract: Program outcomes assessment of learning is mandated in higher education by regional accreditors. The presenter started assessment capacity building at a research university in 2008. This study investigated the extent to which five years of capacity building efforts were successful in achieving capacity building’s goals of integrating assessment so it becomes sustainable and a habit of mind. Faculty members (n=164 from 73 subject areas) who served as degree program ‘assessment coordinators’ completed a survey based on the Evaluation Capacity Assessment Instrument (Taylor-Ritzler, et al., 2013). The presenter will highlight results such as the relationships between faculty thoughts about assessment, assessment knowledge and skill, and use of assessment results. Mediating factors such as years of involvement and attendance at capacity building events will be included. Attendees will leave with an understanding of the necessary, but limited, role of capacity building on faculty engagement with program assessment.

University of Hawai‘i at Mānoa Crawford 230 & 231 Assessment Office manoa.hawaii.edu/assessment

of 25 Monica Stitt-Bergh

Slide 2

Today’s session

1. Context: rationale for capacity building approach2. Capacity building efforts3. Evaluation of efforts4. Results5. Use of findings

Slide 3

B+B- A- AC+

Context

Past: an individual’s outcome

(e.g., course grade)

Current:the program’s learning

outcomes

89% of students met learning outcome #1

In higher education, the traditional indicator of student learning was the individual student’s course grade assigned by the instructor. While that practice continues, colleges and universities must now also provide student learning information at the program level. A program could be a degree program like a bachelor’s degree in chemistry, a collection of courses such as general education courses, or a co-curricular program such as intermural sports.



Slide 4

Context

Current:the program’s learning outcomes

89% of students met learning outcome #1

Findings used for programimprovement or evolution

Current:

In addition to programs having a set of learning outcomes, after finding out the extent to which students achieved the outcomes, the programs needed to demonstrate that they used the findings for program improvement or evolution. Now this sounds pretty simple . . . but it wasn’t. Slide 5

Context

A s s e s s m e n t !!!!!! I’m already over-burdened.

On the surface, the current program assessment mandate does not sound too hard. It asks instructors in a degree program to be clear and explicit about what students should learn, how many learned by graduation, and then what faculty did about it. But in reality, faculty in general in higher education reacted negatively. Two prevailing responses were fear and feeling like this poor donkey here, clearly overworked.



Slide 6

Context

Academic sub-cultures

Academic freedomLess funding

Rituals

Other important factors in higher education that have an effect on assessment practices: • Academic subcultures • Less funding • Rituals • Academic freedom Slide 7

How to overcome barriers and stay true to the culture on campus and in programs?

The question became, how to overcome the barriers to program learning assessment and be true to the culture of the campus?



Slide 8

Evaluation capacity building!

or assessment capacity building in the jargon of higher education

My solution was to start evaluation capacity building (ECB), or in the jargon of higher education, “assessment capacity building.” Labin, Duffy, Meyers, Wandersman, & Lesesne (2012) define ECB as “an intentional process to increase individual motivation, knowledge, and skills, and to enhance a group or organization’s ability to conduct or use evaluation” (p. 308). ECB’s goal is to integrate evaluation practice into program operations and habits of mind so it becomes sustainable (Preskill & Boyle, 2008). ECB typically involves evaluation experts facilitating learning experiences such as training and mentoring as well as written materials and technology. It often takes a participatory approach through which an expert collaborates with program personnel on evaluation activities. By bringing personnel together to participate in evaluation activities and reflect on program implementation, ECB is designed to increase program- and organizational learning. The ECB literature indicates that ECB can result in increased knowledge about evaluation, improved skills related to conducting evaluation, and positive beliefs about evaluation. Evaluation capacity building is suited to higher education because faculty are involved in the assessment which decreases their fear by increasing their understanding. They create assessment plans that fit their academic culture, are manageable, and do not infringe on academic freedom.



Slide 9

Capacity building efforts

On our campus, we have an assessment office with two full time faculty members, myself and a colleague, and a graduate student for 20 hours per week. I’m an educational psychologist with some program evaluation in my background. My colleague is a specialist in second language acquisition, testing, and measurement. Together, we run a variety of assessment capacity building activities: • Workshops • Technical consultations • Meeting facilitation • Events like our annual assessment poster exhibit • An assessment leadership institute in which a cohort of 12 faculty attend a 3-day training,

monthly support group meetings, present at the poster session, and then help us offer capacity building activities. This follows a “community of practice” model.



Slide 10

Some numbers: capacity building, 2008-2014

• 200+ technical consultations• 94 workshops• 5 visiting experts’ workshops• 4 poster exhibits• 4 deans and chairs workshops• 2 leadership institutes

500+ faculty (30%)

“touched”

This is a snapshot of capacity building efforts between 2008 and 2014—over 500 faculty (30% of all faculty) participated in at least one of these. We continue to do all of these.



Slide 11

Success looks like . . . 1. Faculty have positive thoughts about assessment2. Faculty motivated to engage3. Faculty knowledge & skills4. Supportive department climate5. Department resources6. Use of findings

Labin, Duffy, Meyers, Wandersman, & Lesesne (2012)Preskill & Boyle (2008)Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry & Balcazar (2013)

If we were successful in our capacity building efforts, I should be able to find evidence that faculty • Had positive thoughts about assessment • Were motivated to engage in assessment activities • Had the knowledge and skills to conduct or use evaluation • Had a supportive department climate toward assessment • Had resources available • Used findings for program decision making.



Slide 12

Evaluation of our efforts

(A) Workshop & consultation surveysplus (B) a comprehensive survey (C) Annual assessment reports

I looked at three main sources of evidence: • survey results—participants completed surveys immediately following workshops, etc.,

and • in 2014, I asked the lead assessment coordinators from each degree program to

complete a comprehensive survey that was based on the Evaluation Capacity Assessment Instrument by Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar (2013).

• annual assessment reports from 2008 to 2014.



Slide 13

Results

Slide 14

Workshop/consultation survey results

Workshops (n=1,069)95% - useful/very useful89% - effective/very effective in increasing knowledge

85% - answered quiz questions correctly

Consultations (n=161)99% - worthwhile 98% - satisfied/very satisfied with quality of information received

The results from the surveys given immediately after the workshop or consultation showed the ECB activity was highly successful. Attendees perceived them as useful, effective in increasing their knowledge, worthwhile, and high quality of information. The workshop surveys included quiz questions to test knowledge: 85% correct answers.



Slide 15

Comprehensive survey

Thoughts About Assessment*

Motivation*

Knowledge & Skills†

Department Climate*

Department Resources*

Use of Findings†

*Strongly disagree

*Somewhat disagree

*Somewhat agree

*Stronglyagree

† Not at all † To some extent

† To a considerable extent

† To a very great extent

N=161-164 (62%)

The comprehensive survey was administered one time in 2014. The 74-item survey based on Taylor-Ritzler et al.’s survey had 6 sections plus background questions. The 6 dimensions were as follows: • Thoughts about assessment (2.2 mean score) • Motivation to engage (2.1 mean score) • Knowledge and skills (1.8 mean score) • Department climate (1.8 mean score) • Department resources (1.1 mean score) • Use of findings (1.0 mean score) This figure shows the average for each section. The dimensions with an asterisk and blue bars were on an agreement scale. The dimensions with a cross and green bars were on a “not at all” to “very great” scale. 164 faculty who had served as their program’s “assessment coordinator” completed the survey (62% response rate).

• The good news was that they had somewhat positive thoughts about assessment and motivation to engage in assessment. This was very good news given the amount of vocal resistance to assessment by some faculty in 2008 when the assessment office started.

• The faculty also perceived they had decent, but not stellar, knowledge and skills: as a group, they fell just shy of considerable extent of knowledge.

• Department climate was also reasonably positive with the group of respondents somewhat agreeing that the environment in the department was conducive to assessment.

• The two areas that these results showed need great attention were (1) department resources and (2) use of findings.



Slide 16

Annual assessment report analysis

Program activity

Percent of programs

2008 (n=224) 2014 (n=238) % increase

Established measurable learning outcomes

77% 98% +21%

Published learning outcomes for stakeholders

60% 96% +36%

Aligned curriculum and learning outcomes (curriculum map)

41% 87% +46%

The last main source of evidence was the programs’ annual assessment reports. I compared the annual assessment reports from 2008 to 2014. There was a dramatic increase in the number of programs that reported measurable learning outcomes, published outcomes, and aligned curriculum with the learning outcomes via a curriculum map. This was all very good news.



Slide 17

Annual assessment report analysis

Program activity

Percent of programs

2011 (n=230) 2014 (n=238) % increase

Collected evidence 65% 75% +10%

Reported findings 59% 66% +7%

Described use of findings 51% 66% +15%

This slide compares 2011, not 2008, to 2014 because we made a change to the report template that made an accurate comparison to 2008 difficult. Here we see an increase but not as much as the previous slide (learning outcomes and curriculum alignment). The total number of programs that collected evidence, reporting and used findings in 2014 is still below what I would like to see after 5 years of capacity building. The analysis of the annual reports corroborates the findings from the comprehensive survey, especially in the area of use of findings, which was low on both.



Slide 18

Relationships. For example:

Level of engagement Knowledge & skills?

Attended ECB sessions Use of findings?

To further explore the effect of capacity building (ECB), I looked at the relationships between things like faculty reported level of engagement and perceived knowledge and skills and attendance at ECB sessions and use of findings.



Slide 19

One strong relationship

Level of assessment experience

Knowledge and skillsStrong, positiver = .76 p<.01

Only one relationship was strong: faculty level of assessment experience and reported knowledge and skills. Response options for level of experience: "No Experience" "Very Little" "Some" "Quite A Bit" "A Great Deal“ Knowledge and skills: set of questions on the comprehensive survey



Slide 20

Three moderate, positive relationships

Percent of work related to assessment Use of findings

rs = .52, .51, .49, p<.01

Frequency of assessment discussions with colleagues Department climate

Attend non-campus ECB events Knowledge and skills

Three relationships were moderate in strength and positive. Faculty who had frequent assessment discussions with colleagues reported a more supportive department climate. Those who attended evaluation capacity building events off campus – at a mainland conference for example – reported higher levels of knowledge and skills. And the faculty whose percent of work related to assessment was high reported higher use of findings.



Slide 21

Conclusion

We’ve made progress, but we’re not there yet

We have made progress but we are not there yet. Slide 22

Success in . . .• Attitude• Motivation• Knowledge and skills• Department climate

I’ve personally noticed a difference in faculty attitudes, motivation, and their knowledge and skills related to learning outcomes assessment and these results support my observations. I have less push back, faculty volunteer to help with assessment projects, and we do not need to offer as many beginner workshops. These are individual outcomes; only one organizational/program outcome was achieved: department climate.



Slide 23

Mixed success . . .• Department resources• Use of findings

Despite the successes, these two important areas fall short of expectations The lack of use of findings is of most concern because the stated purpose of assessment is for program improvement or program evolution. If findings are not used, the overarching goal is not achieved.



Slide 24

1. Build capacity to discuss assessment

Sessions on Effective department meetings Hallway conversations, etc.

Use of these findings

Here is how we are using the findings from this study of our capacity building efforts. Slide 25

2. Increase faculty-to-faculty interaction Success stories Faculty panel sessions Faculty give mini-workshops Faculty visit other departments


Faculty with mid to high levels of assessment experience can help build capacity and become part of strategy.



Slide 26

3. Encourage attendance at non-campus events Email list of conference opportunities


Slide 27

4. Resources for departments Provide model for infrastructureOffer logistic support Technology solutions




Slide 28

5. Build capacity to use results (!) Success stories Scenario-based workshops List ways to use results Public recognition/awards


Slide 29

My worry

Lack of use will lead to assessment as compliance, not assessment for improvement & evolution.

Lack of use prevents a habit of mind and decreases engagement and motivation.

My worry about the lack of use is real. I think that lack of use will lead to assessment becoming a compliance issue—check the box and submit the report—and then assessment will not be for program improvement or program evolution. The lack of use will prevent assessment from becoming a habit of mind and will, over time, result in lower engagement and lower motivation. We include utilization-focused strategies in our capacity building efforts, but we need more than that. Faculty have to have an experience that shows the value of assessment and an experience that demonstrates that it’s worth the effort to use the results. If I can help them experience the value of assessment, I’ll be another step closer to assessment as sustainable, a habit of mind, and beneficial to them and their students.



Slide 30

Faculty have a positive, successful experience in outcomes assessment – that will encourage assessment as sustainable, a habit of mind, and beneficial to faculty and students.

My additional goal

We have the technical training, fun events, consultations, we include utilization-focused strategies in our capacity building efforts, but the findings here show me that we need more. My additional goal is for faculty to experience the value of assessment so they feel it is worth the effort to use the results. If I can help them experience the value of assessment, I’ll be another step closer to assessment as sustainable, a habit of mind, and beneficial to them and their students.



Slide 31

Capacity Building in Higher Education:

A Small Step Toward an Assessment Habit of MindA Necessary

^

After reflecting on the results, I think this title is more appropriate because tremendous progress has been made, faculty response has been positive, and without opportunities to learn easily available on campus, I don’t think our campus would be where it is today in terms of program and institutional learning assessment.



Slide 32

Useful SourcesBourgeois, I. (2008). Understanding the dimensions of organizational evaluation capacity. (Doctoral

dissertation). Retrieved from ProQuest Dissertations & Theses Global. (Order No. NR48389).Garcia-Iriarte, E., Suarez-Balcazar, Y., Taylor-Ritzler, T. & Luna, M. (2011). A catalyst-for-change approach to

evaluation capacity building. American Journal of Evaluation, 32(2), 168–182.Huffman, D., Thomas, K. & Lawrenz, F. (2008). A collaborative immersion approach to evaluation capacity

building. American Journal of Evaluation, 29(3), 358–368.King, J. A. & Volkov, B. (2005). A framework for building evaluation capacity based on the experiences of three

organizations. CURA Reporter, 35(3), 10–16.Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A. & Lesesne, C. A. (2012). A research synthesis of the

evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338.Leviton, L. C. (2014). Some underexamined aspects of evaluation capacity building. American Journal of

Evaluation, 35(1), 90–94.Preskill, H. & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of

Evaluation, 29(4), 443–459.Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B. & Balcazar, F. E. (2013). Understanding and

measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34(2), 190–206.

Here are useful sources. This study will also be published in a volume of New Directions for Evaluation. I can also send you a copy of the article if you leave your contact information. Bourgeois, I. (2008). Understanding the dimensions of organizational evaluation capacity.

(Doctoral dissertation). Retrieved from ProQuest Dissertations & Theses Global. (Order No. NR48389).

Garcia-Iriarte, E., Suarez-Balcazar, Y., Taylor-Ritzler, T. & Luna, M. (2011). A catalyst-for-change approach to evaluation capacity building. American Journal of Evaluation, 32(2), 168–182.

Huffman, D., Thomas, K. & Lawrenz, F. (2008). A collaborative immersion approach to evaluation capacity building. American Journal of Evaluation, 29(3), 358–368.

King, J. A. & Volkov, B. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10–16.

Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A. & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338.

Leviton, L. C. (2014). Some underexamined aspects of evaluation capacity building. American Journal of Evaluation, 35(1), 90–94.

Preskill, H. & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459.

Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B. & Balcazar, F. E. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34(2), 190–206.



Slide 33

Mahalo (thank you)

Monica [email protected]://www.manoa.hawaii.edu/assessment

Capacity Building in Higher Education: A Small Step Toward an Assessment Habit of Mind

Documents

Transcript of Capacity Building in Higher Education: A Small Step Toward an Assessment Habit of Mind