STUDENT LEARNING:STUDENT LEARNING:

its value and util

isation

its value and util

isation

as indicator ofas indicator of

INSTIT EFFECTIVENE

SSINSTIT EF

FECTIVENESS

Southern African A

ssoc Southern

African Assoc

of Institutional of Institutional

ResearchResearch

BEST PAPER 2009/10BEST PAPER 2009/10

ACKNOWLEDGEMENTACKNOWLEDGEMENTSome of my work is informed by the very inventive

ideas of the following HE & IR expertsHE & IR experts:Banta, Trudy (Indiana Purdue Indianapolis)

Boehmer, Bob (University of Georgia)Chambers, Stephen (Coconino Community College)

Gerek, Mary Louise (Nazareth College of Rochester)Hamilton, Sharon & Evenbeck, Scott (Indiana Purdue

Indianapolis)Knight, Bill (Bowling Green State University)

Lanesskog, Stig (University of Illinois)Leimer, Tina (California State Univ at Fresno)

Manning, Terri & Wells, Denise (Central Piedmont Comm College)Ronco, Sharron (Florida Atlantic Univ)

Subotzky, George (University of South Africa)Volkwein, Fred (Penn State)

Voorhees, Rick (Voorhees Group LLC)Ward, Teresa & Katz, Beth (Georgia State)

I stand

on the

I stand

on the

shoulde

rs of

GIANTSGIANTS…

…

UNIVERSITY OF JOHANNESBURGUNIVERSITY OF JOHANNESBURGAuckland Park Kingsway CampusAuckland Park Kingsway Campus

Established in January 2005 as a Established in January 2005 as a merger between two universities and a merger between two universities and a

technikontechnikon

Soweto campus of UJSoweto campus of UJ

49,80049,800 studentsstudents44 campuses campuses99 faculties faculties

And would you believe…And would you believe…SNOW in SNOW in AfricaAfrica?? Auckland Park Kingsway ?? Auckland Park Kingsway

campus…!!campus…!!

RESEARCH HYPOTHESE

S

1.1. The typical The typical Southe

rn African view of

Southern African v

iew of

IRIR seems “narrower

” than the

seems “narrower”

than the

international IR d

efinition and shou

ld

international IR d

efinition and shou

ld

be reviewed.be reviewed.

2.2. There’s a There’s a gradual

shiftgradual shift from

a from a

performance-based

towards an academi

c

performance-based

towards an academi

c

(less mechanistic)

notion of

(less mechanistic)

notion of

institutional effe

ctiveness, caused

by

institutional effe

ctiveness, caused

by

questions about th

e nature & extent

questions about th

e nature & extent

of

student learning.student learning.

PURPOSEPURPOSE

To reflect on and

to try

and “test” the two

hypotheses

INST EFFECTIVENESSINST EFFECTIVENESS IN IN HIGHER EDUCATION?HIGHER EDUCATION?

SystematicSystematic, ongoing process ofcollecting and analysing valid

informationon extent of progress made towards

mission, goal or outcomes attainmentof an appropriate HE entityentity

and using the results/findingsfor continuous improvementcontinuous improvement purposes.

ESSENTIAL IE ELEMENTSESSENTIAL IE ELEMENTS(Sharron Ronco, 2006)

Linked to: Mission, goals, outcomes, values, etc

Requires: Clear expectationsexpectations

EvidenceEvidence-based

Measures what is meaningful and measurablemeasurable

Directed toward improvementimprovement

Can be “translated”“translated” for all stakeholders

Drives decision-makingDrives decision-making and priority setting

IE CYCLE/LOOPIE CYCLE/LOOP??ENTITYENTITY

Programme, Unit, School, College etc.

ASSESSMENTASSESSMENT Data/Info wrt

mission/goals/outcomes attainmentANALYSISANALYSIS INTERPRETATIONINTERPRETATION

ACTIONSACTIONS Review and Improve

Single, doub

le, triple,

spiral …even

“fruit”

looploop

IE ROLES & FEATURES Connects & aligns: PlanningPlanning & & ResourcingResourcing UGA IE cycle

No common setcommon set of indicators to measure IE

Mission/Goals/Outcomes: Shown to be served

Attainment (or lack thereof) communicatedcommunicated (dashboards)

IE cultureculture of assessment and benchmarkingbenchmarking Illinois

QualityQuality supplements as efficiencyefficiency indicator

(Jacobs & Grobbelaar, SAAIR Forum 2007)

Change in IE focus

IE roles & features

IE roles & features

IE features

IE rolesIE roles

CHANGE IN IE FOCUS?IE focus IE focus used to beused to be on I-P-O:

INPUTSINPUTS: Students, Faculty, Characteristics; financial & physical resources, etc AND

PROCESSESPROCESSES:: Curriculum approval, program planning, self-evaluation reporting, etc

OUTPUTSOUTPUTS:: Employability & employers’ views; Retention & throughput; Dashboard KPIs, etc IE more performanceperformance (bonus)

based?

...increasingly shifted towards OutcomesOutcomes

DEFINING

INSTITUTIONAL

RESEARCH (IRIR)

IRs APPRAISE & SERVE HE(Sidney Suslow, 1972: A (Sidney Suslow, 1972: A

declaration on IR)declaration on IR)“The institutional researcher’s basic role

encompasses the systematic appraisalappraisal of the HE effort. The institutional

researcher servesserves HE and, in turn, his/her institution through critical

appraisal appraisal and careful investigationinvestigation of its programmes and processes…”

DOES IR & IE CONNECT?DOES IR & IE CONNECT? IR: SupportingSupporting & catalyzingcatalyzing IE

IR findings improvementimprovement (eventually…)(eventually…)

Subtle, positive relationshippositive relationship: IR findings and IE actions (should be…)(should be…)

IRIR portfolios varyportfolios vary - across HE sectors

LET’S DEFINE IRLET’S DEFINE IRThe The classicalclassical view view

(Saupe, 1990)(Saupe, 1990)“…research conducted within an institution of higher education to

provide information, which information, which supportssupports institutional planning, policy formation and decision

making…”

BROADER IR NOTIONBROADER IR NOTION(Mary Louise Gerek, (Mary Louise Gerek,

2007)2007)

“…the sum total of allall activities directed at describing the full spectrum of functions (educational, administrative and support) occurring within a college or university. IR activities examine those functions in their broadest definitions, and embrace data

collection and analytical strategies in in support of decision-makingsupport of decision-making…”

FIVE IR FACESFIVE IR FACES (BROADER?)(BROADER?)(Volkwein, Fred: 2005)(Volkwein, Fred: 2005)

The focus is: IMPROVEMENTIMPROVEMENT

The focus is: ACCOUNTABILITYACCOUNTABILITY

ExecutiveExecutive & Admin& AdminAcademicAcademic

& & ProfesProfesData,Data,

informatiinformati

onon&&

technologytechnology

To collect and transform data into institutional knowledge.

IR as Knowledge Manager

To describe the institution:

IR as info authorityinfo authority

To present the best case:

IR as “spin doctor”“spin doctor”To analyse alternatives:

IR as policy analystpolicy analyst

Impartial evidence of effectiveness:

IR as scholar & researcher

DO IRs KNOW WHAT THEY DO IRs KNOW WHAT THEY AREARE OR OR SHOULD BESHOULD BE DOING? DOING?

“If we knew what it was we were doing, it would notwould not be called

researchresearch, would it? ” (Bill Knight, Bowling Green State

University)

Let’s recall real and underlying

IR PURPOSESIR PURPOSES…

Do Do IR PURPOSESIR PURPOSES shed light? shed light? Identifying obstaclesobstacles to achieving goals & efficiencies (Subotzky, 2008)

Rendering the institution intelligiblentelligible to itselfitself (Baijnath, 2008)

Resolving tensiontension: “Bureaucrats” vs “Professionals” TensionTension

Being accountableaccountable on demand Rowing Rowing

Displaying a vast repertoirevast repertoire of competencies Band Band

CONCLUSION: HYPOTHESIS 1CONCLUSION: HYPOTHESIS 1Typical S AfricanS African view of IR view of IR

portfolioportfoliois “narrower” than the is “narrower” than the

abovementioned IRabovementioned IRprofileprofile and should be and should be revisitedrevisited……

IE IE critericriteri

aa

FACULTY HAVE PERCEPTIONS OF FACULTY HAVE PERCEPTIONS OF LESSERLESSER PROMINENCE…? PROMINENCE…?

IR PurposesIR Purposes

The ACCOUNTABILITY dimension of IRWe need performance We need performance now!!!now!!!

Each of us has our own Each of us has our own reason…reason…

IR purposesIR purposes

IR SYNTHESIZER?IR SYNTHESIZER?

IR IR purposespurposes

Demonstrating IE t

o Demonstra

ting IE to

quality/accreditat

ion quality/a

ccreditation

panels has panels has changedchanged

Accrediting agencies areAccrediting agencies are revising their revising their

standards/criteriastandards/criteriaUSA Middle States Commission: New USA Middle States Commission: New

Standard 14Standard 14

Assessment of student learningAssessment of student learning demonstrates that the institution’s students have knowledge, skills, and

competencies consistent with institutional goals and that students

at graduation have achieved appropriate HE goals/outcomesappropriate HE goals/outcomes.

The institution has developed and implemented an assessment plan and processes that evaluate its overall overall

effectivenesseffectiveness via achieving its mission and goals, and using

institutional processes and resources efficiently in support of appropriate appropriate

student leaning outcomesstudent leaning outcomes.

USA Middle States CommissionUSA Middle States Commission New New Standard 7Standard 7 for

demonstrating IEIE

Demonstrating IE requires Demonstrating IE requires answeringanswering 2 “new” questions2 “new” questions

11stst question questionAs an institution, how well are weAs an institution, how well are we

collectively doingcollectively doing what we say we are doing? what we say we are doing?

22ndnd question questionDo we support and can we measureDo we support and can we measure

student learningstudent learningas a as a fundamental fundamental aspect of IE?aspect of IE?

Student learning Student learning assessmentassessment

as an IR challengeas an IR challengeIt seems as ifIt seems as if::Positive qualitative “changes”qualitative “changes” in student learning

is a fundamentalfundamental and most desirabledesirable IE indicator

This brings at least 2 new IR This brings at least 2 new IR challengeschallenges::

1. Assessing student learning 1. Assessing student learning Georgia State Georgia State

2. The Scholarship of Teaching 2. The Scholarship of Teaching ScholarshipScholarship

The degree of competency students had in the The degree of competency students had in the following areas when they entered/graduatedfollowing areas when they entered/graduated

Graph 1 (Entering)Graph 1 (Entering) Graph 2 Graph 2 (Exiting)(Exiting)

Learning OutcomesCommunication TechnologyCollaboration Quantitative Skills

Critical Thinking Contemporary Issues

Teresa Ward & Beth Katz

Georgia State University

IR challengeIR challenge

Enhancing theEnhancing theScholarship of TeachingScholarship of Teaching “Traditionally, IR offices have been treated as a kind of

company auditcompany audit, sitting outside the organization’s inner workings but keeping track of its effectivenesskeeping track of its effectiveness as witnessed by graduation rates, credit hours, faculty workload, etc ” (Hutchings & Shulman)

“Imagine, instead, a kind of IR that asks more central asks more central questionsquestions: What are our students really learning?What are our students really learning? How does teachingteaching e.g. affect their learning? Etc ”

Enhancing the Scholarship of teachingScholarship of teaching= feature of IR Unit that really takes really takes IE seriouslyIE seriously…

EMPIRICAL DIPSTICKEMPIRICAL DIPSTICK Is Is student learningstudent learning currently a currently a

fundamentalfundamental IE component? IE component? Missions of a couple of HEIsHEIs MissionsMissions

UJ FacultyFaculty missions? Not implicit, absentabsent

OIE/IR units’OIE/IR units’ portfolios PortfoliosPortfolios

Questions: AIRAIR members members AIR AIR

In SAAIRSAAIR countries: 2 initiatives2 initiatives

Student learning inStudent learning inHEI missionsHEI missions

ImplicitImplicit - tangible in just a few missions…

The Univ actively facilitates learning through the preservation, discovery, synthesis, and dissemination of knowledge; improve instruction via use of innovative teaching methods that require students to become actively involved in learning and develop the critical thinking skills necessary for life-long learning (Marshall University, USA)

Develop, educate and empower through quality teaching & & learning, well-rounded graduates who are able to think laterally and critically in their service to the country and its people (NWest Univ, RSA)(NWest Univ, RSA)

Promotes access with success in attracting potentially successful students and support them to become employable graduates (Central University of Technology, RSA)Emp Emp

dipstickdipstick

Student learning in Student learning in OIEs/IRsOIEs/IRs

University-wide efforts to improve student improve student learninglearning and to enhance educational effectiveness (California State Univ)

Student LearningStudent Learning is central. The assessment of our students' learning is not only a source of evidence that demonstrates our effectiveness in achieving our mission; it is, more importantly, a process for improving student learning (Texas Christian University)

To empower students to succeedempower students to succeed by providing support, data collection and analyses, communication and program support that facilitates improved retention, graduation, and student success (University of Central Florida)

Emp Emp dipstickdipstick

In increasingly moreincreasingly more IR/IE Unit portfolios/thrusts…

2 questions: 2 questions: AIR membersAIR membersApproached 10 IR Directors10 IR Directors individually

9 responses9 responses eventually1.1. Is student learningstudent learning an important indicator of your institution’s effectiveness? How do you know?How do you know?

2.2. What strategiesstrategies (if any) do your institution employ to assess student

learning?

Is Is student learningstudent learning an important an important indicator of your institution’s indicator of your institution’s

effectiveness?effectiveness?How do you know?How do you know?

All 9 respondentsAll 9 respondents confirmed the prominence of student learning in

IEResp 2: “…student learning is our most most

importantimportant dimension of effectiveness ”Resp 7: “Student success is the President’s President’s

guiding themeguiding theme for the institution and a priority for Academic Affairs ”

What What strategiesstrategies do your do your institution employ to institution employ to assess assess

student learning?student learning?Underlying assumptionUnderlying assumption

Assessment is most effective when it reflects Assessment is most effective when it reflects an understanding of learning as an understanding of learning as

multidimensionalmultidimensional, , integratedintegrated and revealed in and revealed in performance over timeperformance over time

Strategies Strategies authentic assessment authentic assessment Electronic Learning Portfolios Cogn Level & Qual of Writing Assessm (CLAQWA) AACU’s VALUE project (www.aacu.org/value) Online assessment communities

DipstickDipstick

South African Surv

eySouth Afr

ican Survey

of Student

of Student

Engagement (SASSE

) Engageme

nt (SASSE)

INITIATIVE 1INITIATIVE 1

Multi-institutiona

l Multi-ins

titutional

researchresearch

Areas of Areas of investigationinvestigation

Level of academic challenge Active and collaborative learning Student-staff interaction Enriching educational experiences Supportive campus environment

MY 2MY 2NDND HYPOTHESIS HYPOTHESISThere’s a gradual shiftgradual shift froma performance-based towards

an academicacademic (less mechanistic)notion of institutional

effectiveness,dictated by the prominence ofauthentically assessing student authentically assessing student

learninglearning ““You don’t fatten a pig by You don’t fatten a pig by weighingweighing it… it…””

““Data don’t speak to Data don’t speak to strangers…strangers… ” ”

Insanity

princip

le

Insanity

princip

le: Doin

g the sa

me

: Doing

the same

thing we

’ve alwa

ys done,

but

thing we

’ve alwa

ys done,

but

expectin

g expe

cting di

fferent

differen

t result

s...

results

...

IR APPRAISE & SERVE HELet’s revisit Suslow’s declaration on Let’s revisit Suslow’s declaration on

the purpose of IRthe purpose of IR“The institutional researcher’s basic role encompasses the systematic appraisal

of the HE effort. The institutional researcher serves HE and, in turn, his/her institution through critical

appraisal and careful investigation of its programmes and processes…”

SAAIR couldn’t longer ignore this SAAIR couldn’t longer ignore this trend and remained sort of “trend and remained sort of “under under

covercover””

We had to We had to broadenbroaden SAAIR’s focus!!

22ndnd InitiativeInitiative

SAAIR SAAIR Institutes & Institutes & Forum 2010Forum 2010: : Student Student SuccessSuccess

Will I see you in S Will I see you in S Africa?Africa?

IN CONCLUSIONIN CONCLUSIONTo To close the IE loopclose the IE loop, HEIs in , HEIs in

SAAIR have to address the SAAIR have to address the following question:following question:

How well are we How well are we collectively doingcollectively doing what we say we are doing?what we say we are doing?

IRIR is a is a collectivecollective effort, which incorporates IEIE??

IR is HE’s IR is HE’s loyal servantloyal servant…hopefully …hopefully in a synchronised & coherentsynchronised & coherent

manner