Defining, Exploring, and Measuring Relevance in Education

Dissertation

Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy

in the Graduate School of The Ohio State University

By

Megan Sanders, M. A.

Graduate Program in Educational Policy and Leadership

The Ohio State University

2016

Dissertation Committee:

Lynley Anderman, Advisor

Eric Anderman

P. Cristian Gugiu

Bryan Warnick

Copyrighted by

Megan Sanders

2016

ii

Abstract

Within both K-12 and higher education contexts, there is increasing debate about

the relevance of education (Belkin, 2014; Weston, 2015). “Making content relevant” is

framed in K-12 as a solution to the decrease in engagement as students enter early

adolescence (Anderman & Maehr, 1994) and in higher education as a necessary response

to the increasing cost of college combined with graduates’ decreasing skillsets (Arum &

Roska, 2011). However, educational practices at both levels seem grounded in different

conceptions of relevance. These different implicit definitions of relevance are

problematic because they sometimes conflict. Furthermore, most of these definitions are

not grounded in students’ lived experiences in courses that feel relevant, a disconnect that

limits their applicability. Finally, there are no existing survey measures that can be used

to empirically examine which educational practices are more or less effective at

facilitating experiences of relevance. Existing measures do not map onto an appropriate

definition of relevance or have not been rigorously developed and validated. Thus, a

shared definition of relevance that is grounded in students’ lived experiences and

reflected in a rigorously-developed measure is necessary to begin addressing the concern

about the relevance of education.

This dissertation provides preliminary versions of this definition and a

corresponding measure. Chapter 1 provides an overview of the problem. Then, Chapter 2

reviews the educational psychology literature and argues for appreciation (Brophy, 1999,

iii

2008a, 2008b) as the most appropriate conceptualization of relevance. To help

substantiate this definition, Chapter 3 presents interviews with college students about

their experiences in courses that felt relevant to their lives outside of school, analyzed

using a grounded theory approach (Charmaz & Belgrave, 2013; Corbin & Strauss, 2008).

Finally, Chapter 4 presents a preliminary measure of appreciation, developed from the

lived experiences of students explored in Chapter 3 and refined using parallel analysis,

factor analysis, and Rasch analysis. Chapter 5 synthesizes the findings of the three central

chapters.

Taken together, the results of the dissertation suggest that appreciation (Brophy,

1999, 2008a, 2008b) is an appropriate conceptualization of relevance. Examining

students’ lived experience of appreciation suggests that it can be understood in terms of

three dimensions: a cognitive dimension, a behavioral dimension, and an affective

dimension. Finally, exploring the psychometric properties of the preliminary measure of

appreciation suggests that it is reliable and valid, but nonetheless could be further refined.

Taken together, the results provide a starting point for examining what makes education

relevant.

iv

Acknowledgments

I would like to express my sincere thanks to Dr. Lynley Anderman, for both

helping me develop as a scholar and being a role model for the kind of scholar I hope to

become; to Dr. Eric Anderman, for providing insightful feedback and much-appreciated

encouragement; to Dr. Cristian Gugiu, for continually challenging me to become a better

methodologist; and to Dr. Bryan Warnick, for helping me bring a philosophical

perspective to my educational psychology work.

v

Vita

2011 ............................................................... B.S., Psychology, B.A., Plan II Honors,

The University of Texas at Austin

2014 ............................................................... M. A., Quantitative Research, Evaluation,

and Measurement, The Ohio State

University

Publications

Sanders, M., Gugiu, P. C., & Enciso, P. (2015). How good are our measures?

Investigating the appropriate use of factor analysis for survey instruments.

Journal of Multidisciplinary Evaluation, 11(25), 22–33.

Hensley, L., Shaulskiy, S., Zircher, A., & Sanders, M. (2015). Overcoming barriers to

engaging in college academics. Journal of Student Affairs Research and

Practice, 52, 176–189.

Sanders, M., Davis, T., & Love, B. C. (2013). Is better beautiful or is beautiful better?

Exploring the relationship between beauty and category structure. Psychonomic

Bulletin & Review, 20(3), 566–573.

vi

Fields of Study

Major Field: Educational Policy and Leadership

Quantitative Research, Evaluation, and Measurement

vii

Table of Contents

Abstract ............................................................................................................................... ii

Acknowledgments .............................................................................................................. iv

Vita ...................................................................................................................................... v

Table of Contents .............................................................................................................. vii

List of Tables ...................................................................................................................... x

List of Figures .................................................................................................................... xi

Chapter 1 ............................................................................................................................. 1

Chapter 2 ............................................................................................................................. 8

Characteristics of an Appropriate Account of Relevance ............................................. 11

Expectancy-Value Theory and Perceived Task Value ................................................. 14

Expectancy-Value as Relevance ............................................................................... 20

Interest Development .................................................................................................... 27

Interest as Relevance ................................................................................................. 31

Identity .......................................................................................................................... 33

Identity Exploration .................................................................................................. 34

Identity Exploration as Relevance ............................................................................ 37

Identity-Based Motivation ........................................................................................ 40

Identity-Based Motivation as an Account of Relevance ........................................... 42

viii

Transformative Experience ........................................................................................... 44

Transformative Experience as Relevance ................................................................. 48

Appreciation .................................................................................................................. 50

Appreciation as Relevance ........................................................................................ 53

Grounding Relevance in Appreciation ......................................................................... 55

Chapter 3 ........................................................................................................................... 57

Theoretical Accounts of Relevance .......................................................................... 59

Appreciation as an Account of Relevance ................................................................ 60

Method .......................................................................................................................... 64

Participants ................................................................................................................ 64

Procedure .................................................................................................................. 65

Analysis ..................................................................................................................... 70

Results ........................................................................................................................... 73

Cognitive Aspects ..................................................................................................... 73

Behavioral Aspects ................................................................................................... 78

Affective Aspects ...................................................................................................... 84

Variation by Student and Discipline ......................................................................... 87

Discussion ..................................................................................................................... 90

Limitations ................................................................................................................ 93

Directions for Future Work ....................................................................................... 94

Chapter 4 ........................................................................................................................... 95

Conceptual Definitions of Relevance ....................................................................... 95

Measures of Relevance ............................................................................................. 97

ix

Developing a Measure of Relevance ...................................................................... 103

Method ........................................................................................................................ 103

Procedure ................................................................................................................ 103

Instrument ............................................................................................................... 106

Results ......................................................................................................................... 110

Missing Data ........................................................................................................... 110

Nature of the Data ................................................................................................... 111

Construct Dimensionality ....................................................................................... 112

Rasch Analysis ........................................................................................................ 116

Convergent and Discriminant Validity ................................................................... 134

Higher-Order Factor ................................................................................................ 142

Discussion ................................................................................................................... 145

Chapter 5 ......................................................................................................................... 149

References ....................................................................................................................... 160

Appendix A: IRB Approval Letter - Study I .................................................................. 177

Appendix B: Online Screening Survey ........................................................................... 179

Appendix C: Semi-structured Interview Questions ........................................................ 182

Appendix D: IRB Approval Letter - Study II ................................................................. 183

Appendix E: Appreciation Survey Items ........................................................................ 185

Appendix F: Additional Survey Measures ...................................................................... 188

x

List of Tables Table 3.1. Demographics of the University, Full Study Sample, & Interview Sample .... 66

Table 3.2. Interview Participants' Major and Discipline of Worthwhile Course .............. 70

Table 3.3. Cognitive Aspects: Themes and Examples ...................................................... 74

Table 3.4. Behavioral Aspects: Themes and Examples .................................................... 79

Table 3.5. Affective Aspects: Themes and Examples ...................................................... 85

Table 3.6. Occurrence of Themes by Course Discipline .................................................. 88

Table 4.1. Comparison of How Measures Developed ...................................................... 98

Table 4.2. Demographics of the University and Sample ................................................ 105

Table 4.3. Rasch Analysis Results .................................................................................. 120

Table 4. 4. Items of the Three Dimensions of Appreciation Ordered by Difficulty ....... 124

Table 4.5. Raw Score to Rasch Score Conversion for the Three Dimensions of

Appreciation ............................................................................................................ 127

Table 4.6. Correlations Between Measures .................................................................... 141

Table 4.7. Order of Magnitude of Correlations .............................................................. 143

Table 4.8. Factor Loadings for the Seven Composite Variables .................................... 144

xi

List of Figures Figure 4.1. Parallel Analysis Plots for the Appreciation Items. ...................................... 113

Figure 4.2. Category Averages ....................................................................................... 121

Figure 4.3. Category Probability Curves ........................................................................ 122

Figure 4.4. Wright Maps of Three Dimensions of Appreciation. ................................... 123

Figure 4.5. Wright Map of the Cognitive Dimension ..................................................... 130

Figure 4.6. Wright Map of the Behavioral Dimension ................................................... 131

Figure 4.7. Wright Map of the Affective Dimension ...................................................... 132

Figure 4.8. Wright Maps of Additional Measures. ......................................................... 135

Figure 4.9. Wright Map of Initial Interest ....................................................................... 137

Figure 4.10. Wright Map of Triggered Interest .............................................................. 138

Figure 4.11. Wright Map of Situational Interest ............................................................. 139

Figure 4.12. Wright Map of Task Values ....................................................................... 140

Figure 4.13. Parallel Analysis of the Seven Composite Measures ................................. 143

1

Chapter 1

Relevance as an Educational Aim

A growing concern about student motivation seems to undergird discussion at

both the level of K-12 education and of higher education. In the context of K-12

education, several scholars (e.g., Anderman & Maehr, 1994; Ryan & Patrick, 2001) have

reported a troubling decrease in engagement as students enter early adolescence. Trends

in higher education mirror this decrease in motivation. An alarming number of students

drop out of college before finishing their degrees (Carlozo, 2012; Weissmann, 2012),

with the drop out rate estimated at approximately 50% (Symonds, Schwartz, & Ferguson,

2011). Even for students who do graduate, rising tuitions and increasing student loan debt

have prompted closer scrutiny of the college curriculum (Belkin, 2014; Bond, 2015; US

News, 2011; Weston, 2015). This examination suggests not only that many students do

not develop significantly in their critical thinking, complex reasoning, and writing skills

through college (Arum & Roska, 2011), but also that there is a significant disparity

between graduates’ perceptions of their skills and employers’ evaluation of their

readiness for the work world (Hart Research Associates, 2015; Mourshed, Farrell, &

Dominic, 2012).

“Making content relevant” is often framed as a solution to this decrease in student

engagement. In K-12, this manifests in pedagogical practices such as connecting the

content with students’ existing interests (Sparks, 2012), using course content to teach

2

directly transferable job skills (Hasak, 2015), and using social media, blended learning,

and other technology as engaging, real-world vehicles for the content (Benmar, 2015;

Vander Ark, 2014). Solutions that have been proposed to make higher education more

relevant focus on, for example, placing a greater emphasis on transferable job skills

(Jenvey, 2016; Kent, 2016; U.S. Department of Education, 2015; Zernike, 2009),

revitalizing the general education curriculum (Clune, 2015; Lemann, 2016; Lewin, 2013;

Paxson, 2013), measuring essential learning outcomes that cut across disciplines

(Association of American Colleges & Universities, 2014), and adopting a competency-

based curriculum that would allow students to advance as they master material (Berrett,

2015). Thus, a concern about the relevance of education seems to cut across

conversations in K-12 and higher education.

Although this concern is shared, the implicit understanding of what it means for

an education to be relevant is not. Practices such as those described above rest on

different implicit definitions of relevance, and these definitions are at best incomplete and

at worst problematic. For example, proposing an emphasis on vocational training as a

way of making content relevant implies that relevance can be understood in terms of

practical, directly applicable skills. In contrast, connecting with students’ existing

interests seems to frame relevance as related to students’ current goals and priorities.

These represent two very different conceptions of what makes education relevant.

Furthermore, not only do these implicit definitions differ from one another, but they also

seem limited. Under the first definition of relevance as practical skills, for example, it is

unclear how more abstract and less directly transferable content from the arts and

3

humanities would be justified in the curriculum. On the other hand, defining relevance in

terms of students’ existing interests seems to rest on the assumption that individuals will

have a fairly limited set of interests. Students would not be expected to be interested in all

content, so under this conceptualization of relevance, the role of such content is

uncertain. Although the curriculum is under scrutiny, definitions of relevance that cannot

account for the value of the core academic disciplines seem insufficient. Finally, the lack

of an explicit and accepted definition may lead to well-intentioned pedagogical practices

that undermine the very attempt to make content relevant. For example, using technology

as a way to make content relevant may engage students, but their attention may be

focused on the technology rather than the content (Harris, Mishra, & Koehler, 2009;

Papert, 1987). Alternatively, connecting content to students’ existing interests may be an

effective way to help students engage with some content, but may be less effective for

content that cannot be convincingly connected with these interests. Using content to

confer job skills has a similar limitation—if content cannot directly inform their desired

future careers, then students may not see a good reason to value the content. In all three

cases, these practices focus students on the interesting, pleasurable, or useful byproducts

of engaging with the content, rather than the value of the content itself. As a result,

students may not come away being able to bring content to bear on their lives outside of

school, potentially leading to the lack of deep learning (Arum & Roska, 2011) and

preparedness (Hart Research Associates, 2015; Mourshed, Farrell, & Dominic, 2012) that

have been observed in many recent graduates. Taken together, these examples suggest

that the lack of a shared, well-grounded, and operationalized understanding of relevance

4

is problematic. Without a shared conceptualization of relevance, different approaches to

making content relevant may lead to different and even conflicting recommendations for

practice and policy, and outcomes. Thus, a necessary first step is to establish this

theoretical understanding.

Turning to the educational psychology literature suggests a number of existing

constructs that could be used to define relevance. For example, task values within the

expectancy-value framework (Eccles, 2005, 2009; Eccles et al., 1983; Wigfield & Eccles,

1992) reflect different reasons that students might choose to engage with content,

including the importance of the task to their sense of self, the usefulness of the task, or

the interestingness of the task. Two other constructs that seem to reflect the impact that

content can make on students’ lives are interest development (Hidi & Renninger, 2006)

and identity exploration (Flum & Kaplan, 2006; Kaplan & Flum, 2010, 2012; Oyserman,

2007, 2009). Alternatively, perhaps relevance can be grounded in the construct of

transformative experience, which occurs when students apply content to their everyday

lives, see the world in a new or broader way, and value that new perspective (Pugh, 2002,

2004, 2011). As a final alternative, helping students recognize the life application value

of content and, in doing so, appreciate the content’s ability to enrich their lives (Brophy,

1999, 2008a, 2008b) is another possible way to define relevance. All of these constructs

seem connected with students’ lives outside of school and seem to provide good reasons

for engaging with the content.

Although helping students value course content, develop new interests, explore

their identities, and have transformative experiences are worthwhile educational goals,

5

only the concept of appreciation appears sufficiently rich to provide an appropriate

conceptualization of relevance. In the theoretical account of appreciation, Brophy (2008a)

argues that content included in the curriculum should have life application value, which

reflects not only “narrowly construed utilitarian value (helping people to meet their basic

needs and wants) but [also] enriches the quality of their lives by expanding and helping

them to articulate their subjective experiences” (p. 139). When students come to

recognize the life application of school content, they may experience appreciation, which

is characterized by “absorption, satisfaction, recognition, making meaning, self-

expression, self-realization, making connections, achieving insights, aesthetic

appreciation, and so on” (Brophy, 2008a, p. 137). Taken together, appreciation derived

from life application value provides a rich starting point for a shared definition of

relevance. To justify the use of appreciation as the shared definition of relevance, Chapter

2 sets out four criteria for an account of relevance and argues that appreciation is the only

construct among these that fully meets the four criteria.

Establishing a shared theoretical definition is an important first step, but

examining how students experience and describe their encounters with relevant content is

equally important. Although all of the constructs described previously have rigorous

theoretical grounding, many have only been explored using quantitative methods.

Furthermore, the surveys used in these studies were developed by content experts and

grounded in theory (Eccles & Wigfield, 1995; Hulleman, Godes, Hendricks, &

Harackiewicz, 2010; Linnenbrink-Garcia et al., 2010; Pugh, Linnenbrink-Garcia, Koskey,

Stewart, & Manzey, 2010a; Sinai, Kaplan, & Flum, 2012b). Thus, many of the

6

experiences reflected in these constructs have primarily been defined in terms of theory

and studied using the resulting survey measures, with less attention paid to how these

experiences occur in real time for real students. As a result, the theoretical definitions

may not reflect all aspects of these experiences. To ensure a rich understanding of

relevance as appreciation, Chapter 3 takes a qualitative approach, exploring

undergraduate students’ descriptions of their experiences in worthwhile and relevant

courses. The resulting account provides evidence for and helps to illustrate appreciation

as an appropriate conceptualization of relevance.

A clear theoretical definition of relevance substantiated by students’ experiences

provides a starting point for conversations about the relevance of education but cannot

answer questions about the ability of various pedagogical practices, curricula, and

classroom environments to help students recognize the relevance of content. To collect

empirical evidence bearing on these questions, a rigorously developed and validated

measure of relevance as appreciation is required. Several measures of similar constructs

already exist. As argued in Chapter 2, however, appreciation is distinct from these other

constructs in important ways and presents a more appropriate conceptualization of

relevance. Additionally, many of these measures, although well-known and frequently

used, were created before recent advances in survey design, and so now represent less

appropriate and less rigorous methods of survey development. Chapter 4 presents a new

measure of appreciation, developed according to the most current psychometrics methods

and approaches to survey design.

7

Thus, the following chapters aim to clarify the theoretical definition of relevance,

explore how college students describe their experiences in courses that feel worthwhile

and relevant, and develop a preliminary measure that could be used to answer empirical

questions about what makes education relevant.

8

Chapter 2

Relevance in Practice and Theory: Clarifying an Appropriate Conceptualization

Most students will not grow up to be English professors or playwrights, but

Shakespeare is nonetheless required reading in the US high school curriculum (Common

Core State Standards Initiative, 2015). Nor will most students directly apply the things

they learn about rhyme scheme, dialogue, and setting while studying Shakespeare in their

everyday lives. Although some students may leave an English class with a newly kindled

interest in Shakespeare and his plays, many of their classmates will not. As a result, the

value of studying Shakespeare may not be immediately apparent to most students.

Because this is the case with much of the curriculum, helping students recognize good

reasons for engaging with the content may be a particularly powerful way to foster

learning, especially given wide-spread concern about the pattern of decreasing academic

engagement when students reach early adolescence (Anderman & Maehr, 1994; Ryan &

Patrick, 2001). In practice, helping students recognize good reasons for engaging with the

content is often framed in terms of “making content relevant” and manifests in a variety

of different teaching practices. For example, using course content to teach directly

transferable job skills (Hasak, 2015), connecting the content with students’ existing

interests (Sparks, 2012), and using social media, blended learning, and other technology

as engaging, real-world vehicles for the content (Benmar, 2015; Vander Ark, 2014) are

all practices intended to make content relevant to students.

9

Some researchers (Brophy, 1999, 2008a, 2008b, 2010; Deci, Koestner, & Ryan,

1999), however, have suggested that these well-intentioned practices may actually

undermine the attempt to make content relevant. Instead of focusing students’ attention

on the content itself, such practices shift the focus to interesting, useful, or pleasurable

byproducts associated with the content. Although these byproducts may motivate

students to engage with the activity in the short term, students may walk away without

recognizing the broader life application of the content itself. In the long term, this may

limit students’ ability to bring the content to bear on their everyday lives. For example, a

middle school student may be engaged in her Language Arts class because she was able

to write an assignment on a topic she is interested in, has been told that her writing ability

will be important for her future job, and enjoys playing the computer game the class uses

to review vocabulary words. Despite her engagement with these activities, she

nonetheless might not come away with a good sense of how what she has learned can

enrich her inner life, her understanding of the world, and her relationships with others—

all things that Language Arts has the potential to do. Thus, these well-intentioned

practices may actually undermine the connection between school content and students’

everyday lives that they were designed to strengthen.

Although many current practices do not seem like appropriate means, the end goal

of making content truly relevant to students’ current and future lives outside the

classroom is a worthy one. The apparent mismatch between the means and the end may

be the result of two distinct problems. First of all, there is some ambiguity in the term

“relevance” itself. Each of the practices discussed earlier seems to presuppose a

somewhat different conceptualization of relevance: as interest, as practical skills, and as

10

tied to a mode of learning. Ambiguity in the ends leads to a range of sometimes-

conflicting means for achieving that end. Thus, a necessary first step is to establish a

shared conceptualization of relevance.

The second issue that contributes to the mismatch between educational practices

and the goal of relevance is the lack of a clear set of research-based recommendations for

practice. Assuming that a common understanding of relevance can be established, a

rigorous body of empirical research is necessary to establish how effective different

educational practices are as means to relevance. Without a clear conceptualization of

relevance first and research-based recommendations for practice second, current practices

may continue to unintentionally undermine the goal of making school content relevant to

students.

Turning to the literature in educational psychology suggests a number of

theoretical frameworks that may provide appropriate conceptualizations of relevance to

guide the development of more effective teaching practices. Perhaps it is most

appropriate to think about the value students see in the content or task (Eccles, 2005,

2009; Eccles et al., 1983; Wigfield & Eccles, 1992). Alternatively, perhaps the content

can serve as a means for interest development (Hidi & Renninger, 2006), or as a way for

students to explore and act on their sense of identity (Flum & Kaplan, 2006; Kaplan &

Flum, 2010, 2012; Oyserman, 2007, 2009). On the other hand, relevance may best be

conceptualized in terms of helping students apply and see the content in action in the real

world, valuing that transformative experience (Pugh, 2002, 2011). Or perhaps it is best to

conceptualize relevance as helping students develop appreciation for the content’s ability

to enrich their lives (Brophy, 1999, 2008a, 2008b).

11

Although developing teaching practices that help students value course content,

develop new interests, explore their identities, and have transformative experiences are

worthwhile educational goals, I will argue that only the concept of appreciation is

sufficiently rich and educationally useful enough to provide an appropriate

conceptualization of relevance. To do so, I will first argue for a set of criteria for an

appropriate account of relevance. Next, I will review each of the other theoretical

constructs listed, in turn, and highlight important ways they fall short of providing an

adequate account of relevance. I will then describe Brophy’s (1999, 2008a, 2008b)

account of appreciation in more detail, focusing on the way it avoids limitations faced by

other constructs and presents a more compelling conceptualization of relevance.

Characteristics of an Appropriate Account of Relevance

The concept of relevance is fundamentally relational. Something is relevant to

something else, and “relevance” characterizes the nature of the relationship between

those two things. More specifically, the relationship seems to be one of bearing on,

furthering, or contributing to. Something is relevant to the extent that it adds to or

informs the particular subject at hand. When considered in an educational context,

relevance might be framed as the way curriculum and instruction bear on or add to a

student’s education. I make three assumptions about the conceptualization of education

that shape the criteria of relevance I describe. First of all, the process of education seems

to assume that students will undergo a change or be somehow different as a result. If

students leave a learning experience exactly the same as they were before beginning it,

we could say that the educational process failed—it produced no change. Second,

following Dewey (1980), the nature of this change seems to be one of growth or

12

development. This growth may not always be pleasant, as in the struggle to develop a

skill or the realization that a previously held belief is untrue, but with it comes a better

understanding of the world and a greater ability to act within it. Finally, the endeavor of

education also seems to assume that this change is not circumscribed within the

classroom. In other words, the process of education somehow shapes students as people

outside of school. If this were not the case, then education would function only to change

students as students. Any changes they underwent would only impact their experience in

school, without any bearing on life outside of the classroom. Thus, I argue that a

student’s education involves somehow changing the student in ways that promote growth,

with a particular focus on changing the student’s life outside of school. These

assumptions about the nature of education lead to the first criterion of an appropriate

account of relevance: curriculum and instruction would be considered relevant to the

extent that they somehow impact students’ lives outside of school.

Clarifying the nature of curriculum and instruction also suggests criteria for an

appropriate account of relevance. First, the content included in the curriculum plays an

important role in the impact on students’ lives outside of school. Content is not random,

nor is it interchangeable. Presumably, content is included in the curriculum because it is

considered instrumental for a particular type of impact on students’ lives outside of

school. For example, the curriculum might not include both math and English content if

the way students changed by engaging with these two disciplines was the same.

Therefore, the content itself plays a central role in the way students change through

education. In terms of an account of relevance, this suggests that relevance derives

centrally from the content itself, not simply the way it is taught. However,

13

conceptualizing education in terms of both curriculum and instruction suggests that

content with the potential to impact students’ lives outside of school is necessary but not

sufficient. The inclusion of instruction suggests that teachers are instrumental in helping

students recognize the relevance of content. Thus, content included in the curriculum has

the potential to impact students’ everyday lives in particular ways, and instruction can

help students recognize that potential where they may not have seen it before. These

observations lead to two additional criteria for an account of relevance: relevance is

fundamentally grounded in the content and student’s recognition of relevance can be

developed.

Although these criteria seem sufficient for an appropriate account of relevance,

one additional criterion would make the account more educationally useful. An account

of relevance that could be meaningfully implemented on a broad scale—guiding an

individual teacher or a state-level policymaker in choosing content and instructional

methods—would stipulate that most content included in the curriculum reasonably have

the potential to be relevant to most students. Nothing in the conceptualization more

broadly precludes content from being relevant to only one student in a classroom, for

example. However, given the current educational context in which large classes and

standardized curricula are the norm, an educationally useful conceptualization of

relevance would apply to most content and most students.

Taken together, an appropriate and educationally useful account of relevance

would meet these four criteria: relevance would be framed in terms of the impact on

students’ lives outside of school, this impact would derive from content, students’

recognition of relevance could be developed, and most students could come to recognize

14

the relevance of most content. These criteria serve as a basis for evaluating task value,

interest, identity, transformative experience, and appreciation as potential accounts of

relevance.

Expectancy-Value Theory and Perceived Task Value

Expectancy-value theory (Eccles, 2005, 2009; Eccles et al., 1983; Wigfield &

Eccles, 1992) presents one possible conception of relevance. Within this theoretical

framework, students’ behavior in achievement situations can be understood in terms of

two perceptions: expectancies and values (Eccles et al., 1983; Wigfield & Eccles, 1992).

Expectancy reflects the degree of success a student expects to achieve on a particular

task. Value, on the other hand, reflects the degree to which succeeding at the task can

satisfy a student’s needs, help the student achieve important personal goals, or affirm

valued identities (Eccles, 2009; Eccles et al., 1983). Thus, expectancies and values are

subjective, cognitive perceptions, and they are shaped by a range of contextual and

individual factors. Distal factors, such as students’ past experiences, parents’ beliefs and

expectations, and broader cultural norms, influence expectancies and values through

students’ interpretation of and past affective reactions to these factors (Eccles, 2009;

Eccles et al., 1983). Expectancies are influenced most directly by students’ self-concept

of their ability in a task’s domain—their own assessment of their ability to successfully

complete tasks in an area such as science—and their perception of the task’s difficulty.

Perceived self-concept and task difficulty are the result of students’ interpretation of past

successes and failures, the expectations of others, and social stereotypes within the

domain, for example, the stereotype of math as a male domain (Eccles, 2009; Eccles et

al., 1983). Values, like expectancies, are shaped by students’ past experiences with the

15

domain, messages from others about the value or importance of the domain, and

stereotypes that mark the domain as more or less appropriate for individuals like the

students (Eccles, 2009; Eccles et al., 1983). Within the Eccles expectancy-value model,

value is broken down into four, more specific types of task values: attainment value,

utility value, interest or intrinsic value, and cost (Eccles et al., 1983; Wigfield & Eccles,

1992).

Attainment value represents the opportunities that an achievement task presents

for students to act in ways consistent with their personal and social identities (Eccles,

2005, 2009). Within the most recent version of the Eccles model, identities are framed as

collections of self perceptions, including an individual’s perception of his or her own

personality and capabilities, proximal and distal goals, values, more general motivational

goal orientation, ideal future selves, and schema for the values and behaviors associated

with people like the individual (Eccles, 2005, 2009). Identities can be either individual,

reflecting those qualities that make the individual unique, or social, encompassing those

characteristics that tie the individual to valued groups of others (Eccles, 2009). Each

individual holds multiple individual and social identities, and identities that are most

salient to an individual at any given time are influenced by the context.

The primary way individuals enact their identities is through behavior, as

engaging in a particular task provides an opportunity to demonstrate the characteristics

associated with that task (Eccles, 2005, 2009). Thus, attainment value reflects the degree

to which an achievement task allows an individual to confirm that he or she possess the

qualities that are markers of an important identity (Eccles, 2005, 2009). Tasks provide

different degrees of attainment value, and when faced with a choice between two tasks,

16

an individual will choose the task with greater attainment value over the choice with

lower attainment value (Eccles, 2005, 2009). Individuals will try to avoid engaging in

tasks that either are not reflective of or that disconfirm qualities of their valued identities

(Eccles, 2005, 2009). Attainment value is therefore strongly connected with an

individual’s sense of self.

Utility value is the second type of task value within the Eccles expectancy-value

model and captures the value a task holds as a means toward a future end (Eccles et al.,

1983). Importantly, the value does not lie in the process of engaging in the task itself, and

so, in the instrumental nature of the task, utility value is similar to extrinsic motivation

(Deci, Vallerand, Pelletier, & Ryan, 1991; Eccles, 2005; Eccles et al., 1983; Wigfield &

Eccles, 1992). In recent developments of the Eccles expectancy-value model (Eccles,

2005, 2009), however, utility value has been conceptualized as more closely linked with

attainment value. Under this conceptualization, tasks with utility value are not

opportunities to directly enact a valued identity, but may nonetheless provide a way for

the individual to fulfill other important but less central goals. For example, for a student

who hopes to become a doctor, performing well in a math course may not hold attainment

value. Yet if the math course satisfies premed requirements and so indirectly helps the

student achieve a valued identity, the utility value of the course may be more closely tied

to attainment value. Thus, unlike attainment value, which derives from the task itself,

utility value derives from the desired result of engaging in the task. Additionally,

although both types of value are related to identity, utility value is less strongly connected

to identity than is attainment value.

17

The third task value specified in the Eccles model is interest or intrinsic value. In

contrast with utility value, interest value derives from the immediate subjective

experience of engaging in the task itself (Eccles et al., 1983). In elaborating the

experience of interest value, Eccles (2005) explicitly relates it to Csikszentmihalyi’s

(1990) concept of flow, as both are characterized by complete immersion in a task, deep

focus, and loss of self-consciousness. Flow and interest value are also similar in that they

are most likely to result when there is a balance between an individual’s skills and the

challenges presented by the task (Csikszentmihalyi, 1990; Eccles, 2005). Furthermore, as

was the case with utility value, interest value can become more closely related to

attainment value over time. If an individual enjoys engaging in a task, he or she is likely

to repeatedly engage with it, and that repeated engagement might lead the individual to

develop greater competence for the task. Because ability is an identity-relevant self-

perception, developing competence at a task may result in the skill being integrated into

an individual’s identity (Eccles, 2009). Thus, over time, tasks that initially held interest

value can begin to hold attainment value.

Finally, the perceived cost of engaging in a particular task interacts with the

perceived value of the task to shape subsequent achievement behavior. Broadly, cost

represents reasons not to engage with a particular task (Eccles et al., 1983). More

specifically, cost can be shaped by a number of factors, such as the amount of effort

required to be successful on a task, the time and energy students give up to devote to the

task rather than to other valued tasks, the perceived repercussions of failing at the task

(Eccles et al., 1983), and the anxiety students anticipate experience while engaging in the

task (Eccles, 2005). Cost can also be social in nature, both in terms of how important

18

others, such as peers and parents, might react to failure on the task (Eccles, 2005), as well

as the social consequences of success (Eccles, 2009).

Taken together, expectancies and the four task values are shaped by students’

experiences and interpretations and in turn influence students’ behaviors in achievement

situations, impacting behaviors such as choice of task, effort invested, and degree of

persistence (Eccles et al., 1983; Eccles, Adler, & Meece, 1984; Meece, Wigfield, &

Eccles, 1990). A large body of empirical work supports these relationships as presented

in the Eccles model. To begin with, factor analysis of the self-report survey items used to

assess expectancies and the task values indicates that the task values are both distinct

from each other and distinct from expectancies (Eccles & Wigfield, 1995), even in

children as young as first grade (Wigfield, Eccles, Mac Iver, Reuman, & Midgley, 1991).

Empirical studies also support the hypothesized relationships between value, expectancy,

and related perceptions such as task difficulty (Eccles et al., 1984; Eccles et al., 1983;

Eccles & Wigfield, 1995). Expectancy is positively correlated with interest, attainment,

and utility value, suggesting that students may value tasks that they are good at.

Expectancy and task difficulty are negatively correlated, suggesting that students may be

less likely to feel competent at task they perceive as difficult. Finally, value and task

difficulty are also negatively correlated, reflecting that students may be less likely to

value tasks they perceive as difficult (Eccles et al., 1984; Eccles, et al., 1983; Eccles &

Wigfield, 1995).

Expectancies and values, in addition to relating to one another and students’

interpretation of other factors such as task difficulty, are also related to achievement

behavior. Prior work, however, suggests that values and expectancy relate to different

19

aspects of achievement behavior. In a large-scale study of student perceptions and

achievement behavior in math courses, Eccles and her colleagues have found that

students’ value of math is related to their intention to take more math classes and to their

actual enrollment in additional math courses (Eccles et al., 1983; Eccles et al., 1984;

Meece et al., 1990). More broadly, senior high school students’ value of different careers

and of different job characteristics (i.e. helping others or earning a lot of money) also

predicts their career aspirations (Eccles, Barber, & Jozefowicz, 1999). In this study,

students’ expectancy for success in different kinds of careers also predicted their

intention to pursue careers in that field—for example, expectations for success in health-

related occupations predicted plans to pursue such careers. However, expectancy is

generally much more strongly related to students’ actual performance on a task than to

their intention to pursue similar tasks in the future (Eccles et al., 1983; Eccles et al., 1984;

Meece et al., 1990). In the study of math classes, for example, students’ expectation for

success in math was positively correlated with their performance on a math achievement

test the following year.

Finally, a body of work supports the Eccles model’s suggestion that expectancies

and values are shaped by individual and broader cultural factors. In particular, sex

differences exist between girls’ and boys’ valuing of and expectancies for success in

different domains (Eccles & Harold, 1992). In the study of students in math classrooms,

boys reported feeling more competent in math and perceived math as less difficult than

did girls (Eccles et al., 1983). In terms of value, girls perceived math as less useful and

less interesting than boys did (Eccles et al., 1999). These findings can be accounted for

within the Eccles model, given the role of parent’s beliefs and expectations, broader

20

cultural norms, and students’ interpretation of past experiences in shaping their

expectancies and values. Not only do cultural sex roles position math as a masculine

domain, but parents’ and teachers’ response to girls’ performance in math tasks can also

shape the way female students interpret their success and failure in the domain (Eccles et

al., 1983). Expectancies and values are also shaped by development. In general, students’

perceived competence and both attainment and interest value for school subjects tend to

decrease as they move from early elementary school to middle school, but these

perceptions also become more stable and more differentiated over time, both in terms of

different school subjects and relative to students’ peers (Wigfield et al., 1997). Taken

together, these findings provide strong evidence for the overall Eccles model.

Expectancy-Value as Relevance

Because perceptions of relevance are related to the reasons students engage in a

task, the possible conceptualization of relevance presented in the expectancy-value theory

framework will focus on task values, rather than expectancies. Furthermore, because the

social, time, or other costs students perceived to be associated with an achievement task

represent reasons not to engage in a task, rather than good reasons to engage in a task,

cost may be less useful for developing an account of relevance. Thus, the

conceptualization of relevance will be considered in terms of attainment value, utility

value, and interest value.

Attainment value as relevance. When attainment value is considered in terms of

the criteria of an appropriate account of relevance, it seems to meet some criteria but falls

short on others. To begin with, attainment value meets the first criterion of relevance:

connections between the content and students’ existing identities do seem to bear on

21

students’ lives outside of school. Although different identities are more or less salient in

different environments, students carry their set of identities with them both in school and

outside of it. Thus, the impact of achievement value, in terms of providing an opportunity

to confirm valued identities, also transcends school boundaries.

Attainment also seems directly linked with the content itself, the second criterion

of an account of relevance. Individual tasks are the unit of analysis in expectancy-value

theory, and in an educational context, individual achievement tasks are associated with

particular content. Attainment value derives from demonstrating characteristics or

qualities required by the task, and tasks associated with different content are associated

with different characteristics. For example, doing well on assignments in a math course

may say something different about a student than doing well on assignments in science or

English. However, perhaps an individual values being a good student. For this student,

achievement tasks in school could hold attainment value because they offer an

opportunity to confirm the identity of “good student.” In this case, the attainment value of

the task is not necessarily linked with the content—achievement tasks in math, science,

or English could provide equally good opportunities to confirm the “good student”

identity. For this reason, although attainment value generally seems closely associated

with particular content, it is unclear whether that link is necessary. Thus, attainment value

may not sufficiently meet the second criterion of an account of relevance.

There is also some ambiguity about whether attainment value is able to be

developed and so to meet the third criterion of an appropriate account of relevance.

Eccles (2005, 2009) is explicit about the fact that tasks that initially hold interest value or

utility value can eventually come to hold attainment value. A task with utility value can

22

become integrated into a more central goal or, through repeated engagement, an

individual can develop competence, an identity-relevant perception, at a task with interest

value. On the other hand, developing attainment value directly would take the form of

helping students recognize the opportunities achievement tasks offer to confirm their

valued identities. This becomes problematic when there is not an obvious connection to

be made between the material and the way students see themselves. If engaging in a

science class cannot be made to feel like an authentic opportunity for a student who sees

himself as a future novelist to enact his sense of self, the science content will not hold

attainment value for that student. Thus, although there are ways of developing attainment

value, they are limited by students’ existing identities, future goals, or interests. In this

way, attainment value does not seem to fully meet the third criterion of being able to be

developed. In an appropriate conceptualization of relevance, students’ recognition of the

relevance of content would not be limited by their current identities, interests, or sense of

what is useful. Rather, perceptions of relevance could be developed by helping students

recognize the affordances of content—included in the curriculum in order to make a

particular sort of impact on students’ lives outside of school—that they may not have

recognized before.

The limits placed on the development of attainment value by existing identities,

sense of utility, and interests raises the most problematic shortcoming of attainment value

as a conceptualization of relevance: it does not meet the criterion of applying to most

content for most students. By definition, only a subset of all possible content areas or

activities holds attainment value for an individual student. It is unclear whether it is

necessary for all students to see attainment value in all content and whether it is

23

appropriate to focus teaching practices on developing that value. On the other hand, most

content areas included in the curriculum should have the potential to be relevant to most

students. Because of this mismatch, attainment value does not meet the final criterion of

an appropriate account of relevance. In combination with the fact that it is not necessarily

linked with content and shifts the focus to connecting content with students’ existing

identities rather than helping them recognize the potential impact of the content,

attainment value does not meet the criteria for an appropriate and educationally useful

account of relevance.

Utility as relevance. Of the three task values, utility value seems the most similar

to the colloquial conception of relevance: it reflects practical applicability of skills like

literacy and numeracy that act as means to real world ends. As such, it clearly meets the

first criterion of impacting students’ everyday, out of school lives. However, in spite of

its similarity to common understandings of relevance, utility value does not provide an

appropriate and educationally useful account of relevance according to the remaining

three criteria.

Initially, utility value does seem directly linked with content, and for some

content, such as that involved in learning how to read and write, the connection is clear.

The content in this case is instrumental in achieving the desired outcome. However,

utility value can quickly become disconnected from the content associated with the task

itself, because the outcome of a task, rather than engagement in the task itself, is

ultimately the source of utility value (Eccles, 2005; Eccles et al., 1983; Wigfield &

Eccles, 1992), There are many reasons why a student would want to engage in a task that

are unrelated to the task’s content: to receive a good grade, to please parents, to

24

outperform peers, to be accepted into a good school, to get a good job, and so on. In each

of these cases, if engaging in the task results in the desired outcome, then the task holds

utility value, regardless of the content associated with it. Because of this focus on the

outcome rather than the task itself, utility value does not seem well aligned with the

second criterion of an appropriate account of relevance.

Developing utility value, the third criterion, would focus on making explicit the

ways that content could be directly applied to students’ lives outside of school. It seems

possible to develop utility value; however, as was the case with attainment value, this

development could be somewhat limited by the goals students already hold. This is

problematic because it implicitly begins with the goals and outcomes students already

value, rather than with helping students recognize value where they did not see it before.

In other words, to borrow the phrasing of Blumenfeld, Puro, and Mergendoller (1992)

content is brought to students, rather than bringing students to the content. An appropriate

account of relevance would focus on helping students recognize the affordances of the

content rather than molding content to meet students’ current goals or desired outcomes.

Finally, not all content included in the curriculum holds obvious utility value.

Although the usefulness of math and basic literacy is straightforward, the direct

applicability of topics of study like Shakespeare or history is less clear. With more

abstract content such as this, making the case for the content as instrumental to students’

existing goals becomes more difficult without simply reducing content to a means to an

end, which violates the second criterion’s emphasis on content. Thus, utility value may

not apply to all content. It may also be the case that even topics that can be directly useful

in everyday life may not hold equal utility value for all students, given the range of

25

individual experiences and goals they have and the futures they aspire to. This suggests

that utility value does not sufficiently meet the fourth criterion of applying to most

content for most students, and so does not offer an appropriate conceptualization of

relevance.

Interest value as relevance. Interest, the final task value, also falls short of an

appropriate account of relevance. In terms of the first criterion, it is unclear whether

interest value represents an impact on students’ lives outside of school. Interest value

reflects the anticipated or experienced enjoyment of engaging in the task itself, and this

definition does not necessitate that the task can or does also occur in students’ lives

outside of school. It is thus possible that the motivating impact of interest value stays

within school. However, interest value could begin to exert an impact on students’

everyday lives if they began seeking similar tasks outside of school. Alternatively,

repeated engagement with a task based on its interest value could lead an individual to

develop a competency in that task. This competence, as described before, bears on the

student’s identity and so the task could eventually take on attainment value. Because

students carry their identities with them, interest could come to have an impact on

students’ lives through attainment value. Interest value therefore seems to meet the first

criterion of relevance in some cases but not others—all of which are legitimate examples

of interest value—suggesting that it may not provide an appropriate account of relevance.

The theoretical definition of interest value is also ambiguous in terms of whether

it necessarily derives from the content associated with the task. In some instances, it is

clear that the content drives students’ perceptions of interest value, for example, when

starting a unit on World War II for a student with a preexisting interest in the topic. In

26

other cases, however, the interest value of a task can derive from the nature of the task

itself, rather than the content. For example, using a jeopardy game to review content from

this class before a test on WWII may have interest value for a different student because of

the competition and group work, not because of the material being reviewed. The

theoretical definition of interest value does not differentiate between these two instances.

The first instance seems to meet the second criterion, whereas the second instance does

not. If interest value were to be used as an account of relevance, it would be necessary to

focus on content-driven interest value.

Focusing on content-driven interest value becomes problematic when considering

whether interest value can be developed, the third criterion. When it comes to content-

driven interest value, “we know little about the origins of either within-individual or

between-individual differences in interest” and in some ways, content-driven interest is

more closely related to attainment value (Eccles, 2005, p. 111). Thus, there is not a clear

way to develop content-driven interest value, especially without encountering the

shortcomings of trying to develop attainment value that were described earlier. On the

other hand, there is a great deal known about what features of a task increase task-driven

interest value, such as novelty and comprehensibility (Hidi & Baird, 1986). Yet despite

meeting the third criterion of being developable, task-driven interest value does not meet

the second criterion of being content-based. Content-driven interest value faces the

opposite problem, meeting the second by failing to meet the third criterion.

This problem is further exacerbated when content- and task-driven interest value

are considered in terms of the fourth criterion. Although it could be the case that most

content could be taught in such a way that the majority of students experienced task-

27

driven interest value, the nature of content-driven interest value as more similar to

attainment value suggests that most content would not hold content-driven interest value

for most students. In short, neither content-driven nor task-driven interest value fully

satisfies the criteria of an appropriate account of relevance, each falling short on at least

one necessary criterion.

Interest Development

Another construct that may present an appropriate account of relevance is interest.

The construct of interest is similar to interest value in the Eccles model (2005, 2009;

Eccles et al., 1983), but work in this area elaborates the construct more fully and focuses

more closely on the way interest develops (Hidi & Renninger, 2006; Renninger & Hidi,

2011; Schiefele, 2001, 2009). Unlike expectancy-value theory, which is primarily

characterized by the Eccles model (Eccles, 2005, 2009; Eccles et al., 1983), work in the

field of interest is more heterogeneous, representing several distinct by related theoretical

definitions. Interest is alternatively conceptualized as an emotion (Silvia, 2005, 2006), in

terms of value beliefs (Schiefele, 2001, 2009), in terms of the nature of the task (Mayer,

2008; Sansone, 2009), and in terms of its development (Hidi & Renninger, 2006; Krapp,

2002, 2007; Renninger & Hidi, 2011). Among these conceptualizations, Renninger and

Hidi (2011) highlight five qualities of interest that cut across accounts: it is specific to

particular content (Krapp, Hidi, & Renninger, 1992; Krapp & Prenzel, 2011); it

represents a relationship between an individual and the particular content that develops

through interaction (Krapp & Prenzel, 2011); it has both cognitive and affective qualities

(Ainley, Hidi, & Berndorff, 2002; Hidi, Renninger, & Krapp, 2004); it has a biological

basis (Hidi, 2006); and learners are not always aware of their interest while engaged with

28

content (Renninger & Hidi, 2002). These commonalities suggest that the varied accounts

of interest may represent different but complimentary perspectives on the same construct,

rather than definitions of fundamentally different constructs.

Among these perspectives, Hidi and Renninger’s (2006; Renninger & Bachrach,

2015; Renninger & Hidi, 2011) four-phase model of interest development seems the most

promising as a potential account of relevance, given its focus on how interest develops

and the role of teachers, content, and the learning environment on that development.

Within this account, interest is conceptualized as a “psychological state of engaging or

the predisposition to reengage with particular classes of objects, events, or ideas over

time” (Hidi & Renninger, 2006, p. 112), which manifests in degrees across four

sequential stages characterized by different levels of knowledge, affect, and value. First,

interest must be triggered, and this short term, triggered situational interest is

characterized by increased attention and positive affect, but relatively little knowledge.

The next phase of interest development occurs when the activity “holds” students’

interest and triggered situational interest moves into maintained situational interest.

When students’ situational interest is maintained, their attention and persistence with the

task are sustained and may reoccur when re-engaging with the task at another time (Hidi

& Renninger, 2006). Furthermore, maintained situational interest has both affective and

value components: students’ interest can be held by their enjoyment of the task or the

value they place on it (Linnenbrink-Garcia et al., 2010). If students continue to reengage

with content and begin to develop stored knowledge about and value for that content,

maintained situational interest can become emerging individual interest. Compared to

situational interest, individual interest is less dependent on the nature of the task or the

29

learning environment and may start to become self-generated, as students pose questions

about the content and seek opportunities to reengage with it (Hidi & Renninger, 2006).

However, students with emerging individual interest still benefit from external supports,

such as models in peers or experts and encouragement when faced with challenges (Hidi

& Renninger, 2006). As students’ knowledge of, value for, and positive affect associated

with particular content further increases, emerging individual interest can become well-

developed individual interest. Effort associated with reengaging with content begins to

feel effortless, similar to Csikszentmihalyi’s conceptualization of flow (1990) and

students may come to identify with the content (Hidi & Renninger, 2006). Well-

developed individual interest is a relatively stable disposition toward particular content

and can motivate students to reengage with the content (Scheifele, 2009).

Empirical work lends support to the four-phase model. In a study using multiple

exploratory and confirmatory factor analyses, Linnenbrink-Garcia and her colleagues

(Linnenbrink-Garcia et al., 2010) found support for the distinction between triggered and

maintained situational interest, as well as for the distinction between affect-related and

value-related maintained situational interest. To explore the relation between the stages of

interest development, Harackiewicz and her colleagues (Harackiewicz, Durik, Barron,

Linnenbrink, & Tauer, 2008) measured college students’ initial interest at the beginning

of a psychology course; situational and maintained situational interest during the course;

and final grades, intention to major in psychology, and subsequent course enrollment

after the course. In this study, maintained situational interest was treated as a single factor

instead of being divided into affect-related and value-related components. Using path

models, students’ initial interest in psychology was positively related to their triggered

30

situational interest during the course. Both students’ initial interest and triggered

situational interest were positively related to their maintained situational interest at the

end of the course. This result suggests that, for students entering the course with low

initial interest, triggered situational interested in class can lead to the development of

maintained situational interest. Finally, students who reported greater levels of

maintained situational interest at the end of the course were more likely to end up

majoring in psychology, a measure used to gauge individual interest (Harackiewicz et al.,

2008). These findings are consistent with the four-phase model and suggest that

triggered, maintained, and individual interest may represent different stages in interest

development.

Other work has explored ways to catch and hold students’ interest. For example,

in a qualitative study, Mitchell (1993) found that the use of puzzles, computers, and

group work were appropriate strategies to “catch,” or increase students’ triggered

situational interest. Features of text that have been found to increase situational interest

include vividness, concreteness, surprisingness, coherence, and ease of comprehension

(Schraw & Lehman, 2001; Silvia, 2006). In an observational study highlighting the range

of possible triggers of interest, triggers included challenge, hands-on activity, novelty,

group work, computers/technology, autonomy, instructional conversation, and connection

with personal experience (Renninger & Bachrach, 2015). On the other hand, strategies

that hold students’ interest include increasing students’ involvement and emphasizing

how the content is important or related to students’ everyday lives (Mitchell, 1993;

Harackiewicz, Barron, Tauer, & Elliot, 2002; Häussler & Haufmann, 2002; Hoffmann,

Lehrke, & Todt, 1985). Experimental studies suggest that classroom practices designed to

31

catch and to hold students’ interest are more and less effective, depending on students’

level of individual interest (Durik & Harackiewicz, 2007). While students with low

individual interest reported greater interest in a task designed to catch students’ interest

compared to a task without the catch facets, students with high individual interest

reported less interest in the experimental catch task than the control task. The catch

facets, although bolstering low-interest students’ interest, appear to interfere with high-

interest students’ engagement with the task. In a different condition designed to

manipulate hold, the findings were reversed: students with low individual interest

reported less interest in the hold condition compared to a control condition, while

students with high individual interest reported greater interest in the hold condition

compared to the control (Durik & Harackiewicz, 2007). Examined together, these

findings suggest that classroom practices can shape students’ interest development, but

that these practices may work differently depending on students’ existing level of

interest.

Interest as Relevance

When considered in terms of the four criteria of an appropriate account of

relevance, interest seems like a promising potential conceptualization. First of all, interest

does seem to bear on students’ lives outside of school when it develops into emerging or

individual interest. Because individual interest is less dependent on the nature of the task

or the nature of the learning environment in order to be sustained, it is more likely to

shape students’ behavior outside as well as inside of school. Because it makes it more

clear how and when interest can impact students’ everyday lives, the distinction between

situational and individual interest is one advantage of interest development as a possible

32

account of relevance over interest value from expectancy-value theory. Thus, on the first

criterion, interest would be appropriate as long as the focus is on developing individual

interest.

The distinction between situational and individual interest also helps address the

second criterion of being content-focused. Within the four-phase model of interest

development, students begin to value the content itself and to pursue related activities

with less external support when they develop individual interest. Much like the

distinction between task-driven and content-driven interest value in expectancy-value

theory, the distinction between situational and individual interest demarcates

inappropriate from appropriate conceptualizations of relevance. Therefore, the second

criterion of an appropriate account reemphasizes the importance of framing relevance in

terms of individual interest.

Empirical evidence for the four-phase model also suggests that interest can be

developed, meeting the third criterion. However, catch and hold facets are not equally

appropriate ways of developing interest. The teaching strategies used to catch and arouse

triggered situational interest—such as the use of computers, group work, and puzzles

(Mitchell, 1993) or engaging features of text (Schraw & Lehman, 2001; Silvia, 2006)—

rely on the nature of the task, rather than the content itself, to evoke interest. This focuses

both students’ and teachers’ attention away from what makes the content, in and of itself,

worth learning and towards a concern for the way that content is delivered. As Schiefele

(2009) suggests, catch facets “only have an arbitrary relation with a given subject

content” (p. 200). Hold facets, on the other hand, are focused more closely on the content,

highlighting ways the content itself is related to students’ everyday lives. Thus, utilizing

33

strategies designed to hold rather than to catch students’ interest seems an appropriate

way to develop students’ interest and appropriate grounding for an account of relevance.

Although interest would meet the third criterion if strategies focused on hold

facets, it fails to meet the fourth criterion of applying to most content for most students.

To begin with, empirical work indicates that strategies designed to hold students’ interest

are more effective for students with existing initial interest than for students without a

strong initial interest (Durik & Harackiewicz, 2007). This suggests that strategies to hold

students’ interest—the only strategies focused on content and therefore appropriate for an

account of relevance—may not help all students develop individual interest. Secondly,

this account is limited by the fact that not all students will develop an interest for all

required content. Hidi and Renninger (2006) acknowledge explicitly that “even those

students who are highly motivated to achieve generally have interest(s) only for a discrete

set of specific content areas” (p. 112). An account of relevance would help most students

see ways that most required content included in the curriculum bears on their everyday

lives. Thus, although interest development is a worthy educational goal, it is not

sufficiently broad to provide an account of relevance.

Identity

A third body of work that seems related to students’ perceptions of relevance is

that focused on identity. Like the work on interest, the literature on identity is

heterogeneous, representing lines of work developing from theoretical backgrounds

ranging from postmodernist (e.g., Gee, 2000) to sociocultural (e.g., Faircloth, 2012) to

developmental (e.g., Erikson, 1968; Marcia, 1966, 1980, 1993). Because the motivation

34

literature is primarily grounded in Erikson’s (1968) theory of identity and identity

formation, work developing from the psychological perspective will be the focus here.

Identity Exploration

Within Erikson’s (1986) theoretical framework of development, identity plays a

central role throughout life and during adolescence in particular. For Erikson, identity is

shaped both by the intentional decisions of the individual and by the norms and

opportunities afforded by the individual’s social context (1986). Building on Erikson’s

framework, Marcia (1966, 1980, 1993) specifies two processes by which this

development occurs: exploration and commitment. Exploration involves questioning

previously held perceptions and beliefs, seeking out new information, reflecting on the

self, and considering alternate identities. Importantly, exploration involves considering

this new information relative to the self. Commitment involves consolidating, adopting,

and integrating new information, beliefs, and perceptions into the self, and this process is

equally important for adaptive identity development. Taken together, alternating times of

exploration and commitment occur as identity develops throughout life (Marcia, 1966,

1980, 1993).

One line of research stemming from this tradition examines the relation between

identity exploration and students’ academic motivation (Flum & Kaplan, 2006; Kaplan &

Flum 2010, 2012). Because exploration is the necessary first step for adaptive identity

development (Marcia, 1993), particularly in contemporary society where individuals’

roles and careers change frequently, Flum and Kaplan (2006; Kaplan & Flum, 2012)

argue that fostering identity exploration should be a primary goal of education. To

support this argument, they highlight ways that the characteristics of identity exploration

35

mirror the adaptive characteristics of a mastery goal orientation (Ames, 1992; Dweck &

Leggett, 1988). When students adopt a mastery goal orientation towards their learning,

they focus on developing competence on the task at hand, a focus associated with

adaptive outcomes such as academic achievement, self-efficacy for learning tasks, and

the use of learning strategies (see Urdan, 1997, for a review). Kaplan and Flum (2010)

draw parallels between a mastery goal orientation and identity exploration, emphasizing

that both are characterized by an orientation toward growth and development, feelings of

autonomy and self-determination, and seeking out and openness to new information and

experiences. Given both the similarities between identity exploration and a mastery goal

orientation and the adaptive outcomes associated with the latter, Kaplan and Flum (2010)

argue for identity exploration as an appropriate and important educational goal.

Kaplan and Flum also specify the characteristics of curriculum and instruction

aimed at fostering identity exploration, emphasizing the importance of exploration

triggers, a sense of safety, and scaffolding for exploration (Flum & Kaplan, 2003; Kaplan

& Flum, 2006; Sinai, Kaplan, & Flum, 2012a). Exploration triggers are an experience of

cognitive dissonance (Festinger, 1962) when students recognize an inconsistency or

discrepancy in their self-perceptions. Recognizing this inconsistency prompts students to

engage in exploration, seeking new information relevant to the self-perception (Sinai, et

al., 2012a). For example, the experience of struggling in a chemistry class may serve as

an exploration trigger for a student who identifies as a science person, as might the

experience of enjoying the class unit on The Odyssey for a student who does not identify

as someone who likes Language Arts. Because exploration triggers have the potential to

throw valued identities in flux, a sense of safety and teacher caring in the classroom is

36

important when students experience the uncertainty, both positive and negative, of

identity exploration (Flum & Kaplan, 2006; Sinai et al., 2012a). Finally, scaffolding can

help students to engage in effective exploration by guiding their development with

practices such as reflective discussion questions and writing prompts (Sinai et al., 2012a).

A classroom environment characterized by these qualities together would help promote

identity exploration and adaptive identity development.

Although there is somewhat limited empirical work focused on identity

exploration, the studies that have been conducted reinforce both the link between

exploration and motivational processes and the impact of classroom practices designed to

support exploration. Perez, Cromley, and Kaplan (2014) examined the connection

between identity exploration and motivation, focusing in particular on the relations

between undergraduate STEM majors’ degree of identity exploration and their self-

perceived competence in, valuing of, and cost of majoring in STEM. The Eccles

expectancy-value model (Eccles, 2005, 2009; Eccles et al., 1983; Wigfield & Eccles,

1992) was used as the theoretical framework of achievement motivation, and survey data

were collected in three waves across one academic semester. Using path analysis, Perez

et al. (2014) found that the more individuals have explored their identity before the

beginning of the semester, the more competent they felt in their STEM classes, the more

they valued these classes, and the more they perceived the benefits to be worth the costs

at the end of the semester. From the findings, Perez et al. (2014) suggest that identity

exploration leads to increased value for and perceived competence in the domain of the

identity, in this case STEM. These perceptions in turn lead to adaptive outcomes like

persistence, intention to pursue STEM, and greater performance. Thus, this work

37

provides preliminary support for the connection between identity exploration and

motivational processes.

To explore the impact of classroom practices on identity exploration, Sinai and

her colleagues (2012a) used a design-based qualitative study. In collaboration with a 9th

grade literature teacher, Sinai designed curricular activities to prompt exploration using

exploration triggers, emphasizing a sense of safety, and scaffolding exploration. Sinai

conducted classroom observations while the teacher implemented these activities,

afterwards collecting students’ homework and conducting focus groups with students.

From these data, Sinai and her colleagues concluded that the activities did seem able to

trigger identity exploration in some students. Those students who did engage in some

degree of identity exploration also reported feeling more interested in literature, more

engaged during the literature class, and more motivated to engage with literature outside

of class. Taken together, these two studies lend support to Flum and Kaplan’s (2006;

Kaplan & Flum, 2010, 2012) theoretical account of the relationship between exploration

and motivation. Exploration not only appears related to self-perceptions that shape

motivation (Perez et al., 2014), but can also be prompted by classroom practices and in

turn inspire motivated behavior (Sinai et al., 2012a).

Identity Exploration as Relevance

Identity exploration appears to offer a promising account of relevance. First of all,

it is clear that identity exploration bears on students’ lives outside of school. As was the

case with attainment value, students’ identities move with them through both school and

out-of-school contexts, so the results of identity exploration impact students outside of

school, even if the exploration took place within school. Furthermore, in describing

38

identity exploration, Kaplan and Flum (Flum & Kaplan, 2006; Kaplan & Flum, 2010,

20102) emphasize that students undergo this process in their everyday lives anyway and

highlight ways that exploration helps individuals adapt to changing roles in contemporary

society. Thus, identity exploration is explicitly positioned as impacting students’ lives

outside of school.

Identity exploration also seems content-focused, satisfying the second criterion of

an account of relevance. Identity exploration is prompted by new information, considered

relative to the self, that challenges or calls into question existing self-perceptions. Within

an educational context, new information that students encounter and consider in light of

the self is closely linked with content, and different content seems able to trigger different

sorts of exploration. For example, reading a Greek tragedy and discussing characters’

fatal flaws might prompt a student to consider his or her own character strengths and

weaknesses, whereas learning about suffragettes’ fight for women’s right to vote might

prompt a student to consider his or her own values and willingness to advocate for them.

Given these links with content, identity exploration seems appropriate for an account of

relevance in terms of the second criterion.

Additionally, development is built into both the theoretical conceptualization of

identity exploration and the empirical work examining this process. Theoretically,

exploration itself is a process of development, and empirical work suggests that

exploration triggers, a sense of safety, and scaffolding of exploration are teaching

practices that can prompt and support this development (Flum & Kaplan, 2003; Kaplan &

Flum, 2006; Sinai et al., 2012a). For these reasons, identity exploration clearly satisfies

the third criterion of an account of relevance.

39

Thus far, identity exploration appears to be a particularly promising potential

account of relevance, but it is nonetheless limited on the final criterion of being able to

apply to most content for most students. There are three issues here. First and foremost,

given that identity exploration is prompted by discrepancies between new information

and students’ self-perceptions, it is unclear how to conceptualize information that

confirms students’ self-perceptions or helps them develop new identities. An account of

relevance based on identity exploration would not frame this sort of content as relevant,

even though it clearly seems to bear on students’ lives outside of school in a similar way

as does discrepant information. This issue is critical, because it arises from a defining

quality of identity exploration—the role of cognitive dissonance—and severely limits the

range of content that would be considered relevant to any particular student. Thus, for

identity exploration to present a viable account of relevance, the definition of its causal

mechanism would need to be expanded to include extension and development in addition

to dissonance. In its current instantiation, however, identity exploration would not

provide an account of relevance that could reflect the value of most content for most

students.

Second, underscoring this limitation is the fact that individual students vary

widely in their self-perceptions. Given the range of student experiences and identities,

exploration triggers with some content may resonate with some but not all students, while

triggers with other content will not. It may also not be the case that the same content will

be able to prompt identity exploration for all students. If an exploration trigger fails to

resonate with a particular student, an account of relevance as identity exploration does

not provide other good reasons for the student to engage with the content. As was the

40

case with attainment value, this aspect of identity exploration makes it less educationally

useful as a potential account of relevance, given that it is not feasible for a teacher to

tailor content and instruction to each student’s self-perceptions.

Finally, even though content seems to drive identity exploration, not all content

seems equally likely to prompt identity exploration. Although the humanities and social

sciences seem to hold a great deal of potential for prompting identity exploration, given

their emphasis on human culture and experience, the ways in which mathematics and the

natural sciences could prompt identity exploration is less clear. The most obvious way

these disciplines might encourage identity exploration is in terms of interest and

competence, prompting students to ask whether they are the kind of person who is good

at math or the kind of person who is interested in science. Compared to the many ways

the humanities and social sciences could prompt a student to consider their decision

making and way of being in the world, questions of competence and interest seem like a

limited way of conceptualizing the relevance of mathematics and the natural sciences.

These limitations suggest that although identity exploration is an educational goal worth

striving for, it is not an appropriate grounding for an account of relevance.

Identity-Based Motivation

A different account of identity and motivation from that presented by Flum and

Kaplan (2006; Kaplan & Flum, 2010, 2012) is Oyserman’s (2007, 2009) model of

Identity-Based Motivation (IBM). Under this conceptualization, an individual holds

multiple identities, which range in how central they are and which are often not well-

integrated, but together make up that individual’s self-concept (Markus & Nurius, 1986;

Oyserman, 2007, 2009). Each of the individual identities includes a sense of membership;

41

a readiness to see, interpret, and act on the world in ways perceived as congruent with

that identity; and beliefs about what someone like this would or would not believe, value,

and do (Oyserman, 2009). Additionally, individual identities can be of a number of

different types. As was the case with expectancy-value theory (Eccles, 2005, 2009;

Eccles et al., 1983; Wigfield & Eccles, 1992), identities can be either social or individual,

tying the individual to or distinguishing him or her from a particular group (Oyserman,

2009). Identities can also represent past, present, or future selves, with past identities

incorporated into the present self, which can in turn shape the future identities the

individual aspires to or fears (Markus & Nurius, 1986). Finally, identities are extremely

sensitive to context. A particular context can make some of an individual’s identities

more salient than others or can change the meaning or enactment of a specific identity.

Thus, according to the IBM account (Oyserman, 2007, 2009), identities are multiple,

context-sensitive, and of several broad types.

In the IBM model, identities motivate behavior by guiding individuals to act in

ways perceived to be congruent with their current set of valued identities (Oyserman,

2009) and to close the gap between their present selves and desired future selves (Markus

& Nurius, 1986; Oyserman & Markus, 1990). Empirical work supports the theoretical

relations among identities, motivation, and context (Elmore & Oyserman, 2012;

Oyserman, Bybee, & Terry, 2006; Oyserman, Bybee, Terry, & Hart-Johnson, 2004;

Oyserman & Destin, 2010; Oyserman & Markus, 1990; Oyserman, Terry, & Bybee,

2002). For example, in a study with delinquent youth, these individuals imagined far

fewer positive academic possible identities and far more feared possible identities

compared to their non-delinquent peers (Oyserman & Markus, 1990). Working under the

42

hypothesis that the lack of positive academic identities to guide behavior may contribute

to the development of delinquency (Oyserman & Markus, 1990), Oyserman and

colleagues (Oyserman et al., 2006; Oyserman et al., 2004; Oyserman et al., 2002)

designed an IBM-based intervention targeted at low-income and minority youth at high

risk for delinquency. The intervention helped these students develop positive academic

possible selves and cued academic success as congruent with their valued social

identities. Afterwards, students experienced a range of adaptive academic outcomes,

including higher rates of attendance, improved academic self-regulation, and better

academic performance compared to peers who had not participated in the intervention

(Oyserman et al., 2006; Oyserman et al., 2004; Oyserman et al., 2002). Additional work

underscores the impact of contextual cues on the meaning and enactment of identities.

Elmore and Oyserman (2012) designed environmental information to either cue academic

success as being congruent with being male or with being female. When students of both

genders received cues that academic success was congruent with their gender, they

predicted greater future academic success, reported more possible academic identities,

and persisted longer on a challenging task than when academic success was cued as

incongruent with their gender (Elmore & Oyserman, 2012). These findings suggest that

possible identities do guide behavior and that the content and congruence of students’

identities can be impacted by the school context.

Identity-Based Motivation as an Account of Relevance

The focus in Identity-Based Motivation differs somewhat from the focus in

identity exploration, and as a result, the two also differ in the extent to which they satisfy

the criteria of an appropriate account of relevance. Identity-Based Motivation does seem

43

to satisfy the first criterion of impacting students’ lives outside of school. Identity-Based

Motivation is similar to attainment value in that both conceptualize identities as guiding

behavior, and students act in congruence with their existing identities both inside and

outside of school. Furthermore, the inclusion of social identities as well as future possible

identities links Identity-Based Motivation with students’ everyday lives, given that

valued social groups and future selves occur outside of school as well as inside. These

aspects of Identity-Based Motivation help it meet the first criterion of an appropriate

account of relevance.

It is less clear whether Identity-Based Motivation satisfies the second criterion of

being focused on content. Again in ways similar to attainment value, Identity-Based

Motivation applies just as readily to a student’s identity as a “science person” as to the

identity of “good student.” Whereas “science person” is clearly linked with content,

“good student” is not necessarily. Empirical work on Identity-Based Motivation has

largely explored identities linked with more global academic identities, suggesting that

the theoretical framework itself does not distinguish between identities grounded in

content and less content-focused identities. This is not a problem for the theory in

general; however, when the theory is considered as a possible account of relevance, the

lack of a consistent focus on content becomes problematic.

It is also unclear whether Identity-Based Motivation full satisfies the third

criterion of being something that can be developed. Returning again to the similarities

between attainment value and Identity-Based Motivation, the theoretical account focuses

primarily on how existing identities drive behavioral choices. Yet some of the empirical

work has explored ways of helping students develop positive academic possible selves

44

(Oyserman et al., 2006; Oyserman et al., 2004; Oyserman et al., 2002). This mismatch

suggests that development of identities is not entirely outside of the scope of Identity-

Based Motivation’s theoretical framework. There is little discussion of how this

development occurs within the theory, however, with most of the theoretical work spent

on elaborating how existing identities impact motivation and behavior. Therefore,

although development is not inconsistent with Identity-Based Motivation per se, the

theory does not provide an account of how development occurs. As a result, Identity-

Based Motivation is less educationally useful as a potential account of relevance.

Finally, the issues facing Identity-Based Motivation on the fourth criterion mirror

concerns already raised with other constructs focused on the identities or interests of

individual students. Unless a teacher is prepared to tailor content to each student, it may

not be the case that most content will connect with most students’ identities, given the

variety of individual, social, past, and future identities students bring with them to the

classroom. Thus, Identity-Based Motivation does not seem to provide an appropriate or

educationally useful account of relevance.

Transformative Experience

Another construct that seems related to the relevance of the curriculum is

transformative experience (Pugh, 2002, 2004, 2011), sometimes also called aesthetic

understanding (Girod, Rau, & Shepige, 2003; Girod & Wong, 2002). Broadly, these

kinds of experiences may occur when students apply powerful ideas they have

encountered in school to their everyday lives and come to value the new lens through

which to see the world. This account is strongly shaped by John Dewey’s (1938, 1958,

1980) philosophy of education. In particular, the subjective nature of a transformative

45

experience parallels Dewey’s concept of “an” experience, and the qualities of school

content that can promote transformative experiences draw on Dewey’s concept of ideas.

In Dewey’s account, “an” experience is distinguished from other, everyday

experiences by an aesthetic quality and a sense of meaning. Because interactions with

works of art are particularly likely to prompt experiences of this sort, Dewey uses these

interactions to describe the characteristics of “an” experience in terms of anticipation,

cohesion, and meaning. When watching a film, for example, the developing plot creates a

sense of anticipation, the resolution at the climax creates a sense of cohesion and

fulfillment, and if the work is successful, we may also savor a sense of meaning in having

undergone the experience. Dewey goes on to argue that powerful ideas can also prompt

the build and resolution of anticipation, as well as a sense of fulfillment and meaning

(Pugh & Girod, 2007; Wong et al., 2001). Like works of art, ideas can build a sense of

anticipation by suggesting meaning without stating it outright, inviting further

interpretation and exploration. For example, the idea that turmeric, a common spice often

used in traditional Indian cooking, slows the growth of cancer cells more effectively than

many modern drugs is intriguing. Similarly, learning that logging yields decreased when

the early German state cleared the forest underbrush and planted high-yielding trees in

orderly, easy-to-manage rows is counterintuitive. These ideas about cellular biology and

ecosystems produce anticipation, which in turn encourages exploration of these ideas and

may result in an experience.

Building on Dewey’s framework, education scholars (Pugh, 2011; Wong et al.,

2001) suggest that curricular content may be a source of ideas and of experiences, and

that prompting these sorts of aesthetic educational experiences should be one aim of

46

education. Pugh (2011) terms these moments transformative experiences,

operationalizing the construct as “a learning episode in which a student acts on the

subject matter by using it in everyday experience to more fully perceive some aspect of

the world and finds meaning in doing so” (p. 111). More specifically, the process of

undergoing a transformative experience requires three components. The first component

is motivated use, which occurs when students actively and of their own accord seek out

opportunities to apply concepts they have learned in school to their everyday lives. When

students apply concepts, they may come to see the world in new depth or nuance as a

result, and this expanded perception is the second component of transformative

experience. Finally, students may feel a sense of experiential value for the new

perspective that arose from applying school content.

Empirical work focused on transformative experience is still preliminary but

suggests that students do engage in transformative experiences (Girod et al., 2003; Girod,

Twyman, & Wojcikiewicz, 2010), although these experiences are relatively uncommon

without support from teachers (Pugh, Linnenbrink-Garcia, Koskey, Stewart, & Manzey,

2010b). To better understand what aspects of transformative experience are easier or

more difficult for students to engage in, Pugh and colleagues (2010b) surveyed students

in high school biology classes and used Rasch analysis to analyze students’ scores. Rasch

analysis allows the researcher to assign survey items a difficulty. This difficulty

corresponds to the composite score a student would need to achieve on the measure in

order to agree with that item. For example, in the case of transformative experience, only

students with high scores on the transformative experience measure would be expected to

agree with a difficult item, whereas most students, except those with the very lowest

47

scores, would be expected to agree with an easy item. When the transformative

experience survey items were ranked in terms of difficulty, survey items that asked

students to report their motivated use, expanded perception, and experiential value in

class ranked low on the continuum—most students agreed with these items, regardless of

their scores on the measure (Pugh et al., 2010b). On the other hand, items that asked

about students’ engagement with these dimensions of transformative experience outside

of class ranked high on the continuum—only those students with very high scores on the

measure agreed with these items (Pugh et al., 2010b). These results suggest that it is more

difficult for students to engage in motivated use, expanded perception, and experiential

value outside of class than in class. Given that transformative experience is defined

explicitly as occurring outside of school (Pugh et al., 2010b), these results also indicate

that most students do not experience full transformative experiences.

To address the gap between students’ in-class and out-of-class experiences, Pugh

and his colleagues (Girod et al., 2003; Pugh & Girod, 2007; Pugh et al., 2010a, 2010b;

Wong, 2007) have developed pedagogy to foster transformative experience. In this

approach, the instructor models the three dimensions of transformative experience,

sharing examples from his or her own life and providing class time for students to

practice the three components (Pugh et al., 2010b). The empirical work conducted thus

far has focused on science classrooms, and in this context, the pedagogy is termed

Teaching for Transformative Experience in Science (TTES; Pugh et al., 2010b).

Compared to peers in a control group, students taught using TTES report a greater

number of transformative experiences at the end of the intervention and at a one-month

follow up (Girod et al., 2003; Girod et al., 2010; Heddy & Sinatra, 2013; Pugh, 2002).

48

Post-tests of understanding suggest that students taught using TTES also undergo a

greater degree of conceptual change and come away with a deeper understanding of the

content (Girod et al., 2010; Heddy & Sinatra, 2013; Pugh, 2002; Pugh et al., 2010b).

Other factors that emerged as important predictors of engaging in transformative

experiences included having basic knowledge of the subject, endorsing a mastery goal

orientation focused on developing that basic competence (e.g., Ames, 1992; Dweck &

Leggett, 1988), and identifying with science (Pugh, 2004; Pugh et al., 2010b). These

predictors shed light on other factors that might impact the effectiveness of TTES.

Although preliminary, these studies suggest that TTES may help students develop the

skills and dispositions required to engage in transformative experiences, a central goal of

education according to Dewey (1938; 1958; 1980) and this group of scholars (Pugh,

2011; Wong, 2007; Wong et al., 2001).

Transformative Experience as Relevance

Of the constructs considered so far, transformative experience seems the most

appropriate as an account of relevance. First of all, transformative experience is explicitly

focused on making an impact on students’ lives outside of school, to the point that the

process of undergoing motivated use, expanded perception, and experiential value is not

considered a transformative experience unless it occurs outside of the classroom. As

such, transformative experience clearly meets the first criterion of an appropriate account

of relevance.

Secondly, it is also clear that transformative experience is closely connected to

content. Transformative experiences are fundamentally instances of students using the

content in their everyday lives. Whether it is information about turmeric that leads to a

49

transformative experience with cellular biology or learning about German forestry that

prompts a transformative experience with environmental studies, content acts as the

ideas, in Dewey’s terminology (1938, 1958, 1980), that prompt feelings of anticipation

and lead to transformative experience. Thus, transformative experience also meets the

second criterion of an account of relevance.

Additionally, both theory and empirical work on transformative experience make

it clear that the skills and dispositions to engage in transformative experience can be

developed. Theoretically, teachers play an important role in helping students learn how to

engage in transformative experiences, and research on Teaching for Transformative

Experience in Science suggests that sharing examples, modeling the three components,

and giving students an opportunity to practice applying content are effective ways of

helping students engage in transformative experiences (Pugh et al., 2010b). Unlike some

of the other constructs considered thus far, transformative experience does not start with

preexisting interests or self-perceptions the student holds, but rather, like identity

exploration, focuses explicitly on how the ability to engage in transformative experiences

develops. This quality of transformative experience makes it an appropriate account of

relevance according to the third criterion.

Finally, transformative experience is the first construct that seems to apply to

most content for most students. Most content, if included in the curriculum, should have

the potential to impact students’ lives outside of school in some way. Transformative

experience aims to highlight ways students can use the content to expand their perception

of everyday life and experience value in that deepened perspective. Although students

who identify with particular content are more likely to experience transformative

50

experience with it (Pugh, 2004; Pugh et al., 2010b), a student’s ability to engage in

transformative experience is not limited by whether or not they identify with the content.

Unlike the other constructs, transformative experience is not theoretically defined in

terms of students’ individual, existing perceptions. Therefore, transformative experience

seems to meet all four criteria of an appropriate account of relevance.

Appreciation

A final construct that may present an appropriate grounding for an account of

relevance is Brophy’s concept of appreciation. Brophy (1999, 2008a, 2008b) frames the

value of school content in terms of its ability to be authentically applied to students’ lives

outside of school. However, these applications extend far beyond the direct transfer of

practical skills. Instead,

Reading and writing are not just basic skills needed for utilitarian applications but

gateways to interest development, identity exploration, self-expression, and other

enrichments to individual’s subjective lives. Similarly, basic geographical,

historical, social, and scientific understandings are not isolated bits of inert

knowledge but key components of schema networks that individuals use to

understand and respond to the social and physical world. Well-developed K-12

content not only has narrowly construed utilitarian value (helping people meet

their basic needs and wants) but enriches the quality of their lives by expanding

and helping them articulate their subjective experiences. (Brophy, 2008a, pp.

138–139)

Under Brophy’s account, school content that affords outcomes like these has life

application value (1999, 2008a, 2008b). When students recognize the life application

51

value of content, they may then develop appreciation for that content. In contrast with

many accounts of intrinsic motivation, which emphasize the pleasurable, affective

qualities of the subjective experience, Brophy (1999, 2008a, 2008b) argues that

appreciation is more cognitive in nature, better described in terms of “absorption,

satisfaction, recognition, making meaning, self-expression, self-realization, making

connections, achieving insights, aesthetic appreciation, and so on” (Brophy, 2008a, p.

137). Thus, both the nature of content with life application value and the nature of the

subjective experience of appreciation differ somewhat from other conceptualizations of

relevance.

In order for students to recognize life application value and to develop

appreciation, they must possess propositional, procedural, and conditional knowledge of

the content (Brophy, 2008a). Conditional knowledge—knowing where, when, and

particularly why to apply the content—in particular is crucial for the development of

appreciation, but is often lacking from the school curriculum (Brophy, 2008a).

Furthermore, although valuable, the life application value of content is not always

obvious to students, or even to teachers. This is especially true of more abstract

disciplines such as the humanities and social sciences, which do not possess the same

straightforward utility value as basic math or reading (Brophy, 1999, 2008a, 2008b). In

line with this observation, Brophy (2008a) also emphasizes that helping students

recognize the life application of content is not synonymous with trying to tie content back

to students’ current interests. Rather, this account “applies most directly when the value

of the learning is not obvious to the learners” (Brophy, 2008a, p. 134). Although making

connections with interest can be a powerful motivational tool, Brophy’s account (2008a)

52

focuses instead on scaffolding students’ development toward the propositional,

procedural, and especially the conditional knowledge necessary for appreciation. For

these reasons, teachers play an important role in helping students recognize, act upon, and

experience the life application value of curricular content in order to develop appreciation

for it.

Little empirical work has been conducted on appreciation. One study (Knogler &

Lewalter, 2014) examined whether a game-based simulation requiring students to solve

an authentic, real-world problem could increase high school students’ appreciation for

science. T-tests suggested that students’ value of science increased from pre- to post-test

in a student-centered version of the simulation, but not in a teacher-centered version.

Although this finding seems consistent with the theoretical framework of appreciation, it

is not clear whether the items used to measure appreciation mapped well onto the

construct. Half of the items were published in German and so were inaccessible. The

other items were taken from the Changes in Attitude about the Relevance of Science

scale (Siegel & Ranney, 2003), but of these items, some seemed more closely aligned

with utility value (e.g., “science class will help me prepare for college”), others with

attainment value or identity (e.g., “I am interested in a career as a scientist or engineer”),

and still others with interest (e.g., “I am interested in learning more about computer

technology and designing video games”). Given the important theoretical distinctions

between appreciation and these other constructs, the use of items such as these to

measure appreciation is problematic and limits the degree to which this study can inform

the theoretical account of appreciation.

53

Another study that explicitly discusses appreciation (Turner, Christensen, Kackar-

Cam, Trucano, & Fulmer, 2014) took place in a middle school and examined to what

extent an intervention changed the way teachers supported student engagement, as well

as how teacher practices and student engagement interacted with each other over time. In

the conceptual framing of this study, appreciation was grouped with other constructs such

as utility value, identity, and interest under the more general term of “meaningful

learning.” In the observational data that were collected, teacher practices were rated in

terms of how much they supported meaningful learning, as well as belongingness,

competence, and autonomy. Under the coding scheme, practices were given the highest

rating for meaningfulness when the teacher provided rationales for the activities, taught at

a high conceptual level, and connected the activity to big ideas (Turner et al., 2014).

Therefore, both at the conceptual level and at the level of measurement, “meaningful

learning” is too broad to map directly onto appreciation. Even though the final results of

the study are compelling—suggesting distinct patterns of growth in teacher practice and

different practices associated with more and less meaningful learning—they cannot

provide direct evidence to support the theoretical conceptualization of appreciation. In

sum, little if any empirical work has been conducted with an explicit focus on

appreciation.

Appreciation as Relevance

Despite the lack of empirical evidence, appreciation as a theoretical framework

meets all four criteria of an appropriate account of relevance. In terms of the first

criterion, appreciation is defined explicitly in terms of an impact on students’ everyday

lives. Grounding appreciation in the life application value of content makes this

54

connection overt. In this way, appreciation aligns well with the first criterion of an

appropriate account of relevance.

Appreciation is also directly linked with content, deriving from the life

application value of the content itself, rather than arising from a feature of the task.

Brophy (1999, 2008a, 2008b) emphasizes that content must be chosen for inclusion in the

curriculum based on its potential life application value, making appreciation’s grounding

in content clear. As such, appreciation satisfies the second criterion.

Furthermore, Brophy (1999, 2008a, 2008b) frames appreciation as something that

must be developed, meeting the third criterion. To emphasize this aspect of appreciation,

Brophy (1999, 2008a, 2008b) argues that it applies most directly when students do not

already recognize the value of content. More specifically, the development of

appreciation requires students to acquire propositional, procedural, and especially

conditional knowledge of when, where, why, and how to use the content in their

everyday lives, and teachers play a central role in helping students develop these three

types of knowledge.

Finally, appreciation is conceptualized as applying to most content for most

students. Appreciation is grounded in the affordances of the content, and content is

chosen based on its broad life application value. Brophy (1999, 2008a, 2008b) is also

clear that appreciation does not require connecting with students’ existing identities or

interest, emphasizing with this point that content should be chosen so that most students

might recognize its life application value. Given that life application value should have

the potential to be recognized by most students and that content is chosen for its life

application value, appreciation clearly satisfies the fourth criterion of an appropriate

55

account of relevance. Thus, like transformative experience, appreciation appears to offer

a robust conceptualization of relevance.

Grounding Relevance in Appreciation

Of the constructs considered, both transformative experience and appreciation

meet the criteria for an appropriate account of relevance. When the two are compared,

however, appreciation seems to be a broader and thus a more educationally useful

grounding for relevance. Transformative experience focuses ultimately on the outcome of

experiential value resulting from expanded perception, whereas appreciation includes a

much wider range of experiences. Brophy suggests that

powerful ideas expand and enrich the quality of students’ subjective lives. They

provide lenses through which to construe their observations and experiences,

schemas into which they can assimilate novel elements, connections they can

make and draw inferences from, potential for recognizing and appreciating the

aesthetic qualities of the objects or events they encounter. (Brophy, 2008a, p. 140)

Conceptualizing relevance in terms of appreciation would capture a greater variety of

ways that content could impact students’ lives outside of school and so would be a more

educationally useful framing of relevance.

Thus, appreciation appears to reflect a unique experience, different from task

values, interest, identity, and transformative experience, and to present a more

appropriate grounding for an account of relevance. Under this account, content would be

chosen for the curriculum based on its life application value and teachers would aim to

help students recognize this value and develop appreciation. In particular, teaching

56

practices would focus on helping students develop the propositional, procedural, and

conditional knowledge needed to bring the content to bear in their everyday lives.

Appreciation also has the advantage of acting as a precursor to some of the other

constructs considered as possible accounts of relevance. Brophy (2008a) explicitly names

interest development and identity exploration as potential outcomes of engaging with

content that has life application value. In this way, an account of relevance grounded in

appreciation would allow for the development of attainment value, interest, and identity

exploration from an initial recognition of life application value. But rather than beginning

with these more specific and individualized outcomes, an account of relevance based in

appreciation would aim first for life application value that most students would be able to

recognize, even if they did not pursue it to the point of developing an interest or identity

around it.

However, there is still a significant gap between the theoretical account of

appreciation and more concrete recommendations for teaching practices aimed at making

content relevant to students. One reason for this gap, despite the promise of Brophy’s

account, is the lack of empirical work supporting the theoretical account, detailing

students’ lived experiences of appreciation, and exploring teaching practices designed to

highlight life application value and develop appreciation. Without work of this kind,

appreciation will struggle to gain traction in current practice. Thus, in order to tap into

appreciation’s potential, further work must focus on elaborating the construct and

building a rigorous base of empirical support for it.

57

Chapter 3

What College is For: Exploring College Students’ Lived Experience of Appreciation

In the wake of the economic crisis of 2008 and increasing student loan debt,

debate about the worth of a college education has escalated (Belkin, 2014; Bond, 2015;

US News, 2011; Weston, 2015). Recent research has added to this concern. According to

a study conducted across 24 institutions of higher education and 2,300 undergraduates,

36% of the students demonstrated little if any growth across their four years of college in

their critical thinking, complex reasoning, or writing skills—skills considered a basic and

fundamental part of a college education (Arum & Roska, 2011). Furthermore, in the

annual, nationally-representative Gallup-Purdue University survey of 30,000 recent and

older college graduates, only 50% strongly agreed that their undergraduate education was

worth the cost (Gallup, 2015). Among recent college graduates, only 38% strongly

agreed, likely reflecting the fact that they graduated into a less stable job market (Gallup,

2015). Additionally, as individuals’ amount of student loan debt increased, they were less

likely to agree that college was worth it (Gallup, 2015). Thus, both direct measures of

student learning and self-reported satisfaction indicate a growing debate about the value

of higher education.

Although there seems to be agreement about the need to make college more

worthwhile, there is a great deal of disagreement about how to go about doing so. For

example, some call for a greater emphasis on transferable job skills, with more focus on

58

the cost of a degree and the link between that degree and a job (U.S. Department of

Education, 2015)—an emphasis that often results in universities cutting majors that are

less clearly associated with career skills, such as philosophy, classics, and American

studies (Jenvey, 2016; Kent, 2016; Zernike, 2009). On the other hand, critics of this view

argue that a core curriculum designed to help students develop the complex reasoning,

cultural literacy, and writing skills necessary for success in the workplace and as

responsible citizens would need to reinvigorate general education courses and draw from

across the disciplines (Clune, 2015; Lemann, 2016; Lewin, 2013; Paxson, 2013). Other

solutions range from defining and measuring essential learning outcomes that cut across

disciplines (Association of American Colleges & Universities, 2014), to adopting a

competency-based curriculum that would allow students to advance as they master

material (Berrett, 2015), to using online courses and digital badging as an alternative to

traditional credentialing (Bull, 2014).

This disagreement about how to make college more worthwhile, despite a shared

concern with doing so, suggests that the “worth” of a college education may not be well

defined or clearly agreed-upon. “Worth” seems to reflect an evaluative judgment about

the outcomes of a college education, and ambiguity in the understood end of a college

education could lead to the apparent disagreement in the means of achieving that end. For

example, defining the end in terms of job skills might suggest vocational training as a

means, whereas defining the end in terms of a cost-effective credential might suggest

online courses and digital badging as a means. A college education that focuses on

vocational training would be less “worthwhile” if its ultimate goal was to provide a cost-

effective credential, as would a college education that focuses on online courses and

59

digital badging if its desired end was to confer job skills. In short, for the conversation

about making college more worthwhile to be productive, it is important to first have a

shared, explicit, and well-defined understanding of the goal of higher education—what it

is that would make it worthwhile.

Many of the proposed means of making college more worthwhile seem implicitly

grounded in the idea that what students learn in college should be relevant to their lives in

some way: job skills, cultural literacy, critical thinking, complex reasoning, and writing

skills are all outcomes of a college education that bear on students’ lives outside of

school. With relevance as the end of a college education, curricular content would be

considered worthwhile to the extent that it could somehow be made relevant to students’

lives outside of school. Different approaches to making college more worthwhile would

be appropriate means to the extent that they helped students recognize this relevance.

Although relevance seems to be a shared goal among these perspectives, the vast

disagreement in how to achieve this goal suggests that relevance is currently neither an

explicitly held goal nor well defined. Because relevance appears to undergird various

perspectives on the worth of higher education, explicitly framing worth in terms of a

clearly-articulated account of relevance could provide a shared criterion for evaluating

means of making college more worthwhile.

Theoretical Accounts of Relevance

Although there is not currently a theoretical account of relevance per se within the

educational psychology literature, there are several constructs that could provide the basis

for such an account. For example, perhaps it is most appropriate to think about the value

students see in the content or task (Eccles, 2005, 2009; Eccles et al., 1983; Wigfield &

60

Eccles, 1992). Alternatively, perhaps the content can serve as a means for interest

development (Hidi & Renninger, 2006), or as a way for students to explore and act on

their sense of identity (Flum & Kaplan, 2006; Kaplan & Flum, 2010, 2012; Oyserman,

2007, 2009). On the other hand, relevance may best be conceptualized in terms of helping

students apply and see the content in action in the real world, valuing that transformative

experience (Pugh, 2002, 2004, 2011). Or perhaps it is best to conceptualize relevance as

helping students develop appreciation for the content’s ability to enrich their lives

(Brophy, 1999, 2008a, 2008b). Theoretically, these constructs seem to reflect a degree of

relevance or impact on students’ lives outside of school, and many have been used as

proxies for relevance in empirical work. Although this set of constructs certainly shares a

family resemblance around relevance, the construct of appreciation (Brophy, 1999,

2008a, 2008b) offers the most robust account of relevance that could provide a shared,

explicit, and well-defined understanding of the goal of higher education.

Appreciation as an Account of Relevance

According to Brophy (1999, 2008a, 2008b), content should be included in the

curriculum based on its ability to be authentically applied to students’ lives outside of

school. In this account, however, authentic applications extend far beyond practical uses

of skills like reading and writing. Rather, authentic application can also

enrich the quality of students’ subjective lives, providing them with lenses

through which to construe their observations and experiences, schemas into which

they can assimilate novel elements, connections that they can make and draw

inferences from, potential for recognizing and appreciating the aesthetic qualities

of the objects or events that they encounter, and so on. (Brophy, 2008b, p. 40)

61

School content that has the potential to be authentically applied to students’ lives outside

of school is said to hold life application value. Although much school content holds this

value, students often do not initially recognize it (Brophy, 1999, 2008a, 2008b). The

instructor’s role then becomes helping students recognize the life application value of

content. When students come to see this value, they may experience appreciation.

Brophy (1999, 2008a, 2008b) distinguishes appreciation from similar constructs like

interest or intrinsic motivation by emphasizing its cognitive, rather than purely affective,

nature. The subjective experience of appreciation is better described in terms of

“enrichment, enablement, and empowerment” (Brophy, 2008b, p. 40) rather than

pleasure, fun, enjoyment, or interest. Furthermore, the experience of appreciation requires

more investment on the part of the student and is unlikely to result from mere exposure to

content (Brophy, 1999, 2008a, 2008b).

Instead, appreciation is most likely to result when students possess the

propositional, procedural, and conditional knowledge about where, when, how, and

especially why to apply content in their everyday lives (Brophy, 1999, 2008a).

Conditional knowledge is particularly instrumental in the development of appreciation,

but it is often lacking from the curriculum (Brophy, 2008a). Additionally, the reasons

why and opportunities to apply content in everyday life are more straightforward for

some areas of study, like mathematics or literacy, than for other, more abstract disciplines

in the humanities and social sciences. Thus, making curriculum relevant, under this

account, would involve helping students develop appreciation by recognizing content’s

life application value. More specifically, teaching practices would focus on helping

students develop the propositional, procedural, and conditional knowledge about when to

62

use content, with particular attention paid to illuminating the life application value of less

intuitively relevant disciplines.

There are at least four aspects of appreciation that make it a strong grounding for

an account of relevance. First of all, as an account of relevance, appreciation impacts

students’ everyday lives. As a result of engaging with content and developing the three

types of knowledge, students are able to see or act differently in their lives outside of

school. An account of relevance that resulted in no impact on students’ lives after

engaging with content would be difficult to defend. Appreciation presents a convincing

account by focusing explicitly on how students will be different as a result of engaging

with content and positioning this change in students’ everyday lives.

Second, appreciation and life application value are also inextricably linked to

content. Grounding relevance in content, rather than in the nature of the tasks themselves

or qualities of individual students, also makes appreciation a strong account of relevance.

The curriculum is fundamentally part of an education and uncoupling the outcomes of

education from the content would be difficult to justify. By grounding life application

value explicitly in content, appreciation presents an appropriate account of relevance.

A third reason that appreciation offers a robust account of relevance is that it can

be developed. This quality seems fundamental for an account of relevance. Otherwise,

education would be limited to those disciplines that students already value. Brophy

(1999, 2008a, 2008b) explicitly frames appreciation as a value that must be developed,

emphasizing that the argument for appreciation applies most directly when students do

not already recognize the value of content. Given that the value of much of the content

63

included in the curriculum is not immediately obvious to students without the assistance

of the instructor, appreciation also seems apt as an account of relevance for this reason.

A final reason that appreciation seems an appropriate grounding for an account of

relevance is that it applies to most content for most students. Although relevance in its

common usage often refers to connections with an individual’s existing interests, goals,

or priorities, an educationally useful account of relevance would emphasize the potential

ways the content could apply to almost any student’s life. Pragmatically, it is unfeasible

to tailor instruction to every student’s unique set of goals and priorities. Philosophically,

starting with students’ existing values rather than helping them come to recognize the

potential of content to impact their lives would not ground relevance in the content or

frame relevance as something that must be developed. Appreciation, by framing the

potential worth of content broadly in terms of life application value, rather than narrowly

in terms of an individual student, seems applicable to most content for most students.

Taken together, appreciation offers a robust account of relevance because it applies to

students’ lives outside of school, it is derived from the content, it can be developed, and it

applies to most content for most students.

Despite appreciation’s promise as an account of relevance in higher education,

currently little empirical research has been conducted on the construct. First of all, more

work is required to better understand how students experience learning about content

with life application value and what the characteristics of the subjective experience of

appreciation are. Second, if evidence for the benefits of appreciation can be produced, a

subsequent question would explore what sorts of content and teaching practices can

facilitate these experiences of appreciation. Research that details both the subjective

64

experience of appreciation and ways of teaching that help students recognize content’s

life application value would provide both the end of and the means to make higher

education more worthwhile.

This study aims to address the first of these two goals— to develop a deeper

understanding of students’ subjective experience of appreciation, how students recognize

what makes school content worth learning, and how appreciation develops. Because little

is currently known about these experiences, the study took a qualitative approach aimed

at exploring these experiences through interviews with undergraduate students about their

worthwhile educational experiences.

Method

Participants

Undergraduates at a large public university in the Midwest were recruited to take

part in this study (see Appendix A). A total of approximately 1,329 students were invited

to participate in an online screening survey, of whom 127 participated (reflecting a 9.55%

response rate); from these responses, 15 students were invited to participate in individual

interviews. This number was chosen for the interview sample based on guidelines from

qualitative theory and from empirical work. In qualitative methodology, the appropriate

number of participants is determined using the concept of saturation, that is, the point of

redundancy in the information being gathered (Glaser & Strauss, 1967; Lincoln & Guba,

1985) when no new themes emerge from additional data (Jones, Arminito, & Torres,

2006). When participants are sampled based on a shared experience, 10 to 15 participants

are recommended to achieve saturation (Creswell, 2012; Moustakas, 1994; Roulston,

2010; Van Manen, 1990). Empirical work echoes this guideline, suggesting that in

65

studies using in-depth interviews about a shared experience, saturation may be reached

after 12 interviews (Guest, Bunce, & Johnson, 2006). Thus, 15 students were invited to

participate in semi-structured interviews.

Both the sample of individuals who participated in the online survey and the

sample of interview participants were fairly representative of the larger university

population (see Table 3.1). In both the survey and interview samples, however, there was

a greater proportion of women, white students, and education majors compared to the

university as a whole. This sampling bias is likely a result of the method of recruitment

(see below). Of the 226 majors offered at the university, 50 were represented in the

survey sample and 12 were represented in the interview sample. Finally, the average age

of the survey sample was 21.11 (SD = 3.21) and the average age of interview participants

was 20.47 (SD = 2.07).

Procedure

Initial recruitment. Recruitment took place during the Spring 2015 semester and

consisted of two phases. The first phase involved the online survey (see Appendix B).

Students were recruited to participate through announcements about the study in several

classes. Instructors known by the researcher or known through an acquaintance of the

researcher were contacted and asked their permission for the researcher to announce the

study during class. A total of 21 instructors agreed to let the study be announced in class.

Because some of these instructors taught multiple sections, the researcher visited a total

of 24 classes. Instructors emailed out the study announcement to an additional four

classes that were either held online or that the researcher was unable to attend. Education

courses represented 19 of the 28 courses in which recruitment took place. The large

66

Table 3.1 Demographics of the University, Full Study Sample, and Interview Sample

Undergraduate Population (42,843)

Full Study Sample (127)

Interview Sample (15)

Gender Percent N Percent N Percent Female 47.7% 90 70.9% 9 60% Male 52.3% 36 28.3% 6 40% Self-defined No data 1 0.8% - -

Race/Ethnicity White 71.3% 109 85.8% 12 80% Ethnic Minority 17.3% 18 14.2% 3 20%

African American 4.3% 6 4.7% 2 13.3% Asian American 5.7% 6 4.7% - -

Hispanic 3.5% - - - - American Indian/Alaskan Native 0.1% - - - -

Native Hawaiian/Pacific Islander 0.1% 1 0.8% - - Two or More Races 2.6% 5 3.9% 1 6.7%

International 7.7% - - - - Unknown 2.7% - - - -

Year

Freshman 9.3% 14 11% 1 6.7% Sophomore 21.5% 32 24.2% 5 33.3% Junior 25% 40 31.5% 5 33.3% Senior 43.8% 29 22.8% 4 26.7% First Generation Status

First Generation ~21% a 24 18.9% 3 20% Non First Generation ~79% a 103 81.1% 12 80%

Continued Note. Data for undergraduate population from The Ohio State University Enrollment Services (2015). Dashes indicate no participants in the category. a Estimate based on the number of first-generation students in the Autumn 2014 freshman class (The Ohio State University Enrollment Services, 2014)

67

Table 3.1 continued

Undergraduate Population (42,843)

Full Study Sample (127)

Interview Sample (15)

Major Percent N Percent N Percent Education Major 8.4% 43 33.9% 5 33.3% Non Education Major 91.6% 84 66.1% 10 66.7%

Self-reported GPA

M (SD) 3.43 (0.42) 3.52 (0.42) number of education courses represented in the sample and the large percentage of white

women enrolled in education courses may account for the bias seen in the final samples.

The other nine classes in which recruitment took place were offered in psychology (two

courses), physics, anatomy, leadership, geography, health sciences, wellness, and

English. When visiting these classes, the researcher used the first or last five minutes of

class to introduce the study and to answer any questions. Participants were emailed a

reminder about the survey one week and two weeks following the in-class announcement.

After the survey had closed, one winner was randomly selected and given a $100 Visa

gift card for participating in the study.

Selecting interview participants. Interview participants were selected using

purposive sampling (Jones et al., 2006) based on their responses to the online survey.

After students gave their consent to participate in the study, they were asked to provide

demographic information and to indicate if they had taken a course during their time as

an undergraduate that felt worthwhile or relevant to their lives outside of school. Out of

the total sample, 122 participants indicated that they had taken a worthwhile class and 5

68

indicated that they had not. Students who indicated that they had taken a worthwhile class

were then asked if that course had been outside of their major area of study. Students

were then asked in three short answer questions to describe the nature of their experience

(see Appendix B). Students’ responses to these short answer questions were used to select

interview participants.

As survey responses were collected, the data were reviewed and participants were

flagged as potential interview participants using the following criteria. First of all,

participants were prioritized as potential interview participants if they had an experience

that felt worthwhile in a course outside of their major, particularly in a class from a

discipline very different from their own. This was done to reduce the likelihood that

students would describe an experience aligned with an existing interest or a utilitarian use

of the content—experiences that would be less representative of appreciation.

Second, to increase the variation within these similar experiences, participants

were selected to represent a range of majors and worthwhile courses in a variety of

disciplines. In addition to being selected for diversity of contexts, participants were

selected to represent a range of demographic characteristics. Although generalizability is

not necessarily a central goal of qualitative approaches, the interview sample was

selected to mirror the larger university population to help ensure that the variety of

perspectives present at the university were represented. Demographic characteristics were

also used to help counterbalance the perspectives represented by the selection of

interview participants. For example, both a male education major and female education

majors as well as both a female and a male engineering major were included in the

sample.

69

Finally, participants were flagged as potential interview participants if their

written survey responses were somewhat elaborated, suggested some degree of

reflectiveness about the value of the course, and made it clear that they had taken the

survey seriously. This is in contrast with other participants who provided very short

answers, who expressed the value of the course in terms of it being easy or entertaining,

or who did not elaborate on what made the content feel worthwhile.

These criteria were revisited throughout the process of selecting interview

participants and were used to guide subsequent decisions about whom to invite (see Table

3.1 for the demographics of the final interview sample and Table 3.2 for interview

participants’ majors and discipline of the worthwhile courses). Thus, the 15 final

interview participants represented exemplary examples from among the large sample. An

additional 14 participants were identified as possible interview participants based on their

short answer responses. Based on the responses of the other 93 participants who indicated

they had taken a worthwhile course, it is not clear whether they had had similarly rich

experiences but were simply not as articulate or whether their experiences did not reach

the same depth as the exemplary cases.

Interview procedures. Semi-structured interviews were conducted during the

Spring 2015 semester while survey data were still being collected. The same set of

questions was used with each interviewee (see Appendix C), helping to ensure that the

same aspects of experience were discussed in every interview. The structure of the

interviews was also flexible, allowing interviewees to bring in information and

experiences that they felt were relevant and allowing the interviewer to follow up, to ask

70

Table 3.2 Interview Participants' Major and Discipline of Worthwhile Course

Major

Discipline of Course

Arts and Sciences General Information Organic Chemistry

Biomedical Science Comparative Studies – Religious Diversity

Education - Early Childhood Education Theater

Education - Early Childhood Education Communications

Education - Early Childhood Education Biology

Education - Middle Childhood Education Earth Science

Education - Middle Childhood Education History

Electrical and Computer Engineering Russian Literature

Engineering Physics Classics

English History of Art

Health Sciences Program Comparative Studies – Philosophy of Technology

Materials Science and Engineering Comparative Studies – Art Education

Mathematics Biology

Pre-Occupational Therapy Comparative Studies

Psychology Women’s, Gender, and Sexuality Studies for further elaboration, and to clarify the meaning of interviewee’s comments (Charmaz

& Belgrave, 2013). Interviews were conducted individually and audio recorded.

Interviews ranged from 24 to 58 minutes in length and lasted 39 minutes on average. At

the end of the interview, each participant was given $20 for participating in the study.

Interviews were then transcribed, and NVivo 10 was used to code and analyze the

transcripts.

Analysis

The selection of an analytic approach was guided by the exploratory nature of the

research question. Because there is currently very little, if any, empirical literature

71

focused on students’ subjective experience of appreciation, an inductive analysis

approach (Thomas, 2006) rooted in grounded theory (Corbin & Strauss, 2008; Glaser &

Strauss, 1967) was chosen for the current study. In contrast with deductive approaches

that use data to test existing theory, this inductive approach involves thoroughly

exploring the data as a way to develop theory. Thus, an inductive approach is an

appropriate way to explore students’ lived experience of appreciation.

The process of inductive analysis involved a number of steps, which began with

conducting a close reading of the data, with the purpose of orienting the researcher to the

participants’ experience (Corbin & Strauss, 2008) and to major themes in the data

(Thomas, 2006). After a close reading of the data, transcripts were coded, with the goal

of paying close attention to the raw data and resisting the urge to move immediately to

higher-level categories that encapsulate multiple lower-level codes (Berg, 2001). Memo

writing and code comparison were used to fully engage with the data during the process

of coding. Writing memos provided an opportunity for the researcher to reflect on the

interpretation of the code or data, to identify key properties or characteristics of that code,

to ask questions of the data, and to consider how the occurrence of that code compared to

other occurrences (Berg, 2001; Charmaz & Belgrave, 2013; Corbin & Strauss, 2013).

Comparing different occurrences of the same code was another strategy used to engage

with the data (Corbin & Strauss, 2008; Glaser & Strauss, 1967), with the goal of

illuminating codes’ defining features when occurrences were similar or creating new

codes when occurrences of the same code differed in meaningful ways (Thomas, 2006).

Next, codes were grouped together under high-level categories as part of the process of

data reduction and moving from description toward theory (Corbin & Strauss, 2008;

72

Thomas, 2006). This involved consolidating similar codes, clarifying differences in

overlapping but distinct codes, and creating subthemes under broader categories (Berg,

2001; Thomas, 2006). Once the codes were grouped into categories, the categories were

integrated into an overall model, described below.

Finally, two different measures were taken to ensure that the findings were

trustworthy (Lincoln & Guba, 1985) and that the interpretation was justified by the data.

To check the clarity of the codes and the coding of the data, a second coder was given the

set of codes developed by the first coder and asked to apply them to a portion of the

dataset. This reliability sample was made up of 50 randomly-selected instances of coding,

representing 29% of the total number of codes. Thus, the sample met the 50-unit

minimum and exceeded the 10% of the full sample minimum for a reliability sample

(Neuendorf, 2002). The agreement in coding was 72%, which met the 70% threshold for

acceptable agreement defined by some (Multon, 2010) but failed to meet the 80%

threshold set out by others (Miles & Huberman, 1994). The lower agreement observed in

the current study may be due in part to the exploratory nature of the study. In the case of

exploratory studies like this one, the threshold for acceptable agreement may be set

lower, at 70% (Lombard, Snyder-Duch, & Bracken, 2002). Thus, the coding scheme was

considered cautiously appropriate within the context of this initial exploration of

appreciation.

The second measure taken to ensure the trustworthiness of the data was providing

sufficient evidence for the study’s conclusions. This was done to avoid imposing

interpretations that were not warranted by the data; what Glaser and Strauss (1967) term

73

exampling. To provide sufficient evidence, several strong examples were provided to

support every broad interpretation presented in the final write up (Berg, 2001).

Results

Through the process of coding, comparing codes, combining similar categories,

and grouping codes under broad themes, three overarching categories with subthemes

were developed to describe students’ subjective experience in courses that felt

worthwhile to their lives outside of school. The three categories encapsulated cognitive,

behavioral, and affective aspects of students’ subjective experience. These categories not

only reflected the groupings that arose from the data, but they have also been used to

describe other subjective experiences in educational contexts such as engagement

(Fredericks, Blumenfeld, & Paris, 2004) and transformative experience (Pugh, 2011).

This connection lends the categories some theoretical weight, particularly because these

groupings emerged from the data and were not defined a priori.

Cognitive Aspects

The most commonly-reported aspect of students’ experiences in worthwhile

classes had to do with changes in how they saw the world (see Table 3.3). Because these

changes involved new perceptions, new knowledge or understanding, and new ways of

thinking, they were grouped under the broad category of cognitive aspects of students’

experiences in worthwhile classes. Three more specific themes reflected this broader

category.

Developing a broader perspective. The first theme, developing a broader

perspective, captured students’ experience of seeing some aspect of the world in light of

the content of the class. This was also the most commonly reported cognitive aspect of

74

Table 3.3 Cognitive Aspects: Themes and Examples

Theme Interviews Present In Example

Developing a Broader Perspective

13

[The class] gave me new perspective on hope… Prometheus didn’t just give us fire, he gave us hope. Or, as the Greeks interpreted it, false hope, because...when we’re faced with our own mortality…he gave the humans false hope so that we can look beyond that. And so…whenever I’m hopeful about something I always think, “is this false hope or is this something that I actually think could happen?””

Undergoing a Fundamental Change

5 I apply it to everything…I think back then I would just...simply take a side, and kind of be like “he’s wrong,” right and wrong, right there. Now…I analyze it... instead of just taking my view about it, my personal view, I think “how would this person think about it?”

Self-questioning 6 We really talked about what she called “buffet style” religion. So you kind of pick and choose what you believe, and that challenged me personally ‘cause I was like “how am I not doing that when I’m taking this class? When I’m challenged with something and then I make a decision on how I feel on that subject? Is that not a buffet style religion?”

these experiences, occurring across 13 of the 15 interviews. On the most basic level,

some students reported identifying class content in their everyday lives. One occupational

therapy student in a comparative studies course reported that “watching TV shows that I

watch, [I thought] ‘oh, I learned that in class.’” In a similar vein, a mathematics major in

75

a biology course said “I was talking with my grandma on the phone, and she had cancer

last year, and so I was like ‘I finally understand what’s going on in your body.’”

Other students took a more active role in using the content in their everyday lives,

as opposed to some of these connections that seemed to occur spontaneously to students,

without them taking an active role in using the content. For example, a physics major in a

classics course actively thought about hope differently after taking the course. As he

described,

Prometheus didn’t just give us fire, he gave us hope. Or, as the Greeks interpreted

it, false hope, because...when we’re faced with our own mortality…he gave the

humans false hope so that we can look beyond that. And so…whenever I’m

hopeful about something I always think, “is this false hope or is this something

that I actually think could happen?”

Similarly, an English major in a film class pointed out that “I wouldn’t be bringing my

mise-en-scène analysis to [normal movies]…but when I watch pretentious movies by

myself, I will try to use those tools and put it in context of where film was as a medium.”

Students like these took a more active role in making connections between what they

learned in class and aspects of their everyday lives. The greater agentic role these

students took suggests that their experiences in these courses were more powerful—

prompting them to seek connections—than the experiences of students who were not

prompted to seek further connections. Thus, most students valued the connections

between course content and their lives outside of school, but these experiences varied in

the extent to which they compelled students to actively put the content to use in their

everyday lives.

76

Undergoing a fundamental change. Other students reported a fundamental

change in their perspective on the world as a result of taking the course. This differed

from the broader perspective described above in that these students could not imagine

seeing the world in the same way ever again. The change felt irreversible, rather than like

a lens that could be put on or taken off. For a health sciences major who took a

comparative studies class about the history and philosophy of technology, the class

practice of examining different perspectives on an issue fundamentally changed how he

approached everything from “politics,…technology, race, [to] sports.” As he described it,

I apply it to everything…I think back then I would just...simply take a side, and

kind of be like “he’s wrong,” right and wrong, right there. Now…I analyze it...

instead of just taking my view about it, my personal view, I think “how would this

person think about it?”

This student did not simply come out of the class with a deeper understanding of the

ways technology impacts society. Rather, in his own words, “I’m very happy that I took

that class and I went through this transformation.” This is compared to some of his other

classes, in which, as he described, “I didn’t really transform…[and they] didn’t impact

the way that I think about anything else.” An engineering major reported a similarly

broad change in how he approached problem solving after taking a Russian literature

course. He reported that “that class has helped me just to be open to everything and just

really analyze everything critically and bring it all together.” When asked to describe

ways this manifests in his everyday life, he pointed out that “when I go to read an

engineering textbook, now I sort of read it like it’s a literature textbook…I really pick

apart the problems, like how the author approached that problem…I try to pick up on

77

those subtleties.” This student perceives this approach as informing his work as well. As

he accounted,

right now, I’m working on a control panel…with that, you’re working with

different types of components, different types of materials, and to pick out the

right components…you have to be really open to analyzing everything and

learning how to bring it all together again.

These experiences of feeling fundamentally changed seemed to reflect a deeper and more

dramatic impact on students’ lives. As such, they were also less common than the

expanded view of the world that most students reported: only five students described

experiences of being fundamentally changed by a course.

Self-questioning. A final type of changed perspective that students reported was

being prompted by a class to question some aspect of their pre-existing beliefs or

behavior, which occurred for six of the 15 interviewees. For example, a premed student

in a comparative studies course about religious diversity who was “born and raised in

Protestant churches” found herself thinking critically about her own beliefs:

We really talked about what [the professor] called “buffet style” religion. So you

kind of pick and choose what you believe, and that challenged me personally

‘cause I was like “how am I not doing that when I’m taking this class? When I’m

challenged with something and then I make a decision on how I feel on that

subject? Is that not a buffet style religion?”

For the occupational therapy student, a piece of literature in her comparative studies

course “made me think about how I treat other people and how other people treat me,

even though that type of relationship that they had in the story…had nothing to do with

78

relationships in my own life.” Compared with content that helped students develop a

broader perspective or fundamentally changed their view, the content of the classes

described by these students seemed inescapably self-relevant. The degree of questioning

students experienced could also be quite dramatic. In perhaps the clearest example of this

type of experience, a psychology major found herself questioning core aspects of her

behavior and identity while taking a Women’s, Gender, and Sexuality Studies course,

what she described as “an eye-opening experience.” She recalled learning that “the media

plays a big role in how I think and how they portray women…and now, I’m changing

things, like ‘do I really like pink?’ Or ‘do I really love wearing heels and skirts?’…It

makes me rethink everything.” She described asking herself similar questions about other

aspects of her beliefs—”‘do I have a single story for all transgender or all gay people?”

and “I grew up thinking ‘oh, I’ll be submissive to my man, and when he come [sic] home

I’m going to cook and clean’…but now I’m thinking ‘why can’t he cook? Why can’t he

clean?’ [Laughs].” For students like these, something they learned in the course prompted

them to reconsider or critically examine a core belief about themselves or the world. The

depth of these experiences seemed similar to experiencing a fundamental change in

perspective, but the view that changed seemed much more central to the individual’s

sense of self.

Behavioral Aspects

Students also reported behaving differently as a result of taking these worthwhile

courses (see Table 3.4). As was the case with cognitive aspects of the subjective

experience of appreciation, behavioral aspects varied in their degree of change, with

some behaviors requiring significantly more investment from students than others. The

79

Table 3.4 Behavioral Aspects: Themes and Examples

Theme Interviews Present In Example

Talking with Others about the Course

11

My boyfriend came over and I’m like “I can tell you how each commercial has some kind of sexist underlying message to it,” and he was like “no you can’t.” So then I start doing that and he’s like “I didn’t think about that.”

Applying Content Directly

6 I’ve been trying to clean this pan, where I burned um, a lot of sugar and salt into it when I was cooking…I went to Google and they were like “oh, use bicarb and vinegar,” and then…I was trying to think like “oh! What are the actual products of this reaction going to be and what kind of energy release could potentially be playing a role in actually breaking up all the material that’s that’s burned on here?”

Extending Content 4 So, in Russian Literature, … one of their prominent poets is Alexander Pushkin. And I did a little digging on him, and it turned out his great-grandfather was captured by the Ottoman Empire and he was Ethiopian… so this poet was a direct descendant of an Ethiopian. I’m Ethiopian, so that kind of...I didn’t know that my culture was that rich, so it was pretty exciting.

three types of behavioral aspects included talking with others about the course, directly

applying the content, and extending the content.

Talking with others about the course. Of the three behavioral aspects of the

experience of these courses that students reported, talking with others about the content

was by far the most common, with 11 of the 15 interviewees describing at least one

example. These conversations often happened with roommates or peers who were also in

80

the class. For example, the occupational therapy student recounted that “[my] roommate

took a lot of similar classes, [so] I found myself discussing what I had learned in

comparative studies with my roommate who was studying communications.” Other

students did not seem able to contain their excitement about what they were learning. For

example, an engineering major in an art education course recalled that

I would go home, and I would want to talk to my roommates about the things that

we had talked about, just because they were good conversations and they really

applied to your life…I was like “wow! We were talking about feminism and this

girl shared this story about when this happened to her!”

Her enthusiasm was mirrored in the account of an education major taking a

communications course on media and terrorism. As she described,

my roommates would be like “I don’t know that I really care about this,” but I

would want to share it…sometimes if I learn about new things, it’s just exciting to

hear people’s opinions. When we talked about terrorism, I would ask my

roommates “what do you think terrorism is?”…and I’d be like “have you ever

thought that terrorism could be somebody who’s not Muslim?”

In spite of the apparent lack of interest sometimes shown by conversation partners,

students like these seemed compelled to share what they were learning about, in part

because of how exciting the new learning was to them.

The aspect of this learning that students wanted to share often seemed to reflect

the explanatory power of the content. For example, the psychology major describing her

Women’s, Gender, and Sexuality Studies course seemed to want to demonstrate the

power of her new feminist lens on the world when she recounted,

81

My boyfriend came over and I’m like “I can tell you how each commercial has

some kind of sexist underlying message to it,” and he was like “no, you can’t.” So

then I start doing that and he’s like “I didn’t think about that.”

Similarly, the physics major in the classics course drew on a myth from class in a

conversation with his mother. The reading was about

Prometheus, and he’s chained to the rocks and he’s talking to these furies. But I

suggested that to her, just because I read it in that class and I thought “wow, this

actually really applies to your life right now, and I think you’d enjoy reading

this.”

For these students, talking with others about the content did not seem to be simply about

sharing their enthusiasm for interesting content, but rather about sharing the explanatory

power of the content with others.

Applying content directly. Less common than talking with others about the

content were reports of applying the content, which appeared in six of the interviews.

These represented instances in which students somehow used the content in the context

of their everyday lives. The clearest example of this was described by an Arts and

Sciences major in an organic chemistry course:

I’ve been trying to clean this pan, where I burned a lot of sugar and salt into it

when I was cooking…I went to Google and they were like “oh, use bicarb and

vinegar,” and then…I was trying to think like “oh! What are the actual products

of this reaction going to be and what kind of energy release could potentially be

playing a role in actually breaking up all the material that’s burned on here?”

82

This student was actively drawing on content from his organic chemistry class to decide

how best to clean the pan. An education major in an earth science course reported a

similar use of content in her everyday life. After having to visit a ravine near campus as

an extra credit project, this student recounted that “me and one of my good friends in the

class…go back every time it’s nice out.” During these excursions, “we talk about [the

course]…like ‘oh, this is the side, do you see that, where it erodes…this is shale, this rock

is shale,’ so we still bring up certain stuff from the class.” Here, the student was using the

content in the context of these visits, describing what she saw in terms of the course

content. Finally, revisiting the example of the engineering major in the art education

course provides another, slightly different example. As she described it,

I am in an organization, the Girls’ Circle Project…and I met a girl in high school

who was talking about how her best friend was pregnant and she was trying to

figure out how…to help her, without being like “you shouldn’t have done this.”

So I felt like my insight from this class on how to approach people in tough

situations that are very different from mine, I could share that knowledge with this

younger girl.

While not as straightforward an application as the use of organic chemistry to clean a

pan, this example nonetheless seems to represent a direct use of course content to the

engineering major describing the experience. These examples demonstrate ways that

students used the content more actively than simply talking about it with others.

Extending content. Taking students’ use of the content one step further were

examples of students using the content in everyday life, beyond direct applications. This

use of the content encapsulated examples of the students behaving differently after

83

engaging with the content, often in domains of their lives not directly linked with the

content. For some students, this looked like pursuing more information about the content

beyond what was required for their coursework. For example, the engineering major in

the literature class explained that

in Russian Literature, … one of their prominent poets is Alexander Pushkin. And

I did a little digging on him, and it turned out his great-grandfather was captured

by the Ottoman Empire and he was Ethiopian… so this poet was a direct

descendant of an Ethiopian. I’m Ethiopian, so that kind of...I didn’t know that my

culture was that rich, so it was pretty exciting.

Similarly, the occupational therapy student recounted that “having taken comparative

studies and talking to my roommate about the stuff I was learning in comparative studies

and seeing how she could relate to that material, that’s what made me take

communications as another [general education requirement].”

Other examples of using the content, however, represented less literal extension

of the content. For the engineer in the art education course, being one of the only women

in her engineering courses, “I’ve had to become more comfortable vocalizing my

opinion, because I’m the only girl and I felt very, not silenced, but intimidated…[N]ow

that I’m more comfortable with that [after taking this course,] I feel like I am more

myself.” As another example, the health sciences major in the comparative studies course

perceived that the course

really helped me go from simple to complex. I don’t jump to conclusions…when I

witness an argument…even between friends…I kind of just [say] “hey, cool it.

84

Now hey, you, think about why he’s disagreeing with you…he might be right or

wrong on the situation, but this is why he thinks that way.”

When asked how often he has had to break up arguments like this, he said “oh my gosh,

countless—countless times. And I think I wouldn’t have been able to if I hadn’t gone

through experiences like this [course].” Given the greater degree of investment on

students’ part and the presumed greater impact of the course on students who extended

their use of the content, these experiences were reported with much less frequency,

represented in four of the 15 interviews. Students extended the content in their behavior

in different ways, some pursuing more information or exposure to the topic and others

perceiving their behavior in other domains to be drastically different as a result. Thus, the

content had varying degrees of impact on students’ behavior, ranging from low

investment behaviors like talking with others about the content to applying the content

directly to extending their use of the content to gathering more information or acting

differently in another domain.

Affective Aspects

The final aspect of the subjective experience of appreciation that students reported

encapsulated affective experiences, reflecting students’ values and emotional experiences

(see Table 3.5). There were at least two aspects of affect that students reported.

Valuing experience. First, most students ascribed value to their experience in

these courses. For the education major in an earth science course,

I actually really enjoyed it, because a lot of the stuff we talked about…I could

apply to my daily life…We got to look at different earthquakes that already

happened, which was kind of cool to find out what’s going on to cause it.

85

Table 3.5 Affective Aspects: Themes and Examples

Theme Interviews Present In Example

Valuing Experience

15

I notice [plants] in the area differently. It’s fun relating it to our little narrow view of where we come from, because I’m from Ohio, and I’ve always been from Ohio, and [being able to say] “that doesn’t belong,” because we had this whole section over invasive species.

Perceiving New Maturity

9 [The professor] explained the difference between [Suni and Shiite] within the Islam faith, and…it was really eye-opening, because if you talk to anyone about that now, you’re going to get some kind of bias…[but] now I can approach any of those conversations that I have with a more knowledgeable base. Even if I don’t remember all the details, I still am aware that there is actually something more that just politics going on there.

Another education student in a theater course echoed a similar sentiment, suggesting that

“seeing the job of a stage manager or dramaturge…that’s not really something I ever

thought about…so it’s cool to think about what goes on with the lighting and the tech.”

The engineer in the art education course seemed to have an even broader experience,

recounting that “I think that the fact that I was thinking about the class when I wasn’t in

the class and not doing work for it really meant a lot to me.” Finally, the Arts and

Sciences major in the organic chemistry described the value as

the feeling of “oh, I’m somewhat conversant in organic chemistry…that’s in my

tool box now,” and I can pull that out…when I read something or I’m in another

class and I need to think about why something happens the way it does, just the

86

fact that it’s in my toolbox I think is the thing that sticks with me the most…It

was like learning a new kind of math almost, …or like a new language.

All 15 of the students who were interviewed reported valuing their experience in these

courses, and six of the 15 suggested that classes like these are “what college is supposed

to be for,” as the psychology major in the Women’s studies course commented.

According to the male engineering student, the Russian literature course “really

complemented my education,” and for the female engineering student, “the art education

class was a really great first Gen Ed for me…when you think about college…that’s really

what somebody might think of.” Comments like these underscore the depth of the value

students appeared to ascribe to their experiences in worthwhile classes.

Perceiving new maturity. The second affective experience that students reported

was a sense of having grown up or developed a new feeling of maturity as a result of

taking the course, with nine students reporting an experience of this kind. For example,

the education major in the communications course reflected that

I feel like it’s helped [me] to transfer more into an adult…like when I was little, I

never liked to watch the news…so it’s just interesting to see how I’ve become at

least somewhat interested in things that I never would have thought that I wanted

to be interested in.

Being better acquainted with content that seemed important and feeling more confident in

talking about it seemed to be part of the mechanism helping students feel more mature.

The experience of the education major in the history course reflects this as well. As he

described,

87

[the professor] explained the difference between [Suni and Shiite] within the

Islam faith, and…it was really eye-opening, because if you talk to anyone about

that now, you’re going to get some kind of bias…[but] now I can approach any of

those conversations that I have with a more knowledgeable base. Even if I don’t

remember all the details, I still am aware that there is actually something more

that just politics going on there.

Another example underscoring the role of being able to talk with others is the

occupational therapy student, who suggested that having taken the course “helps in

general conversations, just like being knowledgeable about the things that we talked

about…talking to my parents or other adults…I just felt like it was really helpful to be

knowledgeable of that…it was just a building block.” Finally, the engineer in the art

education course underscored how understanding the particular content felt important,

given that the course was “around social constructs…and I think that’s such a big deal in

our society right now…and to learn about it in a formal setting makes you feel more

prepared for what you might encounter in real life.” Thus, students’ affective experiences

included both a sense of value for the content as a lens in their everyday lives and a

newfound maturity that resulted from being familiar with content that felt important.

Variation by Student and Discipline

The cognitive, behavioral, and affective dimensions of worthwhile experiences

emerged as strong themes; however, students and disciplines varied in the extent to which

they reported and prompted these worthwhile experiences. As Table 3.6 makes clear,

some students reported only a few aspects of these worthwhile experiences and others

reported most, if not all, aspects. On the low end of the spectrum, the mathematics major

Tabl

e 3.

6 O

ccur

renc

e of

The

mes

by

Cou

rse

Dis

cipl

ine

Cog

nitiv

e A

spec

ts

B

ehav

iora

l Asp

ects

Aff

ectiv

e A

spec

ts

Br

oade

r Pe

rspe

ctiv

e Fu

ndam

enta

l C

hang

e Se

lf-qu

estio

ning

Ta

lk w

ith

Oth

ers

Appl

y C

onte

nt

Exte

nd

Con

tent

Va

lue

Expe

rien

ce

Perc

eive

New

M

atur

ity

Nat

ural

Sc

ienc

es

Org

anic

Che

mis

try

Bio

logy

B

iolo

gy

Earth

Sci

ence

H

uman

ities

H

isto

ry

Thea

ter

Cla

ssic

s

R

ussi

an L

itera

ture

H

isto

ry o

f Art

Soci

al

Scie

nces

C

omm

unic

atio

ns

Art

Educ

atio

n

C

ompa

rativ

e St

udie

s

Te

chno

logy

R

elig

ious

Div

ersi

ty

Wom

en’s

Stu

dies

88

89

in the biology course only reported experiencing two of the eight aspects, and on the high

end of the spectrum, the health sciences major in the philosophy of technology course

reported experiencing seven of the eight aspects of worthwhile experiences. Most

students reported experiencing about four of the eight aspects. This raises the question of

how individual differences play into students’ experiences in worthwhile classes—how

much of this experience is due to the way the course is taught versus the way students

approach it? To what extent might the same course have a similar impact on different

students? Further research could explore what aspects of students’ approach to learning

might make them more likely to experience these valued outcomes.

Disciplines also seemed to vary in the aspects of experience that students

reported. For example, referring again to Table 3.6, students reported fewer instances of

undergoing a fundamental change in perspective, self-questioning, extending the content,

and perceiving new maturity in natural science courses compared to humanities and

social sciences courses. When it came to a broader perspective and valuing the

experience of taking the course, however, students in classes across disciplines reported

experiencing these outcomes. This suggests that worthwhile courses in general may help

students see the world more broadly and experience value, but that specific disciplines

may be better suited than others for producing some of the other reported outcomes. For

example, it seems possible that humanities and social science courses, with their focus on

human culture and the human experience, may be able to prompt self-questioning in

students more easily than courses in the natural sciences. In contrast, the natural sciences

may lend themselves more readily to helping students apply the content directly, given

their strong ties with the concrete, physical world, than do the more abstract humanities.

90

Discussion

The goal of this study was to develop a deeper understanding of students’

experience in worthwhile courses. This deeper understanding could, in turn, be used to

inform the theoretical account of appreciation and to help clarify the debate about what

makes college worthwhile. Taken together, the findings suggest that students do find

some of their college coursework relevant to their lives outside of school. Furthermore,

students’ experiences in these worthwhile courses share some common qualities.

Cognitive, behavioral, and affective aspects of these experiences were well represented

among the 15 interview participants, as were the eight themes within these three larger

categories (see Table 3.6). As such, these categories and themes, grounded in students’

actual experiences and reflecting aspects of appreciation, offer a starting point for

thinking about the sorts of experiences that make college feel worthwhile.

The results of this study contribute to the educational psychology literature by

providing support for Brophy’s (1999, 2008a, 2008b) account of appreciation. Brophy

(1999, 2008a, 2008b) introduced the theoretical account of appreciation, but as they

stand, his ideas are still largely conceptual. There is little if any empirical work providing

evidence for Brophy’s account, nor has there been additional theoretical work extending

the account. The current study begins to lend some empirical and theoretical support to

Brophy’s argument. Empirically, the findings indicate that students do have experiences

that fit Brophy’s (1999, 2008a, 2008b) account of appreciation: appreciation is not simply

a theoretical construct. The study also contributes additional theoretical nuance to

Brophy’s account. The three aspects of students’ worthwhile experiences that emerged in

this study provide a preliminary picture of what the subjective experience of appreciation

91

looks and feels like. In his account, Brophy (1999, 2008a, 2008b) describes the subjective

experience of appreciation very broadly, as “absorption, satisfaction, recognition, making

meaning, self-expression, self-realization, making connections, achieving insights,

aesthetic appreciation, and so on” (Brophy, 2008a, p. 137). The three aspects of

appreciation and the eight subthemes add detail to Brophy’s broad descriptors.

This study also makes three primary contributions to the debate about what makes

college worthwhile. First of all, the current study provides a framework that can be used

to establish a shared, explicit, and well-defined understanding of the goal of higher

education. Importantly, this framework is grounded in students’ live experiences, lending

validity to the definition. While Brophy’s account of appreciation (1999, 2008a, 2008b)

provides a preliminary understanding of relevance as an educational goal, the results of

the current study help to spell out the defining qualities of experiences that would meet

this goal, according to the students who have experienced them. The findings also

provide evidence that relevance in general and appreciation in particular are an accurate

and appropriate way to frame the goal of higher education. Not only did these qualities

derive from students’ lived experiences, but many participants also explicitly articulated

that experiences in worthwhile courses like these are “what college is for.” Thus, the

current study presents a conceptual definition of the goal of higher education

fundamentally grounded in and validated by students’ lived experiences.

Although the current study suggests that many students do have experiences of

appreciation, it also indicates that these experiences are not ubiquitous. Highlighting the

variation in students’ experience is the second contribution the study makes. Among the

127 students who participated in the study, only 29 students seemed to have had

92

experiences that truly and fully exemplified appreciation—certainly the 15 interview

participants and perhaps the 14 other students identified as potential interview

participants. The other 98 participants either indicated that they had not had a worthwhile

experience (n = 5) or were not sufficiently elaborate or descriptive in their responses to

make clear whether they had had a worthwhile experience (n = 93). Acknowledging the

uncertainty about these responses, only 29, or about 23%, of the students in the total

sample reported a clear experience of appreciation. If the goal of education is framed in

terms of appreciation, this finding underscores the concern about whether students are

having worthwhile experiences in college.

Many students do not seem to experience appreciation in the course of their

undergraduate work, but patterns in the coursework described by the interview

participants point to where these experiences may be occurring—the third contribution of

the study. As highlighted in Table 3.6, the majority of experiences of appreciation

reported by the interview participants occurred in humanities and social science courses.

Furthermore, these experiences overall also seemed richer—reflecting more aspects of

appreciation—than did the experiences reported in the natural sciences. Although only

suggestive, these results are important when framed within the current debate about what

makes college worthwhile. With the growing emphasis on job skills (Jenvey, 2016; Kent,

2016), student debt and future income (U.S. Department of Education, 2015), and

alternatives to traditional credentialing (Bull, 2014), the humanities are often cut from the

curriculum to make room for other, “more worthwhile” courses (Zernike, 2009). Yet it is

these very courses that students describe as what college is for. Thus, many current

approaches to making college worthwhile may in fact be undermining the experiences

93

and value they hope to increase. Taken together, the results of the current study provide

empirical evidence for Brophy’s (1999, 2008a, 2008b) theoretical model, offer a

reasonably specific and concrete picture of what these worthwhile experiences look like,

and inform the debate about what makes college worthwhile.

Limitations

Although suggestive, the results of the current study are limited by the size and

nature of the sample and the reliability of the coding. The interview sample was

intentionally selected to represent a wide range of experiences, but it is possible that the

themes presented here nonetheless do not represent a full picture of worthwhile

experiences in college. This could be the case, first, because participants were drawn

from the undergraduate population of one large public state university. It is possible that

students in other educational contexts, such as small private liberal arts colleges, would

report different or additional aspects of these experiences. Second, the students in the

interview sample reported on existing college courses, which may not necessarily be

optimized for producing experiences of appreciation. It is possible that students might

experience other, additional worthwhile outcomes not represented here if their courses

were designed more explicitly to have value in students’ lives outside of school.

A second limitation of the current study is the low inter-rater coding agreement.

As described before, this may be due in part to the exploratory nature of the study. It is

also possible that in new exploratory work like this, the second coder may require a more

thorough introduction to the codes and their meaning than was given in the current study.

The central implication, however, from the low agreement is that the model of

appreciation developed here could benefit from being further refined and articulated.

94

Directions for Future Work

To further develop the model of appreciation, future work could explore

experiences of appreciation occurring in a wider range of contexts and for a greater

number of participants. For example, examining the experience of appreciation described

by students at liberal arts colleges may highlight aspects of appreciation not articulated

by the participants in this sample, as would interviewing students with different majors

and worthwhile courses than those of the current participants. Additionally, exploring the

differences between the experience of appreciation across disciplines more systematically

could yield insight into the qualities of content or teaching practices that promote these

experiences. Examining the experience of appreciation across contexts and participants

would help solidify the defining qualities of these experiences. A clearer picture of these

qualities would also make it possible to develop a measure of appreciation, which would

be a necessary first step towards conducting quantitative work focused on the construct.

An ultimate goal would be to explore the types of teaching practices and classroom

characteristics that promote experiences of appreciation, with the aim of designing

college coursework to promote the type of experiences that college is intended to provide.

95

Chapter 4

Developing a Measure of Students’ Subjective Experience of Appreciation

Colleges and universities are under increasing pressure to demonstrate their

relevance in response to rising student loan debt (US Department of Education, 2015; US

News, 2011), high-profile examples of successful college dropouts like Steve Jobs, Bill

Gates, and Mark Zuckerberg (Williams, 2012; Zimmer, 2013), and disagreement about

the extent to which college curricula apply to students’ future careers or everyday lives

(Clune, 2015; Lemann, 2016; Paxson, 2013). Exacerbating this debate is the lack of an

agreed-upon definition of relevance, either at the level of public discourse or within the

educational psychology literature. Clarifying the definition of relevance within the

educational psychology literature could help change theoretical questions into empirical

ones, bringing data to bear on the debate about what makes college worthwhile.

However, a conceptual definition is not sufficient. For empirical work on relevance to be

conducted, a well-validated measure of relevance is also required. Thus, the current study

argues for appreciation (Brophy, 1999, 2008a, 2008b) as an appropriate conceptual

definition of relevance and presents the development of a new, preliminary measure of

appreciation.

Conceptual Definitions of Relevance

Within the educational psychology literature, there exist a number of similar

constructs that have sometimes been used as proxies for relevance. These include task

96

values, which represent how useful, interesting, or self-relevant an individual perceives

an achievement task to be (Eccles, 2005, 2009; Eccles et al., 1983; Eccles & Wigfield,

1995); interest, which can reflect the connection between an achievement task and an

existing interest or new interest kindled by the task (Hidi & Renninger, 2006; Renninger

& Hidi, 2011; Linnenbrink-Garcia et al., 2010); identity exploration, which represents

ways an achievement task connects with, affirms, or develops an individual’s sense of

self (Flum & Kaplan, 2006; Kaplan & Flum 2010, 2012); transformative experience,

which occurs when students use school content to see the world in new ways and value

their expanded perception (Girod & Wong, 2002; Pugh, 2011; Pugh & Girod, 2007); and

appreciation, which occurs when students recognize, value, and act on the ways that

school content can be brought to bear on their lives outside of school (Brophy, 1999,

2008a, 2008b).

Of these constructs, Brophy’s account of appreciation (1999, 2008a, 2008b)

appears to offer a particularly promising account of relevance. In this account, content

has the potential to be relevant when it can be authentically applied to students’ lives

outside of school. To use Brophy’s (1999, 2008a, 2008b) terminology, such content holds

life application value. For students to recognize this value, they must possess the

propositional, procedural, and especially conditional knowledge of where, when, how,

and especially why to apply school content to their everyday lives. When students come

to recognize the life application value of school content, they may develop appreciation

for that content. Brophy (2008a) argues that the subjective experience of using these three

types of knowledge and experiencing appreciation can be characterized in terms of

97

“absorption, satisfaction, [and] self-realization” (p. 133). The explicit grounding of

appreciation in life application value suggests that it may provide an appropriate account

of relevance.

Measures of Relevance

Despite appreciation’s promise, however, there has been little if any empirical

work conducted using appreciation as a theoretical framework, in part because of the lack

of a well-established measure of appreciation. The amount of time required to develop

and validate a new measure begs the question of whether appreciation justifies

developing a new set of items, rather than using existing measures of very similar

constructs such as identity (Sinai et al., 2012b), task value (Eccles & Wigfield, 1995),

interest (Harackiewicz et al., 2008; Hulleman et al., 2010; Linnenbrink-Garcia et al.,

2010), or transformative experience (Pugh et al., 2010a). Despite the existence and

frequent use of these measures, however, none were developed and validated in the most

psychometrically rigorous way. Table 4.1 highlights a series of important analytical

decisions made during instrument development and compares existing measures and a

potential new appreciation measure in terms of the appropriateness of their approach.

Development of items. All of the existing measures were developed by content

experts, based on definitions of the construct in the literature. This deductive approach is

the most common method of item development and relies heavily on previous theoretical

conceptualizations of the construct of interest (DeVellis, 1991; Hinkin, 1995, 1998).

Tabl

e 4.

1

Com

pari

son

of H

ow M

easu

res D

evel

oped

Con

stru

ct

Item

D

evel

opm

ent

Cor

rela

tion

Mat

rix

Fact

or

Extra

ctio

n M

easu

re

Ref

inem

ent

Rel

iabi

lity

Iden

tity

Expl

orat

ion

(Sin

ai, K

apla

n, &

Flu

m, 2

012b

)

Expe

rt —

—

—

α

Task

Val

ue

(Ecc

les

& W

igfie

ld, 1

995)

Expe

rt —

K

aise

r crit

erio

n Fa

ctor

load

ings

α

Initi

al In

tere

st

(Hul

lem

an e

t al.,

201

0)

Expe

rt —

K

aise

r crit

erio

n Fa

ctor

load

ings

α

Trig

gere

d &

Situ

atio

nal I

nter

est

(Lin

nebr

ink-

Gar

cia

et a

l., 2

010)

Expe

rt —

Pa

ralle

l ana

lysi

s Fa

ctor

load

ings

α

Tran

sfor

mat

ive

Expe

rien

ce

(Pug

h et

al.,

201

0a)

Expe

rt N

/A

Ras

ch P

CA

Ite

m m

isfit

and

re

dund

ancy

R

asch

Appr

ecia

tion

Gro

unde

d th

eory

Po

lych

oric

Pa

ralle

l ana

lysi

s Ite

m m

isfit

and

re

dund

ancy

Ras

ch

Not

e. D

ashe

s ind

icat

e st

eps i

n m

easu

re d

evel

opm

ent t

hat w

ere

not r

epor

ted

in th

e pu

blis

hed

wor

k.

98

99

Alternatively, a less common approach to item development involves asking individuals

from the target population to describe the relevant aspect of their experience or behavior,

using content analysis or grounded theory to identify themes, and developing items to

reflect these themes (Hinkin, 1995, 1998; Smith, Fischer, & Fister, 2003). One advantage

of this inductive approach is that the resulting items are more likely to reflect most or all

elements of the construct of interest, rather than being limited by a theoretical definition

that may or may not reflect the entire domain of the construct (Smith et al., 2003). A

second advantage is that the items developed through an inductive approach reflect the

way the population of interest conceptualizes the construct, rather than how the construct

is defined by theory (Smith et al., 2003). As such, the inductive approach may result in

measures with greater content validity, compared to the deductive approach (Hinkin,

1995, 1998). Thus, the appreciation measure will be developed using an inductive,

grounded theory approach with the aim of capturing more of the content domain than

other previous measures developed by content experts using a deductive approach.

Correlation matrix. Once items are developed for a measure, the next step is to

compute a correlation matrix that will be used for factor extraction, to assess the

dimensionality of the measure, and eventually for factor analysis, to explore the factor

structure of the measure. The default correlation coefficient used in these analyses is the

Pearson product-moment correlation coefficient, but this correlation coefficient is

problematic when used with ordinal data (Babakus, Ferguson, & Jöreskog, 1987; Carroll,

1961). The Pearson correlation coefficient assumes that the data are continuous, so when

used with ordinal data, a range of latent factor scores are assigned to the same observed

100

variable category, thereby reducing variability. Thus, Pearson correlation matrixes

generated from ordinal data underestimate the associations between variables (Gilley &

Uhlig, 1993) and, when used as input matrixes in factor analysis, result in attenuated

parameter estimates (DiStefano, 2002; Olsson, 1979b). All of these measures collect

ordinal survey data and none specify the correlation coefficient, suggesting that

correlation matrixes used the Pearson correlation coefficient.

The more appropriate correlation coefficient to use with ordinal data is the

polychoric correlation coefficient. By estimating relationships between variables using

the bivariate frequency distribution (crosstab) of the observed ordinal scores, under the

assumption of latent bivariate normality, these relationships are closer to what the

correlations would be if the data were measured on an interval scale (Brown, 2015;

Olsson, 1979a). Therefore, when used for the input matrix in factor analysis, the

polychoric correlation coefficient results in more accurate, less attenuated parameter

estimates (Holgado–Tello, Chacón–Moscoso, Barbero–García, & Vila–Abad, 2010).

Thus, a new measure of appreciation developed using a polychoric correlation coefficient

would reflect a more appropriate approach than existing measures.

Factor extraction. The correlation matrix is then used to extract factors,

indicating how many latent variables define the measure. There are a number of ways to

determine how many factors to retain. Some of the existing measures use the Kaiser

criterion, which is one of the most popular methods of factor extraction and involves

retaining any eigenvalues extracted from the data that are greater than 1.0 (Bandalos &

Boehm-Kaufman, 2008). However, this method tends to overextract factors and has been

101

found to be one of the least reliable factor extraction methods (Hayton, Allen, &

Scarpello, 2004; Velicer, Eaton, & Fava, 2000; Zwick & Velicer, 1986).

A more appropriate method of factor extraction is parallel analysis, which retains

factors that explain more variance than is expected by chance, rather than using an

eigenvalue cutoff like the Kaiser criterion (Brown, 2015). Parallel analysis plots the

eigenvalues calculated from the original dataset against the eigenvalues from a randomly

generated data set that matches important characteristics of the original dataset, including

sample size and number of variables. This process of generating eigenvalues from

random data is repeated between 50 and 1,000 times (Hayton et al., 2004), but instead of

averaging the resulting eigenvalues, the 95th percentile is retained, as parallel analysis

tends to retain one too many factors (Glorfeld, 1995; Harshman & Reddon, 1983). Then,

the eigenvalues are plotted in descending order, and the point at which the lines cross

indicates the appropriate number of factors to retain. When compared to other methods of

factor extraction, parallel analysis has been shown to be the most accurate (Zwick &

Velicer, 1986). Of the existing measures that took a factor analytic approach, only the

measure for triggered and situational interest (Linnenbrink-Garcia et al., 2010) utilized

parallel analysis during measure development (see Table 4.1). The measure of

transformative experience (Pugh et al., 2010a) was developed using principal components

analysis (PCA) for factor extraction, an alternative, Rasch-based method that examines

whether there are patterns in the data not accounted for by an initial factor, which would

suggest additional factors (Brown, 2015; Linacre, 2016). Because the appreciation

measure could be multidimensional and Rasch analysis assumes a unidimensional latent

102

variable (Boone, Staver, & Yale, 2014; Linacre, 2016), a factor analytic approach is more

appropriate for first distinguishing possible factors, which would then be separately

Rasch analyzed. Thus, the appreciation measure will take a factor analytic approach,

using parallel analysis as a more appropriate method of factor extraction compared to

methods used in the development of most of the other existing measures.

Measure refinement. Once the appropriate number of factors is extracted and

factor analysis is conducted, measures can be refined by removing items that do not load

highly and so are not strong indicators of the latent construct (Tabachnick & Fidell,

2001). Most of the existing measures used factor loadings as the sole way of refining the

measure. Another powerful tool for refining measures is Rasch analysis, which provides

more detailed information about the quality of the measure. For example, Rasch analysis

can be used to identify misfitting items, to determine whether the items are able to

accurately measure participants at all levels of the latent variable, and to assess whether

the scaling of the items is functioning appropriately (Boone et al., 2014). Additionally,

Rasch analysis can be used to create final scores on the measure that are linear, rather

than using the averages of raw ordinal data, a common way of creating person scores.

Averaging ordinal data incorrectly assumes that the points on the rating scale are equal

interval and so increases the probability of error (Boone et al., 2014). Therefore, after

removing poorly loading items, the appreciation measure will be refined using Rasch

analysis to take advantage of these benefits, something only one of the existing measured

did—the measure of transformative experience (Pugh et al., 2010a).

Reliability. The reliability of most of the existing measures is estimated using

103

Cronbach’s alpha. However, unless specific conditions hold, such as the absence of

correlated measurement errors and tau equivalence, Cronbach’s alpha misestimates

reliability (Brown, 2015; Miller, 1995). When Rasch analysis is used for measure

refinement, it also provides a reliability estimate, but this estimate is more conservative

and more likely to reflect the true reliability of the measure than is Cronbach’s alpha

(Linacre, 1997). Rasch reliability was reported for only one of the other measures

(transformative experience; Pugh et al., 2010a) and will be reported for the new

appreciation measure.

Developing a Measure of Relevance

Examining the analytic decisions made in the development of existing measure

suggests that there is significant room for improvement. Having a high-quality measure is

especially important given the nature of the question at hand: “what makes higher

education relevant?” The empirical data gathered with such a measure could inform this

question at all levels, from individual instructors’ pedagogical choices, to colleges’ and

universities’ curricula, and even to higher education policy. Given these stakes, having a

soundly developed and well-validated measure is crucial. Thus, the current study aims to

develop a new measure of appreciation, both because appreciation is theoretically distinct

from similar constructs in important ways and because of the opportunity to create the

measure according to the most psychometrically rigorous methods.

Method

Procedure

A total of 8,000 undergraduates at a large public institution in the Midwest were

104

emailed an invitation to participate in the study (see Appendix D). This sample was

randomly selected and represents approximately 20% of the undergraduate population. A

large sample was chosen based on the extremely low response rate (3.5%) of prior

research with a similar design, conducted with this same population (Nadel, 2014) and

given the desired final sample size of at least 250 participants. Students were emailed an

invitation to participate near the end of the Spring 2015 semester so that they would have

a well-developed impression of their courses for that semester. The recruitment email

included a link to an online consent form and the survey (see Appendix E and F). The

survey was intended to close after 400 students participated; however, students responded

much more quickly than anticipated, and 88 additional students participated in the study

before it could be closed. From this sample of 488, one winner was randomly selected

and received a $100 Visa gift card for his participation.

Participants

The final sample included 488 undergraduates, and the demographics of this

sample approximately mirrored the demographics of the university’s undergraduate

population (see Table 4.2). In the study sample, however, there appeared to be a

somewhat larger proportion of women, first year students, and Asian American students

compared to the university as a whole. Participants’ self-reported GPA ranged from 1.0

to 5.0 (M = 3.33, SD = 0.50), and participants’ majors represented 110 of the 185

undergraduate majors offered at the university. Finally, the courses that participants

reported on were also drawn from 110 of the 185 major areas.

105

Table 4.2

Demographics of the University and Sample

Undergraduate

Population (42,843)

Full Study Sample (488)

Gender Percent N Percent Female 47.7% 285 58.4% Male 52.3% 200 41.0% Self-defined No data 3 0.6%

Race/Ethnicity White 71.3% 375 76.8% Ethnic Minority 17.3% 110 22.5%

Black/African American 5.3% 15 3.1% Asian 5.7% 68 13.9%

Hispanic/Latino 3.5% 11 2.3% American Indian/Alaskan

Native 0.1% 1 0.2% Native Hawaiian/Pacific

Islander 0.1% 1 0.2% Two or More Races 2.6% 14 2.9%

International 7.7% - - Unknown 2.7% 3 0.6%

Year

Freshman 9.3% 118 24.2% Sophomore 21.5% 106 21.7% Junior 25% 124 25.4% Senior 43.8% 140 28.7% Note. Data for undergraduate population from The Ohio State University Enrollment Services (2015).

106

Instrument

Basis of appreciation measure. The appreciation items were developed based on

a qualitative study conducted with undergraduates (see Chapter 3). Given how little

empirical work has been conducted on appreciation, as well as how little the theory has

been explored or extended by others beyond Brophy (1999, 2008a, 2008b), a qualitative

approach was an appropriate means of soliciting the important characteristics of

appreciation from the population likely to have experienced them: college

undergraduates. The study involved semi-structured interviews that focused on what

about participants’ courses felt worthwhile and relevant to their lives outside of school,

their subjective experience in worthwhile courses, and the valued outcomes of taking the

course. The 15 participants were selected from a larger sample of participants who took

an online screening survey. Interview participants were chosen based on the elaboration

and reflectiveness of their responses to short answer questions about their worthwhile

courses.

The transcripts from these interviews were analyzed using an inductive analysis

approach (Thomas, 2006) rooted in grounded theory (Charmaz & Belgrave, 2013; Corbin

& Strauss, 2008; Glaser & Strauss, 1967). This approach uses the data to generate theory,

rather than testing the data against existing theory (Thomas, 2006). This approach was

appropriate given the exploratory nature of the study. The analytical process involved a

close reading of the data, iterative coding, and organizing codes into broader themes

(Charmaz & Belgrave, 2013).

Three primary themes emerged from this analysis, each encapsulating a set of five

107

subthemes: aspects of students’ experience in worthwhile courses could be grouped into

cognitive, behavioral, and affective elements. Cognitive elements reflected changes in

how students saw the world. Because these changes involved new or altered perceptions,

beliefs, knowledge, understanding, and ways of thinking, they were grouped together

under cognitive elements of appreciation. The second set of elements were grouped

together under behavioral aspects of appreciation, and these elements represented

changes in some aspect of students’ behavior or actions as a result of taking the course.

Finally, affective aspects of appreciation reflected students’ reports of their emotional

experience in the course of value associated with the experience. Subthemes represented

a range of variation within the broader dimension. Example subthemes included

“developing a new understanding” within the cognitive dimension, “talking with others

about the class” within the behavioral dimension, and “feeling more confident” within the

affective dimension.

Developing the appreciation items. These themes served as the basis for item

development for the appreciation measure. A total of 15 items were developed for each of

the three primary themes; this number of items was chosen for several reasons. First,

during item development, it is important to measure the entire domain space of the

construct (Churchill, 1979; Hinkin, 1995). Generating a large number of items helps to

ensure that the content domain space is covered (Clark & Watson, 1995) and increases

the likelihood of establishing content validity (de Vaus, 2002). In addition to measuring

the entire domain space, it is also important that each aspect is measured with multiple

items, because all items are measured with error. Using multiple indicators increases the

108

likelihood that the random error in the set of items cancels out (Bohrnstedt, 1983). Thus,

three items were used to measure each of the five subthemes of each dimension. Finally,

15 items were developed for each dimension because having a large number of initial

items per dimension is necessary in order to refine the measure down to the strongest

indicators (Fowler, 2009; Spector, 2014). If there is redundancy in the initial items, the

redundancies can be eliminated with minimal impact on the content validity of the final

measure of appreciation (DeVellis, 1991).

Reviewing and refining the appreciation items. To help increase the likelihood

of creating a strong final measure, the quality, clarity, and wording of the items were

assessed using an expert panel before administering the appreciation measure (DeVellis,

1991). Advanced graduate students and professors in Educational Psychology, subject

matter experts, were asked to review the appreciation items, with particular attention

given to whether the items appeared to measure appreciation and whether all important

aspects of appreciation had corresponding questions. Item wording was revised based on

their suggestions.

Finally, item wording was altered to help ensure that the set of items reflected the

entire range of possible degrees of experience of appreciation. The range of experiences

of appreciation can be conceptualized as a continuum, from little experience of

appreciation to an intense and full experience of appreciation. Questions are located

along the continuum relative to how strong an experience of appreciation is required for

participants to agree with the question. Most people will be able to agree with questions

on the low end of the continuum, but only individuals who have had a full experience of

109

appreciation are likely to agree with questions higher on the continuum. Therefore, the

set of items included some items that most participants would be likely to agree with,

some items that about half of the participants would be likely to agree with, and some

items that only a few participants would be likely to agree with. Altering item wording in

this way was also done to help differentiate between individuals within the range, with

the goal of creating a more precise measure that is better able to distinguish between

individuals. Thus, the preliminary appreciation measure was made up of three

subsections with 15 items each (see Appendix E).

Additional measures. In addition to the appreciation measure, demographic

questions and other measures of similar, existing constructs were included in the survey.

These included measures of initial interest (Hulleman et al., 2010), triggered interest,

maintained situational interest grounded in affect, maintained situational interest

grounded in value (Linnebrink-Garcia et al., 2010), and task values (Eccles & Wigfield,

1995) (see Appendix F). All measures were assessed using a 5-point Likert scale in order

to be consistent with the appreciation measures and to avoid possible method effects,

which could occur if measures share variance based on the scaling used (DeVellis, 1991).

These additional measures were included in order to explore the relationship between the

new measure of appreciation and measures of existing constructs. Although Cronbach’s

alpha is not appropriate for ordinal data of this kind (Brown, 2015; Miller, 1995), it is the

only measure of reliability reported in the published work focused on these measures.

These alpha values are reported to give a general sense of the reliability of these

measures in past work.

110

Initial interest was measured using Hulleman, et al.’s (2010) five item measure.

The measure was reworded to apply to any college course rather than to math or

psychology, the two disciplines studied in the Hulleman et al. paper. In the two studies

conducted using the original measure, the reliability was reported in terms of Cronbach’s

alpha, with values of .92 and .93.

Triggered interest, maintained situational interest grounded in affect, and

maintained situational interest grounded in value were measured using Linnebrink-Garcia

et al.’s (2010) scales, each with four items. In the two studies reported, Cronbach’s alpha

values ranged from .81 to .92 for triggered interest, from .88 to .92 for maintained

situational interest based in affect, and from .88 to .91 for maintained situational interest

based in value.

Finally, the three task values were measured using Eccles and Wigfield’s (1995)

scale, with two items for utility value, three items for attainment value, and two items for

intrinsic value. The reported Cronbach’s alpha coefficients for the three types of task

value were .76 for the two intrinsic value items, .70 for the three attainment value items,

and .62 for the utility value items.

Results

Missing Data

One participant with more than 30% missing data was deleted from the sample, as

was one participant with all missing data on three of the scales. After removing these

participants, missing data per variable ranged from 0 to 2 (0.6%). Multiple imputation

was used to replace missing data. Although missing values are often replaced with the

111

variable mean or a single regression-imputed value, these simple imputation methods

tend to underestimate variance and overestimate covariance (Brown, 2015). Multiple

imputation, however, reintroduces random variance into regression imputation by

imputing multiple data sets with slightly different estimates (Little & Rubin, 1987).

Following this method, five imputations (Yuan, 2000) of the survey data were conducted

using the expectation maximization algorithm and fully conditional specification (van

Buuren, 2007), a method appropriate for ordinal data. Because the between-sample

variance among the five imputations was negligible, only 0.000005 at most, the five

imputed datasets were averaged into a single dataset used for all subsequent analyses.

Nature of the Data

After imputing missing values, the normality and continuity of the data were

assessed at both the latent and observed levels. These characteristics shape the type of

analyses that may be appropriately conducted. At the latent level, these constructs would

be expected to be continuous and normal—individuals probably differ in their

appreciation, interest, and task value along a continuum, and individuals’ feelings about

these constructs are likely to be unimodal and symmetrically distributed about a mean. To

assess the latent characteristics of the data, PRELIS was used to test the bivariate

assumption for each correlation. At the observed variable level, however, each item

displayed significant skewness, kurtosis, or both, suggesting that the data’s observed

distributions differed significantly from normal. Thus, it is reasonable to conclude that

the data were approximately normal and continuous at the latent level and non-normal

and categorical at the indicator level.

112

Construct Dimensionality

After imputing missing data, ordinal exploratory factor analysis (EFA; Jöreskog

& Moustaki, 2006) was used to establish the dimensionality of the construct of

appreciation, as well as to assess the dimensionality of the three types of interest and the

three task values. The first step was defining the input correlation matrixes to be modeled

with latent factors. The polychoric correlation coefficient was chosen instead of the

Pearson coefficient because it yields more accurate, less attenuated parameter estimates

when used with ordinal data, the type of data collected in the current study (Holgado–

Tello et al., 2010). Next, parallel analysis (Horn, 1965) with principal components as the

method of extraction was used to determine the appropriate number of factors to retain

(Gugiu, Coryn, Clark, & Kuehn, 2009; see Figure 4.1). Finally, using the Gugiu SAS

macro for parallel analysis (Gugiu et al., 2009) provides additional information in the

form of a confidence interval around the eigenvalues calculated from the randomly-

generated data.

After conducting parallel analysis, ordinal EFA was used to explore the factor

structure and to modify the models. Two criteria were used for modifications. First, items

with factor loadings below .3 were eliminated (Tabachnick & Fidell, 2001). Below this

threshold, the latent factors account for less than 10% of the variance in the item. As

such, these items were not strong indicators of the latent factor and could be eliminated.

Second, items were removed if their loading onto a factor was not interpretable or

consistent with theory. Separate parallel analyses and ordinal EFAs were conducted for

each measure in the current study.

113

Figure 4.1. Parallel analysis plots for set of 45 appreciation items.

Appreciation. All 45 items of the appreciation measure were included in the

same parallel analysis, because with 486 participants in the full sample, there were at

least 10 participants per item on the measure, enough to exceed the ratio of three to five

participants per item suggested by some (Cattell, 1978; Gorsuch, 1983) and to meet the

ratio of 10 participants per item recommended by others (Nunnally, 1978). The parallel

analysis suggested that three factors be retained (see Figure 4.1).

Ordinal EFA was then conducted and the factor structure and reference structure

were examined. The factor loadings in the factor structure represent the correlation

114

between a particular item and its associated factor, including the influence of other

factors. In the reference structure, factor loadings represent the unique correlation

between the item and the factor, controlling for the influence of the other factors. The two

output tables thus present slightly different information that is useful for interpretation. In

the reference structure, the factor loadings of three items, two from the affective

dimension and one from the cognitive dimension, were less than .3, suggesting that none

of the three factors accounted for more than 10% of the unique variance in any of these

items (Tabachnick & Fidell, 2001). When these items were examined in the factor

structure, the magnitude of their factor loadings was approximately the same across the

three factors. This suggests that these three items were not a good reflection of any of the

three factors, and so they were deleted (see Appendix E).

Aside from these three items, the reference structure showed a relatively clear

grouping of items onto factors. Eleven of the 19 items loading onto the first factor were

from the affective dimension of the appreciation measure, all but two of the 12 items

loading onto the second factor were from the behavioral dimension of the appreciation

measure, and all 11 items loading onto the third factor were from the cognitive dimension

of the appreciation measure. The items on each factor were carefully reviewed to

determine whether they could theoretically hang together as an interpretable factor.

Because each item had been explicitly designed to map onto one of the three dimensions,

factors were not clearly interpretable when they including all the items, from all three

factors, that loaded onto them in the ordinal EFA. Using interpretability as the criterion

for elimination, a total of 12 additional items were dropped, so that each factor would

115

only include items corresponding to the same dimension. This resulted in 11 cognitive

items, 10 behavioral items, and 11 affective items. To make the scales even, one

cognitive item that cross-loaded onto the affective factor was dropped and one affective

item that cross-loaded onto the cognitive factor was dropped. The only other cross-

loading item, a behavioral item that cross-loaded onto the cognitive factor, was retained

so that each of the three dimensions would be defined by 10 items.

Additional measures. The other measures included in the study were made up of

between four and 12 items and so exceeded the 10 participant per item ratio required for

measured items to be analyzed together (Nunnally, 1978). Thus, the task value items

were assessed together, the initial interest items were assessed together, and the triggered

and situational interest items were assessed together. Initial interest was assessed

separately from the other interest measures because it had a different stem, which could

have created a method effect in which these items shared variance and split into their own

factor based on the shared wording of the items (DeVellis, 1991). The triggered and

situational interest items were assessed together to mirror the way the items were treated

during the development of the scale (Linnenbrink-Garcia et al., 2010).

The parallel analysis for the task value items suggested retaining one factor,

despite these three values being defined theoretically as three distinct factors. However,

given the small number of items defining each dimension (two each for utility and

interest value and three for attainment value), as well as other previous empirical work

that shows these items loading onto one factor (Artino & McCoach, 2008; Eccles,

Wigfield, Harold, & Blumenfeld, 1993; Parsons, 1980; Perez et al., 2014), these results

116

are not surprising. Furthermore, forcing the items to load onto three separate factors

would reduce the variability within each factor, given that each would be defined by so

few items. This reduced variability would in turn threaten any future correlational

analyses, which rely on covariance, and may potentially misrepresent the relationships

between the individual task values and other measures. Thus, one factor was retained for

the task values items, and this single factor was utilized in subsequent analyses. In the

ordinal EFA, all task value items loaded above .3.

Parallel analysis suggested that one factor be retained for initial interest,

consistent with how the construct is theoretically conceptualized. All items loaded above

.3 in the ordinal EFA.

Finally, the parallel analysis suggested retaining two factors for the triggered and

situational interest items, whereas Linnenbrink-Garcia and her colleagues (2010) retained

three factors. However, relatively little empirical work has been conducted using

Linnenbrink-Garcia et al.’s (2010) interest measure and so the factor structure is still not

well established. When ordinal EFA was conducted, the two types of situational interest

items loaded together on one factor and the triggered interest items loaded together on a

second factor. No items loaded below .3. Both because of the results of the parallel

analysis and because of the relative lack of other empirical work demonstrating a three-

factor solution, two factors were retained.

Rasch Analysis

Rasch analysis (Rasch, 1980) was used to refine the appreciation measure and to

translate the ordinal scores of the final appreciation measure and the additional measures

117

into continuous latent outcome measures. Rasch analysis is a powerful tool for measure

refinement in part because it expresses the difficulty of items and the ability of

individuals in the same terms: logits (Boone et al., 2014). Person measures, in terms of

logits, are conceptualized as where the individual falls along the continuum of the

unidimensional latent construct, from low to high. Put differently, item measures can be

conceptualized as the difficulty of the item and person measures can be conceptualized as

an individual’s ability (Boone et al., 2014).

Rasch analysis can be run either as a rating scale model (RSM) or as a partial

credit model (PCM). The RSM forces the distance between answer choices (i.e., between

“Agree” and “Strongly Agree”) to be the same for all items. In other words, all items

share the same rating scale structure (Linacre, 20000). In contrast, the PCM allows each

item to have a unique rating scale structure (Linacre, 2000). If the items behave similarly,

then there is little to be gained from running the Rasch analysis as a PCM, which is less

parsimonious than a RSM (Wright, 1998). On the other hand, if the items do not behave

similarly, then running the Rasch analysis as a PCM can result in a better fit of the data to

the model (Wright, 1998). The decision to run RSM or PCM can be made empirically by

examining whether the chi-square difference in model fit between a RSM and PCM is

significant (Linacre, 2016). For each of the three dimensions of appreciation, a RSM and

a PCM Rasch analysis were run and the fit compared. None of the chi-square differences

were significant, Cognitive: χ2 (62) = 73.20, p = ns; Behavioral: χ2 (164) = 160.18, p = ns;

Affective: χ2 (90) = 104.08, p = ns. This suggests that the improvements in model fit were

not sufficient to justify the use of PCM. Thus, all Rasch analyses were run as RSM.

118

Next, item difficulty was computed with participants with extreme scores

excluded. These are participants who scored the lowest possible score on a measure or

the highest possible score on a measure. In the case of the measures in this study,

participants would have an extreme score if they answered 1 (“Strongly Disagree”) or 5

(“Strongly Agree”) for all items on the particular measure. Participants with extreme

scores made up a small portion of all participants: Cognitive n = 12 (2.4%), min = 4

(0.8%), max = 8 (1.6%); Behavioral n = 21 (4.3%), min = 18 (3.7%), max = 3 (0.6%);

Affective n = 24 (4.9%), min = 8 (1.6%), max = 16 (3.3%). The reason that scores like

these are problematic is because they provide no information for estimating item

measures (Linacre, 2016) and so they are not used to compute item difficulty. Finally, the

Rasch logit scores of all person and item measures were rescaled to a 0 to 100 scale to aid

interpretability.

Misfitting items from appreciation measure. Each of the three dimensions of

appreciation was analyzed in a separate Rasch analysis, because Rasch assumes that the

items being assessed reflect a unidimensional construct (Boone et al., 2014).

Additionally, the Rasch analysis produces fit statistics that can be used to identify

misfitting items. In particular, Item Mean-Square (MNSQ) values reflect the amount of

distortion in the model, due to the data overfitting the model, causing inflated reliability

statistics, when MNSQ is less than 1.0 and due to unmodeled noise when MNSQ is

greater than 1.0; values between 0.5 and 1.5 are considered productive for measurement

(Linacre, 2002; Wright & Linacre, 1994). Of the two Item MNSQ values reported in the

Rasch results, Item Outfit MNSQ is recommended over Item Infit MNSQ for

119

interpretation because it is more sensitive to outliers (Boone et al., 2014; Linacre, 2016).

After excluding persons with extreme scores, one item on the cognitive dimension and

one item on the affective dimension demonstrated misfit (see Table 4.3 and Appendix E).

Each of these items was removed and the respective Rasch analyses were rerun. The

revised Rasch analyses showed no additional misfitting items. All five subthemes of the

cognitive dimension were represented by at least one item, as were all five subthemes of

the affective dimension. Four of the five subthemes of the behavioral dimension were

represented with at least one item. The subtheme that was not represented in the final set

of behavioral items was “applying course content outside of class.” It is possible that this

subtheme dropped out because it was less concrete than other behavioral indicators of

appreciation, such as talking with others about the class or explicitly changing one’s

behavior, and therefore did not hang together well with the remaining items on the factor.

Finally, the point by measure correlations between each of the remaining items

and its associated latent variable were all positive and greater than .3: all items on the

cognitive dimension correlated at .73 or greater; all items on the behavioral dimension

correlated at .65 or greater, and all items on the affective dimension correlated at .72 or

greater. This suggests that the remaining items meaningfully reflected the latent variable

(Linacre, 2016).

Scaling of appreciation measure. Two methods were used to assess whether the

scaling of the items behaved appropriately. First, the plot of category averages was

examined for each dimension (see Figure 4.2). This table plots the average difficulty

associated with each possible answer choice, for each item of the scale. Items are

120

Table 4.3 Rasch Analysis Results

Measure Misfitting Items (MNSQ, ZSTD)

Person Reliability

Person Separation

Item Reliability

Item Separation

Appreciation

Cognitive Item 9 (1.59, 7.8) .89 2.83 .98 7.60

Behavioral None .85 2.38 .98 6.99

Affective Item 14 (1.85, 9.8) .86 2.46 .95 4.32

Other Measures

Task Value Item 5 (1.94, 8.3) .81 2.08 .99 10.76

Initial Interest None .84 2.26 .97 6.07

Triggered Interest None .84 2.30 .99 8.26

Situational Interest

Item 7 (1.81, 8.1) .88 2.70 .96 5.02

considered to be functioning correctly when the order of difficulty matches the order of

the answer choices. As an example, Figure 4.2 provides the plot of category averages for

the items of the cognitive dimension of appreciation. When plotted against their

difficulty, the answer categories are ordered as expected: with 1 or “Strongly Disagree”

showing the lowest difficulty and each subsequent category showing increasing

difficulty. For all three dimensions of the appreciation measure, the category averages

were ordered appropriately.

The second method for assessing item scaling involves examining plots of the

121

0 10 20 30 40 50 60 70 80 90 100 |-----+-----+-----+-----+-----+-----+-----+-----+-----+-----| NUM ITEM | 1 2 3 4 5 | 3 C_6 | | | | | 1 2 3 4 5 | 7 C_11 | 1 2 3 4 5 | 4 C_7 | 1 2 3 4 5 | 1 C_2 | 1 2 3 4 5 | 10 C_15 | 1 2 3 4 5 | 2 C_3 | | | 1 2 3 4 5 | 5 C_8 | 1 2 3 4 5 | 9 C_14 | 1 2 3 4 5 | 8 C_13 |-----+-----+-----+-----+-----+-----+-----+-----+-----+-----| NUM ITEM 0 10 20 30 40 50 60 70 80 90 100

Figure 4.2. Category averages for the nine items of the cognitive dimension. category probability curves for each item (see Figure 4.3). These plots represent the

likelihood that a participant of a given ability selects each of the categories (Linacre,

2006). The x-axis represents the item difficulty and the y-axis represents the probability

of participants selecting the answer category. When each category is the most probable in

some portion of the plot, as it is for item B_2 in Figure 4.3, this suggests that the number

of categories is appropriate. In contrast, when one or more categories is “buried,” some of

the categories may be redundant and could be collapsed or eliminated (Liu, 2010). None

of the items from any of the three dimensions displayed buried category probability

curves, suggesting that the 5-point Likert scale functioned appropriately.

122

Figure 4.3. Category probability curves for item 2 of the behavioral dimension of appreciation. The plot of item 2 shows scaling behaving appropriately. A “buried” category would indicate that the scaling should be adjusted to include fewer answer categories.

Wright maps of dimensions of appreciation. Next, Wright Maps were produced

for each dimension of appreciation (see Figure 4.4). These maps plot person scores

against item difficulties and are a way to assess gaps in item difficulty—places where

people were not well differentiated because they fell in a gap between items—and item

redundancy—places where there were multiple items of similar difficulty (see Table 4.4

for item difficulties). The Wright maps for the appreciation dimensions indicated both

123

Cognitive Dimension

Behavioral Dimension Affective Dimension MEASURE PERSON - MAP - ITEM <more>|<rare> 100 .## + | | | | .# | 90 + | | . | | 80 . + T| .### | | ### | .### | 70 + .##### | .##### | S| .########## | .##### | 60 .######### + .######## | C6 .####### |T .######### | .########### M|S 50 .########## + C11, C7 .###### |M C2 .####### | C15, C3 ############ |S C14, C8 .#### | C13 .##### |T 40 .##### S+ ### | .### | .## | .### | 30 . + .# | .# T| | . | . | 20 + . | | . | | | 10 # + | | | | 0 + <less>|<freq> EACH # IS 3: EACH "." IS 1 TO 2

MEASURE PERSON - MAP - ITEM <more>|<rare> 100 . + | | | | | 90 . + | | | . | 80 + | . | | . | . | 70 + T| . | .# | #### | .## | 60 ### + .##### S|T B14 .## | ######### |S #### | B13, B15 50 ############# +M B10, B12, B2, B6 .########## | B11 .##### M|S B5 .######### | .##### |T B4 ######## | 40 .##### + .#### | .### | ### S| .### | 30 ### + ## | | .## | T| .# | 20 + .# | | | | # | 10 + | | | | 0 + <less>|<freq> EACH # IS 4: EACH "." IS 1 TO 3

MEASURE PERSON - MAP - ITEM <more>|<rare> 100 #### + | | | | | 90 .## + | | | .# T| 80 + .## | | .#### | | .#### | 70 + .#### S| .###### | | .######## | | 60 .########## + .######### | M| .####### | ####### | 50 .##### +T A13, A7 #### |S .##### |M A6, A9 .#### | A11, A2, A4 .######### |S A10 .# S|T A5 40 .#### + .# | .## | . | .# | 30 .## + . T| . | . | | . | 20 . + | . | | | | 10 . + | | | | 0 + <less>|<freq> EACH # IS 4: EACH "." IS 1 TO 3

Figure 4.4. Wright maps of three dimensions of appreciation. ceiling and floor effects. In other words, there were a large number of people whose

estimated ability far exceeded the difficulty of the hardest item and people whose

estimated ability was far below the easiest item. This is problematic because it means that

124

Table 4.4

Items of the Three Dimensions of Appreciation Ordered by Difficulty

Cognitive Behavioral Affective

Item Item

Difficulty Item Item

Difficulty Item Item

Difficulty C_13 43.55 B_4 43.24 A_5 41.58 C_14 45.04 B_5 46.14 A_10 43.92 C_8 45.58 B_11 48.25 A_4 44.38 C_3 47.40 B_12 49.44 A_11 45.04 C_15 47.54 B_6 50.19 A_2 45.12 C_2 48.37 B_10 50.52 A_6 46.47 C_7 49.91 B_2 50.86 A_9 46.78 C_11 50.65 B_15 51.39 A_13 49.72 C_6 57.66 B_13 51.93 A_7 50.19

B_14 58.02

the measure does not provide sufficient coverage at the high and low ends of participant

ability, resulting in a decrease in the precision of the measure (Boone et al., 2014).

Additionally, there was also redundancy in items—clusters of items with very

similar difficulty, as can be seen with items 11 and 7, 15 and 3, and 2 and 8 on the

cognitive dimension, for example (see Figure 4.4 and Table 4.4). Clusters of items are

appropriate for measures that aim to determine whether participants meet a cut-off at the

difficulty at which items are clustered; for example, for licensure tests that determine

whether participants pass. For measures like this appreciation scale that aim to assess

participant ability across the full range of possible values, a variety of items with

different difficulties is more appropriate.

Finally, the appreciation dimensions also exhibited a few gaps between items, for

example, between item 6 and items 11 and 7 on the cognitive dimension and between

125

item 14 and items 13 and 15 on the behavioral dimension (see Figure 4.4 and Table 4.4).

When there are gaps between items, the precision with which persons in the gaps are

measured decreases, and their scores are estimated with greater error (Salzberger, 2003).

Reliability and separation of appreciation measure. After removing misfitting

items, assessing scaling, and examining Wright maps, person and item separation and

reliability were calculated for each of the three dimensions of appreciation. The Rasch

reliability index ranges from 0.0 to 1.0, with values closer to 1.0 representing greater

reliability (Boone et al., 2014). Rasch analysis produces reliability estimates both for

person measures and item measures. Person reliability corresponds with traditional test

reliability, like more commonly-used measures such as Cronbach’s alpha; item reliability

does not have a traditional equivalent, but reflects the stability of the item difficulty

estimates (Linacre, 2016). The person and item reliabilities for the three dimensions of

appreciation appear robust, with all person reliabilities at or above .85 and all item

reliabilities at or above .95 (see Table 4.3).

In addition to reliability, separation was also used to assess the dimensions of

appreciation. Separation is the ratio of “signal to noise,” representing the ratio of true

variance to error variance. More specifically, it represents the ability of the measure to

distinguish between people, in the case of person separation, and to confirm the hierarchy

of item difficulties, in the case of item separation (Linacre, 2016). Separation values

range from 0 to positive infinity, with higher values representing better separation (Boone

et al., 2014). In terms of criteria for assessing person separation, 1.5 is considered

acceptable, 2 is good, and 3 is excellent (Fisher, 1992; Wright & Masters, 1982). When it

126

comes to item separation, 1.5 is required but closer to 2.5 is better (Tennant & Conghan,

2007). For the three dimensions of appreciation, both the person separation and the item

separation for each dimension are very good (see Table 4.3). Taken together, the strong

reliability and separation indexes for the three dimensions of appreciation suggest that the

measure is reliable and is capable of distinguishing between people, at least in the middle

of the continuum, and item measures. Table 4.5 presents a key for converting raw scores

on the three measures into continuous Rasch measures.

Revisions to the measures. Although the appreciation measure appears to have

acceptable reliability and separation, each of the issues that were highlighted by the

Wright maps could be addressed in subsequent versions of the measure. Adding or

revising the measure’s items would remedy many of the limitations of the original

measure. For example, to counteract the ceiling and floor effects, additional items could

be added to the measure, with some of a much greater difficulty than the hardest item on

each dimension and some that are much easier than the easiest item on each dimension.

For example, the most difficult item on the cognitive dimension was item 6, “I see my

everyday experience in a completely new light.” A harder item might ask students how

strongly they endorse the statement “I have fundamentally changed the way I see every

aspect of the world.” On the other hand, the easiest item on the cognitive dimension was

item 13, “I have become more familiar with different viewpoints after taking this class.”

An easier item might ask students how strongly they endorse the statement “I have seen

at least one example of how content from this class applies to everyday life.” By adding

127

Table 4.5

Raw Score to Rasch Score Conversion for the Three Dimensions of Appreciation Raw Score

Cognitive Rasch Score

Behavioral Rasch Score

Affective Rasch Score

9 0 — 0 10 9.23 0 10.03 11 15.07 11.25 16.08 12 18.89 18.16 19.82 13 21.92 22.51 22.61 14 24.54 25.81 24.88 15 26.91 28.52 26.83 16 29.10 30.86 28.56 17 31.16 32.93 30.13 18 33.10 34.79 31.58 19 34.93 36.49 32.94 20 36.66 38.05 34.24 21 38.30 39.51 35.49 22 39.87 40.87 36.71 23 41.38 42.16 37.91 24 42.84 43.38 39.10 25 44.28 44.56 40.30 26 45.71 45.69 41.51 27 47.14 46.79 42.76 28 48.59 47.88 44.05 29 50.09 48.94 45.41 30 51.65 50.00 46.86 31 53.29 51.06 48.42 32 55.05 52.13 50.14 33 56.94 53.21 52.07 34 59.00 54.31 54.24 35 61.22 55.45 56.70 36 63.59 56.63 59.45 37 66.09 57.85 62.40

Continued Note. The cognitive and affective dimensions have nine items each, resulting in a minimum possible score of 9 (participant indicated 1 or “Strongly Disagree” to all questions) and a maximum score of 45 (participant indicated 5 or “Strongly Agree” to all questions). Because the behavioral dimension had 10 items, the minimum possible score on that dimension ranged from 10 to 50.

128

Table 4.5 continued

Raw Score

Cognitive Rasch Score

Behavioral Rasch Score

Affective Rasch Score

38 68.66 59.14 65.45 39 71.30 60.51 68.52 40 74.04 61.96 71.63 41 76.98 63.52 74.89 42 80.29 65.22 78.52 43 84.40 67.07 82.96 44 90.54 69.13 89.61 45 100 71.46 100 46 — 74.17 — 47 — 77.46 — 48 — 81.82 — 49 — 88.74 — 50 — 100 —

easier and more difficult items, the measure would be able to provide a more accurate and

precise measure of participants with a high and low ability. To address item redundancy,

the wording of items of similar difficulties could be altered to make some items more

difficult and others easier, which would contribute to greater item spread as well.

Additionally, gaps between items could be reduced by altering existing items or creating

new items to fill the gaps

Another approach to revising the measure, beyond adding or adjusting items,

would be to use a different type of scale. With the Likert scale used in the current study,

participants endorsed items much more strongly than expected: for two of the three

dimensions, the mean person ability measure was greater than the mean item difficulty

(see Figure 4.4). This could be due in part to social desirability, with participants

129

endorsing these items because of the belief that one is supposed to have valuable and

transformative experiences in college—a theme that arose in the qualitative interview

study that informed the measure development. Using a frequency scale that asked

students to report how often they had the experiences reflected in the items might result

in less redundancy among items, fewer ceiling effects, and a better spread of items

relative to person measures.

Examining Wright maps with item thresholds mapped (see Figures 4.5, 4.6, and

4.7) suggests that the issues described above may not be as problematic as they first

appear. The item difficulties plotted on the original Wright map (see Figure 4.4)

correspond with the point where the probability curves for the lowest category and the

highest category overlap (Wright & Masters, 1981). Using item B_2 in Figure 4.3 as an

example, this would be the point where the leftmost curve and the rightmost curve

intersect, just above the “0” on the x-axis. This is a vestige of using Rasch analysis for

dichotomous test questions with two possible categories—correct and incorrect. In a

dichotomous category, the item difficulty on the Wright map represents the point at

which the participant has a 50% chance of answering the item correctly. For a rating

scale, however, there are multiple thresholds, corresponding to k – 1 of k category answer

choices (Bond & Fox, 2015). In the case of the appreciation measure, with five answer

categories (1, “Strongly Disagree,” to 5, “Strongly Agree”), there are four category

thresholds. In the case of item B_2 in Figure 4.3, these thresholds are represented by the

four points where neighboring curves intersect. When plotted on a Wright map, these

thresholds represent the point at which the probability of a participant endorsing the

130

Cognitive Dimension MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 .# + | | | . | | 80 . + T| .### | | ### | .### | | 70 .##### + | .##### S| .########## | | .##### | C_6 .5 60 .######### + | C_6 .4 .######## |T C_6 .3 .####### | C_6 .2 C_11 .5 .######### | C_7 .5 .########### M|S C_11 .4 C_15 .5 C_2 .5 C_3 .5 50 .########## + C_11 .3 C_7 .4 C_8 .5 .###### |M C_7 .3 C_15 .4 C_13 .5 C_2 .4 C_14 .5 C_3 .4 .####### | C_11 .2 C_15 .3 C_8 .4 C_7 .2 C_2 .3 C_3 .3 .###### |S C_15 .2 C_14 .3 C_14 .4 C_3 .2 C_8 .3 .##### | C_2 .2 C_13 .3 C_13 .4 C_8 .2 ########## |T C_14 .2 40 .##### + C_13 .2 ### S| .### | .## | .### | | 30 . + .# | .# | T| . | . | | 20 . + | | . | | | 10 # + | | | | | 0 + <less>|<freq> EACH "#" IS 3: EACH "." IS 1 TO 2

Figure 4.5. Wright map of the cognitive dimension with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

131

Behavioral Dimension MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 + . | | | | | . | | 80 + | . | | . | | # | 70 + T| . | | .## | .## | .## | .### | 60 #### + B_14 .5 .## | .#### S|T B_14 .3 B_14 .4 ### | B_14 .2 .##### | .###### |S B_10 .5 B_13 .5 .############ | B_13 .3 B_13 .4 B_12 .5 B_15 .5 B_2 .5 B_6 .5 .########## | B_13 .2 B_10 .3 B_10 .4 B_11 .5 B_15 .3 B_15 .4 B_2 .3 B_2 .4 B_6 .3 50 .####### +M B_15 .2 B_12 .3 B_11 .4 B_12 .4 B_6 .4 .###### | B_10 .2 B_11 .3 B_5 .5 B_2 .2 B_6 .2 ####### M|S B_12 .2 B_5 .4 B_4 .5 .############ | B_11 .2 B_5 .3 ####### | B_5 .2 B_4 .3 B_4 .4 .##### |T B_4 .2 .##### | | 40 ####### + ###### | .#### | #### S| | ##### | #### | 30 + .## | | | .### | T| ## | | 20 + | .## | | | | | .# | 10 + <less>|<freq> EACH "#" IS 3: EACH "." IS 1 TO 2

Figure 4.6. Wright map of the behavioral dimension with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

132

Affective Dimension MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 .## + | | | | .# | T| | 80 + .## | | | .#### | | .#### | 70 + .#### | S| | .###### | | .######## | | 60 .########## + | .######### | | .####### M| | A_7 .5 ####### | A_13 .5 |T 50 .##### + A_13 .4 A_11 .5 A_7 .4 A_6 .5 A_9 .5 #### |S A_13 .3 A_2 .5 A_7 .3 A_4 .5 .##### | A_13 .2 A_9 .4 A_10 .5 A_7 .2 .#### |M A_9 .3 A_11 .4 A_5 .5 A_6 .4 .#### | A_11 .2 A_11 .3 A_2 .4 A_9 .2 A_6 .3 A_4 .4 ##### |S A_4 .2 A_10 .3 A_10 .4 A_6 .2 A_2 .3 A_4 .3 .# S| A_2 .2 A_5 .3 A_5 .4 ## |T A_10 .2 A_5 .2 40 .## + .# | # | .# | . | .# | .# | 30 .# + . | T| . | . | | . | | 20 . + | | . | | | | | 10 . + <less>|<freq> EACH "#" IS 4: EACH "." IS 1 TO 3

Figure 4.7. Wright map of the affective dimension with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

133

category below the point equals the probability of the participant endorsing the category

above (Linacre, 2016). Higher thresholds correspond with greater item difficulty and

participant ability. In other words, it is more difficult to endorse an item with a 5,

“Strongly Agree,” than with a 1, “Strongly Disagree.” Because they have more thresholds

than dichotomous items, polytomous rating scale items also provide more information

along the continuum of the Wright map (Van Wycke & Andrich, 2006). When the Wright

maps for the three appreciation dimensions are recreated with these thresholds (see

Figures 4.5, 4.6, and 4.7), the items appear to provide somewhat better coverage of the

range of person measures. Thus, the appreciation measure may be functioning more

appropriately than it first appeared to be, but revisions suggested above should still be

made to further improve the measure.

Rasch analysis of additional measures. Rasch analysis was also used to assess

the measures of task value, initial interest, triggered interest, and situation interest. When

the Rasch analyses were examined for misfitting items, one item from the task value

scale (item 5) showed misfit, as did one item from the situational interest measure (item

7; see Appendix F and Table 4.3). In order to preserve the original measures, misfitting

items were not removed, but item anchoring was used to decrease the impact of the

misfitting items. Anchoring involves removing the misfitting item, running the Rasch

analysis to estimate the difficulty of the remaining items, fixing these item difficulties,

and rerunning the Rasch analysis with the misfitting item added back in (Boone et al.,

2014). Person measures estimated with the item difficulties of well-fitting items anchored

are less impacted by the misfitting item (Linacre, 2010). Thus, anchoring was used to

correct for misfitting items on the task value scale and situational interest scale while

134

preserving the original measures.

In terms of the scaling for these additional measures, none of the items showed

inversions in category averages or buried probability curves. This suggests that the 5-

point scaling was functioning appropriately for these additional measures. Wright maps

using item difficulty indicated large ceiling and floor effects for these additional

measures, as well as significant amounts of item redundancy (see Figure 4.8). Because of

how few items many of these measures contained, item redundancy sometimes resulted

in the measure only being able to capture three distinct levels of person ability. The non-

redundant items also clustered together, meaning that only a small range of person

abilities could be measured with any accuracy. When Wright maps were recreated using

thresholds (see Figure 4.9, 4.10, 4.11, and 4.12), these issues were mitigated somewhat,

but the maps still indicated ceiling and floor effects and large gaps between items.

Despite the problems highlighted with the Wright maps, the reliability indexes for

these additional items suggested that the measures were acceptably reliable: all person

reliability indexes were above .8, item reliabilities were .96 and above, all person

separation indexes were at least 2.0, and all item separation indexes exceeded 5.0.

Convergent and Discriminant Validity

Continuous Rasch person scores for each dimension of appreciation, as well as

the additional measures, were output for use in analyses of validity (see Table 4.6). All

correlations were significant at the .01 level. The three appreciation dimensions

correlated highly with each other, ranging from r = .68 between the cognitive and

behavioral dimensions to r = .78 between the cognitive and affective dimensions. Of the

135

Initial Interest Triggered Interest MEASURE PERSON - MAP - ITEM <more>|<rare> 100 .#### + | | | | | 90 + T| .# | | | 80 .## + | | .## | | S| 70 .## + | | .############ | | | 60 + .##### | M| |T II_4 .### | 50 .## +S | II_5 .## |M .## | II_1, II_2, II_3 .# |S S| 40 .# +T .# | | .# | | 30 .## + | # | T| | . | 20 + . | | | . | | 10 + | | | | 0 + <less>|<freq> EACH "#" IS 9: EACH "." IS 1 TO 8

MEASURE PERSON - MAP - ITEM <more>|<rare> 100 ######## + | | | T| | 90 .##### + | | | | 80 .###### + | | | .######## S| | 70 + | .############ | | | | 60 .########### + |T M| TI_4 ####### | |S 50 + .##### | |M .###### | TI_3 | TI_1, TI_2 |S 40 ##### + | .#### S|T | .# | 30 + .###### | | | .# | | 20 + ## | T| | | | 10 .# + | | | | 0 + <less>|<freq> EACH "#" IS 5: EACH "." IS 1 TO 4

Continued Figure 4.8. Wright maps of additional measures.

136

Figure 4.8 continued.

Situational Interest Task Value MEASURE PERSON - MAP - ITEM <more>|<rare> 100 .### + | | | | | 90 + .# | T| | ## | 80 + ## | | .# | | .## | 70 S+ .### | | .### | | .########### | 60 + | .##### M| .#### | |T 50 ### + SIF_2, SIV_2 .## |S SIF_1 .## | .## |M SIV_1 .# |S SIF_3, SIF_4, SIV_3, SIV_4 .## | 40 .### S+T .# | . | . | ## | 30 .# + . | . | T| . | . | 20 . + | . | | | . | 10 + | | | | 0 + <less>|<freq> EACH "#" IS 7: EACH "." IS 1 TO 6

MEASURE PERSON - MAP - ITEM <more>|<rare> 100 .#### + | | | | | 90 + | .#### | | | 80 + T| ### | | | .###### | 70 + .#### | S| .##### | | ###### | 60 + .########### |T | .########## | ########## M|S UV_2 50 .######## + UV_1 ###### | IV_1 ###### | AV_2 ###### |M .##### | AV_1 ## | IV_2 40 .### + .### S|S .## | .# | .### | 30 # +T AV_3 ## | | .# T| .# | | 20 . + | . | | | | 10 + | | | | 0 + <less>|<freq> EACH "#" IS 4: EACH "." IS 1 TO 3

137

Initial Interest MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 .# + | | | .## | | 80 + .## | | | S| .## | | 70 + | | .############ | | | 60 + .##### | II_4 .5 M| | .### |T II_4 .4 II_5 .5 | II_4 .3 II_2 .5 50 .## +S II_1 .5 II_3 .5 | II_5 .3 II_5 .4 .## |M II_4 .2 .## | II_5 .2 II_1 .4 II_2 .4 II_3 .4 |S II_1 .3 II_2 .3 II_3 .3 .# |T 40 S+ II_1 .2 II_2 .2 II_3 .2 .# | .# | | .# | | 30 + .## | | | # | T| | 20 . + | | . | | | 10 + . | | | | | 0 + <less>|<freq> EACH "#" IS 9: EACH "." IS 1 TO 8

Figure 4.9. Wright map of initial interest with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

138

Triggered Interest MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 T+ | .##### | | | | | | 80 + .###### | | | | | S| 70 .######## + | | | | | .############ | | 60 + TI_4 .5 | | .########### |T | TI_4 .4 | M| TI_4 .3 |S TI_4 .2 50 ####### + | TI_3 .5 | TI_2 .5 .##### |M TI_1 .5 | | TI_3 .4 .###### | TI_3 .3 TI_1 .4 TI_2 .4 |S TI_1 .3 TI_2 .3 40 ##### + TI_1 .2 | TI_2 .2 TI_3 .2 | .#### | S|T | .# | 30 + | .###### | | | | .# | | 20 + | ## | | T| | | | 10 .# + <less>|<freq> EACH "#" IS 5: EACH "." IS 1 TO 4

Figure 4.10. Wright map of triggered interest with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

139

Situational Interest MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 .# + | T| ## | | | 80 ## + | .# | | .## | | S| 70 .### + | .### | | | .########### | 60 + .##### | M| | .#### | SI_F_2 .5 SI_V_2 .5 ### | SI_F_1 .5 50 +T SI_F_2 .4 SI_V_1 .5 .## |S SI_F_2 .3 SI_F_1 .4 SI_F_3 .5 SI_V_2 .3 SI_V_2 .4 SI_F_4 .5 SI_V_3 .5 SI_V_4 .5 .## | SI_F_1 .3 .## |M SI_F_1 .2 SI_V_1 .4 SI_F_2 .2 SI_V_2 .2 .# |S SI_F_4 .3 SI_F_3 .4 SI_V_1 .3 SI_F_4 .4 SI_V_3 .4 SI_V_4 .4 .## | SI_V_1 .2 SI_F_3 .3 SI_V_3 .3 SI_V_4 .3 40 .## S+T SI_F_3 .2 SI_F_4 .2 SI_V_3 .2 SI_V_4 .2 .# | .# | . | . | .# | 30 . + .# | . | . | T| . | . | 20 + . | | . | | | 10 . + | | | | | 0 + <less>|<freq> EACH "#" IS 7: EACH "." IS 1 TO 6

Figure 4.11. Wright map of situational interest with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

140

Task Values MEASURE PERSON - MAP - ITEM - Andrich thresholds (modal categories if ordered) <more>|<rare> 90 + | .#### | | | | | T| 80 ### + | | | .###### | | | 70 .#### + | S| | .##### | | ###### | | 60 +T .########### | | .########## | UV_2 .5 ########## M| UV_1 .5 |S UV_2 .4 IV_1 .5 .######## | UV_1 .3 UV_1 .4 UV_2 .3 ###### | UV_1 .2 AV_2 .5 UV_2 .2 50 + IV_1 .3 IV_1 .4 AV_1 .5 ###### | IV_1 .2 AV_2 .3 AV_2 .4 ###### |M AV_2 .2 IV_2 .5 .##### | AV_1 .3 AV_1 .4 ## | AV_1 .2 IV_2 .3 IV_2 .4 | IV_2 .2 .### S| .### |S 40 .## + | .# | .### | # | AV_3 .5 |T ## | AV_3 .3 AV_3 .4 30 + AV_3 .2 .# | T| .# | | | . | | 20 + . | | | | | | | 10 + <less>|<freq> EACH "#" IS 4: EACH "." IS 1 TO 3

Figure 4.12. Wright map of task values with Andrich thresholds indicated. Breaks in the latent variable indicate redundancy in the difficulty of neighboring items.

141

Table 4.6 Correlations Between Measures 1 2 3 4 5 6 7

1. Cognitive —

2. Behavioral .686** —

3. Affective .781** .712** —

4. Task Values .652** .719** .775** —

5. Initial Interest .433** .400** .500** .571** —

6. Triggered Interest .520** .469** .557** .548** .352** —

7. Situational Interest .713** .676** .827** .848** .653** .637** —

** p < .01, two-tailed. four additional measures, the three dimensions of appreciation correlated the most

strongly with situational interest, followed by task values (except in the case of the

behavioral dimension, which correlated more strongly with task values than situational

interest), then by much lower correlations with triggered interest and initial interest (see

Table 4.7). This suggests that appreciation may be more closely associated with students’

judgment of value than with their affective enjoyment in the moment. The items of the

triggered interest measure reflect this affective enjoyment, for example, with items like

“This class is often entertaining” (see Appendix F). On the other hand, the situational

interest items focus on students’ evaluation of the content, with items like “I find the

class material we are covering in class interesting” (see Appendix F). Although the

142

appreciation dimensions do correlate significantly both with measures focused on

affective experience and measures focused on the value of the content, the stronger

relationship between appreciation and value reinforces the theoretical account of

appreciation presented by Brophy (1999, 2008a, 2008b).

Among the three dimensions of appreciation, the affective dimension correlated

the most strongly with all four additional measures (see Table 4.7). The cognitive

dimension correlated the next strongest with all dimensions except task value, and the

behavioral dimension correlated the least strongly with all additional measures except

task value. Given that the additional measures focus primarily on students’ affective

experience and judgments of value, the stronger relationship between the affective and

cognitive dimensions and these other measures is not surprising. The behavioral

dimension of the appreciation measure also appears to tap into an aspect of students’

experience not well assessed by the additional measures. This suggests that the

behavioral dimension may be a unique contribution of the appreciation measure. Thus,

the current study provides only preliminary evidence for the validity of the appreciation

measure, but the relationships suggested by the correlations are in line with what would

be predicted by theory.

Higher-Order Factor

The strong positive correlations among the appreciation measures and the other

related constructs suggests that there may be a higher order factor to which all seven

constructs relate. To assess this, parallel analysis was used to extract the appropriate

number of factors that characterized the composite scores of the seven constructs. The

parallel analysis indicated that one factor should be retained (see Figure 4.13). When a

143

Table 4.7

Order of Magnitude of Correlations Dimension of Appreciation Additional Measures Cognitive (C) SI, TV, TI, II Behavioral (B) TV, SI, TI, II Affective (A) SI, TV, TI, II Additional Measures Dimensions of Appreciation Task Value (TV) A, B, C Initial Interest (II) A, C, B Triggered Interest (TI) A, C, B Situational Interest (SI) A, C, B

Figure 4.13. Parallel analysis of the seven composite measures, indicating that one factor should be retained. single factor EFA was conducted with the composite measures, all seven loaded above .3

and so were retained in the model (see Table 4.8). The pattern of the loadings

144

Table 4.8

Factor Loadings for the Seven Composite Variables Variable Factor Loading

Situational Interest 0.94

Appreciation – Affective 0.90

Task Values 0.88

Appreciation – Cognitive 0.80

Appreciation – Behavioral 0.77

Triggered Interest 0.63

Initial Interest 0.59 underscores some of the findings described earlier and helps characterize the nature of

the higher-order factor. First, the two constructs with the lowest loadings were triggered

interest (.63) and initial interest (.59), suggesting that students’ existing interests and

purely affective enjoyment in the moment are less closely related to this higher-order

factor than the other constructs, which seem more closely tied with emerging judgments

of value. Experience value seemed most closely associated with the higher-order factor,

based on the strong factor loading of situational interest (.94), the affective dimension of

appreciation (.90), and task value (.88). Finally, as suggested by previous analyses, the

behavioral aspect of appreciation was not as strongly associated with the higher-order

factor (.77) as the other aspects of appreciation, but nonetheless was more strongly

associated than were triggered interest and initial interest. Taken together, these results

suggest that the constructs are very closely related. They also suggest that the higher-

order factor is best characterized as content-derived value. The items from the measures

145

more closely related to the higher-order construct were more focused on the role of the

course content in the valuable outcomes students experienced, and items from the

measures less closely related to the higher-order construct were more focused on things

the instructor did to make the course valuable or on more general valuable outcomes.

Discussion

This study sought to create a new, psychometrically rigorous measure of

appreciation that avoided the limitations of related measures and had the potential to

inform the debate about what makes higher education relevant. Overall, the results

indicate that this newly-created measure is a good starting point, but further refinement

and validation are necessary. More specifically, the findings provide initial evidence for

the operationalization of appreciation into cognitive, behavioral, and affective

dimensions. Items from these three dimensions generally loaded as expected in the factor

analysis. Additionally, the reliability and separation of the three dimension measures are

generally very good (see Table 4.3), suggesting that these measures, even in the first

iteration, are robust. Finally, the correlations between the three dimensions of

appreciation and the other constructs reinforce the conceptual similarity of these

constructs. Of particular note is the stronger relationship between appreciation and value-

related constructs such as task value and situational interest, which provides empirical

support for Brophy’s (1999, 2008a, 2008b) theoretical conceptualization of appreciation.

Furthermore, the weaker relationship between the behavioral aspect of appreciation and

these other constructs suggests that this aspect of relevance in particular may not be well

represented by existing measures. Taken together, these correlations suggest that

146

appreciation is related to these other constructs, while still representing a distinct

experience.

Thus, the results of this study make a theoretical contribution to the educational

psychology literature and have the potential to make an empirical contribution to the

debate about what makes college relevant. Theoretically, the results provide evidence for

Brophy’s account of appreciation (1999, 2008a, 2008b) and highlight important ways it is

distinct from other constructs. This study also makes a valuable contribution to the

literature by providing a template for psychometrically rigorous measure development

within the field and illustrating the iterative nature of measure development.

In terms of the debate about what makes college relevant, the current study brings

to light the types of experiences students have in their college courses that feel

worthwhile and relevant. One of the most valuable things about the experiences reflected

in the appreciation measure is that they were described by college students themselves.

The inductive, grounded theory approach taken in this study helped bring students’ lived

experiences, a valuable and often overlooked perspective, into the debate. The study also

contributes a preliminary measure that could be used by educators, administrators, and

policy makers at various levels to collect empirical data. Like students’ perspectives,

empirical data also help inform the discussion and subsequent decisions, and the

appreciation measure offers a first step towards gather that data.

Despite the promise of the appreciation measure, it is still in its preliminary

stages, in part due to limitations during the process of development. One area of concern

is the population used to develop the measure. All of the participants were drawn from

one large public university, and it is possible that there are aspects of appreciation not

147

represented in their experiences. It could be the case that students in other higher

education contexts, such as community colleges or small liberal arts colleges, might

report additional aspects of appreciation. The measure would be more robust if it

represented a greater variety of perspectives within the target population, so future

research should explore students’ experience of appreciation in other realms of higher

education.

Another concern is the spread of items relative to people (see Figure 4.4).

Although the thresholds do seem to mitigate the item clustering somewhat (see Figures

4.5, 4.6, and 4.7), the measure would be improved by adding more items, particularly

very difficult items and very easy items, or changing the scaling and then reanalyzing the

measure.

Finally, the current study did not validate the new measure of appreciation in an

independent sample. Validation seems premature, given the range of ways that the

measure could be refined and modified, but it is nonetheless necessary for a measure to

be considered robust. Once a revised version of the appreciation measure is developed,

analyzed, and refined, future work would administer the measure to an independent

sample and conduct a confirmatory factor analysis to assess the fit of the measure.

Thus, the appreciation measure in its current form is limited, but it sets a

trajectory for other exciting work. The first steps are to interview a wider range of college

students, refine the measure, and validate it, in order to have a robust measure of

appreciation. This measure could then be used to explore the relationship between

appreciation and other constructs not included in the current study, such as identity

exploration (Flum & Kaplan, 2006; Kaplan & Flum 2010, 2012; Sinai et al., 2012b) and

148

transformative experience (Girod & Wong, 2002; Pugh, 2011; Pugh & Girod, 2007).

Having a measure of appreciation would also allow for exploring what kinds of

pedagogical practices, curricular content, and classroom environments promote

experiences of appreciation. Across time, this measure could also be used to examine

educational outcomes associated with experiences of appreciation. Taken together, a

theoretical definition of relevance as appreciation and a well developed and validated

measure of appreciation could provide a foundation for answering the question, “what

makes higher education relevant?”

149

Chapter 5

Laying the Groundwork and Setting the Trajectory for Future Work on Relevance

At the heart of debate about the relevance of education are questions like: How do

we understand what it means for education to be relevant (Lemann, 2016; Paxson, 2013)?

Given that definition, is college worth it, considering the enormous investment of time

and money it requires students to make (Belkin, 2014; Bond, 2015; Weston, 2015)? And,

what sorts of pedagogical practices would help instructors communicate, and students

recognize, content’s ability to bear on their lives outside of school (Benmar, 2015;

Berrett, 2015; Hasak, 2015; Sparks, 2012)? In this dissertation, I sought to begin laying

the theoretical and empirical groundwork necessary to answer these questions. First,

Chapter 2 clarified an appropriate definition of relevance. Next, Chapter 3 substantiated

and expanded this definition by exploring college students’ lived experiences in courses

that felt worthwhile and relevant, but that were not aligned with their previously existing

interests. Finally, Chapter 4 presented a preliminary measure of relevance that could be

used in future work. Taken together, the three chapters present a set of findings that

provide a starting point for answering questions about the relevance of education.

First of all, the results of these studies suggest that students do have experiences

that feel worthwhile and relevant. In the qualitative study of Chapter 3, students

described worthwhile experiences and explicitly made the point that these types of

experiences were “what college is for.” The results of the measure development study in

150

Chapter 4 underscores this conclusion. Many students in that sample scored high on the

measure of appreciation, indicating that they too perceived themselves to have had

experienced courses that felt worthwhile and relevant.

The results of both empirical studies also provide evidence for appreciation as an

appropriate conceptualization of relevance. The analysis provided in Chapter 2 argued for

important theoretical distinctions between appreciation and other related constructs, and

in Chapter 3, the characteristics of the experiences students described overlapped with the

broad theoretical account of appreciation (Brophy, 1999, 2008a, 2008b). For example,

these interview participants did not come into the courses in question with a strong

existing interest in the content, nor did they develop an enduring interest in the content,

which could have indicated that interest value (Eccles, 2005, 2009; Eccles et al., 1983;

Wigfield & Eccles, 1992) or interest development (Hidi & Renninger, 2006) were

primarily at play. On the contrary, Brophy (1999, 2008a, 2008b) argues that appreciation

applies most readily when students do not previously recognize the value of the content.

Brophy (1999, 2008a, 2008b) also suggests that the life application value captured in

appreciation applies broadly, beyond the more specific applications associated with

utility value (Eccles, 2005, 2009; Eccles et al., 1983; Wigfield & Eccles, 1992).

Reiterating this, the students in the sample brought the content to bear on their lives in

ways beyond the direct application of practical skills and also without fundamentally

changing their academic or individual trajectories, which may have indicated identity

development (Flum & Kaplan, 2006; Kaplan & Flum, 2010, 2012). Similarly, these

students also reported other cognitive, behavioral, and affective experiences in addition to

aspects of motivated use, expanded perception, and experiential value reflected in

151

transformative experience (Pugh, 2002, 2004, 2011; Pugh et al., 2010a, 2010b),

suggesting that appreciation may describe a broader range of experiences than captured

by transformative experience. Finally, the value students ascribed to their experiences in

the course went beyond affective enjoyment to include higher-level cognitive outcomes

as well. This is corroborated by Chapter 4 as well, which found stronger correlations

between the appreciation measure and similar constructs focused on value than between

the appreciation measure and similar constructs reflecting affect or enjoyment. This

pattern of findings ties in with Brophy’s assertion that the value of engaging with relevant

content “is best described using terms such as enrichment or empowerment, not pleasure

or fun” (2008a, p. 137). In these ways, the two empirical studies provided evidence for

the argument in Chapter 2 that appreciation is theoretically distinct and an appropriate

conceptualization of relevance.

The findings of these studies did not simply confirm the theoretical account of

appreciation, but also added new depth and detail to the definition. Chapter 3 highlighted

three distinct dimensions of appreciation that emerged from students’ descriptions of

their lived experiences. The cognitive dimension reflected changes in students’

perceptions, beliefs, or knowledge. Changes in students’ actions were captured by the

behavioral dimension. Finally, the affective dimension represented students’ values and

emotional experiences. These three dimensions represent extensions of Brophy’s (1999,

2008a, 2008b) initial theoretical definition. Although the theoretical definition does

recognize the importance of attending to the cognitive as well as the affective dimensions

of appreciation, the definition of these valuable outcomes is sketched in otherwise broad

terms like “absorption, satisfaction, recognition, making meaning, self-expression, self-

152

realization, making connections, achieving insights, [and] aesthetic appreciation”

(Brophy, 2008a, p. 137). The three dimensions in Chapter 3 help operationalize these

outcomes in more concrete terms. Chapter 4 further translated these three dimensions into

a Likert-type survey measure, and the results suggest that the behavioral dimension in

particular may represent a unique contribution of this new measure and, by extension, the

elaborated definition of appreciation. Though correlated at a significant level with the

measures of interest and task values, the behavioral dimension was much less strongly

related to these other constructs than were the affective and cognitive dimensions. Thus,

the three chapters help make Brophy’s (1999, 2008a, 2008b) account of appreciation

more concrete and highlight the perhaps under-explored role of behavioral changes in

these experiences.

Finally, these studies also point to the types of courses that may be particularly

apt at facilitating experiences of appreciation. In Chapter 3, more students reported

experiences of appreciation in humanities and social science courses than in natural

science courses, and these experiences also reflected more components of appreciation

than the experiences reported in the natural sciences. These results are particularly

interesting in light of the fact that much of the work exploring similar constructs has

taken place in math and science classrooms (i.e., Eccles et al., 1983; Eccles & Wigfield,

1995; Renniner, Nieswandt, & Hidi, 2015; Pugh et al., 2010a, 2010b). Brophy (1999,

2008a, 2008b), in contrast, contextualized most of his examples of appreciation within

social studies. This may help explain why appreciation is better able to capture students’

experiences with both concrete and abstract content than are similar constructs that have

primarily been explored with relatively concrete content. Clearly, further work is needed

153

to bolster the finding that humanities and social science courses seem well-suited for

helping students develop appreciation, but the findings nonetheless raise interesting areas

for greater exploration. Thus, the three chapters provide evidence that appreciation is an

appropriate way to conceptualize relevance, that it is different in important ways from

related constructs, and that it can be reliably measured—all of which are necessary steps

for exploring what makes education relevant.

One area of disagreement between the results of the two empirical studies

reinforces that there is still more work to be done. The point of conflict is in the

frequency of appreciation suggested by each study. In Chapter 3, participants considered

all of their college coursework to date when choosing and describing a course that felt

worthwhile and relevant. Despite the range of courses that participants considered, only a

small subset of the entire pool of possible interview participants reported experiences that

seemed sufficiently rich and sufficiently different from, for example, interest or utility to

be considered experiences of appreciation. This would suggest that experiences of

appreciation do not happen frequently and perhaps do not even occur for all students. The

results of Chapter 4, however, contrast with this interpretation. In that chapter, most

students in the survey sample reported high levels of appreciation in at least one of their

courses during the semester in which the survey was administered. If this study were used

to evaluate the frequency of appreciation, the findings would suggest that appreciation

occurs quite often for most students. Thus, the two studies suggest either that

appreciation does not occur in every course and perhaps not even for every student, or

that appreciation occurs frequently for most students. Three possible limitations could

have caused these contradictory findings. First, it could be the case that examples of

154

appreciation were underrepresented in the pool of possible interview participants. This

would lead to the faulty conclusion that appreciation is a relatively uncommon

experience. This conclusion may also have been an artifact of the study design. Because

the depth of students’ experience was judged based on their written responses, it is

possible that students who either did not take the time to provide detailed responses or

were otherwise limited by their ability to express themselves in writing did in fact have

experiences of appreciation, but were not included in the estimate. Finally, it could be the

case that the newly developed measure of appreciation was not precise enough to capture

experiences of appreciation while excluding similar positive experiences such as interest

or value.

My impression from selecting interview participants is that appreciation is not a

common experience and that the measure is detecting “noise” from similar but distinct

experiences. Because of the short answer questions used to select interview participants,

students in the qualitative study were able to provide richer descriptions of their

experiences, even when they were not ultimately interviewed, compared to the students in

the measure development study. Students’ responses to these short answer questions

suggested that they had had valuable experiences in their college courses, but this value

often derived from the easiness of the course, having an entertaining professor, the pure

utility value of the content, or being able to pursue an existing interest. These types of

experiences are similar to appreciation, as suggested by the significant positive

correlations between appreciation and the other measures found in Chapter 4, but the

theoretical account of appreciation and the interviews with students about their

experiences of appreciation suggests it is a distinct construct and experience.

155

This point of conflict highlights important directions for future work. For

example, refining the appreciation measure following the suggestions described in

Chapter 4 could help the measure better distinguish among different kinds of valuable

educational experiences. Additionally, measures of other related constructs, such as

transformative experience (Pugh et al., 2010a) and identity exploration (Sinai et al.,

2012b) could be included in future versions of the survey to explore how appreciation is

or is not distinct from these constructs. Although transformative experience is closely

related to appreciation, the 28-item measure of transformative experience was not

included on the survey administered in Chapter 4 because of concerns about the length of

the final survey. Future work could assess the differences between these two constructs

by including both measures in a survey and conducting parallel analysis and exploratory

analysis to determine whether the constructs are distinct. In short, the current set of

studies cannot directly address the question of the frequency of appreciation, but they

suggest future work that could better speak to this question.

Despite the ambiguity in this finding, the three chapters nonetheless make

important theoretical, methodological, and practical contributions. At the level of theory,

the studies provide evidence for and add nuance to Brophy’s (1999, 2008a, 2008b)

account of appreciation. In providing conceptual clarity and a preliminary measure of

appreciation, they help pave the way for more work using this construct.

The studies also make two methodological contributions. First, they illustrate the

value of taking a qualitiative, and particularly a grounded theory (Charmaz & Belgrave,

2013; Corbin & Strauss, 2008; Thomas, 2006), approach to both theory development and

measure development. On one level, this approach helped to elaborate and extend

156

Brophy’s (1999, 2008a, 2008b) theoretical account, and on another level it provided a

basis for the development of the appreciation measure. Grounding theory and

measurement in students’ lived experiences adds authenticity and validity to our work,

but the review of the theoretical literature and the development of measures in the field in

Chapter 4 suggests that this valuable approach is under-utilized. Thus, the two empirical

chapters of this work not only demonstrate but also provide models of this approach.

The second methodological contribution these chapters make is to survey

development. As was suggested in Chapter 4, most measures of constructs similar to

appreciation were not developed in the most psychometrically rigorous way. High quality

measures are important, especially when seeking answers to high stakes questions like “is

college worth it?” With the availability of increasingly rigorous methods of survey

development and validation, continuing to use poorly developed measures and

inappropriate metrics of reliability and validity becomes harder to justify. Chapter 4

models appropriate measure development and illustrates the iterative nature of this

process, in part with the aim of providing a template that others could follow to improve

the measures used in the field.

Finally, these chapters also make practical contributions to the debate about what

makes education relevant. First, Chapter 3 suggest that experiences of relevance do occur

and can be characterized by the three dimensions of appreciation, a claim grounded in

students’ own experiences. Additionally, the rarity of these experiences also suggests that

there may be some legitimacy in the call to make college more relevant (Belkin, 2014;

Bond, 2015; US News, 2011; Weston, 2015). Despite lending credence to this claim, the

results of Chapter 3 caution against solutions that cut the arts and the humanities in favor

157

of vocational training (Jenvey, 2016; Kent, 2016; Zernike, 2009). Not only were most

experiences of appreciation reported in courses in these disciplines, but they were also

characterized by qualities other than transferable job skills or immediate utility. This

suggests that many of the proposed solutions to make college more relevant would not

only be ineffective but possible also harmful. Stripping content down to its purely

practical applications would leave little room in the curriculum for the very content that

appears most likely to facilitate experiences of appreciation. The nature of this content is

also important, both because of its ability to encourage appreciation but also because its

value is not always readily apparent or easily accessible. As Brophy (2008a) points out,

it can be challenging to enable [students] to recognize and appreciate the value of

many of the experiences afforded by…content…These experiences are potentially

very compelling and highly valued, but they usually do not emerge spontaneously

upon mere exposure to the content or even involvement in application activities.

(p. 137)

In other words, content with the most power to facilitate experiences of appreciation is

also content most likely to require significant scaffolding on the part of the instructor.

Taken together, these results not only highlight the critical role of the arts and humanities

in facilitating experiences that make college feel worthwhile, but also emphasize the

importance of instructors who can help students come to recognize and appreciate the life

application value of the content. Thus, by enhancing the definition of appreciation,

modeling rich and rigorous methods of grounded theory and measure development, and

speaking to key issues in the debate about the relevance of education, the three chapters

make important theoretical, methodological, and practical contributions.

158

The current studies also suggest several trajectories for future research. One of the

most limiting features of the two empirical studies is the samples that were used. Because

the samples were relatively small and drawn from one context, future work should

examine whether other populations of students describe their experiences of appreciation

in similar ways, as well as whether a revised measure of appreciation can be validated in

an independent sample. Work like this would help bolster the theoretical account of

appreciation and further improve upon the measure of appreciation.

Beyond examining appreciation in college populations, it would also be valuable

to explore whether and to what extent students in K-12 experience appreciation. In

particular, it would be important to better understand how appreciation manifests at

different stages of development. For example, some of the qualities of appreciation that

emerged from students’ experience, such as developing a new maturity or seeing the

world with a critical eye, may be particularly reflective of the developmental tasks

(Erikson, 1950, 1968) facing college students. This raises the question of how

appreciation might manifest for younger students at different stages of development.

Exploring appreciation across different levels of development is necessary if pedagogical

practices are to be effective at facilitating appreciation, because these practices likely

look very different in elementary school, middle school, high school, and college. The

current studies provide an initial understanding of how college students in particular

experience appreciation, and future work should examine this experience in other ages as

well.

Another direction for future work is examining the types of content, pedagogical

practices, and educational environments that facilitate appreciation, using a further

159

refined measure of appreciation to conduct this larger scale empirical work. Although the

results of the current studies suggest that the humanities and social sciences may be a

particularly fruitful place to start, these findings are preliminary and should be

considered, at best, suggestive. Future work could compare disciplines more

intentionally, with the goal of eventually conducting intervention work and comparing

different practices in terms of their ability to bring about appreciation.

References

Ainley, M., Hidi, S., & Berndorff, D. (2002). Interest, learning, and the psychological processes that mediate their relationship. Journal of Educational Psychology, 94, 545–561.

Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of

Educational Psychology, 84, 261–271. Anderman, E., & Maehr, M. L. (1994). Motivation and schooling in the middle grades.

Review of Educational Research, 64, 287–309. Artino, A. R., & McCoach, D. B. (2008). Development and initial validation of the online

learning value and self–efficacy scale. Journal of Educational Computing Research, 38, 279–303.

Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college

campuses. Chicago, IL: University of Chicago Press. Association of American Colleges & Universities (2014). Essential learning outcomes.

Retrieved from http://aacu.org/leap/essential-learning-outcomes Babakus, E., Ferguson, Jr., C. E., & Jöreskog, K. G. (1987). The sensitivity of

confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research, 24, 222–228.

Bandalos, D. L.& Boehm-Kaufman, M. R. (2008). Four common misconceptions in

exploratory factor analysis. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends (pp. 61–87). New York: Routledge.

Belkin, D. (2014, September 4). For some graduates, college isn’t worth the debt. The

Wall Street Journal. Retrieved from www.wsj.com/articles/for-some-graduates-college-isnt-worth-the-debt-1409803261

Benmar, K. (2015). My favorite teachers use social media: A student perspective.

Education Week, 34. Retrieved from http://www.edweek.or /ew/articles/2015/04/22/my-favorite-teachers-use-social-media-a.html

Berg, B. L. (2001). Qualitative research methods for the social sciences. Boston, MA:

Pearson. Berrett, D. (2015, October 28). How a 40-year-old idea became higher education’s next

big thing. The Chronicle of Higher Education. Retrieved from chronicle.com/article/How-a-40-Year-Old-Idea-Became/233976

Blumenfeld, P. C., Puro, P., & Mergendoller, J. R. (1992). Translating motivation into

thoughtfulness. In H. Marshall (Ed.), Redefining student learning (pp. 207–241). Norwood, NJ: Ablex.

Bohrnstedt, G. W. (1983). Measurement. In P. H. Rossi, J. D. Wright, & A. B. Anderson

(Eds.), Handbook of survey research (pp. 70–122). San Diego, CA: Academic Press.

Bond, C. (2015, September 4). Why college isn’t worth the money. Forbes. Retrieved

from http://www.forbes.com/sites/caseybond/2015/09/04/why-college-isnt-worth-the-money/

Bond, T., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in

the human sciences. New York: Routledge. Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences.

Dordrecht, The Netherlands: Springer. Brophy, J. E. (1999). Toward a model of the value aspects of motivation in education:

Developing appreciation for particular learning domains and activities. Educational Psychologist, 34(2), 75–85.

Brophy, J. E. (2008a). Developing students’ appreciation for what is taught in school.

Educational Psychologist, 43(3), 132–141. Brophy, J. E. (2008b). Scaffolding appreciation for school learning: An update. In M.

Maehr, S. Karabenick, & T. Urdan (Eds.), Advances in motivation and achievement, Vol. 15, Social psychological perspectives (pp. 1–48). Bingley, UK: Emerald.

Brophy, J. E. (2010). Motivating students to learn (3rd ed.). NY: Routledge. Brown, T. A. (2015). Confirmatory factor analysis for applied research. New York:

Guilford Publications.

162

Bull, B. (2015, September 14). Why colleges should support alternative credentials. The Chronicle of Higher Education. Retrieved from chronicle.com/article/Why-Colleges-Should-Support/232965

Carlozo, L. (2012, March 27). Why college students stop short of a degree. Reuters.

Retrieved from www.reuters.com/article/us-attn-andrea-education-dropouts-idUSBRE82Q0Y120120327

Carroll, J. B. (1961). The nature of the data, or how to choose a correlation

coefficient. Psychometrika, 26, 347–372. Cattell, R. B. (1978). The scientific use of factor analysis. New York: Plenum. Charmaz, K., & Belgrave, L. (2012). Qualitative interviewing and grounded theory

analysis. In J. F. Gubrium, J. A. Holstein, A. B. Marvasti, & K. D. McKinney (Eds.), The SAGE handbook of interview research: The complexity of the craft (2nd ed., pp. 347–367). Thousand Oaks, CA: Sage.

Churchill, G. A., Jr. (1979). A paradigm for developing better measures of marketing

constructs. Journal of Marketing Research, 16, 64–73. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale

development. Psychological Assessment, 7, 309–319. Clune, M. W. (2015, December 6). Degrees of ignorance. The Chronicle of Higher

Education. Retrieved from chronicle.com.proxy.lib.ohio-state.edu/article/The-Gutting-of-Gen-Ed/234453

Common Core State Standards Initiative (2015). English language arts standards »

Reading: literature » Grade 11-12. Retrieved from http://www.corestandards.org/ELA-Literacy/RL/11-12/

Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and

procedures for developing grounded theory (3rd ed.). Thousand Oaks, CA: Sage. Creswell, J. W. (2012). Qualitative inquiry and research design: Choosing among five

approaches. Thousand Oaks, CA: SAGE. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York:

Harper Perennial. de Vaus, D. A. (2002). Surveys in social research (5th ed.). London: Routledge.

163

Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125, 627–668.

Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and

education: The self-determination perspective. Educational Psychologist, 26(3 & 4), 325–346.

DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA:

Sage. Dewey, J. (1938). Experience and education. New York: Macmillan. Dewey, J. (1958). Experience and nature. New York: Dover. Dewey, J. (1980). Art as experience. New York: Perigee. (Original work published

1934). DiStefano, C. (2002). The impact of categorization with confirmatory factor

analysis. Structural Equation Modeling, 9, 327–346. Durik, A. M., & Harackiewicz, J. M. (2007). Different strokes for different folks: How

individual interest moderates the effects of situational factors on task interest. Journal of Educational Psychology, 99, 597–610.

Dweck, C., & Leggett, E. (1988). A social-cognitive approach to motivation and

personality. Psychological Review, 95, 256–273. Eccles, J. S. (2005). Subjective task value and the Eccles et al. model of achievement-

related choices. In A. J. Elliott & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 105–121). New York: Guilford.

Eccles, J. S. (2009). Who am I and what am I going to do with my life? Personal and

collective identities as motivators of action. Educational Psychologist, 44(2), 78–89.

Eccles, J. S., Adler, T. E., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., &

Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and Achievement Motives (pp. 75–146). San Francisco: W. H. Freeman.

Eccles, J. S., Adler, T. E., & Meece, J. L. (1984). Sex differences in achievement: A test

of alternate theories. Journal of Personality and Social Psychology, 46, 26–43.

164

Eccles, J. S., Barber, B., & Jozefowicz, D. (1999). Linking gender to education,

occupation, and recreational choices: Applying the Eccles et al. model of achievement-related choices. In W. B. Swann, J. H. Langlois, & L. A. Gilbert (Eds.), Sexism and stereotypes in modern society: The gender science of Janet Taylor Spence (pp. 153–192). Washington, DC: APA Press.

Eccles, J. S., & Harold, R. D. (1992). Gender differences in educational and occupational

patterns among the gifted. In N. Colangelo, S. G. Assouline, & D. L. Ambroson (Eds.), Talent development: Proceedings from the 1991 Henry B. and Jocelyn Wallace National Research Symposium on Talent Development (pp. 3–29). Unionville, NY: Trillium Press.

Eccles, J. S., & Wigfield, A. (1995). In the mind of the actor: The structure of

adolescents’ achievement task values and expectancy-related beliefs. Personality and Social Psychology Bulletin, 21, 215–225.

Eccles, J., Wigfield, A., Harold, R. D., & Blumenfeld, P. (1993). Age and gender

differences in children’s self‐ and task perceptions during elementary school. Child Development, 64, 830–847.

Elmore, K. C., & Oyserman, D. (2012). If ‘we’ can succeed, ‘I’ can too: Identity-based

motivation and gender in the classroom. Contemporary Educational Psychology, 37, 176–185.

Erikson, E. H. (1950). Childhood and society. New York: Norton. Erikson, E. H. (1968). Identity: Youth and crisis. New York: Norton. Erikson, E. H. (1968). Identity: Youth and crisis. New York: Norton. Faircloth, B. S. (2012). “Wearing a mask” vs. connecting identity with learning.

Contemporary Educational Psychology, 37, 186–194. Festinger, L. (1962). Cognitive dissonance. Scientific American, 207(4), 93–106. Fisher, W. P. (1992). Reliability statistics. Rasch measurement transactions, 6(3), 238. Flum, H., & Kaplan, A. (2003). Training Jewish and Arab school counselors in Israel:

Exploring a professional identity in a conflicted society. In D. M. McInerney & S. Van Etten (Eds.), Research on sociocultural influences of motivation and learning. Sociocultural influences and teacher education programs (Vol. 3, pp. 135–163). Greenwich, CT: Information Age.

165

Flum, H., & Kaplan, A. (2006). Exploratory orientation as an educational goal. Educational Psychologist, 41(2), 99–110.

Fowler, F. J., Jr. (2009). Survey research methods (4th ed.). Thousand Oaks, CA: Sage. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential

of the concept, state of the evidence. Review of Educational Research, 74, 59–109.

Gallup (2015). Great jobs, great lives. The relationship between student debt,

experiences and perceptions of college worth. Gallup-Purdue index 2015 report. Retrieved from http://www.gallup.com/file/services/185924/GPI_Report_2015_09.24.2015.pdf

Gee, J. P. (2000). Identity as an analytic lens for research in education. Review of

Research in Education, 25, 99–125. Gilley, W., & Uhlig, G. (1993). Factor analysis and ordinal data. Education, 114, 258–

264. Girod, M., & Wong, D. (2002). An aesthetic (Deweyan) perspective on science learning:

Case studies of three fourth graders. The Elementary School Journal, 102, 199–224.

Girod, M., Rau, C., & Schepige, A. (2003). Appreciating the beauty of science ideas:

Teaching for aesthetic understanding. Science Education, 87, 574–587. Girod, M., Twyman, T., & Wojcikiewicz, S. (2010). Teaching and learning science for

transformative, aesthetic experience. Journal of Science Teacher Education, 21, 801–824.

Glaser, B. & Strauss, A. (1967). The discovery of grounded theory: Strategies for

qualitative research. New York: Aldine Publishing Company. Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for

selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377–393.

Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum. Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An

experiment with data saturation and variability. Field Methods, 18(1), 59–82.

166

Gugiu, P. C., Coryn, C., Clark, R., & Kuehn, A. (2009). Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument. Chronic Illness, 5, 268–276.

Harackiewicz, J. M., Barron, K. E., Tauer, J. M., & Elliot, A. J. (2002). Predicting

success in college: A longitudinal study of achievement goals and ability measures as predictors of interest and performance from freshman year through graduation. Journal of Educational Psychology, 94, 562–575.

Harris, J., Mishra, P., & Koehler, M. (2009). Teachers’ technological pedagogical content

knowledge and learning activity types: Curriculum-based technology integration reframed. Journal of Research on Technology in Education, 41(4), 393–416.

Harackiewicz, J. M., Durik, A. M., Barron, K. E., Linnenbrink-Garcia, L., & Tauer, J. M.

(2008). The role of achievement goals in the development of interest: Reciprocal relations between achievement goals, interest, and performance. Journal of Educational Psychology, 100, 105–122.

Harshman, R. A., & Reddon, J. R. (1983). Determining the number of factors by

comparing real with random data: A serious flaw and some possible corrections. Proceedings of the Classification Society of North America at Philadelphia, 14–15.

Hart Research Associates. (2015). Falling short? College learning and career success.

Washington, DC: Association of American Colleges and Universities. Hasak, J. (2015). Make K-12 skills relevant to students. Education Week, 34. Retrieved

from http://www.edweek.org/ew/articles/2015/01/21/make-k-12-skills-relevant-to-students.html

Häussler, P., & Hoffmann, L. (2002). An intervention study to enhance girls’ interest,

self-concept, and achievement in physics classes. Journal of Research in Science Teaching, 39, 870–888.

Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in

exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191–205.

Heddy, B. C., & Sinatra, G. M. (2013). Transforming misconceptions: Using

transformative experience to promote positive affect and conceptual change in students learning about biological evolution. Science Education, 97, 723–744.

Hidi, S. (2006). Interest: A motivational variable with a difference. Educational Research

Review, 7, 323–350.

167

Hidi, S., & Baird, W. (1986). Interestingness—A neglected variable in discourse

processing. Cognitive Science, 10(2), 179–194. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest

development. Educational Psychologist, 41(2), 111–127. Hidi, S., Renninger, K. A., & Krapp, A. (2004). Interest, a motivational variable that

combines affective and cognitive functioning. In D. Y. Dai & R. J. Sternberg (Eds.), Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development (pp. 89–115). Mahwah, NJ: Erlbaum.

Hinkin, T. R. (1995). A review of scale development practices in the study of

organizations. Journal of Management, 21, 967–988. Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey

questionnaires. Organizational Research Methods, 1, 104–121. Hoffmann, L., Lehrke, M., & Todt, E. (1985). Development and change of pupils’

interest in physics: Design of longitudinal study (grade 5–10). In M. Lehrke, L. Hoffmann, & P. L. Gardner (Eds.), Interests in science and technology (pp. 71–80). Kiel: Institute for Science Education.

Holgado–Tello, F. P., Chacón–Moscoso, S., Barbero–García, I., & Vila–Abad, E. (2010).

Polychoric versus Pearson correlations in exploratory and confirmatory factor analysis of ordinal variables. Quality & Quantity, 44, 153–166.

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis.

Psychometrika, 32, 179–185. Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing

interest and performance with a utility value intervention. Journal of Educational Psychology, 102, 880–895.

Jenvey, N. (2016, February 10). Why the OECD wants a global effort to measure student

learning. The Chronicle of Higher Education. Retrieved from chronicle.com.proxy.lib.ohio-state.edu/article/Why-the-OECD-Wants-a-Global/235264

Jones, S. R., Arminio, J., & Torres, V. (2006). Negotiating the complexities of qualitative

research in higher education: Fundamental elements and issues. Florence, KY: Brunner-Routledge.

168

Jöreskog, K. G. & Moustaki, I. (2006). Factor analysis of ordinal variables with full information maximum likelihood. Hillsdale, NJ: Lawrence Erlbaum Associates.

Kaplan, A., & Flum, H. (2006, July). Facilitating an exploratory orientation in school.

Paper presented at the 4th international biennial SELF research conference, Ann Arbor, MI.

Kaplan, A., & Flum, H. (2010). Achievement goal orientations and identity formation

styles. Educational Research Review, 5, 50–67. Kaplan, A., & Flum, H. (2012). Identity formation in educational settings: A critical

focus for education in the 21st century. Contemporary Educational Psychology, 37(3), 171–175.

Kent, C. (2016, February 7). To solve the skills gap in hiring, create expectations in the

classroom. The Chronicle of Higher Education. Retrieved from chronicle.com/article/To-Solve-the-Skills-Gap-in/235206

Knogler, M. & Lewalter, D. (2014). What makes simulation games motivating? Design-

based research on learner’s motivation in simulation gaming. In W. Kriz, T. Eiselen, & W. Manhal (Eds.), The shift from teaching to learning: Individual, collective and organizational learning through gaming simulation. (pp. 150–162). Dornbirn, Austria: ISAGA.

Krapp, A. (2002). An educational–psychological theory of interest and its relation to

SDT. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self- determination research (pp. 405–427). New York, NY: University of Rochester Press.

Krapp, A. (2007). An educational–psychological conceptualization of interest.

International Journal of Educational and Vocational Guidance, 7, 5–21. Krapp, A., Hidi, S., & Renninger, K. A. (1992). Interest, learning, and development. In

K. A. Renninger, S. Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 3–25). Hillsdale, NJ: Erlbaum.

Krapp, A., & Prenzel, M. (2011). Research on interest in science: Theories, methods, and

findings. International Journal of Science Education, 33, 27–50. Lemann, N. (2016, January 8). What should graduates know? The Chronicle of Higher

Education. Retrieved from chronicle.com.proxy.lib.ohio-state.edu/article/What-Should-Graduates-Know-/234824

169

Lewin, T. (2013, October 30). As interest fades in the humanities, colleges worry. The New York Times. Retrieved from www.nytimes.com/2013/10/31/education/as-interest-fades-in-the-humanities-colleges-worry.html

Linacre, J. M. (1997). KR-20 / Cronbach alpha or Rasch person reliability: Which tells

the “truth”? Rasch Measurement Transactions, 11, 580–581. Linacre, J. M. (2000). Comparing “partial credit models” (PCM) and “rating scale

models” (RSM). Rasch Measurement Transactions, 14, 768. Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean?

Rasch measurement transactions,16(2), 878. Linacre, J. M. (2006). Demarcating category intervals. Rasch Measurement Transactions,

19(3), 10341–10343. Linacre, J. M. (2010). When to stop removing items and persons in Rasch misfit

analysis? Rasch Measurement Transactions, 23, 1241. Linacre, J. M. (2016). A user’s guide to Winsteps Ministeps Rasch-model computer

programs [version 3.92.0]. Retrieved from http://www.winsteps.com/winman/index.htm

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. Linnenbrink-Garcia, L., Durik, A. M., Conley, A. M., Barron, K. E., Tauer, J. M.,

Karabenick, S. A., & Harackiewicz, J. M. (2010). Measuring situational interest in academic domains. Educational and Psychological Measurement, 70, 647–671.

Little, R. J., & Rubin, D. B. (1987). Statistical analysis with missing data. New York:

Wiley. Liu, X. (2010). Using and developing measurement instruments in science education: A

Rasch modeling approach. Charlotte, NC: Information Age Publishing. Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content analysis in mass

communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587–604.

Marcia, J. E. (1966). Development and validation of ego identity status. Journal of

Personality and Social Psychology, 3, 551–558. Marcia, J. E. (1980). Identity in adolescence. In J. Abelson (Ed.), Handbook of

adolescent psychology (pp. 159–187). New York: John Wiley.

170

Marcia, J. E. (1993). The ego identity status approach to ego identity. In J. E. Marcia, A.

S. Waterman, D. R. Matteson, S. L. Archer, & J. L. Orlofsky (Eds.), Ego identity: A handbook for psychosocial research (pp. 3–21). New York: Springer-Verlag.

Markus, H. R., & Nurius, P. (1986). Possible selves. American Psychologist, 41, 954–

969. Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the

design of multimedia instruction. American Psychologist, 63, 760–769. Meece, J. L., Wigfield, A., & Eccles, J. S. (1990). Predictors of math anxiety and its

influence on young adolescents’ course enrollment intentions and performance in mathematics. Journal of Educational Psychology, 82, 60–70.

Miles, M. B, & Huberman, A. M. (1994). Qualitative data analysis. Thousand Oaks, CA:

Sage. Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of

classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255–273.

Mitchell, M. (1993). Situational interest: Its multifaceted structure in the secondary

school mathematics classroom. Journal of Educational Psychology, 85, 424–436. Mourshed, M., Farrell, D. & Dominic, B. (2012). Education to employment: Designing a

system that works. Washington, DC: McKinsey Center on Government. Moustakas, C. (1994). Phenomenological research methods. Thousand Oaks, CA:

SAGE. Multon, K. D. (2010). Interrater relaibility. In N. J. Salkind (Ed.), Encyclopedia of

research design (pp. 626–629). Thousand Oaks, CA: Sage. Nadel, S. (2014). Developing a social support measurement instrument: A

methodological approach to measuring undergraduate perceptions of social support. (Unpublished master’s thesis). The Ohio State University, Columbus, OH.

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw Hill.

171

Olsson U. (1979a). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443–460.

Olsson, U. (1979b). On the robustness of factor analysis against crude classification of

the observations. Multivariate Behavioral Research, 14, 485–500. Oyserman, D. (2007). Social identity and self-regulation. In A. Kruglanski, & T. Higgins

(Eds.), Handbook of social psychology (2nd ed., pp. 432−453). New York: Guilford Press.

Oyserman, D. (2009). Identity-based motivation: Implications for action-readiness,

procedural-readiness, and consumer behavior. Journal of Consumer Psychology, 19, 250–260.

Oyserman, D., & Destin, M. (2010). Identity-based motivation: Implications for

intervention. The Counseling Psychologist, 38, 1001–1043. Oyserman, D., & Markus, H. R. (1990). Possible selves and delinquency. Journal of

Personality and Social Psychology, 59, 112–125. Oyserman, D., Bybee, D., & Terry, K. (2006). Possible selves and academic outcomes:

How and when possible selves impel action. Journal of Personality and Social Psychology, 91, 188–204.

Oyserman, D., Bybee, D., Terry, K., & Hart-Johnson, T. (2004). Possible selves as

roadmaps. Journal of Research in Personality, 38, 130–149. Oyserman, D., Terry, K., & Bybee, D. (2002). A possible selves intervention to enhance

school involvement. Journal of Adolescence, 25, 313–326. Papert, S. (1987). A critique of technocentrism in thinking about the school of the future.

Retrieved from http://www.papert.org/articles/ACritiqueofTechnocentrism.html Parsons, J. E. (1980). Self-perceptions, task perceptions and academic choice: Origins

and change. Unpublished final technical report to the National Institution of Education, Washington, DC. (ERIC Document Reproduction Service No. ED186477.)

Paxson, C. H. (2013, August 20). The economic case for saving the humanities. New

Republic. Retrieved from https://newrepublic.com/article/114392/christina-paxson-president-brown-humanities-can-save-us

Perez, T., Cromley, J. G., & Kaplan, A. (2014). The role of identity development, values,

and costs in college STEM retention. Journal of Educational Psychology, 106,

172

315–329. Pugh, K. J. (2002). Teaching for idea-based, transformative experiences in science: An

investigation of the effectiveness of two instructional elements. Teachers College Record, 104, 1101–1137.

Pugh, K. J. (2004). Newton’s laws beyond the classroom walls. Science Education, 88,

182–196. Pugh, K. J. (2011). Transformative experience: An integrative construct in the spirit of

Deweyan pragmatism. Educational Psychologist, 46(2), 107–121. Pugh, K. J., & Girod, M. (2007). Science, art, and experience: Constructing a science

pedagogy from Dewey's aesthetics. Journal of Science Teacher Education, 18, 9–27.

Pugh, K. J., Linnenbrink-Garcia, L., Koskey, K. L., Stewart, V. C., & Manzey, C.

(2010a). Motivation, learning, and transformative experience: A study of deep engagement in science. Science Education, 94, 1–28.

Pugh, K. J., Linnenbrink-Garcia, L., Koskey, K. L., Stewart, V. C., & Manzey, C.

(2010b). Teaching for transformative experiences and conceptual change: A case study and evaluation of a high school biology teacher's experience. Cognition and Instruction, 28, 273–316.

Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests.

Chicago: University of Chicago Press. Renninger, K. A., & Bachrach, J. E. (2015). Studying triggers for interest and

engagement using observational methods. Educational Psychologist, 50(1), 58–69.

Renninger, K. A., & Hidi, S. (2002). Student interest and achievement: Developmental

issues raised by a case study. In A. Wigfield & J. S. Eccles (Eds.), Development of achievement motivation (pp. 173–195). New York, NY: Academic Press.

Renninger, K. A., & Hidi, S. (2011). Revisiting the conceptualization, measurement, and

generation of interest. Educational Psychologist, 46(3), 168–184. Renninger, K. A., Nieswandt, M., & Hidi, S. (Eds.). (2015). Interest in Mathematics and

Science Learning. Washington, DC: American Educational Research Association. Roulston, K. (2010). Reflective interviewing: A guide to theory and practice. London:

Sage.

173

Ryan, A. M., & Patrick, H. (2001). The classroom social environment and changes in

adolescents’ motivation and engagement during middle school. American Educational Research Journal, 38, 437–460.

Salzberger, T. (2003). Item information: When gaps can be bridged. Rasch Measurement

Transactions, 17, 910–911. Sansone, C. (2009). What’s interest got to do with it?: Potential trade-offs in the self-

regulation of motivation. In J. P. Forgas, R. Baumiester, & D. Tice (Eds.), Psychology of self-regulation: Cognitive, affective, and motivational processes (pp. 35–51). New York, NY: Psychology Press.

Schiefele, U. (2001). The role of interest in motivation and learning. In J. M. Collis & S.

Messick (Eds.), Intelligence and personality: Bridging the gap in theory and measurement (pp. 163–194). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Schiefele, U. (2009). Situational and individual interest. In K. R. Wentzel & A. Wigfield

(Eds.), Handbook of motivation in school (pp. 197–223). New York: Taylor Francis.

Schraw, G., & Lehman, S. (2001). Situational interest: A review of the literature and

directions for future research. Educational Psychology Review, 13(1), 23–52. Siegel, M. A., & Ranney, M. A. (2003). Developing the Changes in Attitude about the

Relevance of Science (CARS) questionnaire and assessing two high school science classes. Journal of Research in Science Teaching, 40, 757–775.

Silvia, P. J. (2005). What is interesting? Exploring the appraisal structure of interest.

Emotion, 5, 89–102. Silvia, P. J. (2006). Exploring the psychology of interest. New York: Oxford University

Press. Sinai, M., Kaplan, A., & Flum, H. (2012a). Promoting identity exploration within the

school curriculum: A design-based study in a junior high literature lesson in Israel. Contemporary Educational Psychology, 37, 195–205.

Sinai, M., Kaplan, A., & Flum, H. (2012b, April). Students’ achievement goals and

identity exploration in the classroom context. Paper presented at the annual convention of the American Educational Research Association, Vancouver, BC, Canada.

174

Smith, G. T., Fischer, S., & Fister, S. M. (2003). Incremental validity principles in test construction. Psychological Assessment, 15, 467–477.

Sparks, S. D. (2012). Studies find payoff in “personalizing” algebra. Education Week, 32.

Retrieved from http://www.edweek.org/ew/articles/2012/09/26/05personalize_ep.h32.html

Spector, P. E. (2014). Survey design and measure development. In T. D. Little (Ed.), The

Oxford handbook of quantitative methods (pp. 170–188). New York: Oxford University Press.

Symonds, W. C., Schwartz, R., & Ferguson. R. F. (2011). Pathways to prosperity:

Meeting the challenge of preparing young Americans for the 21st century. Cambridge, MA: Pathways to Prosperity Project, Harvard University Graduate School of Education.

Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Boston: Pearson. Tennant, A., & Conaghan, P. G. (2007). The Rasch measurement model in rheumatology:

What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Care and Research, 57, 1358–1362.

The Ohio State University Enrollment Services (2014). Applications, enrollment and new

freshman profile comparisons. Retrieved from https://news.osu.edu/assets/uploads/Projected_Profile14_Media_Day_081114.pdf

The Ohio State University Enrollment Services (2015). Highlights of fifteenth day

enrollment for the spring term 2015. Retrieved from https://oesar.osu.edu/pdf/student_enrollment/15th/enrollment/15THDAY_SPRING_2015.pdf

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation

data. American Journal of Evaluation, 27, 237–246. Turner, J. C., Christensen, A., Kackar-Cam, H. Z., Trucano, M., & Fulmer, S. M. (2014).

Enhancing students’ engagement: Report of a 3-year intervention with middle school teachers. American Educational Research Journal, 51, 1195–1226.

U.S. Department of Education (2015). College scorecard. Retrieved from

https://collegescorecard.ed.gov/ Urdan, T. (1997). Achievement goal theory: Past results, future directions. In M. Maehr

& P. Pintrich (Eds.), Advances in motivation and achievement (pp. 99–141). Greenwich, CT: JAI.

175

US News. (2011, November 17). Is a college degree still worth it? US News. Retrieved

from http://www.usnews.com/debate-club/is-a-college-degree-still-worth-it van Buuren, S. (2007). Multiple imputation of discrete and continuous data by fully

conditional specification. Statistical Methods in Medical Research, 16, 219–242. Van Manen, M. (1990). Researching lived experience: Human science for an action

sensitive pedagogy. London, Canada: The University of Western Ontario. Van Wyke, J., & Andrich, D. (2006). A typology of polytomously scored mathematics

items disclosed by the Rasch model: Implications for constructing a continuum of achievement. Unpublished report, Perth, Australia.

Vander Ark, T. (2014). How next-gen learning boosts rigor, relevance, and relationships.

Retrieved from http://blogs.edweek.org/edweek/on_innovation/2014/08/how_next_gen_learning_boosts_rigor_relevance_relationships.html

Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or

component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment (pp. 41–71). Boston: Kluwer.

Weissmann, J. (2012, March 29). Why do so many Americans drop out of college? The

Atlantic. Retrieved from www.theatlantic.com/business/archive/2012/03/why-do-so-many-americans-drop-out-of-college/255226/

Weston, L. (2015, October 5). Why college is still worth it even though it costs too much.

Time. Retrieved from http://time.com/money/4061150/college-degree-worth-it/ Wigfield, A., & Eccles, J. S. (1992). The development of achievement task values: A

theoretical analysis. Developmental Review, 12, 265–310. Wigfield, A., Eccles, J. S., Mac Iver, D., Reuman, D. A., & Midgley, C. (1991).

Transitions during early adolescence: Changes in children's domain-specific self-perceptions and general self-esteem across the transition to junior high school. Developmental Psychology, 27, 552–565.

Wigfield, A., Eccles, J. S., Yoon, K. S., Harold, R. D., Arbreton, A. J., Freedman-Doan,

C., & Blumenfeld, P. C. (1997). Change in children's competence beliefs and subjective task values across the elementary school years: A 3-year study. Journal of Educational Psychology, 89, 451–469.

176

Williams, A. (2012, November 30). Saying no to college. The New York Times. Retrieved

from www.nytimes.com/2012/12/02/fashion/saying-no-to-college.html Wong, D. (2007). Beyond control and rationality: Dewey, aesthetics, motivation, and

educative experiences. Teachers College Record, 109, 192–220. Wong, D., Pugh, K. J., the Dewey Ideas Group at Michigan State University. (2001).

Learning science: A Deweyan perspective. Journal of Research in Science Teaching, 38, 317–336.

Wright, B. D. (1998). Model selection: Rating scale model (RSM) or partial credit model

(PCM)? Rasch Measurement Transactions, 12(3), 641–642. Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch

Measurement Transactions, 8(3), 370. Wright, B. D., & Masters, G. N. (1981). The measurement of knowledge and attitude

(Research Memorandum No. 30). Chicago, IL: University of Chicago, MESA Psychometric Laboratory.

Wright, B. D., & Masters, G. N. (1982). Rating scale analysis (Rasch measurement).

Chicago: MESA Press. Yuan, Y. C. (2000). Multiple imputation for missing data: Concepts and new

development (SAS Tech. Rep. No. P267-25). Rockville, MD: SAS Institute. Zernike, K. (2009, December 29). Making college “relevant.” The New York Times.

Retrieved from http://www.nytimes.com/2010/01/03/education/edlife/03careerism-t.html?pagewanted=all

Zimmer, R. J. (2013, March 1). The myth of the successful college dropout: Why it could

make millions of young Americans poorer. The Atlantic. Retrieved from www.theatlantic.com/business/archive/2013/03/the-myth-of-the-successful-college-dropout-why-it-could-make-millions-of-young-americans-poorer/273628/

Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the

number of components to retain. Psychological Bulletin, 99, 432–442.

177

Appendix A

IRB Approval Letter – Study I

178

179

Appendix B

Online Screening Survey 1. Please select your gender. (Male, Female, Transgender, Self-defined)

2. Please select your race/ethnicity. (Hispanic/Latino, American Indian or Alaska

Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, White)

3. Please select your undergraduate status. (First-Year, Second-Year, Third-Year,

Fourth-Year, Fifth-Year or Higher) 4. Are you a first generation college student? In other words, is it the case that neither of

your parents/legal guardians earned a bachelor’s degree? (Yes, No) 5. Please select your major. (Drop down menu; populated with official list of undergrad

majors) 6. If you are double majoring, please select your second major. Otherwise, please select

N/A. (Drop down menu; populated with official list of undergrad majors) 7. Please enter your cumulative undergraduate GPA. (Open response) 8. Please enter your age. (Open response) 9. In your studies, what is your primary area of interest? For example, if you are a

psychology major, your primary area of interest may be in counseling psychology, in experimental psychology, in school psychology, or in another area. (Open response)

10. Please enter your OSU email address if you are willing to be contacted for an

interview. Entering your email does not guarantee that you will be selected, nor does it bind you to participating in an interview. If you are selected, you will be given an opportunity to decide if you would like to participate. If you are selected for and choose to participate in an interview, you will be compensated $20 in cash. (Open response)

In this study, I am interested in learning more about your experiences in courses that have felt especially worthwhile and/or relevant to your life outside of school; in other words, the kind of courses that really stand out in your memory. I am particularly interested in

180

courses that have felt worthwhile for reasons beyond the job skills you acquired or the degree requirements you were able to fulfill. Please take a moment to think about the classes that you have taken as an undergraduate that have felt worthwhile, at OSU or any other institutions you have attended. 11. Have you taken any courses during undergraduate that have felt worthwhile and/or

relevant to your life outside of school? (Yes, No)

à If #11 is yes…

12. Were any of these courses outside of your dominant area of interest, which you listed as _______ ? (Populated with answer to question 9). Courses that are required for your major but are NOT focused on ______ (question 9 answer) would be considered OUTSIDE of your dominant area of interest. (Yes, No)

à If yes… Please consider the undergraduate course outside of your

primary area of interest that felt the most worthwhile. à If no… Please consider the undergraduate course that felt the most worthwhile.

13. When did you take this course? (Drop down menu; semester and year)

14. What discipline was this course in? (Open response)

15. What was your experience in the course like? Please elaborate and include specific examples if you are able. (Open response)

16. What about the content of the course felt worthwhile or relevant? Please elaborate and include specific examples if you are able. (Open response)

17. What sorts of things did the instructor do that made the course

content feel worthwhile or relevant? Please elaborate and include specific examples if you are able. (Open response)

End of survey for participants who answered YES to #11

181

à If #11 is no…

12. What were you hoping for in your classes, beyond fulfilling degree requirements or acquiring job skills? Please elaborate and include specific examples if you are able. (Open response)

13. What would have made your classes more worthwhile or relevant to your life outside of school? Please elaborate and include specific examples if you are able. (Open response)

End of survey for participants who answered NO to #11

182

Appendix C

Semi-structured Interview Questions

• Which course did you think about when you filled out the online survey? What was the name of the course?

• What about the course felt worthwhile? Could you give an example?

• How did you choose to take this course?

• What did you expect when you enrolled in the course? What was the course actually like? How did it meet or differ from your expectations?

• Could you tell me about your experience in the course? Did your experience

change at all from the beginning to the end of the course? How so?

• If you were to choose a few adjectives, how would you describe the way that you felt while completing the assignments?

• What did you find most challenging in the course? Best or favorite part?

• What sorts of things did you take away from the course, if anything? Did you

think about what you were learning outside of class while you were taking it, not counting homework and such, or did you think about the content mostly just during class? Do you ever still think about things you learned?

• How did the instructor teach the course? What sorts of things did he or she do to

make the content relevant to you? Could you give an example?

• In what ways was your experience in this course similar to and different from your experiences in courses in your area of study?

• Have you taken other courses like this? What brought this course in particular to

mind when you were completing the survey?

• What was your academic trajectory like before taking the course? What have your studies been like after taking the course?

183

Appendix D

IRB Approval Letter – Study II

184

OfficeofResponsibleResearchPractices300ResearchAdministrationBuilding

1960KennyRoadColumbus,OH43210-1063

Phone(614)688-8457Fax(614)688-0366www.orrp.osu.edu

ProtocolTitle: DevelopingAppreciationforCollegeCourseworkProtocolNumber: 2015E0168PrincipalInvestigator: LynleyAndermanDetermination: TheOfficeofResponsibleResearchPracticeshasdeterminedthe

abovereferencedprojectexemptfromIRBreview.DateofDetermination: 03/23/2015QualifyingCategory: 02Attachments: None Dear Investigators,Please note the following about the above determination:

• Retain a copy of this correspondence for your records.• Only the Ohio State staff and students named on the application are approved

as Ohio State investigators and/or key personnel for this study.• No changes may be made to exempt research (e.g., personnel, recruitment

procedures, advertisements, instruments, etc.). If changes are needed, a new application for exemption must be submitted for review and approval prior to implementing the changes.

• Records relating to the research (including signed consent forms) must be retained and available for audit for at least 5 years after the study is closed. For more information, see university policies, Institutional Data and Research Data.

• It is the responsibility of the investigators to promptly report events that may represent unanticipated problems involving risks to subjects or others.

This determination is issued under The Ohio State University’s OHRP Federalwide Assurance #00006378. Human research protection program policies, procedures, and guidance can be found on the ORRP website. Please feel free to contact the Office of Responsible Research Practices with any questions or concerns.

Jake Stoddard, QI Specialist, ORRP

185

Appendix E

Appreciation Survey Items

Original Appreciation Items

Please consider the course you selected (name of course provided in question 8 or 9) and your experience thus far. For each question, indicate the degree to which these statements reflect experiences you have had as a result of being in that course. (5-point Likert scale: 1 = Strongly Disagree, 5 = Strongly Agree) Cognitive Items

1. I have a thorough understanding of the material that was taught in this class.* 2. I comprehend many aspects of the world at a deeper level than I did before. 3. I better understand some part of my everyday experience. 4. I occasionally see examples of what we discussed in my life outside of school. † 5. I see some aspects of the world differently than before. ‡ 6. I see my everyday experience in a completely new light. 7. I am less likely to take things on face value than I was before. 8. I consider the ideas I encounter more deeply than I did before. 9. I analyze every aspect of my everyday experiences. ° 10. I see a few ways in which what I have learned could apply to my life. † 11. I perceive opportunities every day to use what I have learned in my life in general. 12. I recognize ways I can use what I have learned in my everyday life. † 13. I have become more familiar with different viewpoints after taking this class. 14. I see the world from more than one perspective after taking this class. 15. I now see the world from a different perspective than my friends who have not

taken this course. Behavioral Items

1. I have had at least one conversation with a friend or family member about what I am learning. †

2. I can’t stop talking about what I am learning. 3. I sometimes bring up what I am learning in conversations with other people. † 4. I invest a little more effort into this class than I usually do in other classes. 5. I participate much more in this class than I usually do in other classes. 6. I spend more time reading course material for this class than what is minimally

required. 7. I occasionally apply what I have learned in class to my life outside of school. †

186

8. I seek out examples in my everyday life of things we discussed in class. † 9. I have used a skill I learned in this class in a different context. † 10. I have changed how I behave in my personal life. 11. I have significantly changed the way I approach my studies. 12. I have somewhat changed how I interact with others. 13. This class convinced me to major in this discipline or reinforced my decision to

stay in this discipline. 14. I joined an extracurricular related to the discipline of this class. 15. This class convinced me to pursue a career in this discipline or reinforced my

prior decision to pursue a career in this discipline. Affective Items

1. I have matured as a person. * 2. Learning more about the topic of this class has made me a more well-rounded

person. 3. I have been transformed in important ways by this class. * 4. Now that I have taken this class, I can see why this knowledge is important. 5. I can see why people who study this discipline find it worthwhile. 6. I see value in the discipline of this course that I did not fully recognize before. 7. Learning the material covered in this class makes me feel slightly more prepared

in the rest of my life. 8. Taking this class has made me feel more self-reliant in solving problems. † 9. Being able to talk about the topic of this class with others makes me feel more

confident. 10. Taking this class has made me aware of topics that are important to know about. 11. This class has contributed to my education in meaningful ways. 12. My college experience would not have been complete without taking this class. † 13. I feel that what I have learned in this class will be valuable in my everyday life in

the future. 14. I would take this course again. ° 15. This class has significantly enriched my understanding of the world. ‡

* Items removed because they loaded equally across all three factors † Items removed because they loaded on the wrong factor ‡ Items removed because they cross-loaded onto two factors ° Items removed because of misfit in the Rasch analysis Final Appreciation Items Cognitive Items

1. I comprehend many aspects of the world at a deeper level than I did before. 2. I better understand some part of my everyday experience. 3. I see my everyday experience in a completely new light. 4. I am less likely to take things on face value than I was before. 5. I consider the ideas I encounter more deeply than I did before.

187

6. I perceive opportunities every day to use what I have learned in my life in general. 7. I have become more familiar with different viewpoints after taking this class. 8. I see the world from more than one perspective after taking this class. 9. I now see the world from a different perspective than my friends who have not

taken this course.

Behavioral Items 1. I can’t stop talking about what I am learning. 2. I invest a little more effort into this class than I usually do in other classes. 3. I participate much more in this class than I usually do in other classes. 4. I spend more time reading course material for this class than what is minimally

required. 5. I have changed how I behave in my personal life. 6. I have significantly changed the way I approach my studies. 7. I have somewhat changed how I interact with others. 8. This class convinced me to major in this discipline or reinforced my decision to

stay in this discipline. 9. I joined an extracurricular related to the discipline of this class. 10. This class convinced me to pursue a career in this discipline or reinforced my

prior decision to pursue a career in this discipline.

Affective Items 1. Learning more about the topic of this class has made me a more well-rounded

person. 2. Now that I have taken this class, I can see why this knowledge is important. 3. I can see why people who study this discipline find it worthwhile. 4. I see value in the discipline of this course that I did not fully recognize before. 5. Learning the material covered in this class makes me feel slightly more prepared

in the rest of my life. 6. Being able to talk about the topic of this class with others makes me feel more

confident. 7. Taking this class has made me aware of topics that are important to know about. 8. This class has contributed to my education in meaningful ways. 9. I feel that what I have learned in this class will be valuable in my everyday life in

the future.

188

Appendix F

Additional Survey Measures

Demographic Questions 1. Please select your undergraduate status. (First-Year, Second-Year, Third-Year,

Fourth-Year, Fifth-Year or Higher)

2. Please select your gender.

3. Please select your ethnicity. 4. Please select your major. (Drop down menu; official list of undergrad majors) 5. If you are double majoring, please select your second major. Otherwise, please select

N/A. (Drop down menu; official list of undergrad majors) 6. Please enter your cumulative undergraduate GPA. 7. Are you taking a General Education (GE) course during the current (Fall 2015)

semester? (Yes, No)

If #7 is yes à

If you are taking more than one GE course this semester, please consider the GE course you have found the most worthwhile.

8. What is the name of that course? (Fill in)

9. What discipline is this course in? (Drop down menu of official program

areas)

10. What is your anticipated grade in this course? (A, A-, B+, B, B-, C+, C, C-, D+, D, F)

189

If #7 is no à

Of the courses you are currently taking, please consider the one you have found the most worthwhile.

8. Is this course required for your major? (Yes, No)

9. What is the name of that course? (Fill in)

10. What discipline is this course in? (Drop down menu of official program

areas)

11. What is your anticipated grade in this course? (A, A-, B+, B, B-, C+, C, C-, D+, D, F)

Interest Please indicate your agreement with the following statements. (5-point Likert scale: 1 = Strongly Disagree, 5 = Strongly Agree)

Initial Interest (Hulleman et al., 2010)

When I chose to enroll in this course…

1. I thought this was an interesting subject. 2. I was not interested in the subject. (Reversed) 3. I thought I would like learning about the subject in this course. 4. I thought the subject would be interesting. 5. I had always wanted to learn more about this subject.

Triggered Situational Interest (Linnenbrink et al., 2010) 1. My instructor is exciting. 2. My instructor does things that grab my attention. 3. This class is often entertaining. 4. This class is so exciting it’s easy to pay attention.

Maintained Situational Interest – Feeling (Linnenbrink et al., 2010) 1. What we are learning in this class is fascinating to me. 2. I am excited about what we are learning in this class. 3. I like what we are learning in this class. 4. I find the material we are covering in class interesting.

190

Maintained Situational Interest – Value (Linnenbrink et al., 2010) 5. What we are studying in this class is useful for me to know. 6. The things we are studying in this class are important to me. 7. What we are learning in this class can be applied to real life. 8. We are learning valuable things in this class.

Task Value (Eccles & Wigfield, 1995) Please indicate your agreement with the following statements. (5-point Likert scale: 1 = Strongly Disagree, 5 = Strongly Agree)

Intrinsic Value 1. In general, I find working on assignments in this class very interesting. 2. I like the subject of this class very much.

Attainment Value 3. The amount of effort it will take to do well in this class is worthwhile to me. 4. I feel that, to me, being good at this subject is very important. 5. It is very important to me to get good grades in this class.

Utility Value 6. This class is useful for what I want to do in the future. 7. What I learn in this class is useful for my daily life outside school.