An Exploration of Epistemological Understanding and Participation inOnline Knowledge Building Communities

By

Bruce Cameron Forrester

A thesis submitted in conformity with the requirementsfor the degree of Doctor of Philosophy

Graduate Department of Curriculum, Teaching, and LearningUniversity of Toronto

© Copyright by Bruce CameronForrester (2006)

iiLibrary andArchives Canada

Published HeritageBranch

395 Wellington StreetOttawa ON K1A ON4Canada Canada

NOTICE:

The author has granted a non-exclusivelicense allowing Library

and Archives Canada to reproduce,

publish, archive, preserve, conserve,

communicate to the public bytelecommunication or on the Internet,

loan, distribute and sell theses

worldwide, for commercial or non-

commercial purposes,in microform,

paper, electronic and/or any other

formats.

The author retains copyrightownership and moral rights in

this thesis. Neither the thesisnor substantial extracts from it

maybeprinted or otherwisereproduced without the author's

permission.

Direction du

Patrimoine de I'édition

Bibliotheque etArchives Canada

395, rue WellingtonOttawa ON K1A ON4

Yourfile Votre référence

ISBN: 978-0-494-21814-3

Ourfile Notre référence

ISBN: 978-0-494-21814-3

AVIS:

L'auteur a accordé unelicence non exclusive

permettant a la Bibliotheque et Archives

Canada de reproduire, publier, archiver,

sauvegarder, conserver, transmettre au public

par telecommunication ou parI'Internet, préter,distribuer et vendre des theses partout dans

le monde, a des fins commerciales ou autres,

sur support microforme, papier, électronique

et/ou autres formats.

L'auteur conservela propriété du droit d'auteur

et des droits moraux qui protege cette these.Ni la these ni des extraits substantiels decelle-ci ne doivent étre imprimés ou autrement

reproduits sans son autorisation.

In compliance with the Canadian

Privacy Act some supportingforms may have been removed

from this thesis.

While these forms may be includedin the document page count,

their removal does not representany loss of content from the

thesis.

Canada

Conformémenta la loi canadienne

surla protection dela vie privée,

quelques formulaires secondairesont été enlevés de cette thése.

Bien que ces formulairesaient inclus dansla pagination,

il n'y aura aucun contenu manquant.

AN EXPLORATION OF EPISTEMOLOGICAL UNDERSTANDING AND

PARTICIPATION IN ONLINE KNOWLEDGEBUILDING COMMUNITIES

Doctor of Philosophy (2006)Bruce Cameron Forrester

Graduate Department of Curriculum, Teaching, and Learning

University of Toronto

ABSTRACT

As knowledgeis the main resource of the knowledge age, we, as a society, need to

understand whattools and skills can support developmentof this resource. A learner's personal

epistemology affects knowledge work, making the assessment and understanding oflearner's

epistemologies and its impact on learningcritical. Traditionally, entry into the world of

knowledge creation has been available only after years of advanced study. However, in the

knowledgeage,entry into this world can be available at a much younger age. Knowledge

building is a promising pedagogy that could be the key to opening this door. This design research

study explored the effects of epistemological view on participation in two online courses (n=12,

n=15). A pre- and post-essay was given to measure changein levels of epistemological

understanding and useofjustification, specifically dialectical reasoning. Directed learning logs

were used to encourage and code metacognitive and epistemological thought. An indication of

level of participation in online courses was producedusing analytic tools underlying the online

environment. Changesin specific aspects of epistemological understanding were identified.

Results indicated that participants with a high participation rating in belief mode epistemology

were more likely to use sophisticated justification, such as dialectical reflection. Additionally,

these participants showed greater indication of metacognitive activity within the learning logs

(when prompted and when not prompted), were more collaborative, and were more often rated as

an evaluativist (the highest level of epistemological understanding in Kuhnet al’s. model(2000))

in the post-essay. As anticipated, design mode epistemology - a primary indication of knowledge

building - was underrepresented even in the highest levels of belief mode epistemology.

Widespread change occurred in the acceptance of peers as sources of knowledge. This has

significant implications for the practice of teaching. Participation patterns did not depend on

numberof graduate courses taken, numberof previous online courses, nor previous familiarity

with the course platform. Future research should explore whether learning logs can facilitate

metacognition, encourage greater epistemological sophistication, and lead to augmented

knowledge building.

ii

AKNOWLEDGEMENTS

Manythanks to my entire committee Clare Brett, Marlene Scardamalia, Jim

Hewitt, Dan Keating and my external Kevin O’Neill.

Clare, you provided great insight and moral support through the entire process — I

could not have finished without you! Malrene, your guidance brought my thesis to new

heights and has opened myeyes to a world of new and exciting research. Jim, your

careful attention to methodological detail ensured a solid underpinning to mythesis.

Dan, you provided meinsights that could only come from your ability to see the big

picture. Finally Kevin, your wise input helped to round out my thesis and allowed me to

further see howit fits into the entire literature.

To my family, I owe many hours of payback for the time I spent away during the

writing of this thesis. Myléne, merci pour ton support et compréhension — je t'aime. To

my boys — William, Andrew, and Maximeyouare always mygreatest inspiration. Your

curiosity and joy of life keeps mestrong.

To my parents, Jim and Ruth, I want to thank you both for your unending

encouragement and love. You have both had the utmost influence on mylove of learning

and critical stance towards knowledge. You started me onthe trajectory oflifelong

learning and your ongoing example continuesto provide mestrength today.

Finally, many thanksto all of the friends I met along the way. You havetruly

madethis an enriching experience. Long live the Duke.

ill

TABLE OF CONTENTS

ADSUIACE ..sssessensrsessonevssssensssoessassaensvnsssesevsessensosasassserasnsvessecnssssvasssonevsssesenenessnenaseten ii

Aknowledgements .0..........:cccsscesssecssceesceeeseesenecseerssecenseensecsueeeavecseesenseceneeeesnseeareees ill

Table of Contents... cescesscrecensesssesecesecssscnascnecssesesseneseeersceraeesseesterenseeneenasonaes iv

List Of Tables ....... cee eeccscssccssecsnecseececesceseeesecseeeseeesceeseeaeesseceecsueeasenseeseaeseeeateaten ix

List Of Figures ..........ccccscccsccesscsssessesssescessscesecssessnseecssesesesseeesesseesstseseseaaseneetaeersseonss xi

CHAPTER 1: LITERATURE REVIEW........cccscsssssescscsesectecseceeseaeecesesessesereensees 1

Knowledge Building: Design-Mode Epistemology.............:csscsscestseeressseesreeees 5

Belief, Argument, and Warrant: Belief-Mode Epistemology...............ss0000 8

William Perry’s Ethical and Intellectual Development Scheme................... 11

Belenkyetal.’s Five Epistemological Categories ...........ccsessseecrseteeeneeeeeenes 13

Baxter Magolda’s Epistemological Reflection Model............cccssesseeeeeeeetees 16

King & Kitchener’s Reflective Judgment Model...........cceccssscsesessresenseeeeses 18

Kuhn’s Levels of Epistemological Understanding .......... ec ceseeeseeceseneeteeees 20

Metacognition ..........cccccssccssessestecscssecssecesecsecssccesecsesesessessssceseeseessreseneesaseenseensses 23

Online COMMUNITIES 0.0.0... ce eceeceeteceneceseeeseseceseeseeeseeeeessseesaneceaseesesesesneseeeeoes 27

My Framework .........:ccscccsecesscesesecesceseceeesceceaeeeseeeeseeeseeeseeesseeseesesaeeeseeeenaeetsneens 29

Research Questions ...........cccsssccssssesssssessesscessncesssseecssseseasecesnseesesseesseseseeseesesenes 31

QUESTIONS ........cesecesecesecesscesscecneveesesnsesscessnecescesssesteeceaeesaecsescesaeecssesessessnes 31

Whywasthis Research Important?....0.....eeeeseesssesseecsesceseesecseeeneseneeeeeaeens 32

CHAPTER2: PILOT STUDY 0...cc eeescssecssessceeesseeeseesecssssnesatenesaesnesnseneatens 33

Advance Look at Overall Research Methods...........essscssseserseceesserseacenecerenes 33

Pilot Framework.0.......ccescesseesseesseecescecseeceneessncessecsseceeeeeeaeeseasceseecessaeenneeseneees 34

Survey DeSigQn.........sccesccssesssecssrecsncecsesenesssecesececseessecssseseseesaseneeseeasesenssennsenes 36

Pilot Method .0.......eeceeeceeseeceseceseceescecnceceeeeeeeessecesaeeeseceeneeeseeeeeeeseecesaeesnneeesetens 39

SULVEY oo. eceesseesetesecsseeseensccsesesecesecseeeseeessenerseesseeeseaesraseeeesnesaeesesenesatensesss 39

Conference Notes........cccsescscsssscssceseessesecneeseseassessesenessecaesnecseecaeseaeeresaeeeees 39

Participation and Interaction Patterns.............ccsccscssssscesscenesseeesseessseeseeensens 40

Learning LOgS ........ ce essesecsseesceeccesceeseessecascaeeesesesseaeecesseecseesssessceeeeeaeeeneeens 40

iv

Analysis Procedure .0.....sseessssssssssecssesessresecsassseseseesseeaesseraeesesanesaeesesnseaseaees 42

Pilot Study Results and Discussion ...........:cccsescceessescesreteeensesceneeaeeeneraeeeenaees 42

How Participants Were Chosen for Detailed Analysis... eeeeeeeeeeeeerees 43

Classification of Participants into Group..........ssceseseseeseeceseeseeseeeeteteneeenes 45

Individual Analysis... scsseesesescsssessesecsseeesacesesseseesesseeseeeaacsseaeccatenesenenaeeas 46

Analysis of Participant 25.0.0... ceeseecseesccsesseeteceeecseceesescescesceesaeseneeseenecees 46

Analysis of participant 12 0.0... ceeessecssecesseteneccsecesseesasecseeensececnnsessnteneeetes 50

Analysis of Participant 7.0.0... .ecceccescesecssecsccesecesecceeseeeeessesseeseeeseatesneeenernees 54

Analysis of Participant 6.0.0.0... ccsesccessccessessnecesecesectsneseseeeneeceasesseneeseasecneenes 60

Other Results that Affected the Main Study... cccsescsstesssseesesesseeeseeessneees 65

Framework.........ceseecccsesescesserseeeceeseeseeeseseseseesoasesesseseseeenecsaceaessseesentsessenesens 65

Pre and Post-Survey 00... esseescsscssessecseeseseeseeseccsesseeseseeseneenesatenatessenseeseeases 66

Learning LOgS........cesssscesssesessncestcesscsnceencesecnsecseessesseensssesesenesseseguessnssnesensents 66

Pilot Conclusions..........ccccsccesssesesseeeseceseceseeseceascnscesaecseeseeesecenesseessrseesessateenseseees 67

CHAPTER3: METHOD 10.0.0... esccescscesrceceseeesetseesecstenecsceaeesecsrseeessseaceaeeesenesseseees 68

Literature Review for Methods...........eccscsscsssesseesserecsecenecssecenecsasseeesseceseseneees 69

Design Research.......cccccsssssssssssssecssersssessesssssssessssesssesesssenecassesesersesesesenssaeens 69

Participant Reflection 00... eesecscsseeseeeseceeecnecerececsseessesececseesseeeesesesseaeeas 72

Moshman’s Levels of Justification...eeeeseeseeseseseceeneeseeneeseesersesenseatens 75

PartiCipamts...........sccecccecssecesccssecetseeessecesecessceseeseseessnerssessnesseeeesteeenseeseseecseeeseaeees 78

Course DeSCTription ..........ccccssccssesssssecesscsesessesseessseseesssssecssecuesseesaeesetsessesersesseass 79

Data Sources and Collection Methods...........cecescssesccseeeneeseersesesseceeseenseneeves 79

Learning LOgS .........escsccssscsseseessceeseseaesseeessceesesessressesssessessausesseseessaeeseeateess 81

Pre and POSt-CSSAY........ se eescesescsceseeeeeeeseeseeeseeseescseeaeesseeasessessnecesensarseeeeeeees 83

Participation and Interaction Patterns ............:csscsscssecsceseeteseseeseeeeseeeeseeneens 84

Qualitative Analysis...sesceesseeceesesseessessrsesesceessceseseersseesceseetseesesaresseentes 87

COIN...eeeeeseceeseessacecscecseeeeseeeacenseecsaeeesacecsaeceeeesesesseeceseersnesstseeceateesaeesaes 88

Inter-Rater Reliability... cccesssccssscscsescssecsccsssecscssecsseseneseeesseeserscssssensesaeses 89

SUMMALY ........cccccesseccestsccsseeseeececsseccssseecessueeeeseeeesesesseeesssseeseausesesassesesanseseneees 90

CHAPTER 4: RESULTS AND DISCUSSION...cceceesesseesretecesneteeeeeeseseeess 91

Description Of Database ............ccccccssscsesssseesnseeesauseseaesesnseeeseeeeseseecsscsnaseesauecensaes 92

Description of Participants...........csessssesseseescessenessssesssseesseseteseesesresseesssasesseees 93

Inter-Rater Reliability 2.0... eccssccsssseesessnecesnessneeesesssesesseessesessneessaeessnseseneees 94

Are There Changesin Participants’ Level of Epistemological Understanding

Observable over the Duration of the Courses? 0.0.0... ccsscssssessercessscessseetareneeees 95

Pre- and Post-Essays - Theories of Learning......... cc cssscssseserecesteceseeesseeeens 95

Levels of Epistemological Understanding Determined from Pre- and Post-

Essays on Theories of Learning...........cccccsccsssssrssssesseessesseesessseessseeseessnsesseaes 96

Explicit Examples of Changes in Belief-Mode Epistemological Thinking 100

Use of Justification Within the Pre- and Post-EssayS...........:cccscsssessseereeess 104

Use of Metacognition in the Learning Logs...........csccsseesssseessecstseeseeeneses 106

Preference of Directed Questions Vs Non-Directed Questions Within

Learning LOgS...........:ceccesscssscesesscesseesscsseeesecesseseessesseeesevensseeessesesseseseeenesnees 109

Use of Self-Directed Questions in Learning Logs During Last Three Weeks

sesecacesevecesesevasscesetsceacseesceatesesacstenesesaeneraceasuaescenssaenssacsnenesseuaeeesarsaseseeneeanensenes 111

Preference for Private Versus Public Learning Logs...........cceeeseeeeeeeees 112

HowDoes Belief-Mode Epistemology Relate to Participants’ Online

DiSCOUTSE? .....sscssscssssceseeresesssesetsesecssteessesseasecesessessesessstetsessesaeseseesenaeesessetassaees 113

Participation and Interaction Measues...........ccsccsesccseseesssesecesreesersceneessens 113

Activity Rating...ccccssscsscsssessesssessecssssesesssessesseessesecssseeeseesssesessesessareeges 114

Interactivity Rating 0...eecesesscesescecessesseecesecesecssescssecesseeeseeessseesssesenees 114

Participation Rating ........cssssesesessseeserseeseseceecaceeesesssesesesseeseeesaeeaeenenees 115

Overall Collaboration 00... eecesessceseeeseeesessnsceeesseeseeesecesesaeeeaeceneseseeeeessns 116

Overall Edits 00.0... eeccsscesecssecsreesscessesseeeeeaecsaeesseseecssceseseneseeesseeseseeenesnees 118

Use of Authoritative Sources ....... cc cecssessecensetcesetseessesenessesesssseesssnsesseseees 119

Who’s Read Whose Notes 0.0... ccsscsesccsesecssessssseeseesseecesesseenssenseeseresassaes 121

Who Built on to WhOM? 00.eeeeseecsseseeeeseeeecteessesessseaesasesseseceesaeenesees 123

Week-By-Week Metacognitive Activity ..........cccscssssssessessssersecssseneeeeeeees 124

First Half Versus Second Half Comparison .............ccccssccsseetseesseeseeeneeenees 127

Case Analysis for Participants Who Showed — Change in Rated Belief-Mode

Epistemology ........cccscessssscssecseecssseceessessecssecsseeseessecsesssseeeecssesaeeeaeessacseesatenas 128

2_F36-40:HIgn 0... essceseseesseeccssceesesecscenecsecsscsesseeeesesessessesseecsuesseeasensees 128

vi

6F26-30:High cccccsccccssssssssssssssssssssssssssseccsssesessvessssssssasssesesssesesceseeeeeseese 130

11M26-30:LOW ues ececeseescesescesceeeeccnscoseeeneeseceseeneneeeeeaseaseeesaeaceeeensensaees 132

13_M_41-45LOW0cccecescesenscecseeeseesesecseeaeescasaceeesesseeaseceseeatereseeaeeaseees 133

14F26-30:LOW 20... eesescsecceseseesceseeessceseeseeesceseeeaeeaeetseeaseaeceeeaueaeeseetares 135

Summary of Case Analyses ........ cc cscescesesscsseeecessesetsseseseseesenaseaeecseeseessnes 137

CHAPTER5: CONCLUSIONS,IMPLICATIONS, AND FUTURE RESEARCH

secaneacesecsecaccuscsseesessecscessessessuseescussesacsessaesseasenerseesecaneseseasseessenaeeasanecsecsnssanesseasents 139

Are Changesin Belief- or Design-Mode Epistemology Evident over the

Duration of the Courses? .0..... cc ccesessseessseesssecsseseceecesceeseresesenesseestenaeeerseaseneens 141

Implications of Level of Epistemological Understanding for Practice....... 143

Implications of Level of Epistemological Understanding for Theory........ 144

What Were Some Manifest Characteristics of the Epistemological Levels? . 146

Implications of Characteristics of the Epistemological Levels for Practice 148

Implications of Characteristics of the Epistemological Levels for Theory 150

Howdoes Epistemological Understanding Relate to Participation and

Interaction Patterns in Online Discourse?..........cceseccssscescesseneeesesereeesrseenesses 152

Implications of the Relationship of How Epistemological Understanding

Relates to Participation and Interaction Patterns in Online Discourse for

PLAClICE.... cece cesecsecessesserseesseetaseecssevsnessnseseecnersnecssessessesceseaessateaeesseesneraee® 153

Implications of the Relationship of How Epistemological Understanding

Relates to Participation and Interaction Patterns in Online Discourse for

TROOTY .0... eee eeccecesecesecsceeneeeseessceceeseseascssesseeeseensecseseceeeseatsedsenssesnsseseceneesaes 154

Were There ChangesIn The Interactions Between Individuals?.................. 154

Implications of the Changes In The Interactions Between Individuals for

PLACTICE.... eee eeeeseceecneecncesseeseerseenesscecseesssensecseeeseenseecenasesessasensseseeeseeseessnees 155

Implications of the Changes In The Interactions Between Individuals for

TROOTY .... cs eeseeeeseecesesecessceesssecscecstenseesaceseceseeesesessenssecessesenesseesatesesesensessseenss 156

Closing Remarks 0000.0... cccescsssecssssssccsseessesesssecssesesssnecseseseesssesesseeseneesasenseesees 157

REFERENCES.0.0...ecceccsessceessecseesecreseensesscesessesanesssesseseeseeeesesseenessuseseseessaesaes 159

APPENDIX A - Perl script for Pilot post-Survey .........:ceccssscesessseeeresseenseeeeets 165

APPENDIX B — Sample of pre-course Pilot Survey........:.cccscccsscssstesssteesseeesneees 175

vii

APPENDIX C - Protocol for Scoring Levels of Epistemological Understanding

sesseseecensesssseucesaseseuaeseeasscenesseesecsssseaseseneeseesesacseeseeaseaeecesseaeaceseeeesaeeteaseneeneeseeaseeene® 180

APPENDIX - Coding chart for Metacognition...........csscessessessestsessnseneeeneees 184

APPENDIX - Anearly determination of Epistemological levels................... 188

APPENDIXF - Ruberic used for self-evaluation of learning Logs................... 195

viii

LIST OF TABLES

Table 1 Description of Baxter Magolda’s Epistemological Reflection Model................ 16

Table 2 Levels of Epistemological Understanding (D. Kuhn et al., 2000)...........cssssees 22

Table 3 Chronological Overview of Data Collection .....0eccesscseesecteeeeesenessetaeeerenes 33

Table 4 Knowledge Building Elements and Related Questions «0.0.00... ceseeeseeeteeeeens 35

Table 5 Calculation of Basic Knowledge Building MeasuresperParticipant................ 45

Table 6 Experience Level of Participants ...0..........cscsssccsscssesseeserscsscsseseeessesesesseseeeasens 46

Table 7 Breakdown ofInstances of Belief-Mode Epistemological Levels for Participant

25 per Week within the Learning LOgS ........... cc cccsseesescesseseesecsecsscescseseessesseseenees 48

Table 8 Breakdown ofInstances of Belief-Mode Epistemological Levels for Participant

12 per Week within the Learning Logs and Weekly Notes...........c:ccssccsseeeseeneeees 52

Table 9 Breakdown ofInstances of Epistemological Levels for Participant 7 per Week

within the Learning Logs and Weekly Notes..........ccccccssssssessessessesseesseseeeseesesaees 56

Table 10 Breakdown ofInstances of Epistemological Levels for Participant 6 per Week

Within the Learning Logs and Weekly Notes.0.........csesscsssececssecnseseeseeesseeessesees 62

Table 11 Moshman’s (1998) Levels of Justification ............cccssecssscssseeerscessrtessrseesneeenes 76

Table 12 Chronological Overview of Data Collection for Main Study.............ccceeee 80

Table 13 Thesis Questions with Related Data Sources ..........c:cscessssesscsseseeseseeeesseensessens 91

Table 14 Participant Biographical data...eeeeeeseecscetecessensesecssseenscesecseecsssseseeenessnes 94

Table 15 Levels of Belief-Mode Epistemological Understanding and Type of

Justification Used for the Pre- and Post Essays .........ccccssccsssscessessseeeseeeessseesseessneee 99

Table 16 Descriptions of Areas of Belief-Mode Epistemological Change and Overall

NumberofPassages Coded .........:ccsccssscsssecssscsssceseceesteseeessecessecseteassaseesesesstenses 103

Table 17 Metacognitive Activity: Total, Knowledge, and Regulation.................cc 109

Table 18 Directed Versus Non-directed Learning Logs ..........cssssseeesesesecsseteeeeeeeeees 110

Table 19 Comparison of Post Level of Epistemological Understanding with

Metacognitive Activity and Self-Questions for the Final Three Weeks............. 112

Table 20 Activity, Interactivity, and Participation Ratings with Their Corresponding

SCOPES... cecescesscsseesscsseesecesensseneeseesserseseaeseassesesecsaecseessecseenseeesevacseseessuessnsesseensess 116

Table 21 Overall Collaboration Score ..........ccccccessssecsresssessssseeseesseessecsseeseeesssseneesseessees 118

ix

Table 22 Overall Edits .........ccecccesssscecsssesessecseesecsnessseessssssessessssssesesseenesensersceeessesneenses 120

Table 23 Use of Authoritative Sources... ceescsssssescesscressensecssesesteesssensesscnasenesanenees 120

Table 24 Who’s Read Whose Notes by Percentage High vs Low Participation Rating and

Percentage of Professor’s Notes Read ...........:cccscssccssecsessesssesseessseeaeeenseeseeeseeseees 122

Table 25 Who Built on to Whom by Category of High Versus LowParticipation, Build-

ons to the Professor, and Ratio of Build-ons to Notes...........ccccccsecccccsssersresesseres 124

Table 26 Week-by-Week Analysis of Metacognitive Activity Within the Learning Log

sesaeseeseeconsecsacucensensssesnesavasscesevsccassseusesseassaenseseuassceseeseneessesecsssesaeessssssesaeeeseesacsasenseaeeseeesersas 126

Table 27 First Half Versus Second Half Comparison of Revisions, Views Worked On,

Notes Read, and Notes Created 0.0... cccssssccsscssscsccssecescesecseteenessessesseseseressneenes 127

Table 28 Comparison of Participants Who ShowedSignsofbelief-mode

Epistemological Change ..........csccsccsscssssscsecesesesessssessecsesseesessessecsesseseeeeeeessenscssens 137

Table Al Characteristics Based on Level of Epistemological Understanding............... 181

LIST OF FIGURES

Figure 1. Number of Words Created per View..........csccsscssssssscssceseesscessesrsenseeesneesasensesnees 93

Figure 2. Explanation of How Edits Were Counted. ............c:csscsssscessscenseseseceeessestatenss 119

xi

CHAPTER 1: LITERATURE REVIEW

Imagine a world where people have access to unlimited knowledge resources and

where ideas are continually created and improved upon. How wethink about knowledge

is one factor that can restrict our entry to this place of unlimited resources. This research

is about gaining entry to such a world. Bereiter (2002) argues the need for better

conceptual tools in the knowledge age. Hestates: ““The most basic of tools are our

conceptions of knowledge and mind”(p. 4). If we are to be able to create and work with

knowledge, we must go beyondourfolk theories of knowledge. These folk theories “all

have their source in conceptions of mind and knowledge that we acquire as children and

never think to examine — because they seem to be given directly by experience and

because no alternatives have been presented”(p. 7). Further, and more importantly, “it is

a deceptively simple step, down a slippery slope, from the belief that everyone has a right

to their opinionto the belief that all opinions are equally right. Tolerance of multiple

positions, in other words, becomes confused with discriminability among them” (Kuhn,

Cheney, & Weinstock, 2000, p. 326). This research will explore students’ examination

and restructuring of beliefs about learning and knowledge. Such research can also inform

design of pedagogical techniques for encouraging metacognitive reflection, crucial in

dialectical reasoning. This research looks at how students think about these alternatives.

Today’s current learning theories emphasizethat social interactionis critical to

learning. Collaborative learning requires knowing about and caring for one another’s

needs. Helping individuals feel comfortable in a learning community, feel they want to

stay, and workto create new knowledge requires a respect for others’ ideas

and a willingness to work in constructively critical ways.

Assuming we develop a pedagogy that supports knowledge building and that such

online communities begin to flourish, how will they be sustained? Do we continueto

rely upon a champion,a key facilitator? Or, must we build into the community of

learners a sense of shared responsibility, a collective cognitive responsibility

(Scardamalia, 2002)? In a community that knowsand practices shared responsibility and

goals, individuals could come and go; the community would remain strong and continue

to grow.

However,the idea that the acquisition of knowledge aloneis sufficient to sustain

the group is not adequate; “Cooperation and trust help build online communities”

(Preece, 2000, p. 170). What people’s contributions are and howtheir needsare satisfied

are of prime concern to an online community (Preece, 2000). Individual strengths and

preferences needto be identified and acknowledged.

Whatare the conditions under which participants engage in purposeful dialogue

and the creation of knowledge? Can they take charge at the highest levels or is a

champion or permanentfacilitator required? If such a person started with the group,

could they be replaced from within, as needed? Can individuals who make up the

community sustain the discussion and progression of knowledge? Suchself-intentioned

purposeful work in knowledge advancementis the knowledge building ideal. This

research is aimed at exploring changesthat occurin learners en route to such an ideal,

specifically their epistemological changes. Theintentis to see if there is a link between a

person’s level of epistemological developmentand the sustainability of online

communities. For example, does a person need to have a view of knowledge such that

they are able to accept others’ views of knowledgebutalso be able to judgeits validity?

According to Bruffee (1999), “knowledge is what we can explain to one another’s

satisfaction about what we believe in commonaboutthat [physical] world. Knowledgeis

a conversation in progress among people who understand one another — people who

speak the same language”(p. 118). What level of epistemological understandingis

needed for members to becometrue contributors in a place ofknowledge creation?

It seemsthat a multipilist’s view of knowledge is gaining acceptance. Such a

view accepts all opinions as equally valid; there is no understanding as to what counts as

justification to judge one opinion “more correct” than another. Multipilists believe that

reality is not directly knowable, that knowledge is generated by human mindsandis

uncertain (Kuhn etal., 2000). Lee Knefelkamp (1999)states, “We are well aware ofthe

socializing effects of school systems, and that many (perhaps even most) students are

educated in environments that Freire would characterize as ‘banking models’, with their

emphasis on authority, information exchange, and the quest for right answers”(p.xvi).

Further, Bruffee (1999) states, “given most students’ almost exclusive experience of

traditional classroom authority, many haveto learn, sometimes against considerable

resistance, to grant authority not to the teacher alone but to a peer ... instead of the

teacher” (p.14). The majority of today’s learners have likely experienced, in the main,

teacher-directed classroom learning. This traditional style generally leads to an

absolutist’s view of knowledge. With this view,reality is directly knowable, knowledge

comes from an external source andis certain (Kuhnetal., 2000). It may well be that to

be successful online learners, many students will need to develop a different order of

epistemological understanding and change their views on whatis valuable in learning.

This literature review will examine epistemological frameworksthat support

different educational models. A full, multifaceted account of epistemological

developmentis, of course, beyond the scope ofthis thesis. I review epistemological

development from the perspective of a common and general framework that underlies a

broad range of developmental studies and educational practices. I characterize models

within this framework as belief mode epistemologies, following from Bereiter and

Scardamalia’s (Scardamalia, 2003; Bereiter & Scardamalia, 2006) distinction between

beliefmode and design mode. They argue that schoolworktypically deals with

knowledge quite differently from the way knowledgeis dealt with in the working world.

The school learning orientation is dominated by beliefmode and students receivelittle if

any experience in knowledge building, which is a design mode concept. Both modesare

important, and knowledge workers tend to move easily between the two.

The concern in belief mode is with whatis believed or ought to be believed - with

agreement, disagreement, warrant for or against propositions, weight of evidence,

arguments and counter arguments. Educational programs equip students to think in belief

mode. They learn concepts and whatthey and their peers believe, they use evidence and

logic to help them takea critical stance to these beliefs and to evaluate truth claims, they

weigh evidence for different claims. Through sharing ideas, weighing evidence for one

or another belief system, achieving consensus, participants move from personal belief

systems to more canonical or shared systemsofbelief, which are typically the target of

instruction.

Design mode, by comparison, focuses on idea improvement- the adequacy and

improvability of ideas, and whethercurrent ideas are really leading to a resolution of

problemsin a field. What cannot be explained is as important as what can be explained;

problemsare guides to where new breakthroughs maytake place. Overall the work is

guided by anticipation that a better idea will be forthcoming. Accordingly, participants

engage idea diversity, out-of-the-box thinking, and tolerate more seemingly “off topic”

ideas than school environmentstypically do, with the goal of rising above that diversity

to some new,higherorder construction or breakthrough. The goal is not consensusbut

“inference to the best explanation” (Lipton, 2004). The insight from the philosophy of

science andhistory is that “a theory is rarely abandoned just because it faces empirical

difficulties; rejection of the problematic theory comes only when a new theory comes

alongthat is visibly superior in that it can explain most of what the previous theory did

and more” (Thagard & Millgram, 1997, n.p.). Processes for sustained idea improvement

are informed by work in knowledge-creating organizations where the cumulative effects

of idea improvementis evident. Design modeis essential for continually advancing

beyond whatis known.It is interspersed with belief mode efforts to check the soundness

of assumptions, beliefs to be taken into account, andreliability of data, but the creation of

new knowledge, not acceptanceor rejection ofbeliefs,is the goal.

Good educational programs generally equip students to think in belief mode - to turn

a critical eye on beliefs, to use evidence andlogic, to resist propaganda, to evaluate truth

claims, and so forth. Bereiter and Scardamalia (in press) indicate that given the

dominanceofbelief modein education,“it is possible for discourse to go on indefinitely

within belief mode, never venturing into design.It is characteristic of much academic

discourse (in fact, that is what we imply when wesay a question is ‘purely academic’).”

With respect to the framework for this thesis, the argumentis that in an active,

knowledge-engaging online community, we maywell find epistemological change within

belief mode, even in the relatively short time span of a university course. However,

design moderequires enculturation into quite a different educational culture. It isa

perceptibly different culture, and can be noticed and engaged in even by young students.

For example, a student from grade 4 who moved from an elementary school with a

traditional learning culture to a school that operates as a knowledge building community

was asked to explain the difference in her school experiences between these two contexts.

The question wasposed to her after about three months in the new environment. She

explained the difference in the knowledge building context as follows:

It is not so much about “true” and “false” - there may be right or wrong

ideas but you don’t know until you really do research and understand, and

as you work toward greater understandingyou discover that the more youknow, the more you know what you don’t know.

In the current research we did not presumeto be able to take a well-honed university

learning environmentandturn it into a knowledge building community operating in

design mode. This is not to say that advancesrelevant to design mode could not be made

in the term of a university course - that would be a design challenge in its own right. But

that wasnot the design goal of this study. First, the belief mode/design modedistinction

is relatively new, and wasnotasclearly a part of the knowledge buildingliterature at the

time data were collected. Second, the goal was to explore what waspossible - and not

possible - within current course frameworks, and to use results to inform nextsteps.

Finally, it is quite possible that there are multiple paths to epistemological development,

that are not present in the body ofliterature, that include social epistemologies andthat

these likely change overtime.

Knowledge Building: Design-Mode Epistemology

The essence of the design mode is idea improvement. Engineers don’t think in

terms of building the ultimate computer, but rather advancingthe state of the art. Each

advance opensnewpossibilities, and the sameis true with ideas. Knowledge building

aimsto enculturate students into a knowledge-creating culture - bringing design modeto

the heart of education, along with the dynamicof continual idea improvementthat

underlie the production of new knowledge.

Scardamalia and Bereiter (2003) define knowledge building “as the production

and continual improvementofideas of value to a community, through meansthat

increase the likelihood that what the community accomplisheswill be greater than the

sum of individual contributions and part of broader cultural efforts” (p. 1370).

Knowledge building is a process, and as such, can be practiced by even youngstudents.

Second grade students can advance their communities’ knowledge on the digestive

system of wormsin a similar fashion to cancer researchers advancing their knowledge

about how genesaffect susceptibility to cancer. Perception and experience play major

roles in determining what constitutes ideas of value to a community. Potentially, anyone

could participate in a knowledge building community.

Students in knowledge building communities ideally see the production of new

and improvedideas - not learning - as their primary goal. Bereiter and Scardamalia

distinguish between learning and knowledge building to highlight different

epistemological frameworks within constructivist approaches that support learning versus

knowledge building environments. In knowledge-creating organizations people are

engaged in producing new knowledge. Their products may be scholarly things like

theories or proofs or more practical things like plans and inventions. These serve as new

ideas used to produce more new ideas. Knowledge buildingis this process of idea

creation, development, and refinement that keeps ideas on a continual improvement

trajectory. In the process of knowledge building, knowledge workers naturally learn, and

such learning is essential to their careers as knowledge builders, but learning is not the

goal. The goal is idea improvement and the production of new knowledge.

Advancingthestate of the art of online environments will require greater clarity

regarding possibilities and designs underlying epistemological frameworks. While new

knowledge media open up new meansand opportunities for addressing fundamental

problemsin education, the poweris not in the learning environmentitself, but in a

complex interaction between environments, conditions, and epistemologies of use. The

feasibility of knowledge building as an educational goal has been demonstrated in a wide

variety of contexts (e.g., (Bereiter, 2002; Scardamalia, 1991)) but we are yet to

understand the conditions that make it possible - and those that preclude it.

A knowledge building environmentis different from online environments

designed more specifically for course delivery and distance learning. Although a

knowledge building environmentcan be used for these purposes,its distinctive strengths

emerge in contexts - educational and other - where the emphasis is on knowledgecreation

and sustained idea improvement(Scardamalia, 2002). The following list of socio-

cognitive dynamics of knowledge building are taken from that article.

Socio-Cognitive Dynamics ofKnowledge Building

e direct engagement with problems of understanding

e work with emergentrather than fixed goals

e evolution of goals toward higher-level formulations of problems

e self-organization around promising new directions rather than mandated work

on other-directed and scripted activities

e workat the edge of competence

e self-monitoring and self-correction, without undue dependence onexternal

evaluation

e® engagement with knowledgeintensive processes that lead deeper and deeper

into the field of inquiry

e productive use of idea diversity

e risk taking

e responsibility for high level socio-cognitive activities such as setting and

refining goals, providing resources, and identifying different perspectives

Cultures of innovation enable the processes listed above and encourage

participants to take collective responsibility for knowledge creation. The environment

additionally provides support for knowledge building discourse, not simply

argumentative discourse. Thus, for example, it includes a knowledge building theory

scaffold that encouragesparticipants to enter theories, identify gaps in understanding,

create improvedtheories, and so forth. The reflective processes required for continual

idea improvementhelp ensure innovation becomesa cultural norm within the knowledge

building community.

Popper(1994; 1999) has described subjective and objective knowledgeandits

relation to what he called the three worlds: world 1 - the physical, world 2 - our minds,

and world 3 - the products of our minds. His epistemological concept cleverly allowed

for separation of the products of our minds (world 2), thinking about things derived

empirically (world 1), or written thoughts about these things or products (world 1 & 2)

into a world 3 object that could then be used for further improvement and advancement.

Knowledge building, and the environmentsthat support it, aim to create a public, shared

space whereideasrepresentthe collective space of understanding and memberstake

responsibility for the continual improvementofthe ideas in this public resource. Popper

believed that once theories or hypothesis were recorded, they becameartifacts for

discussion and testing. As such, they took on a life of their own; one that could exist

separately from the originator. There are many examplesofinteresting problems and

questions that have continued over centuries, such as evolution and where knowledge

exists (See Popper, 1994). This continuity describes the essence of rise above; “There is

no real endpoint in knowledge building but rather a continual discovery and improvement

or the deeper interconnections betweenall ideas” (Austin, 2002, p.16)

Questions that characterize work in design modeinclude the questions that would

be askedin a real-world knowledge-based organization. They are questionslike the

following:

e Whatis this idea good for?

e What doesit do andfail to do?

e How could it be improved?

This difference in emphasis will be elaborated in Chapters 3 and 4, where

epistemological changein belief mode is contrast with change in design mode.I

highlight the difference betweentreating ideas as fixed entities, to be accepted or rejected

and sometimes to be applied, as opposed to improvable objects in a design context that

leads a knowledge-based society forward.

Belief, Argument, and Warrant: Belief-Mode Epistemology

The epistemological models discussed below all concern themselves with an

individual’s belief about the nature of knowledge. The general and commonthread from

a developmental perspective is that one’s beliefs are egocentric or in other ways limited

at first, and with development comesthe ability to provide warrantforbeliefs, to

recognize counterbeliefs, and/or to construct integrated conceptual frameworksthat take

into account multiple perspectives and the evidentiary basis for them. One gets a sense of

the extent to which belief mode epistemologies dominate workin the field from thetitle

of an article by (Bell, 2002) “Science is Argumentation”and a large,related literature on

argumentation as the foundation of science and “argumentative discourse” (D. Kuhn,

1991) as the primary meansofengagingin scientific work. In turn, belief-based

frameworksare used to induct students into what is termed “scientific discourse” by

encouraging them to identify their beliefs, provide warrant for them, and consider

alternate beliefs and warrant (Andriessen, Baker, & Suthers, 2003; Driver, Asoko, Leach,

Mortimer, & Scott, 1994; D. Kuhn, 1991; Takao, Prothero, & Kelly, 2002) The emphasis

on argumentation is further reflected in the models of epistemological development

presented below. Overall, the work is characterized as belonging to belief mode because

growth is along a dimension ofindividual, independent belief systems that come into

alignment with or are juxtaposed with belief systems held by others, with the evidentiary

basis for personally held ideas and those of others yielding increasingly complex systems

of belief. As suggested above, design-mode epistemologies are more relevant to

knowledge building and are more concerned with usefulness, potential, or strengths and

weaknessesofideas.

The recent increase in online courses for adults provides the researcher with a

unique opportunity to study what Vygotsky (1978) would call ‘fossilized behaviour’.

Thatis, “processes that have gone though a very longstage of historical development and

have becomefossilized” (p. 63). This fossilized behaviour could also be called habit, as

we no longer need to provide voluntary attention to the task. This thesis examined

students’ beliefs about learning, teaching, and knowledge. These beliefs and behaviours

wereinitially formed in the early school years, but, as the following developmental

models will show, changes occur throughout university and beyond. Theparticipants for

this research were at the master and doctoral level. Typically, these students were

returning to school after someperiod in the workforce. Further, since online learningis a

recent phenomenon,online courses were notlikely part of their schooling to date. During

those years of work,their beliefs about knowledge and how they learn have probably

become habit. This was certainly the case in my own experience. I found that online

learning caused meto re-think myold habits. I preferred to learn by studying the

textbook and did not find group work particularly productive for my learning. Does the

constructivist pedagogy most used by online learning require a fundamental change in

10

students’ epistemology? The emphasis on more collaborative work and thus greater

engagementwith the beliefs of peers may well lead to a shift in belief-mode

epistemology, but this in its own right is unlikely to lead to design-mode epistemology.

Nonetheless, online learners are exposed to a very different medium which often employs

markedly different pedagogies from face-to-face pedagogies, and so the goalofthis

research is to look closely at the type of epistemological change that they foster,

specifically within a well designed community framework aimed at giving students

greater responsibility for their knowledge work.

William Perry Jr (1970)states, “structurally different epistemological

assumptions imply different forms of teaching and learning congruent with them”(p. 43).

The inverse was one of the basic assumptions underlying this research — that structurally

different learning and teaching environments imply different forms of epistemology.

King and Kitchener (1994) would agree, stating, “We argue that educators cannot ignore

individuals’ epistemic assumptionsif they wantto help their students becomereflective

thinkers” (p.13). An implicit goal in this research was exploring how individuals become

their own ‘educators’ and hence cometo understand their own epistemic assumptions.

In developing the belief-mode epistemological framework for this research, five

major theories were examined. Perry wasthe pioneerin intellectual development

research and, despite excellent research, his study sample waslimited exclusively to men.

However, he did provide the base upon which other models were built. Belenky,

Clinchy, Goldberger, & Tarule (1986), initially using Perry’s scheme, conducted research

exclusively on women and found that his model did not adequately capture the

epistemologies of their female group. Marcia Baxter Magolda (1992) conducted research

on a combined sample of male and female college students and has developed a model

that is more inclusive than the previous two.

The above three models focus on how individuals interpret their educational

(formal or informal) experiences. The following two models are more interested in the

effect of epistemological beliefs on students’ reflective and reasoning processes (Hofer &

Pintrich, 1997). These processes were the focus of this research. However, a review of

the above modelsis beneficial in understanding the roots of the next two models.

King and Kitchener (1994) also build upon the work of Perry (1970) but added

11

the influence of reflective thinking by Dewey (1933). Their stage model is supported by

over 15 years of research on high school aged through to middle-aged participants. They

proposed a definite endpoint to the developmentof reasoning andjustification.

Kuhn (1991; 2000) took a different approach, and proposed a model based on the

coordination of objective and subjective views of knowledge. She useda participant

group that widely spanned age, education, and backgroundas well as varied ethnicity.

The simplicity of this model plus the well-justified combination of theory and empirical

evidence madeit the best choice as the belief-mode frameworkfor this research. Each of

these five models will now be discussed.

William Perry’s Ethical and Intellectual Development Scheme

Perry (1970), in his book, Forms ofIntellectual and Ethical Developmentin the

College Years, studied randomly selected college students (n=109), from Harvard and

Radcliffe, over a 4-year period. His study took place between 1954 and 1967. He

initially administered a measure called Checklist ofEducational Views (CLEV) to

determine where students’ thinking lay on an epistemological development continuum,

whichhecalled the ‘main line of development’. The CLEVplaced students’ current

thinking in oneofthree positions (the modifying of dualism,the realizing of relativism,

or the evolving of commitments), which were further segmented into three more

positions each. The main data collection method used was 366 interviewsincluding 67

complete four-year reports. Judges were used to rate the interviews. Perry’s initial

model was modified and fine-tuned during this process. Excerpts were used to categorize

student’s thinking in particular positions. Freshmen,at the end offirst year, were

typically in positions 3, 4, or 5. Most seniors ended up in positions 6, 7, or 8 with very

few reaching position 9. These positions were markers on a continuum from viewing the

world as absolute at one end (1) to seeing the world contextually relative at the high end

(9). The continuum wasa blend of epistemological views and an exploration of values.

Assuch, this model had a pragmatic flavourin line with the notions of Dewey (1938).

Towards the absolute end (position 1, 2 &3), knowledge wasfactual; there were right and

wrong answersthat could be found in authoritative sources. Authorities had the answers

12

and it wastheir job to help students to find these answers. Gradually as one movedalong

the continuum,diversity and uncertainty appeared. In about the middle (position 4),this

diversity mushroomedinto multiplicity where everybody’s opinion was equally valid and

the students believed that they needed to give authorities ‘what they want’. Movement

from this position towards higher levels required contextualization of knowledge and was

a real turning point for growth. From this point on the dualism of absolutism became a

mere special case of relativism. Further movement(positions 6, 7, & 8) required that

students orient themselves through some form of personal commitment. Commitment, in

turn, had its own set of dualistic tensions with which students struggled. At the far end of

the continuum, commitments were realized as continuing activities that expressed a style

of life and existential being.

Perry admitted that the positions towardsthis end of the scheme were ‘rough and

crude’. However, because the participants in this research were likely to be concentrated

at this higher end of development, for my purposes, Perry’s schemewasbest used as a

guide to understanding the theories that follow. Nonetheless, I remained opento seeing

positions as they may appearin the research.

In Perry’s research, moving betweenpositions of the scheme usually required

participants to show a greater accommodation ortransformation in assigning meaning to

events. Old structures needed to be redefined and reformed in order to grow or develop.

However, there wasa parallel path that subjects could take that did not lead to growth.

They could suspend, nullify or even reverse the trend towards growth. However, Perry

was not clear on how one might moveonto this negative path.

In more than 25 years of investigations using the Perry scheme, Knefelkamp

(1999) found an interesting occurrence called functional regression — “the phenomenon

seen when adult learners undertake new learning in a new learning environment and

‘functionally’ regress to multiplistic thinking until they feel comfortable in the new

environment (wefirst observed this with graduate students)”(p. xviii). In multiplistic

thinking everyoneis considered to have an equal right to an opinion and no judgments

can be made betweentherelative values of those opinions. From this perspective,

starting graduate school might be likened to another freshman year of university. Many

students return after several years of work, probably having readjusted their valuesto suit

13

the work environment. The workplace often values solid (read black and white) answers

in order to make quick decisions; time and moneyare of the essence. In contrast, values

in the academic world can bestrikingly different. Value is placed on quality of research

and the depth of the argumentdespite the time it takes. The workplace often limits the

pursuit of divergent ideas whereas academia embracesthis diversity. Hence,it is

possible that new graduate students must once again struggle with the relative nature of

knowledge.

Thus, while it was possible that some graduate students in this thesis study would

still possess a multiplistic (position 4 or 5) worldview,it was morelikely they would be

in a position of commitment(position 6, 7, or 8). However, possible functional regression

combined with the diverse and often contradictory readings presented in introductory

courses could potentially push many back into multiplicity, position 5. Alternatively,

students might becomelost in the relativity of the diverse and dialectic positions

presented in readings. Further, they could easily resort to expression of personal

experiences to help resolve complex issues and opposing views, but mightfindit difficult

to know with certainty why they believe what they believe. Some may turn to

temporizing or escape. Onthe other hand, others might start to commit and find strength

and voice in this commitment.

Belenky et al.’s Five Epistemological Categories

In Women’s Ways ofKnowing, Belenky, Clinchy, Goldberger, & Tarule (1986)

interviewed in depth (2-5 hours) 135 womenand produced over 5000 pages of

transcribed text. The interview started with a very non-leading question: “Looking back,

whatstands out for you over the past few years?” (p. 11). Their female participants,

mostly worlds apart from Perry’s participants, included a very non-homogeneous group

of students, recent students, and non-traditional students (mothers seeking information

about parenting from agencies). This last group wasparticularly interesting because the

authorsfelt that,

most organizations in our society are shaped and directed by men andmasculine perspectives, the agencies andprogramsfor very young

14

children andprogramsto support mothers in caringfor children are likelyto be among thefew exceptions, these are usually organized andstaffed bywomen. We wanted to know what kinds ofinstitutionsfor promotinggrowth and development women would createfor themselves ifthey werenot so dominated by masculine images, theories, founders, oradministrators. (p. 13)

The researchersinitially tried to code the data using Perry’s scheme, however,

they found that the data did not fall neatly into his categories (not surprising considering

the genderof participants in Perry’s research). Belenky et al. describe five

epistemological categories: silence, received knowing, subjective knowing, procedural

knowing, and constructed knowing. Unlike Perry, the five epistemological categories did

not necessarily have progressive qualities. Belenky et al. (1986) were not able to

determine, based on the mostly single interviews, whether movement from one category

to another happenedin a particular sequence.

Silence was a place where women werecontrolled by external authority, had no

voice, and were “mindless”. In the category of received knowledge, women wereable to

acquire and reproduce knowledge from external authorities but were not able to create

knowledge. These previous two categories could be roughly equated to Perry’s positions

1 and 2. As a subjective knower, all knowledge becameprivate and personal. In contrast,

procedural knowing used objective procedures for obtaining and communicating

knowledge. These were Perry’s positions 3 through 5 respectively. Finally, Belenky et

al’s category of constructed knowledge incorporated both the subjective and objective

ways of knowing, and wasa position in which womenwere creators of knowledge.

Constructed knowledge wasparallel to Perry’s higher positions.

Someofthe moresalient features of subjective, procedural, and constructed

knowing will be described. Subjective knowing tendedto be a very liberating stage.

Womenwerefinally finding their own voice after having beenin silence or solely a

receiver of knowledge. Subjective knowing tended to be held in much higher esteem in

many non-Western and non-technological societies. However, “In a world that

emphasizesrationalism andscientific thought, there are bound to be personal andsocial

costs of a subjectivist epistemology” (Belenkyet al., 1986, p. 55).

Procedural knowing had two, seemingly contrasting perspectives: connected and

15

separate knowing. Connected knowersstarted out with an interest in people’s lives that

eventually developed into an interest in their thinking. They felt that a person’s

personality added great richness to the overall knowing. In contrast, separate knowers

tried to filter out this personality ‘noise’ in order not to skew their perception. Separate

knowerslearned to look at the world through different lenses, much in the waythat a

discipline teaches one to see the world in particular ways. In contrast, the connected

knower’s lens was the person at whom they were looking.

Constructed knowersbelieved that the world was complex and contextual. Their

understanding and assumptions about knowledge were continually being re-evaluated.

Belenkyet al. (1986) uncovered that womenin this category believed that “experts must

reveal an appreciation for complexity and a sense of humility about their knowledge”(p.

139). Further, “when asked how they feel about experts disagreeing, many

constructivists say that they are challenged, not daunted, by contradiction and conflict”

(p. 140). Constructed knowers were passionate and, I believe, exemplified ‘life long

learners’.

Aninteresting distinction made by constructed knowers was between‘really

talking’ and ‘didactic talk’. In didactic talk, experiential evidence was reported by

participants without the attempt being madeto join this evidence to form new

understanding. Conversely,

‘Really talking’ requires careful listening; it implies a mutually shared

agreement that together you are creating the optimum setting so that half-baked ideas or emergent ideas can grow. ‘Real talk’ reaches deep into the

experience ofeach participant; it also draws on the analytical abilities ofeach. Conversation, as constructivists describe it, included discourse andexploration, talking andlistening, questions, arguments, speculation, andsharing. (p. 144)

A final important aspect of constructed knowers wastheir ability to look back and

describe how they got to where they were. Further, this knowledge was shared with

others so that they too could somedayarrive at this level of knowing.

The above study contributes to a deeper understanding ofthe differences in the

development of epistemological views and values between women and men. Further, this

study looked at a broad population in terms of socio-economic status, educational and

16

cultural background, and age group, and in so doing helped to validate Perry’s scheme.

Baxter Magolda’s Epistemological Reflection Model

Marcia Baxter Magolda (1992), developed her Epistemological Reflection Model

through a longitudinal study of 50 men and 51 womeninfirst year college. Of the

original 101, 80 students remained through 4 years of college and 70 for the fifth-year

interview. She finished with 37 participants in post college follow up (10 years total). Of

these final 37, 12 were in advanced academicsettings and 13 had pursued advanced

education. She used annual open-ended telephoneinterviews and the Measure of

Epistemological Reflection (Baxter Magolda, 2001) during the college years, and a

modified interview afterwards, as measuresofparticipants’ epistemic assumptions.

Her Epistemological Reflection Model (Baxter Magolda, 1992) consisted of four

ways of knowing. Three of these ways had genderspecific patterns. While these

patterns were not exclusive to one gender, one used them more often than the other.

Baxter Magolda’s four ways of knowing are summarized and presented below in Table 1

for easy comparison.

Table 1

Description of Baxter Magolda’s Epistemological Reflection Model

Absolute Transitional Independent Contextual

Knowing Knowing Knowing Knowing

What is it? Factual Uncertainty exists Uncertainty, Reasonedinformation only insome areas, everything is uncertainty.is non- whereascertainty relative.

changing. remains elsewhere.Information

is either rightor wrong andexists in allareas ofknowledge.

Whohasit? Instructors Instructors, Equity between Self, peers and

and experts authorities. Peers authorities, peers, authorities have

are the help in the process and themselves. validholders of the of knowledge Students see their knowledgeif

17

Absolute Transitional Independent ContextualKnowing Knowing Knowing Knowingtruth. acquisition butdo viewsasvalidfor they can

Peers are not, notreally possess the first time. All support theirbut can pass valid knowledge. personal views are stance.on valid. Contextualknowledge knowersarelearned. unwilling toNostudent rely solely on

voice. their own

perspective orrely solely on

the perspectiveof others.

How does Readingand Shift to trying to Instructors provide One constructsone getit? listening to understand. Can context to explore. knowledge

lectures. share ideas with Must express throughInstructors peers. viewsand hear consultingshould Like to be other’s views. experts andclearly help challenged. processingthem to evidence.

acquire Concurrently,knowledge. one values

working

through his/herown perspective

by assessingone’s ownexperience andothers’

perspectives.(reasoning) Mastery Interpersonal (used Interindividual One moves

Patterns of (used more more by women)— (used more by toward harmonyknower by men) — tend to focus on women) — varying in communion

active uncertain areas. perspectives arise (connections

approach to Likes discussion from different andlearning. with peers and interpretations of relationshipsAsks lots of opportunity to knowledge claims. with others) and

questions express own Theylisten to agency(not content views. Resolves others and (separateness

to just listen). uncertainty themselves in with others).through personal forming their own

Receiving judgment. perspective. But(used more may havenovalidby women) — reason for theirmainlylisten, Impersonal (used view.

18

Absolute Transitional Independent ContextualKnowing Knowing Knowing Knowingnot much more by men) —

interaction wants to be forced Individual (used

with to think. more by men) —

instructors. Exchangesideas while believing inwith instructors the exchange ofand peers through views, theydebate. Resolves focused ontheiruncertainty by own views.logic and research. Changing others’

perspectivesis notrequired.

Howisit Instructors. Through Individual thinking Self-evaluated.evaluated Standardized measurement of and ideas are Thinkingis

tests. understanding. rewarded, Not connected to

penalized for self and one can

holding different stand outside

views from one’s

instructor. experiences toreflect.

Abstractprocessing ofexperiences and

information.Explanation Caused by Caused by the Representthe Used to shape

of the nature of the variety of views one’s own

differences inadequacies knowledge/subject. possible in perspectiveof the teller. (i.e. calculus is not uncertain world. through

discussed whereas judgment of

art and be evidence.discussed)

King & Kitchener’s Reflective Judgment Model

King & Kitchener (1994) proposed a seven stage model that had distinct levels of

knowing and meansof reasoning aboutill-structured problems. Each level represented

an increasing complexity of understanding andjustification, with stage 7 representing the

endpoint of cognitive development. “This modelis particularly noteworthy forits

elaboration of the upper levels of Perry’s scheme andfor the specification of dimensions

of epistemic cognition” (Hofer & Pintrich, 1997, p. 102).

19

There have been longitudinal (1977-1987) and numerouscross-sectional studies

conducted using the Reflective Thinking Interview with approximately 1,700

participants. Participants included almost 200 (in the longitudinal studies), 150 high

schoolstudents, 1,100 college students (both traditional and non-traditional age), 150

non-student adults, and about 200 graduate level students.

In this model, the first three levels were labeled pre-reflective and were similar to

Perry’s dualism and Baxter Magolda’s absolute knowing. In the pre-reflective levels

knowledge wasdirect, observable, and concrete (stage 1), moving slightly to not always

observable but still concrete (stage 2), to still concrete but could be temporarily uncertain

(stage 3). Justification for the first three stages remained either unexaminedor based on

reference to the authority.

With quasi-reflective thinking knowledge became uncertain. First it was

uncertain due to problemsin reporting or access (stage 4) and then movedto uncertainty

due to personal interpretations (stage 5) similar to Perry’s relativism and Baxter

Magolda’s independent knowing. At stage 4 a shift occurred in that justifications used

evidence and reasons, but they were incoherent. Justification at stage 5 was hampered by

context and wasfiltered by personal perception.

Uponreaching reflective thinking one becamea constructor of contextualized

knowledge and opento reconstruction. At stage 6, justification used evidence and

opinion (personal and from others) across different contexts and was evaluated using

specific criteria (weight of evidence,utility of solution, pragmatic need for action). At

stage 7, knowledge wasthe product of reasoned inquiry. Justifications were based on

probabilistic evaluation with the addedcriteria of risk of erroneous solutions,

consequencesofalternative judgments, and possible combinationsofthe criteria. Here

re-evaluation and possible reconstruction were mandatory when newevidence,

perspectives, or tools were introduced.

According to Hofer and Pintrich (1997) this seven-stage model provided “the

most extensive developmental scheme with epistemological elements” (p.102).

However, while there was a good range of scores for high school and college students, a

leveling off occurred in students following advance graduate degrees (masters/early

doctoral: M 4.62 SD .81 n=126 and advance doctoral > 3" yr M 5.27 SD .89 n=70)(King

20

& Kitchener, 1994, p. 283 Table B6.6). In fact, in the longitudinal study, this was the

only group that consistently reached scores in stages 6 and 7 (King & Kitchener, 1994,p.

133) and the differences were subtle. As the participants in this thesis study fall into the

former two categories (masters and early doctoral students) similar ceiling effects could

be expected.

Interestingly, Hofer and Pintrich (1997) reported little was known about how

reflective judgment developed in context. King and Kitchenerattributed this to the belief

that movement between stages happened very rapidly with plateaus of differing length

within stages. They argued that this was consistent with Fisher’s model of cognitive

development (Fisher & Pipp, 1984) in which development was “marked by periods of

relatively abrupt or fast growth followed by periodsofrelative stability” (King &

Kitchener, 1994, p. 29). As measurementofreflective judgment happened only

sporadically the chances of catching someone in mid-change wasrare.

For King and Kitchener’s Reflective Judgment Model, it was recommended to

have experienced administrators and face-to-face interviews. The standard problems

included: 1) differing claims about the pyramids; 2) bias in the reporting of news;3)

religious versus scientific views of evolution; 4) the safety of chemical additives in food;

and 5) the safety of nuclear energy. Further, scoring of the interviews required two

certified raters. Despite this complication, the level of detail in description provided at

the upper levels seemed to make it a very useful model for students in higher education,

as in this research. All the same, as wewill see in the final model below,very similar

results in classification of levels are possible using a much simpler process.

Kuhn's Levels ofEpistemological Understanding

Kuhn’s epistemological understanding model (D. Kuhnet al., 2000) was a simple

modelthat captured the essence of a very complex notion. This simplicity was perhaps a

responseto her earlier statementthat, “The study of more comprehensive kinds of

cognitive change, however, especially those involving change in knowledge acquisition

strategies themselves, poses serious methodological challenges” (D. Kuhn, White,

Carver, & Klahr, 1995, p. 8). This model advances the conceptual notions presented in

21

the other two models and is conceptually simpler. Kuhn et al. (2000) proposedthat

the developmental task that underlies the achievement ofmature

epistemological understanding is the coordination ofsubjective andobjective dimensions ofknowing. Initially, the objective dimension

dominated, to the exclusion ofsubjectivity. Subsequently, in a radicalshift, the subjective dimension assumes an ascendant position and theobjective is abandoned. Finally, the two are coordinated, with a balance

achieved in which neither overpowers the other. (p. 310)

Aswith the other models, Kuhn’s model was validated through empirical

research. In order to counter someofthe shortfalls of other models (Hofer & Pintrich,

1997), Kuhn ensureda cross section of participants. They included: three younger

groups (median age 10 years, n=20; median age 13 years, n=21; median age 17 years,

n=21;) all from low to middle class Caucasian population; young adult undergraduate

students (18 - 21 years, n=20); two mature adult groups (mid 20 — late 30) from a

community college (n=20) mainly Hispanic, and professional group (n=18) enrolled in an

exec MBAand mainly Caucasian;and finally an expert group (mid 20 — late 30, n=5)

Ph.D. candidates in educational philosophy.

The characteristics of knowledge represented by these categories are similar to the

other models discussed previously. The levels of epistemological understanding are

presented in Table 2. What makes this modelvery attractive is that Kuhn (2000) reported

a high correlation of assessed levels using this model compared to more complex

measurement instruments. There were no specially trained analysts required or overly

complex interviews. The instrument used has 15 items and waseasily administered in 10

— 20 minutes. Unlike other models, Kuhn claimed that the transitions from level to level

vary dependingonthe particular intellectual domain. She further explored the orderin

whichtransitions occurred across different domains (personal taste, aesthetics, value,

physical truth, and social truth). She found that the order that subjectivity and objectivity

were acknowledged and reintegrated atthe two majortransitional levels (Absolutist to

Multiplist, Multiplist to Evaluativist) differed. For instance, personal taste and aesthetic

judgments tended to be easily accepted as subjective and so objectivity had to be

introduced to progress from Absolutist to Multiplist levels. In contrast, objectivity was

accepted for physical truths and subjectivity had to be introduced at the Multiplist level.

22

Oncethis was dominant, objectivity had to be reintroduced at the Evaluativist level and

brought into balance with the subjective.

Table 2

Levels of Epistemological Understanding (D. Kuhnetal., 2000)

Level Assertions Reality Knowledge Critical thinkingRealist Assertions are Reality is Knowledge Critical thinking

COPIESof an directly comes from an is unnecessary.externalreality. knowable external source

and is certainAbsolutist Assertions are Reality is Knowledge Critical thinking

FACTSthatare directly comes from an __is a vehicle forcorrect or knowable. external source comparing

incorrectin their and is certain. assertionsto

representation of reality andreality (possibility determining their

offalse belief). truth or falsehood.Multiplist Assertions are Reality isnot Knowledgeis Critical thinking

OPINIONSfreely directly generated by is irrelevant.chosen by and knowable. human mindsaccountable only and is

to their owners. uncertain.Evaluativist Assertions are Reality isnot Knowledgeis Critical thinking

JUDGMENTS directly generated by is valued as athat can be knowable. human minds vehicle thatevaluated and and is promotes soundcompared uncertain. assertions and

according tocriteria of

argument andevidence.

enhancesunderstanding.

Olson (in press) discussed the implications of such an epistemological view with

regard to metacognition and normativity. He also advocated a balance between the

objective and subjective. The objective was comparedto the normsandstandardsthat

wereinternal to the practices themselves (Dewey, 1998), and derived from and embodied

in the institutions and schools. These institutions and schools determined what was

acceptable in a discipline, in part, through the awarding ofcredentials and degrees (and

all that that implies). While Olson did not mentionthe role that peer review and

23

conferencesplay in determining the norms,their importance seemsconsistent with his

argument. Metacognition played an important part in understanding and, consequently in

making judgments based on these normsand standards. These rules and norms were

based on reason. For Olson, the essence of the subjective was represented by experiences

of epiphany and perplexity. He argued “these feelings, feelings of puzzlementat a

problem andsatisfaction at its solution,are critical to all cognition” (n.p).

Analogous to Kuhn’s harmonizing of the subjective and objective, Olson suggests

that “coordinating goals and standards for their achievementare not only at the heart of

the educational problem; they are a primary source of metacognitive resources of

learners” (n.p.). Further, that “it is metacognition, an awareness of one’s own cognition

in relation to a norm orstandard previously agreed to with the teacher or other expert”

(n.p.). It is important to understand howthe subjectivity of metacognitive activity comes

into balance throughthe critical analysis of objective norms and standards becoming the

essence of a person’s ability to make judgment based on reason. The nextsection will

describe metacognition in moredetail.

Metacognition

‘Thinking about thinking’ is a commondefinition of metacognition. Flavell

(1979; 1987), a pioneerin this construct, contended that metacognition wasbest

understood as metacognitive knowledge (knowledge of person variables, task variables

and strategy variables) and metacognitive regulation (overseeing, planning, monitoring,

and measuring outcomesof cognitive activities). According to Hacker (1998),

To differentiate metacognitive thinkingfrom other kinds ofthinking,it isnecessary to consider the source ofmetacognitive thoughts:

Metacognitive thoughts do not springfrom a person's immediate externalreality; rather, their source is tied to the person's own internal mentalrepresentations ofthat reality, which can include what one knows aboutthat internal representation, how it works, and how onefeels aboutit.

(np.)

In addition, metacognition involved purposeful, goal-directed, and future-looking

behaviours (Flavell, 1971). Moshman (2003) stated, “At advanced levels, metacognitive

24

developmentinvolves the developmentof explicit understanding about the fundamental

nature andjustifiability of knowledge and reasoning. These are matters of what

philosopherscall epistemology, the study of knowledge”(n.p.).

Regulation often manifests itself in the form of questions to oneself that provide

oversight on the learning process. However, in characterizing cognition as

metacognitive, one must be careful concerning automated and nonconscious thoughts.

These are thoughts about procedures or knowledge that have been mastered at some point

in the past and no longer need metacognitive regulation. Hacker (n.d.) provided an

excellent conceptual summary;

The notion ofself-efficacy is echoed by Paris and Winograd (1990) whobelieve that most researchers now recognize a definition ofmetacognition

that “captures two essentialfeatures ofmetacognition—self-appraisal and

self-management ofcognition”(p. 17). Self-appraisals are people'spersonal reflections about their knowledge states and abilities, and theiraffective states concerning their knowledge, abilities, motivation, andcharacteristics as learners. Such reflections answer questions about“what you know, how you think, and when and whyto apply knowledge or

strategies" (Paris & Winograd, 1990, p. 17, italics in original). Self-

managementrefers to "metacognitions in action," that is, mental

processes that help to "orchestrate aspects ofproblem solving"(p. 18).Focusing on self-appraisal and self-managementhelps in the

conceptualization oflearners as individuals who needto be activelyinvolvedin the orchestration oftheir knowledge construction. (n.p.)

Thuswecan seethat the notion ofself-efficacy, the belief that you can influence

your own thoughts and behaviour,relates strongly to the notions of epistemic agency - a

core knowledge building principle - and dialectical reasoning, to be elaborated below.

“A strong case can be madethat the emergence of increasingly sophisticated

metacognitive understandingis a central aspect of advanced cognitive development”

(Moshman,1998, p. 963) with interesting and important correspondences to knowledge

building

Progressive change has been characterized for each of the epistemological models

presented above (Baxter Magolda, 1992; Belenkyet al., 1986; King & Kitchener, 1994;

D. Kuhn et al., 2000; Perry Jr., 1970). Most models are based on stages (see King &

Kitchener (1994) for a discussion on what constitutes a stage model) or levels that one

25

movesthrough, each with defined endpoints. It is the endpoints that are contentious in

most models. The endpoints are fuzzy at the highest stagesor levels, leading to the

question, “What happensafter one reaches the defined endpoint — does cognitive

developmentstop?” Insightfully, Moshman (1998) argued that endpoints are irrelevant to

identifying cognitive progress. In his seminal chapter Cognitive Development beyond

Childhood, he used a biological analogy, puberty, as a guide in setting parameters for

cognitive development. However, he believed that cognitive development“differs in

important ways from prototypical examples of developmentsuch as puberty”(p. 950).

He arguedthat there are three types of change that are noteworthy in both cases:

qualitative, progressive, and internally directed.

Piaget (1972) claimed that the transformation from logical deduction to formal

reasoning, or hypothetico-deduction, at about age 11 or 12 represented such a qualitative

change. Metacognitive reflection is another example of such change. Moshman(1995)

proposed that qualitative change could be viewedin the form of four types of reasoning;

cased-based, law-based, coherence-base, and dialectical. He fully acknowledged that

these forms of reasoning could be combinedto form other types. Further, that while

there is evidence that these formsare general acrossintellectual domains, there could be

other types identified that are specific to domains. Details on the types of reasoning are

provided in the methodssection.

Moshmanstates, “The emergence of thinking involves the deliberate coordination

of inferences and thusrepresents a higherlevel of intentionality and complexity. The

emergence of reasoning involves increasingly explicit knowledge about the epistemic

properties of one’s inferences, thus representing a higherlevel of reflection” (p. 973). So

it is possible to see cognitive progress thorough analysis of the process as well as

examination of the endpoints.

Internally directed changeis typically interpreted as change caused by the genes.

Suchis the case for puberty. However, there is no evidencethat this is true for cognitive

change (Moshman,1998, p. 952). Epistemic agency (Scardamalia, 2002) has individuals

actively knowledge building based on their own initiative and goals. These goals and

initiatives are derived interactively with, rather than solely caused by, the environment. It

is in this sense that internally directed change occurs rather than being genetically driven.

26

Moshman (1998) elegantly described the progression from inference to reason

and howthis related to cognitive development. He defined “thinking as an advanced

39 66.form of inference”, “reasoning as an advanced form of thinking” and suggested that

“reasoning is the primary locus of late developmental changesin cognition” (p. 952). He

next described reasoning in greater detail and related reasoning to metacognition and

cognitive development:

T suggest that three suchforms ofreasoning — case-based, law-based, anddialectical — can be distinguished and that each continues to develop long

beyond childhood. (p. 954)

A deliberate effort to constrain one’s thinking on the basis ofwhat one

believes to be justifiable epistemic norms constitutes reasoning, even ifthe

normsare not successfully applied or are demonstrably inappropriate. (p.953)

Individuals may, for example, achieve levels ofmetacognitiveunderstanding about the nature ofinference that transcend domains ofknowledge, types ofthinking, andforms ofreasoning. (p. 954)

I have proposed that advanced cognitive developmentis in large part the

development ofreasoning — that is epistemologically self-constrainedthinking. Reasoning, thus defined, is done by a rational agent — that is, anindividual whose thinking is rooted in epistemicforms ofmetacognitiveunderstanding. (p. 965)

To the extent that the individual perceives difficulties with his or hercurrentset ofprinciples and intentionally coordinates and reconstructs

themfor the purpose ofachieving a higher level of[] understanding, wemayposit a process ofdialectical reflection that is simultaneously aprocess ofreasoning and aprocess ofdevelopment. (p. 967)

Finally,

With the rise ofdialecticalforms ofreasoning, the study ofreasoningbecomesindistinguishablefrom the study ofdevelopment. Dialecticalreasoningis, infact, usefully construed as an effort to take control ofone’s cognitive development. (p. 968)

Hence,in simple terms,dialectical reasoning is recognizing through a

metacognitive process that you have gapsor incoherence with your understanding and

doing something aboutit (thus combining epistemic agency as found in knowledge

27

building and metacognition) to increase coherence and understanding — cognitive

development occursas a result of and interestingly enough wastheinstigator of the

process. This study looks for changesin students’ reasoning and levels of metacognition

overtime asa result of learning.

While movementthrough inference andreflection can be consideredasa natural

progression during intellectual development, reasoning, especially dialectical reasoning,

requires a consciouseffort. Learners move towardsdialectical reasoning through

effortful engagementin learning; knowledge builders engage in creative work with ideas

that expandsthe space of inventiveness and dialectical reasoning, with learning as an

important by-product.

This intermingling of concepts played a key role in the developmentof a

framework and the resulting methodsusedin this research. However, before introducing

this frameworkit is importantto briefly discuss aspects of community, especially online

communities, as the concepts introduced thusfar all have an inherent social aspect, which

will now be elaborated.

Online Communities

Woodruff (2001) states, “a communityis held together by four cohesion factors:

1) function, 2) identity, 3) discursive participation, and 4) shared values” (p.158). There

are perhaps other cohesive forces that weave together thus influencing the online

community (Woodruff, 1999). This research focused on epistemological development

andits relation to participation patterns within online communities, specifically how

epistemologyrelates to interaction patterns. Austin (2002) claimed that knowledge

building would not occurif taken out of a social context because the process was

inherently social. From a social cultural perspective, knowledge cannotbe built outside

of a social context (Vygotsky, 1978). Hence, sustainability of a community may be

deeply connected to the sustainability of knowledge building - especially design-mode

dynamics where participants take collective responsibility not just for their own

knowledge advancement but for the community as a whole.

Preece (2000) discussed the life cycle of a community. She used four stages,

28

prebirth, early life, maturity, and death, to describe this cycle. In maturity, communities

were often able to function independently with little need for manager oversight. Death

occurred when the discussions slowed below critical mass. Critical mass was the number

of users or people that was required to make the discussion viable.

McLoughlin & Marshall (2000) discussed the importance of scaffolding for new

learners in an online teaching environment. They believed that there were four skills that

werenecessaryfor effective online learning: 1) Articulation — this was being aware of

how onethinks and one comesto conclusions; 2) Self regulation — this was how one

plans and adjusts the strategies for learning; 3) A repertoire of learning strategies; and 4)

Self assessment/self evaluation — this was anticipation of problemsand finding help when

needed. Theystate,

A key part ofthe student experience in an online community is to learn tocommunicate andparticipate. Opportunitiesfor interaction are created

throughout the online module by allowing scopefor students to work in

groups, commenton each other’s work and offer suggestions, advice and

feedback to each other. (n.p.)

In this research I looked at how modeling and scaffolding by more experienced online

learners influenced the new learner.

For this thesis, a new learner is defined as someone whohasnevertaken an online

course. In my experience most course participants, especially new learners, started out

online by “talking”. That is, discussing how their experiences were related to the course

material or even just chatting about the topic in general. If knowledge building occurred

at this early stage it was likely by chance, or perhaps by a more experienced online user

who wasin the course and able to model techniquesthat lead to knowledge building.

The hypothesis to be investigated in this thesis is that in order to move beyond “online

talking” towards knowledge building, learners mustbe at a certain level of

epistemological development.

Finally, the instructor’s epistemological views normally have a large influence on

what occurs and on how this type of community develops. Belenkyetal. (1986) stress,

“Reliance on authority for a single view ofthe truth is clearly maladaptive for meeting

the requirements for a complex, rapidly changing,pluralistic, egalitarian society and for

29

meeting the requirements of educational institutions, which prepare students for such a

world”(p. 43). This position is consistent with the view that movement from absolutism

to contextual relativism is important if we want to develop in our students the ability to

function in a liberal society.

My Framework

The framework'ofthis thesis consists of four dimensions — knowledge building

(particularly epistemic agency) (Scardamalia, 2002), levels of epistemological

understanding (Kuhn etal., 2000), metacognition and forms of reasoning (Moshman,

1998). The level of epistemological understanding forms the base against which the

other aspects were compared. The aim wasto identify relationships between

epistemological level, metacognitive activity, forms of reasoning, and knowledge

building. Asthis research was exploratory, strong and interesting relations between these

dimensionswill require additional study that specifically targets these relations and their

cause and effect.

Of the epistemological models presented above, Kuhn’s model appearedto be the

most appropriate for analysis of belief-mode epistemologies. First, her participants

represented a good fit. Second, her model wasinclusive of gender, and third, it seemed

conceptually clear. Despite this, it was not clear that defined movement by students

between levels of epistemological understanding would be observable, because this was

only a 13-week period of virtual observation. Nor wasit clear that this would reflect

growth in design-mode epistemology. Thus welookedat the data separately for

indications of attention to design modeissues and continual improvement. To provide a

further measure of discrimination, Moshman’s forms of reasoning were not only used in

their own right, but were used to help make distinctions between multiplist and

evalyativist levels of epistemological understanding. Observing an increased

sophistication in the types ofjustification used between pre- and post-tests might be due

1 My research wasa based on design research. It involved pilot testing of all aspects and

continual tweaking of instruments. This framework represents the one used after the pilot was conducted

and analyzed. Many changes were made asa result of pilot testing. See Chapter 2 for the framework usedfor the pilot.

30

to increased levels of epistemological understanding. Other expected variations included

differences in knowledge building activity between students depending on their use of

justification and their explicit use of metacognition and design-mode. Metacognitive

activity was a dimension that provided an added level of confirmation to results as it was

intricately related to the other three dimensions. I expected that students with high levels

of metacognitive activity would also show higherlevels of sophistication in justification.

Individual measures only showedpart of the overall picture, and problems

inherent in correlational data are well known. As discussed in the literature review,

knowledge building, epistemology, and cognitive developmentare inherently social in

nature. Hence, the thesis also examined how specific social interactions that occurred

within the online environment might be interpreted on these various dimensions. Data on

basic social interactions, such as who reads whosenotes and whobuilds on to whose

notes, were automatically captured by the Knowledge Forum software used for the

courses understudy. In addition, both Kuhn and Moshmanhighlight the importance of

argumentation as a meansto progressto higher levels. ““Argumentation is usefully

construed as a processofdialectical reasoning in which two or moreindividuals

coordinate multiple cases and laws in a shared effort to make conceptual changes”

(Moshman,1998, p. 691). Hence, argumentation was also examined by lookingto see if

there were examples where argumentation played a role in the discussions. Beyond these

correlational results I also looked for aspects of epistemological developmentthat|

expected to be absent evenat the highest levels of belief-mode epistemology; namely,

evidence of student work in design mode. Asindicated earlier, this is a dramatically

different epistemology than the one students are introduced to in school and the current

13-week course was not designed to impact that particular facet of epistemological

development. So the prediction was that there would be limited advance alongthis

dimension, and the goal wasto characterize both the advances andlimitations.

Finally, it is important to note that that the highest levels of the forms of reasoning

or epistemological understanding are not being presented in this thesis as representing the

endpoints of a cognitive developmenttrajectory; in fact, the authors of the models do not

suggest this either. Rather, they represent the current edge of understandingin this

domain andin the case of the belief-mode work are supported by strong empirical

31

evidence. Hence they provide excellent models for this research.

Research Questions

In developing my ideasfor this research, I reflected upon my experience with

knowledge building communities. Through discussion I found that others have had

similar experiences and feelingsin their struggles with the medium ofonline

collaborative communities. Grappling with the reasons why online communities seem to

have a hard time sustaining life outside an established structure, like a course,is

challenging. Insufficient time is often presented as a reason for lack of participation, yet

in reality, time is a matter of priorities. Because reciprocity has an affect on

sustainability, Preece (2000) offered hope whenshestated, “For various reasons having

to do with personal and group psychology, reciprocity can be strong, even whensocial

ties are weak... As long as people return to the group, whichis often achieved by

ensuring fresh information. Reciprocity is likely to occur, which will benefit both the

group andthe individuals in it” (p.181). How does one encourage noteworthy

engagementin such an environment? This research may shed somelight on a possible

cognitive-developmental connection to personal engagementand sustainability of the

community.

This research was exploratory in nature and began with individuals and their

interactions within an online community. It examined whether epistemological changes

occurred. If changes occurred, how were they manifested within the community’s

database? Again, the aim wasto identify relationships between online participation

patterns, epistemological level, metacognitive activity, and forms of reasoning.

Questions

1. Are changesin belief- or design-mode epistemology evident over the duration of

the courses?

a. Whatare characteristics of student work in learning logs?

b. How does epistemological understanding relate to participation and

interaction patterns in online discourse?

32

2. If there are signs of epistemological change within individuals, how do these

changes manifest themselvesin the interactions amongst learners in the online

learning conference as measured through changesin the reading and responding

patterns of users? Is there a changein the reading/responding patterns of users

over the course?

Why wasthis Research Important?

Bereiter (2003) stated that

Preparing studentsfor the Knowledge Age is indeed a new challenge. Noone knowsfor certain whatit entails. Educators and researchers needto

collaborate and invent together. We need to build knowledge ofhow toeducatefor knowledge building. Thisis a difficult problem, but the stakesfor solving it are high. Nations able to solve the problem will gain a

significant economic advantage over those that arestill struggling withhow to solve the older problems. (n.p.)

If knowledge is the main resource of the knowledge age, then weasa society will

need the tools and skills to work with this resource. If one’s epistemology effects one’s

ability to work with knowledge, then understanding this phenomenawill be importantin

the knowledge age. “Beliefs about knowing are potentially important determinants of

intellectual performance” (Kuhn & Pearsall, 2000, p 309). It could be important to

society’s transformation from industrial-based to knowledge-based.

In more practical terms, I believe that there will be an ever-increasing demandfor

online learning to help re-tool for the knowledge age. Adult learners, with many

competingpriorities in their busy lives, will insist on well thought out, high quality

educational opportunities. This research can help instructional developers and online

facilitators to get at the root of learner participation, and hence, help get the most out of

an already powerful knowledge building pedagogy. Further, the transition from

multiplist to evaluativist is difficult and requires a considerable effort by educators

(Kuhn, 2003; Kuhn & Dean, 2004). Indications are that the shift to design-mode could

be harder yet, as well as underrepresented in the currentliterature.

CHAPTER2: PILOT STUDY

Advance Look at Overall Research Methods

This design research was conducted in two phasesas outlined in Table 3. The

pilot phase was conducted from January- June 2003. It included data collection and

analysis. As a result of the pilot work, there were changesto the theoretical framework

and methodsfor the main study. This pilot phase was conductedto test the

operationalisation of the framework and for the researcher to learn how to code

qualitative data. Phase two wasthe main study and data collection was conducted from

September to December 2003.

Table 3

Chronological Overview of Data Collection

When Collected Data Collection Method What used for

Pilot Study

Jan 03 Permission to use Online survey Ethics reviewdata

Jan 03 Pre-epistemological Online Measure baseline

views (see Appendix questionnaire epistemologicalB) views

Jan — April 03 Epistemological Leaning Logsviews

Jan — April 03 Analytical Took Kit Automatically Indication of

measures recorded by participation anddatabase software interaction patterns

March- April 03 Post-epistemological Online Correlationsviews questionnaire

March- April 03 Biographical Online Correlationsquestionnaire

Main StudySept 03

Sept 03Sept 03

Permission to use

dataBiographicalEpistemologicalviews

Justification

Online survey

Theories of learningpre-essay

33

Ethics review

Determineinitial

level of

epistemologicalunderstanding

Measure ofdialectical reflection

34

Sept — Dec 03 Metacognition Learning Logs Measure ofJustification dialectical reflection

Sept — Dec 03 Analytical Took Kit Automatically Indication ofmeasures recorded by participation and

database software interaction patternsDec 03 Post-epistemological Theories of learning Determinepost

views pre-essay level ofepistemologicalunderstanding

Measure ofdialectical reflection

Jun 05 Coded passagesfor Inter-raterreliability

metacognition and

dialectical reflection

Glesne (1999) described a pilot as a chance totest out certain aspects of a

research study. It could be interview questions, surveys, observation techniques, or

whateverneedsto betrialed before the ‘real research’ starts. She recommended using

participants from the actual population wheneverpossible. After conducting mypilot

research, I found that I needed to improve my overall framework, data collection

methods, and analysis techniques.

The theoretical basis for the pilot research was discussed in chapter one.

However,as a result of the pilot, an additional review ofthe literature was necessary, in

order to understand online surveying methods. While the research questions remained

the same, modification to the framework wasrequired. This chapter will explain what

wasdone during the pilot testing. First will be a brief literature review of the methods

used (methodsthat were different from the subsequent main study), followed by a

description of the specific tools I used to collect data. Next, I discuss the results and the

effect they had on the main study.

Pilot Framework

The pilot framework focused on two specific dimensions from the larger study —

selected knowledge building elements (Scardamalia, 2002), Table 4, and ways of

knowing, Table 1. While it is understood that the knowledge building elements operate

as a system,it was decided to target specific elements that seemed to morereadily relate

35

to the epistemological models. The knowledge building elements and myrelated

questions were used to structure survey and learning log questions and to help identify

progress towards a knowledge building community. The epistemological reflection

model was used to determine approximately where participants started and ended on an

epistemological continuum.

Ofthe first three epistemological models (Baxter Magolda, 1992; Belenkyetal.,

1986; Perry Jr., 1970) presented in chapter 1, Baxter Magolda’s model appeared to be the

most applicable for my research. Thefinal two models discussed in chapter | (King &

Kitchener, 1994; Kuhn et al., 2000) were reviewed and addedas result ofthe pilot

research. However,for the pilot, Baxter Magolda’s participants (especially the ones who

continued the study after college) seemed to represent the bestfit.

Table 4

Knowledge Building Elements and Related Questions

Knowledge Building Related QuestionsElements

Real ideas & authentic From where and from whom dothe problems come?problems Whatare the types of problems discussed or questions

posed? (This measurecangetat the belief that one’s own

ideas and problemsare important and the extent to whichthey lead to the generation of authentic problemsto driveidea improvement.)

Idea diversity & constructive Are ideas explored from different angles and contrasting

use of authoritative sources viewpoints? To what depth are problems explored?Howare sources used? Do participants enterauthoritative sources into their notes? (These measures

explore the value placed on diversity and authority.Whichis given more weight? Howare conflicts inopinion resolved?)

Improvable ideas Howare ideas treated? Do membersfeel safe to expressideas that are not ‘fully developed’ (if this is possible)?Whenare ideas treated as ‘finished’? Do they engage in

design with ideas, aimed at their continual improvement?

Epistemic agency Whotakes the lead in generating notes and ideas? Howinvolvedis the instructor? Do students convey a sense ofagencywith respect to ideas, finding ways to explorethem in greater depth, identifying new problems,monitoring their work and collaborating with others, andin other ways identifying new steps rather than being

36

Knowledge Building Related QuestionsElements

dependenton the teacherto identify next steps for them?

Embedded and Howdoesthe individual/group monitorits progress?transformative assessment What happensasa result of this assessment?

Symmetric knowledge Whereelse (other than class mandated material) doadvancement students get information? How doesthe progress of the

grouprelate to the progress of individuals and progress

of the discipline as a whole? Are outsideresources/communities accessed?

Knowledge building Whois addressedin the notes (peers, instructor)? What

discourse is the tone of the notes? Is there evidence of knowledgeadvancement?Whatare the goals of the discussions?

Whoadds value to the discourse?

Pervasive knowledge Doesthinking about ideas occur outside of class?(i.e. is

building there any evidence that students are transferring ideas toout-of-school contexts, and bringing ideas in from

diverse contexts?)Do students express a desire to continue with the ideas,

media, and challengesraised in the course, after courses

are over?

Survey Design

This section presents a literature review of the web-based survey methodspecific

to the pilot research. It does not include theories or models previously discussed as part

of the literature review in Chapter 1, nor those that will be discussed as part of the

methods in Chapter3.

Surveys have been used for data collection in the social sciences for manyyears.

There is a plethora of types and uses. As a result, surveys tend to be taken for granted.

They are seen as easy to design and administer, yet many are poorly designed and

administered. This is especially true for surveys administered through the internet (web

surveys) (Dillman & Bowker, 2001; Solomon, 2001). Dillman, Tortora, Conradt, &

Bowker(1998)state, “The ability to collect thousands of responses at no morecost than

collecting dozens of such questionnaires has enticed a culture of web surveys to develop

whichtendsto ignore the scientific underpinnings of surveys. Large numbers of

responses are sometimes viewedasindicative of the quality of the survey”(p. 1). Using

the web has greatly reduced the cost and effort required to administer surveysas neither

37

mailing nor processing of large quantities of paper-based surveys are required. However,

care must be taken in production of quality surveys.

Dillman and Bowker (2001) have identified four important sources oferror that

are associated with surveys: coverage, sampling, measurement, and nonresponse. This

pilot research dealt with a population that had access to both a computer andthe Internet.

Further, the research was exploratory in nature and was not immediately concerned with

generalizing the results to a large population. So, the problems associated with coverage

and samplings were not considered applicable. However, nonresponse and measurement

error categories were considered germane.

With a small sample population, participants who are unable to properly

download and respondto a survey could seriously affect the results. This is nonresponse

error. Therefore, the design ofmy survey was very simple. Simple formats reduce

transmission andload timesinto the participants’ browsers, thus increasing the chance of

successful completion. Dillman et al. (1998) did an experiment that compared the

completion rates of plain vs. elaborate web surveys. The plain surveys used black

lettering on a white background and placed the answers onthe left-hand side (similar to

paper-based surveys). In contrast, the elaborated survey used HTMLtables, sophisticated

graphics, bright colors, and placed the answers on the far right side. They found that the

plain surveys took less time to complete, were likely completed in one session, were

more fully completed, and had a higher completionrate.

Poor wording and complex web form buttons can cause measurementerror; that

is, measuring something other than whatis anticipated. I used a small group of fellow

researchersin order to verify the wording ofmy questions. Several questions were

modified for clarity and someofthe text boxes were increased in size. Further, for

multiple-choice questions, participants were provided with the opportunity to supply their

own answers. This was intended to reduce error caused by a lack of appropriate choice in

the proposed responses.

To encourage maximum participation, questions for my survey were designed to

keep the time to complete to approximately 20 minutes. Indeed, average completion time

during the pilot was just under 20 minutes. This required the use of mainly closed-ended

questions. Open-ended questions were used only where deemed absolutely necessary.

38

However, this meant the use of several types of web form buttons. Participants had to be

able to properly navigate and correct answers, using radio buttons, check boxes, drop

down menus,etc. Fortunately, the participants in the pilot were web-savvy and did not

need to be shown howthese web-surveyfeatures functioned. To avoid the problems of

multiple submissions, survey fatigue, and non-completion as identified by Dillman &

Bowker (2001) and Anderson & Kanuka (2003, Ch. 11), participants were required to log

into the survey (using a login name and password). Questions were scrollable. The

survey was presented on one web page with a simple color scheme, and security allowed

for only one submission perparticipant. All this was achieved using a self-designed Perl

script. Appendix A showsthescript for the pilot post-survey. A different script was used

for the pre-survey, another for the consent/bio data, and anotherscript for analysis.

Dillman, Sinclair, and Clark (1993) conducted research that showed the effects on

responserates of survey length, respondent-friendly design, and objectionable questions.

Not surprisingly, shorter surveys that are respondent-friendly produce higher response

rates. Surveys with objectionable questions decrease the rates of return. Whileit is

sometimeshardto tell what would constitute an objectionable question, I believe that I

did not ask anything embarrassing or personal and additionally, my questions had passed

ethical review. Further, my self-imposed timerestriction obliged meto be verycritical

about the number of questions and the design of an easily usable form. While I included

all the questionsthat I felt were important to help answer myresearch questions,I limited

open-ended questions and used multiple-choice questions to reduce completion time. As

well, participants were able to provide their own answersforall questions by using an

“other” option.

Anderson & Kanuka (1997) used a 43-question survey when analyzing a 3-week

asynchronous computer conference of 18 experts in adult education and community

development. The survey was used to measure the effectiveness and value of the forum

as perceived by the participants, as well as to gather some demographic data. This

research was an example of a successful survey because 15 out of the 18 participants

completed them. I considered this research very similar to my pilot research, in that

participants were adults working online at a distance, interested in education, and with

access to a computer andthe Internet.

39

Pilot Method

The pilot research consisted of 4 main data collection methods: 1) a pre and post

survey; 2) the conference notes; 3) participation and interaction patterns derived from the

analytic tool kit; and 4) learning logs. These complementary sources were chosen based

on the exploratory nature of the research questions. Each method will be described

below,followed bya brief description of my analysis procedure.

Survey

Pre- and post-course web-based surveys were usedto collect baseline and end

state views. Survey questions were based onthe pilot framework outlined above. They

were designed to help understand where students werein their epistemological

development. The survey questions used principles from knowledge building as a

context in order to add another dimension for analysis. A sample of the pre-course

survey questions can be found in Appendix B.

The design of a web-based survey involved a numberofsecurity issues. A login

and password, special program codethat allowed only one submission perparticipant,

and a secure database all helped to ensure data integrity and security. Questions were

designed to be a mix of multiple choice and short answer. All multiple choice questions

had the option of “other”. Participants’ answers were rated using Baxter Magolda’s

Epistemological Reflection Model as a guide to scoring (Baxter Magolda, 1992).

Responseswere sorted into one of her four categories (Absolute, Transitional,

Independent, or Contextual Knowing) and a composite score was determined.

Conference Notes

Note analyses produced a huge amountof data. All notes within the course

conference were read. Analysis was conducted using the knowledge building

determinants (Scardamalia, 2002), and the Epistemological Reflection Model (Baxter

Magolda, 1992), as outlined in the literature review from Chapter 1. A first reading was

madeto highlight notes and groups ofnotesofinterest, in particular, notes that

demonstrated a knowledge building principle or displayed clear evidence ofa particular

40

way of knowing. The analyses were inherently iterative (deeper analysis of notes based

on what was found during the previous pass) and included the use of the Analytic Tool

Kit, which is described briefly in the next section.

Participation and Interaction Patterns

The Analytic Tool Kit for Knowledge Forum is an automated data tracking

system which can compile a variety of reports based on participants’ performancein the

online databases. It is described more fully in Chapter 3. The Analytic Tool Kit was

used to analyze participation within the database. Individual and group measures were

used to investigate overall indications of collaboration. Comparison and analysis of notes

from the beginning and end of the courses were made. Participants were divided into

three groups to better inform the analyses. Thefirst group consisted of participants new

to both online knowledge building communities and the masters’ programme. The

second group ofparticipants had taken previous courses within their degree programme

but was new to online knowledge building communities. The third group was made up of

experienced users. It was expected that differences amongst these groups might emerge.

For example, the more experienced group’s notes might provide useful models of

participation for the newerparticipants, and so might be read more by membersofthat

group. Using Analytic Tool Kit measures that show who hadread whose notes, who had

built on whom,and who had referenced whom,I hoped to get a deeper understanding of

the dynamicsofthe conference.

Learning Logs

Learning Logs were special notes in which students kept an ongoingreflective

journal of thoughts. These thoughts were about course content and process. They could

have been metacognitive (Hacker, n.d.) in nature. In a similar fashion to the regular

conference notes, learning logs were analyzed for knowledge building and

epistemological content.

Design research involvesiterative intervention sequences. One such example of

intervention from the Learning Log data wasthe use of prompts or questions to motivate

the direction of the weekly log. To facilitate epistemological reflection, participants were

4]

periodically encouraged to use specifically designed questionsfortheir learning logs. All

questions were providedin a note at the beginning of the course. These questions,

grouped by knowledge building determinants (Scardamalia, 2002), were:

Real Ideas, Authentic Problems:

e Where did the problems/questions come from this week?

e How relevant were the questions posed this week?

e What can you do to make things morerelevant?

Idea Diversity & Constructive Use of Authoritative Sources:

e Have you used an idea from someone else?

e Did someoneelse’s idea spark a new idea for you?

e How haveyoujustified your opinions and ideas?

Improvable Ideas:

e How do you decide whento contribute to the discussion?

e Is there a point at which no more can besaid in a discussion?

Epistemic Agency:

e What are your goals for this course?

e How do youensure that your questions and concernsare being met in the

discussions?

e What do you do when youare notinterested in the week’s topic?

e When do you look for more research or articles on a topic?

Embedded and Transformative Assessment:

e How doyoufeel that online evaluation/assessment should happen?

e Explain how you evaluate your own learning throughouta course.

e How do you value/judge the feedback from various sources(peers, instructor,

tutor)?

Symmetric Knowledge Advancement:

e How do you go about looking for information concerning questions or

problemsarising from the week’s readings?

e How doyoureconcile information presented in other classes or readings with

what you are learning in this course?

42

Knowledge Building Discourse:

e Doyou think that discussion is an important part of this course? Why? Isit

important in all courses, i.e. are there content areas where discussion in class

is less important?

e Howis new knowledgecreated?

e Do youthink that any new knowledge has been created in this course?

Pervasive Knowledge Building:

e How hasthis online pedagogy changed yourthinking?

e What would you recommend for someone to do well in a similar online

course?

It was not expected that many explicit examples of epistemological development

would be seen within the conference notes because of the normally elusive nature of

cognitive development. Instead, I plannedto rely on abstractions from the experiential

discussions within the conference. However, with the help of the above questions to

focus student reflection, I hoped to see a clearer indication of their current

epistemological viewsin the learning logs.

Analysis Procedure

In an attemptto classify participants’ survey responses into epistemological

framework,the first three weeks of the conference entries, as well as the Learning Logs

and Pre-survey responses were read and then reread. The array of data sources provided

a broader and more comprehensive view of a participant's reasoning. From this review, a

participant’s epistemological category was assigned. A parallel process was carried out

on the last three weeks of the conference entries, as well as the learning Logs and Post-

survey responses. The weeksin the middle were used to identify any apparent points of

change.

Pilot Study Results and Discussion

This section provides the data analysis from the Pilot Study conducted between

January and May 2003, The course wasan Introduction to Computer Applications in

Education, conducted over a 13-week period. The overview read,

43

Computer technologies are influencing many aspects ofeducation. Thiscourse offers critical overview ofthe different ways that computers areused in education. Topics include computers in the classroom, interactive

instructional technologies, learning environments, communications

technologies and Internet applications at K-12 and beyond. (Brett, 2002)

While this was not a mandatory course, students in the Computer Applications

Programmewereencouragedto take this course. It was also very popular with students

in other curriculum programmes. The entire course was held online and there were no

face-to-face meetings.

How Participants Were Chosenfor Detailed Analysis

The participants, n=12, were masters and doctoral candidates (masters = 10,

doctorate = 2) between the ages of 25 and 50. Of the 18 students whogave full consent

to participate in this pilot study, 17 completed a pre-survey but only 12 completed the

post-survey. Only participants who had completed both pre and post surveys were

included in the analysis.

For the pilot research, I wanted to understandif the data collected would help me

to answer my research questions. WasI missing key data necessary to be able to make

conclusions? Would I be able to clearly analyze and categorize the responses from the

survey questionnaires? Four participants were selected for in depth analysis. They were

selected to ensure a wide range of ‘participation indicators’ (the two extremelevels of

participation within the database as well as the average for both male and female) using

the Analytic Tool Kit’s basic participation and interaction measures on the conference

viewsof the main discussions. It was decidedto limit the database analysis using the

Analytic Tool Kit to simply the weekly notes, i.e. none of the assignment or coffee house

views were used. This was done because, in the views where assignments were presented

and discussed, participants were expected to contribute and were graded accordingly.

While the main views were part of the overall course performance and thus graded, it was

felt that a more accurate accountofparticipation could be garnered by using only the

weekly views that focused on the course content. The overall assessment value of these

main views was 10% of the overall course mark. However,for analysis of

epistemologicallevels, all views were used as participants were not graded in this area

44

and henceit wasfelt that no bias would exist.

Analysis of the data from the Analytic Tool Kit, Table 5, provided a clear

indication of the average and extremes. The four participants (6, 7, 12, & 25) were

chosento represent the two extremelevels of participation within the database as well as

the average for both male and female. For example, participant 25 had both created and

read the fewest numberofnotes, participant 7 had created and read closeto the greatest

numberofnotes, and participants 6 and 12 were close to the average and median in note

creation and reading. Choices werelimited due to the availability of post survey data as

regrettably, not all participants completed the post survey. Nevertheless, all 18 students

(17 participants and | non- participant) who finished the course were included in the

Analytic Tool Kit analysis which provided a goodindication of levels of participation.

Unfortunately, some features of the Analytic Tool Kit were not available to be

used with these data as this course used an older version of the Web Knowledge Forum

software (Version 2), which contained fewer of the more sophisticated knowledge

building features available in Version 4. The average, median and standard deviation are

provided in Table 5.

It must be noted that survey questions were amended and improvedaspart of the

iterations in the pilot study. Five questions were either reworded or had their response

options reworded (questions 4, 9, 12, 14, and 15) and 3 additional questions were added

to the post survey. There were also different biographical questions asked between the

surveys. A total of 18 questions remained identical between surveys.

All of the participants’ notes and learning logs were read at least twice during the

coding processin addition to once or twice during the course itself. This increased toat

least seven or eight readings during coding for the main research. The nature ofthe

content waslatent, as opposed to manifest (content residing on the surface). Hence,

interpretive analysis was required to determineclassification of latent content. This type

of analysis can be contrasted to overtly counting instances that were easily observable

(how manytimesa student used the word ‘knowledge’ for instance). The unit of

analysis, or what was codedas an instance, varied from one sentence to several sentences

and can be considered a “thematic unit” (Garrison & Anderson, 2003, p.144). The unit of

analysis had to be flexible as contextual meaning varied. An effort was madeto capture

45

enough content to represent the classification for the epistemologicallevel.

Table 5

Calculation of Basic Knowledge Building Measuresper Participant

Participant #ofnotescreated %ofnoteslinked %ofnotesread # of revisions

ID

0 44 59 44 01 76 82 81 132 58 79 68 73 87 85 27 304 172 92 93 1646 75 77 74 467 120 89 93 228 62 84 52 109 69 81 93 810 97 79 43 72

11 106 79 92 5312 94 87 80 9813 52 56 92 9914 45 64 92 2016 157 92 80 3318 101 76 37 5725 24 50 21 1130 113 89 57 99

Average 86 78 68 78

Median 82 80 77 32SD 39 13 25 44

One very important componentoflatent analysis is using multiple raters to ensure

reliability. This was not donefor the pilot analysis because the goal of the pilot study

wasto help test data gathering techniques and to hone the methodsofanalysis, rather

than establish reliability at this point. It should be noted that two raters were used for the

coding of data from the main researchstudy.

Classification ofParticipants into Groups

Participants were classified into three categories for group analysis. It was

hypothesized that participants who have nevertaken an online course would interact

differently from those who had. Hence,participants were considered new online learners

46

if they had not experienced a fully online course previously, regardless of the point they

had reached in their masters’ program. Participants who had experiencedat least one

fully online course were considered to have some experience. There were no participants

considered experienced (greater than 2 online courses completed) in this sample. Table 6

showsthe results of this classification.

Table 6

Experience Levelof Participants

New Online Learner (Online — 0) Some Experience (Online — < 1)

ID Knowledge Online Courses ID Knowledge Online CoursesForum Forum

3 0 0 0 2 2 2 06 0 0 0 10 1 1 >47 0 0 >4 12 1 1 18 0 0 >4 14 0 1 >49 1 0 1 16 2 1 >425 0 0 0 18 2 1 4

Individual Analysis

The following section discusses the analysis of four individuals as described

above. The detailed individual analyses enabled a greater understanding of the

effectiveness of the chosen methods, data gathering tools, and analysis techniques.

Reflection on these areas would ensure improvements for the main research.

Analysis ofParticipant 25

This participant was considered to be a new online learner (had not experienced

any online course) as per Table 6.

Biographical data

Age 36-40; Female

Q.Briefly describe your experience with computers, online learning, and

collaboration.

A. “I have good knowledge and experience about computer applications and withvarious programming languages- This is myfirst experience with online learning and

47

collaboration” (Participant 25, Survey, Pre).

Self-introductionfrom Database

I am very excited to be a part ofsuch diverse class setting. I did myundergraduate in Electronics and Communications engineering. I workedfor a large telecom company in the area ofswitching technology and IT(particularly application development and database technologies) and hadenjoyed. But at one point Ifelt that I need something morein life and

migrated to Canada. I did my master in Information Systemfrom

University ofToronto last year and also have learnt aboutmulticulturalism. It is a very rich experience in addition to academic.Still

Ifeel that I need to learn more and decided to cometo the exciting andpromising CTL program. In this process I have developed myinterestin

integration oftechnologies into constructive learning environment anddistant learning. CTL 1602is the start.Tamafull time student ofMA (CTL), specialization in ComputerApplications. This is myfirst on line course andalsofirst coursefor theprogram.Lookingforward to working and learningfrom everyone. (Participant 25,Views, Intro)

Participation and Interaction Patterns

This participant created the least numberofnotes (24, avg = 86), linked the fewest

notes (50%, avg=78%), read the least numberofnotes (21%, avg = 77%), and made only

11 revisions to notes (avg 78). Participant 25 displayed a low level of knowledge

building characteristics for this database.

Learning Log Analysis

Belief-Mode Analysis

This participant did not use any of the questions provided for the learning logs.

Unfortunately this resulted in only 8 coded segments (Independent Knowing = 6,

Transitional Knowing = 2). There were no codedsections during the first 3 weeks and

there was a greater concentration of coded segmentsin the last 4 as shownin Table 7.

48

Table 7

Breakdown ofInstances of Belief-Mode Epistemological Levels for Participant 25 per

Weekwithin the Learning Logs

Week Absolute Transitional Independent Contextual

Knowing Knowing Knowing. Knowing

1

234 15 167 1

89 1 110 11112 1 1

Thefirst instances of belief-mode epistemological level came in week 4. This

example of Independent Knowing showedherview on equity ofability to know,

I believe it depends on each person's (boy or girl) potential, interest and

determination. I did my engineering whichis still a male dominatedprofession, my experience says that girls are equally competent and arehard working. No doubt there are some exceptions. (Participant 25, LL,Week 4)

Week 9 showed a greater appreciation for the role that peers play in the creation of

knowledge,

I am glad and thankful to my classmates that class colleagues respondedon the questions in very constructive way which gave me new dimensionandbetter clarity about the subject. (Participant 25, LL, Week 9)

However,this participant never indicated how a peers’ contribution would be judged or

evaluated. All opinions were taken at face value and, hence,this participant remained at

an Independent Knowinglevel.

49

Design-Mode Analysis

There wasnoindication of design-modeactivities from this participant in either

her learning log or conference notes. Comments were madeas to her agreement or

disagreement with, like or dislike of ideas or theories, as shown below in the example in

the next section. However, there is no suggestion in this statement of an idea to be

improved.

Survey Analysis

Pre- and post-survey showed slight movement towards a more complex

epistemological view. Interestingly, there was movementin both directions, that is, from

more complex to less complex and vice-versa. There were several contradictory

responses. For example, the participant jumped back and forth concerning the role of

peers in their own learning. In the case of classroom/online discussion, peers were

important, however, with respect to learning, the peers were not important. It seemed

that peers were important for sharing ideas as shown in this Independent Knowing

example from the learninglogs,

CMCprovides an option to the individuals who are less confident and shy,to express themselves. This is an excellent ideafor knowledge creation inthe workplace / school. From mypast experience that wonderful ideas

comefrom different stakeholders andifthey are channeledproperly, and

become new knowledgefor the organization. (Participant 25, LL, Week12)

This participant started the course believing that students could learn just as much from

their peers as from the instructor but finished the course by notbelieving this to be the

Case.

Note Analysis

There were very few contributions (24 notes). No inferences to epistemological

levels in the notes were found.

50

Analysis ofparticipant 12

This participant had experienced one previousonline course as per Table 6.

Biographical data

Age 36-40; Female

Q. Briefly describe your experience with computers, online learning, and

collaboration.

A. “T use a computerfor e-mail, grades, making handoutsfor my students online

learning - this is my second online course collaboration - very little experience”(Participant 12, Survey, Pre).

Self-introductionfrom Database

Wow, what a varied group ofpeople we are! It’s great to read everyone's

bio and get afeelingfor who I will be spending the nextfew months with.

I live in Japan (Thailand before here) and teach English at a women’suniversity here. My students are rather unmotivated and I’m hoping thatgetting them into a CMC environment might motivate them. So I’m taking

this course to learn more about that. I think my thesis will be about CMC

environments and the crazy things that can happenin them.I have nofree time but when I did liked traveling, scuba diving, reading,

running, and eating greatfood(lots ofgarlic andolive oil) with red wine.

Okay, I still eat and drink well, but myfree time is totally consumed by my1 1/2 year old son Bob so no morereading (well, I do love Red Hat Green

Hat). My husband Frank is an internet guy who consumesthe rest ofmymeagerspare time, what can I say, I’m busy but loveit, life is great...Lookingforward to what looks like a great course with an interestingbunch ofpeople! (Participant 12, Views, Intro)

Participation and Interaction Patterns

This participant created a sizable numberofnotes (94, avg = 86), linked the fifth

greatest numberofnotes (87%, avg=78%), read an above average numberofnotes (80%,

avg = 77%), and made a large numberrevisionsto notes (98, avg 78). Participant 12

displayed an average example of knowledge building characteristics for this database.

51

Learning Log Analysis

Belief-Mode Analysis

This participant showed great commitment and tenacity towards learning as

shown in this example of Independent Knowing,

Also the one posting I made (whichfor me was groundbreaking) had no

responsefrom anyone. I had thoughtit original and thought provoking

but... no response. Anyway the point is thatfor me a newfield ofresearchopened up - transformational/critical pedagogy and the Internet andithappened because ofmy participation in this course. So, nofeedbackisokay. (Participant 12, LL, Week 3)

But at the sametimethis student displayed a belief of certainty of knowledge in some

areas, Transitional Knowing,

And,

Asfor whether online discussion is appropriatefor all subjects, I guess it

depends on the subject matter. I can't imagine much discussion happeningin a Statistics course, but then again I haven't taken one yet... we'll see!

(Participant 12, LL, Week 10)

The problem comes when the words "new knowledge" come up. For me,new knowledgeis as I have describedit above - it is new, unique, and hasnever been expressed by an individual before. To create new knowledge,one would have to be an expert, a genius, or an artist.That's why Iprefer to say that we shared our knowledge, and that we builtour ownindividual knowledge as well as our group knowledgein thiscourse - but no, I do not believe that we came up with anything unique.(Participant 12, LL, Week 13)

52

Table 8

Breakdown ofInstances of Belief-Mode Epistemological Levels for Participant 12 per

Weekwithin the Learning Logs and Weekly Notes

Learning Log Distribution Weekly Note Distribution

Transitional Independent Contextual Transitional Independent ContextualWeek Knowing Knowing Knowing Week Knowing Knowing Knowing

1 12 2 1 2 13 1 3 14 45 1 56 67 1 7 18 1 89 1 9 210 1 10 111 1 1112 1 12 113 2 13 3

Design-Mode Analysis

Again with this participant, the focus was not idea improvement. For example, in

a discussion of gender differences in computer mediated communicationsshestated,

I remembera graduate class years ago where one student carefully kepttrack ofhow much timefemale students talked, and how much time malestudents talked. The male students talked 5 times as much as thefemalestudents, even though there were only 5 males and 20females. The thingis, after she reported herfindings, the class atmosphere changed. Themen, who were ourfriends, felt guilty, and the womenfelt guilty about

being the oppressed ones, whereformerly there had been no oppressed. Is

it better to recognize these things? At what cost? Asfar as I could see thewomendidn’t talk any morethan they did before the observation was

made, infact, no one wanted to talk because everyonefelt guilty. Therewas a rift between us where there hadn’t been one before. The men had

53

become the enemy somehow and we missedthe oldfriendly classroom.(Participant 12, LL, Week 4)

There is recognition of problems caused by the theory however, no suggestions

for advancesrelated to that field were made. Theissue of the concepts of knowledge

building and “new knowledge” wereraised,

The kind ofknowledge building that Scardamalia describes seems a bit

advancedfor our intro level class. I think that this kind ofknowledgebuilding - creation and advancementofknowledge - would occur morereadily once we know more about the issues involved. Ifeel like a

beginner, this is my second online course at OISE and although I amlearning a lot, I don'tfeel like I could contribute muchto the creation of"new" knowledge... The kind ofknowledge creation that goes on in

cutting-edge research and developmentteamsis a different kind ofknowledge creation. There, all participants are experts. An idea can be

tossed about in an objective and constructive way. By dissecting it as a

team, development teams can and do come up with new ideas, new ways of

doing things - create "new" knowledge.(Participant 12, LL, Week 13)

However,the ability to improve or create new knowledge wasrelegated to experts. No

alternative definitions of “new knowledge” or what it means to be an expert were

explored. While this participant did write some notes where she discussed what ideas

failed to show, her mindset, as described in the quote above, prevented her from going

any further and making suggestions as to how to improvethe ideas.

Survey Analysis

Participant 12 responded at the Independent/Contextual level in the pre-survey

and showed movement towards the Contextual level in 4 questions in the post-survey.

Note Analysis

This participant seemed to be all over the belief-mode epistemological map,as

seen in Table 8. There were Contextual Knowing examples, where reasoningandcritical

analysis are important,

54

Students must be taught how to evaluate websites and others’

commentaries. There is a lot ofcrap out there as Clare noted. One ofthemost important skills we can help our students develop is the ability to

critically analyze what theyfind on the net. Whois the writer, what are hisor her biases, what are their qualificationsfor writing this article? Bit bybit, reading this article and that, having group discussions ofwhat theyhavefound, students can educate themselves in a brave new way.(Participant 12, Views, Week 2)

The preceding excerpt showsthat the participant realized the importanceofcritically

analysing the web sites to determine their merits; specifically that not all views are

equally valid. But then, the same note suggests that such judgments werenot valid and

stated that “everything is relative”. This is a good example of Independent Knowing,

Imagine the revolution that could take place in schools. No more

textbooks! Gofind outfor yourselfwhat happened, what is happening. Nomore standardized tests because everything is relative (well, the socialsciences anyway). The information is there, the computers are there, areteachers going to use this new power? Are students? (Participant 12,Views, Week 2)

Analysis ofParticipant 7

This participant had not experienced a fully online course as per Table 6.

Biographical data

Age 31-35; Female

Q.Briefly describe your experience with computers, online learning, and

collaboration:

A. “My experience with computers is improving on a daily basis and I am

learning something new everyday. This is myfirst on-line course and I lookforward to

collaborating and learningfrom everyone.” (ID 7, Survey, Pre)

Self-introductionfrom Database

Iam very excited to be starting myfirst on-line course through OISE! I amlookingforwardto participating in a course that allows me to not onlylearn but to also enjoy the comforts ofhome.I have been teachingfor 8 years and currently Iam a Grade 6 teacherat

55

Forest Manor School (TDSB). This year is the veryfirst year that lam

teaching computers to my students. I was apprehensiveatfirst about thisnew venture, but now with the supportofstaff, I have learneda lot.Although I have been using a computerfor quite some time, I still considermyselfa novice. Therefore, I hope that by taking this course my

knowledge, as well as my confidence level, improves and expands.

This is my 7th course towards my MA in education. I have been pursuingmy Masters on a part-time basis and I hope tofinish by Dec. 2003.

Iam a mother oftwo daughters ages 5 1/2 and 3 years. In addition to

spending as much time with my daughters as possible, I also enjoyworking out at the gym, reading, watching movies and spending time withfamily andfriends.I lookforward to working and learningfrom all ofyou! (Participant7,Views, Intro)

Participation and Interaction Patterns

This participant created a large numberofnotes (120, avg = 86), linked the forth-

greatest numberofnotes (89%, avg=78%), read the largest numberofnotes of any

participant (93%, avg = 77%), but made only 22 revisions to notes (avg 78). Participant

7 displayed excellent knowledge building characteristics relative to others in this data set.

Learning Log Analysis

Belief-Mode Analysis

There wasa slight increase in the epistemological levels found from the beginning

of the course to the end. There seemed to be a breakthroughin thinking that occurred in

week 7 as seen in Table 9.

56

Table 9

Breakdown ofInstances of Epistemological Levels for Participant 7 per Week within the

Learning Logs and Weekly Notes

Learning Log Distribution Weekly Note Distribution

Absolute Transitional Independent Contextual Transitional Independent ContextualWeekKnowing Knowing Knowing Knowing Knowing Knowing Knowing

—het

DDDOPOIAKNAWN 1

1 1 2 21 2 4

1 2

2 1 1

2 2 4

2 2

1

1 1 1

1 1

The following example, from week 7, wasrated at the Contextual Knowing level

as the participant wasself-evaluating and also looked for and valued the input from peers,

I went out on a limb this week and shared my experiences with

constructivist teaching methods and said how I had incorporatedthismethod in myprimary classroom but had not used in myjunior classroom.I expected a responseto this and I gotit! It's not easy as a teacher to say

that you are not using an effective teaching strategy in your classroom

when you know thatit can work. However, Ifelt that it was important toillustrate howjunior teachers don't use the constructivist method as oftenas primary teachers. I didfeel a little "exposed"!

This admission was goodfor me because I realized that I need to relook at

my teaching strategies. I need to go back to using more constructivistapproaches whentackling the curriculum with my students. (Participant7,LL, Week 7)

Another passage from the same week wasrated at the Independent Knowinglevel

because she seemed to be waiting for direction from others and notself-directing as

would be the case at the contextuallevel,

57

I generally like to contribute to discussions after others have begun to

share their thoughts. I like to see where the conversation is going to go. As

well, I reflect on what others have said, and I determine what personal

experiences andpersonal opinions I have to add. (Participant 7, LL, Week

7)

Finally, this example,I felt, exemplified the Independent Knowinglevel in thatit

appreciated the varying perspectives but emphasizedthatall opinions are equally valid,

Question: Do you think that discussion is an important part ofthis course,

why?

This week's discussion questions took a different turn. An interesting oneto say the least! I thought that the idea ofhaving us respond to the

readings and questions as someone else was a goodone.Ifelt that theexperience made me read the papers in a different, more in-depth manner.Having to respond as someone else made me look at the readings with anew set ofeyes and felt that I came away having learned more about thereading in this way than in the past weeks. Once the personal response

began, Ifelt that a bit more relaxed because it was my opinion not

someoneelse's so I couldn't go wrong! (Participant 7, LL, Week 13)

Design-Mode Analysis

Participant 7 did a good job at identifying strengths and weaknesses in most

theories as shown in this excerpt,

When I read the statement:"The traditional classroom...is singularly ill suited to producinglifelonglearners: 'Right now, you've got 31 little workers who comeinto a room,sit in rows, follow instructionsfrom a boss, and can't talk to one another.

Schoolis the last time they'll ever see that model" (Law et al., 2001, citedin Corcoran 1993)

I wastotally taken aback! When wasthe last time that Law et al were ina

classroom? Obviously not recently! In my classroom at least, my studentsare not in rows (unless they're writing the EQAOtest and that's anotherissue!), my students interact with their peers and myselfon a daily basisbut there are also times when the do work independently (this helps withindividual assessment)... Law et al need to revisit their article, as well as,

as a real classroom and see that teachers do an amazingjob ofgivingtheir students the best that they can regardlessifthey have 20 or 35

students! (Participant 7, Views, Week 7)

58

However, except for the obvious statement about“revisiting their article”, no actual effort

to improve the idea was offered. In the next passage, she also identifies several problems

regarding student access to computers within her school. This time, the problem is just

too insurmountable. She seemed to accept the problem and doesnot feel any

responsibility for helping to produce solutions.

I agree with Warschauer & Meskill when they say: "Technologies which

support a cognitive approach to language learning are those which allow

learners maximum opportunity to be exposed to language in meaningfulcontext and to construct their own individual knowledge." The idea of

using the Web to connect my class with another sounds great! What agreat learning experience it would be ifmy students could interact withotherfrench students andparticipate in activities on-line! I think that ifI

were to integrate the use ofcomputer activities into myprogram it wouldwin over my reluctant learners. However, the problem I would have isaccessibility. I only have access to the computer lab once in a 6-day cycleandthis time is onlyfor my homeroom class. In my classroom, I only have

2 computers! I don't know how I would be able to do what the authorshave suggested. Accessibility poses such a huge stumbling block! .

(Participant 7, Views, Week 8)

Survey Analysis

There was some movement towards higher epistemological levels (from

Transitional to Independent) in about 8 of the survey questions. The other questions

remained fairly constant. Of particular note was the changein preferred problems. This

participant moved from preferring textbook problems to problems proposed by the

instructor and peers. In addition summarization of important ideas at the end ofthe

discussion moved from just the instructor to include students and the instructor.

Note Analysis

See Table 9 for a breakdown offrequencies. This participant showed a strong

tendency toward self-assessment in many notes, a trait of Contextual Knowing,as this

example showed,

I always am intrigued to learn more about how to become a better

59

educator. The information presented in Volman & Eck's article has helpedme as an educator evaluate my methodofteaching and has helped mepositively critique my lessons. I believe that educators need to do that as

often as possible in orderfor professional growth to occur. (Participant7,Views, Week 4)

Here again,is the need for feedback. This showsthe value of others’

perspectives, and strong commitment for improvementcharacteristic of the Contextual

Knowinglevel and of constructed knowers (Belenkyet al., 1986),

I think thatit's so important as educators that we listen to our students and

get continualfeedbackfrom them.It is only through this process that welearn, change and grow in our profession. (Participant 7, Views, Week 12)

This Contextual Knowing excerpt exemplified howthis participant’s view of

valid sources of knowledge included grade 6 students,

My students are asked to problem-solve, ask questions and respond toissues with supportfrom the teacher and their peers. They are respectedand viewed as equals. My students also have the benefit ofinteracting

with students that are representative ofour true multicultural society. I do

present instructionsfor activities but I certainly do not consider myselfthe

boss but rather afacilitator andpresenter offacts! (Participant 7, Views,Week 4)

Participant 25 was extremely supportive to other students throughoutthe

database. Comments ranged from ‘pat on the back’ type,

Participant 10,

Your point about improved writing makes total sense! You're right that weare continually looking at how to say what we want clearly and succinctlyin our writing all the time! (Participant 7, Views, Week 12)

To in-depth supportive feedback,

Great activity!Participant 10,There's something to be said about using a child's imagination and drama

together to experience a variety ofdifferent situations. The activity you

presented is an excellent example ofthat!T agree with you that simulations do not allow children to imagine a lot ofdifferent circumstances but wouldn'tyou agree that the use of "real-life"

60

simulations can allow people to experience situations that they wouldnever have had the chance to do?

Ijust think that simulations can sometimes be a great experience that can

enhance the learningprocess in a classroom. I'm not saying though thatithas to be the only one but it can sure be an interesting one! (Participant 7,

Views, Week 10)

Feedback to others, and helping guide others to follow their learning journey are

dominanttraits of constructed knowers (Belenkyet al., 1986) which correspondsto the

Contextual Knowing level. Overall, this participant’s notes were very well written. It

wasevident that entries were carefully thought out and constructed.

Analysis ofParticipant 6

This participant was considered to be a new online learner in online KB as per

Table 6.

Biographical data

Age 26-30; Male

Q. Briefly describe your experience with computers, online learning, and

collaboration.

A. “This will be myfirst online course at the masters level although I have done

one correspondence course using the computerfor my additional teaching qualifications.

Iam comfortable using computers and the world wide web/internet.” (Participant6,Survey, Pre)

Self-introductionfrom Database

Hello allfrom sunny South Africa. I'm in the last week ofmy holiday, (just

finished climbing Table Mountain infact) and mustsay that sitting in this

smokeylittle internet cafe is not my idea offun at the moment! Soforgive

me ifI keep this short. Next week I'll be back in Vienna, Austria and willhave much more time to devote to my computer and myfirst Masters ofEd. course.

I'm currently in myfourth year ofinternational teaching. Infact, I startedin Singapore with Nancy (Hi Nancy, small world!!!) and then went toCairo, and am now in Vienna. In all the schools I've taught at integratingcomputers into the curriculum has been a top priority and I've always

tried to incorporate them as much as possible, though not always as wellas I'd wantedto. I'm lookingforwardto exploring the research thatis out

61

there with regards to using computers effectively in the classroomespecially with ESL students and students with special needs.

Asfor myfree time, when I'm notsleeping, I enjoy skiing, hiking, baking

and watching Trading Spaces on BBC (one ofour three English channels).Cheers and enjoy the rest ofyour week!” (Participant 6, Views, Intro)

Participation and Interaction Patterns

This participant created slightly below the average numberofnotes (74, avg =

86), linked an average numberofnotes (77%, avg=78%), read slightly below the average

numberofnotes (74%, avg = 77%), and made 46revisionsto notes (avg 78). Participant

6 displayed average knowledge building characteristics for this database.

Learning Log Analysis

Belief-Mode Analysis

This Participant used the provided reflection questions from week 7 to 13 (except

week 9). See Table 10 for the breakdown of frequencies. Here wasa classic example for

Transitional Knowing, where the knowledge of some domainsis thought to be absolute

while in other domainsit is consideredrelative,

Sometimesit would be nice ifthere were definite answers to what we werediscussing, but ifthat were the case, there would be no discussion and notnearly as thought provoking!! (Participant 6, LL, Week 7)

In this example of Independent Knowing,the fear of being let loose by the

instructor wasillustrated,

Leading the discussion this week was certainly a new experience. Nothaving someoneto discuss the questions with before posting them, notknowing whatdirection to take the discussion in, which questions would

be relevantto all, and which would generate the most discussion, was alittle worryingfor me. (Participant 6, LL, Week 7)

In week 12, the participant went little further and talked about self-assessmentin this

Contextual Knowing example, (Contextual Knowing is where learners take responsibility

62

for assessing their learning),

In terms ofthinking about online pedagogy with regards to my teaching,

this course has reinforced that students need to be involved in their own

learning to becomelife long learners. (Participant 6, LL, Week 12)

This point was also backed up by a corresponding change in epistemology as measured

using the pre and post survey.

Table 10

Breakdown ofInstances of Epistemological Levels for Participant 6 per Week Within the

Learning Logs and Weekly Notes

Learning Log Distribution Weekly Note Distribution

Absolute Transitional Independent Contextual Absolute Transitional Independent ContextualWeek Knowing Knowing Knowing Knowing Knowing Knowing Knowing Knowing

1

2 1 1 2 2 l

3 1 2 1 1 1

4 1 1

5 2 l 2

6 1 2

7 1 2

8 2 l

9 1 1

10 1 1 1 2

11 1 2

12 2 1

13 1 1

Design-Mode Analysis

In a similar mannerto the others in the pilot research, participant 6 does not seem

to feel that it is his job to go any further than understanding theories and ideas from

various perspectives,

Theoretical issues, concepts, and ideas require discussions where differentpoints ofview, understanding andreflection can occur. Through a gooddiscussion a person can becomesless ignorant and can then build more

63

insightful perceptions. (Participant 6, LL, Week 10)

However, unlike mostofthe other participants, he does venture into the possibility of

using an instructional strategy — Learning Circles — in his teaching. No mention is made

of how this concept might be improved or modified to better meet his students’ needs.

This week gave us a chance to look at three novel ways ofusing theinternet in relevant and authentic ways and the discussions lead to a

greater understanding ofeach ofthese programs. Ofthe three, I can seemyselftrying out Learning Circles. This seems to be a program wherewith the right approach, you canreally motivate students and get theminterested in more thanjust the material they are meant to learn. In mymind, one can’t really go wrong with the concept oflearning by teaching.I think students would really enjoy this type ofonline learning/teaching.(Participant 6, LL, Week 9)

The next excerpt seemsto be the closest to a design-mode statement found

amongst participants,

How can technologyfacilitate this type ofCollaborative Learning?

Is there a type ofsoftware / learning environment or web-site that canfacilitate this learning? Are there any challenges / shortcomings when

using technology tofacilitate this approach?Is there anyway that one canovercomethe challenges / shortcomingsofthis approach?Cooperative Learning

-I have had students teach the entire class how to create a website,for example;

-Stronger students could team up with ones whoarelessexperienced with computers;

-Our on-line courses are a good example ofcooperative learningusing technology.

-Group research projects on different topics using a variety ofweb

sites to gather information. Students do their research separately ortogether, then collaborate theirfindings and decide on what's importantand how to presentit. I have the students assign themselves group member

roles before we begin the projects. -A shortcoming ofour web-basedcourses is that we don’t get the person-to-person contact that weotherwise would be experiencing in a classroom situation. (Participant 6,Views, Week 3)

He asks somepertinent questions and challengesthe role of technologyin collaborative

learning. He then goes on to provide examples and suggestions of how collaborative

learning looks, or might look. But the notion of a dynamic interplay between these

64

designs, results, and next-generation possibilities is missing.

Survey Analysis

This participant showed slight movementin 3 questions. Other questions seem to

be answered from an Independent to Contextual Knowing viewpoint.

Note Analysis

See Table 10 for a breakdown offrequencies. This first example showed the

black and white thinking at the Absolute Knowinglevel,

As so many havestated sofar, it’s simply a tool. Ifthey are used atappropriate times and in appropriate ways, computers can diversify and

enhance our teaching andperhaps motivate afew morechildren in waysthat other methods cannot. (Participant 6, Views, Week 2)

This view changed in week 3 to a Transitional view, where both absolute andrelative

knowledge domainsexist,

These edutainment programs reinforce concepts discussed. It doesn’t

usuallyfoster “real” questions in which there really isn't an answer. Dueto this, there is very little creativity expressed when responding to aquestion that has a definitive answer. (Participant 6, Views, Week 2)

Finally, by weeks 10 and 11 this participant's entries had reached the Contextuallevel,

whereself-directed learning is the norm,

Participant 8 brings up a great point when he stated “Perhapsifall oftheconstructivism I have been reading aboutin thisforum takes hold,

graduate students will be more comfortable with less structure!” I can’tagree more. Although weall have different learning styles, as students

become more responsiblefor their own learning at a younger age, theywill be able to more readily and independently seek the information theyneedfrom whatis provided. Andifthey don’tfind it at one website, theywill know how try at another. When constructivist teaching will becomethe norm is another question. (Participant 6, Views, Week 11)

65

This ends the discussion of specific results that relate to individuals. This analysis

was useful because it showed that variability in epistemological levels and movement

could indeed be seen within a 13-week course. Further, the above analysis showed the

need for an epistemological modelthat could be moreeasily operationalised. The next

section will discuss how the individual analysis influenced revisions to the main research

methodsand analysis. Specifically, the framework, the pre and post-survey, and the

learning log questions are discussed.

Other Results that Affected the Main Study

Framework

Thefirst analysis looked at the pre-course survey data. However, while using

Baxter-Magolda's framework there were so many nuancesand conflicting examples

where participants were at multiple levels that it made classification into one overall

category extremely difficult and speculative.

Hence, analysis was altered to look for change in responses over time. The pre-

survey wasre-read, as were the first 2-3 weeks of the conference and learning logs.

Similarly, the post-survey and the last 2-3 weeks of the conference and learning logs

were re-read together. Participants’ responses were then categorized into belief-mode

epistemological levels. There appeared to be a positive relationship between

epistemological level and the basic participation and interaction measures as produced by

the Analytical Took Kit, suggesting that it might be possible to increase likelihood of

knowledge building through epistemological development. It seemed that those

participants with responsesrated at a Contextual or Independent Knowledgelevel also

demonstrated higher participation measures(i.e. read and wrote more notes, revised their

own notes moreoften etc.). However, the ambiguities involved in categorizing

participants madethis conclusiontentative. Applying the Baxter-Magolda model proved

too problematic to form any clear conclusionsusing the pilot framework. As a result,

additional literature that dealt with cognitive developmentand other epistemological

models were explored which lead to the updated framework described in Chapter1.

66

Pre and Post-Survey

Manyidiosyncrasies were discovered through using an online methodologyfor

survey delivery. However, for this research, these discoveries were to prove partially for

naught. In consultation with my thesis committee, it was decided that the data gathered

through close-ended questions was too ambiguous. The committee members basedthis

conclusion on the uncertainty of classification plus their own detailed review. The

methodology was changed for the main research. A pre- and post-test essay was

designed based on four questions. Details of this revised method are found in Chapter3.

Learning Logs

The final method that was changed from the pilot to the main study concerned the

learning logs. First, it was questionable whetherthe reflection questions used for the

learning logs helped to identify belief-mode epistemological levels. The epistemological

model chosen did makeit hard to differentiate between levels, though, morelikely it was

the actual use of the questions by participants. Only 2 participants (n=18) used the

learning log questions until prompted during weeks 7 and 13. In week7, participants

werespecifically asked to use the questions and this prompted 10 participants to do so.

In fact, two participants did not use the questionsat all during the course, which seriously

affected their value for analysis. While these questions were designed to elicit deep

reflection about epistemological issues, their lack of use resulted in a majority of logs that

concentrated on course content rather than also including metacognitive reflection.

It was decided that several changes would be madefor the main study. First, to

increase participant use of the questions they were made mandatory. Additional

comments and reflection were always encouraged, but use of the questions wasthe basis

of each week's learning log entry. Second,to increase participants’ motivation towards

journaling their thoughts, logs were graded through self-assessment using a group-

negotiated rubric. Third, learning log questions were produced on a weekly basis. These

questions were designed to producereflective responses andto take into consideration the

content of the discussions from each week. As such these weekly questions were

relevant to the current discussions. Finally, for the pilot the learning log questions were

67

based on all twelve of Scardamalia’s Knowledge Building principles. This proved too

ambitious in a short 13-week course. It was decided to concentrate on fewerprinciples

for the main study; ones that seemed to correlate well with Kuhn’s and Moshman’s

theories.

Pilot Conclusions

This pilot research was invaluable for honing the framework, data collection

methods, and analysis techniques for the main study. The notion ofa pilot study alsofit

extremely well with the overall design research philosophy ofiterative design phasesthat

was used for both studies. Additionally, the pilot results indicated the operational

weaknesses with the theories that had been choseninitially and resulted in the choice of

more effective and developed theories and thus a stronger overall framework.

Three other important modifications included (a) the focusing of learning log

questions and the paring down oftheir number,(b) the changingofthe learning log

questions to mandatory, and (c) the refining of pre- and post-survey questions and format.

However, despite the above refinements, new questions arose. It seemed that using “the

numberoffully online courses taken” as a way of grouping might not have been

appropriate for this study. A better way might be to consider the epistemological levels

and then comparethis to the knowledge building principles and levels of participation

within the database. As such,it was difficult to define a “new online learner”. Should it

be someone whowasnewto online learning or someone who wasnewto the masters’

level (and the epistemological developmentthat occursat that level)? Should more

weight be placed on the Knowledge Forum/Online componentand how does the number

of masters’ courses completed affect participation.

CHAPTER 3: METHOD

This research was exploratory in nature and began with a group ofdistance

learners and their interactions within an online class. The class used Knowledge Forum

software which supports an approach to pedagogy that encourages students to take charge

of their own knowledge building. The software includes multiple supports to the

cognitive and social processes of knowledge building. The aim ofthis research wasto

identify relationships between epistemological levels, participation patterns,

metacognitive activity, and forms of reasoning, and to relate them to advancesin

knowledge building. There were two main research questions: 1) Are changesin belief-

or design-mode epistemology evident over the duration of the courses? That is, do

learners change their assumptions aboutthe nature, certainty, and justification of

knowledgeand their understanding of ideas as continually improvable? 2) If there are

signs of epistemological change within individuals, how do these changes manifest

themselvesin the interactions amongst learners in the online learning conference as

measured through changesin the reading and responding patterns of users? Is there a

changein the reading/responding patterns of users over the course?

This chapter starts with a discussion andliterature review of the design research.

This is followed by a description of the participants, the course under study, and the

research questions. Finally, a description and review ofliterature for the data sources

complete this chapter. The data sources were:

e Web-based survey. The web-based survey gathered ethical consent and

biographical data;

e Participant reflection via learning logs. Learning logs were usedto investigate

metacognitive activity and self-identified examples of epistemological

change;

e Pre and post-essay on theories of learning. The theories of learning essays

were used to assess epistemological levels, and the use ofjustification; and,

e Participation and Interaction Patterns. An Analytic Tool Kit underlying

Knowledge Forum automatically records user contribution and interaction

68

69

e patterns. This data was used to measure Overall Participation Patterns, Active

and Interactive Ratings, Collaborative Activity, Edit Rating, and to provide a

week-by-weekparticipation analysis.

These complementary data sources were chosen based on the exploratory nature

of the research questions and developmental nature of what was being studied.

Conference notes were also used during the study to develop the reflection questions used

for the learning logs and they formed the bulk of the notes used for the quantitative

analysis. To develop reflection questions, the main themesderived from the week’s

discussions were used as the context and epistemological models as the framework.

Literature Review for Methods

Design Research

Design research (also known as design experimentation) in education evolved

from traditional psychological experimentation as a response to the difficulty of

controlling variables in real life settings, such as schools, in contrast to traditional

laboratory settings (Collins, 1999). Design research takes into account the complexity of

real life settings in order to gain a deeper understanding of phenomenain context. This

often takes the form ofiterative cycles of design, intervention, analysis and redesign.

Design research is consistent with constructivist approaches and, as such, is concerned

with multiple realities and interaction with the participants (Glesne, 1999). Lincoln &

Guba(2000, see Table 6.4) would furtherclassify this research method as bordering on

participatory in that knowledgeis seen as residing in the participants and “embedded in

communities of practice” (p.170).

Design research in education brings together the reformer from the classroom and

the researcher from the lab. Wherein the past these two worked apart, now close

collaboration and innovative methodologies have emerged (University of California

Berkeley, 1997). Neumann,Pallas, & Perterson (1999)state, “Unlike the disciplines,

educationis a field centered on professional practice, whereas education research derives

from and contributes to both theory and practice” (p. 247). This point is further

emphasized by Collins (1988) who believes that educational research needs to become

70

more akin to a design science like “aeronauticsorartificial intelligence” (p. 15).

Design research’s closest neighbour would be action research. Similar to action

research (Kember, 2000), design research focuses on concrete and practical issues or

problems of immediate concern to particular groups or communities. Further,it is

conductedin naturally occurring settings andis iterative in nature. With action research,

the researcher is normally the classroom teacher. In design research, the researcheris

often an outsider working in very close partnership with the teacher and perhaps other

experts. The major difference between these two types of research is that, while results

from action research are often taken and fed back into practice and sometimes into new

research studies, with design research there is reciprocity between design, theory, and

implementation, with advancesofone sort contributing continuously in advancesto the

others (Scardamalia, 2004).

‘Design studies have been characterized, with varying emphasis depending on the

study, as iterative, process focused, interventionist, collaborative, multileveled,utility

oriented, and theory driven” (Shavelson, Phillips, Towne, & Feuer, 2003, p. 26). While

not defined by any particular methodology, sustained innovation - the goal of design

research - has several characteristics (Bereiter, 2002). First, the researcher and the

designers mustbe in close collaboration. “Design researchis part of the design process;

if separated from it, it ceases to be design research” (Bereiter, 2002, p 326). Second,in

stark contrast to positivist methods, design researchis ‘inherently interventionist’. To

affect the design, one must be diligent in striving for improvement. Third, problem

solving is a main focus. Finally, “Design research is guided by some vision ofas-yet-

unrealized possibilities and is characterized by emergent goals — that is, goals that arise

and evolve in the course of cycles of design and research”(Bereiter, 2002, p 326).

Ann Brown (1992), a pioneer of design research in education, believed that the

complexities of the classroom and the inadequacies of the experimental method could not

satisfactorily explain classroom actions. Asa trained positivist psychologist, she

reflected upon the changesin her thinking and methods. Shestarted her career

performing decontextualized and content-poor experiments. Later, her studies became

rich in content and context for participants. This shift inherently led to more of her

studies occurring in the messy and complex classroom environment and fewerin the

71

laboratory. Brown considered that far too many interwovenvariables exist in a real

classroom to be controllable.

Inevitably, problemsarise from a majorshift in research philosophy. Such is the

case with design research whichisstill an emergent paradigm. Much debate about

validity remains as researchers and policymakerstry to apply the tenets of traditional

research paradigmsto design research (Neumann etal., 1999; Shavelsonet al., 2003; The

Design-Based Research Collective, 2003). However, design research is slowly starting to

makeits way into the main stream of educational research philosophies, as shown by the

Educational Researcher devoting a full edition to the topic in January/February 2003.

What excited myinterest was its participatory, collaborative nature and its goal of

sustainable innovation. Both were important factors in my research. Bereiter (2002) has

emphasized the need to change how wedoresearch in education. He believed that most

innovations in educational design have failed, not because they were poorideas, but

because their success was not properly measured. Traditional research paradigmsdo not

lead to innovation; instead they tend to prevent it. However, design research has as a

goal, perpetual improvement. This allowsfor real-time,iterative design improvements

and a fighting chance to compete with the status quo. Bereiter advocated the need to

have patience when judging innovation. There is often strong resistance to change the

existing (traditional) process or method due to years of refinement (For details see T. S.

Kuhn, 1970). So for example, didactic pedagogies almost always seem better than a new

and innovative pedagogy when measuredor researched usingtraditional methodologies

(such as standardized exams).

The design research aspectof this dissertation study concerned the pedagogic

iterations of the learning logs. Numerousaspects of the learning logs changedas result

of the pilot research or were designed and redesigned “on the fly” as my understanding of

the phenomena changed. However, the four following changes were the main design

researchiterations:

1. During the pilot study learning logs were graded by the instructor whereas

for the main research learning logs were self-assessed by the students using a rubric. The

instructor initially built the rubric but all participants were able to influenceits design

during the course;

72

2. The learning log questions started out as non-mandatory but were made

mandatory for the main research;

3. Forthe pilot, all learning log questions were presented at once at the

beginning of the semester. Howeverfor the main research, the questions were designed

and presented weekly based uponthe context of that week’s discussion. The questions

were also designed toelicit responses based on the research framework; and,

4, Thelast iteration happened nearthe end of the main research. In

conjunction with the instructor, participants were encouragedto design their own learning

log questionsfor the last three weeks.

Positivists might argue that this type of research is too subjective and will bias

results. In response, I leave them to ponder Perry’s words,

We take it that the act ofobservation always influences the events

observed. Ifthis proposition holdsfor the physical sciences (Kuhn, 1962;Heisenburg, 1952; Polanyi, 1958), it surely holdsfor the social sciences

(Erikson, 1964; Kaplan, 1964). In previous studies analogous to our own,somestartling effects have been documented. In one (Heath, R., 1964),the investigator “followed” a representative sample of31 college

participants through theirfour years, meeting with them individually andin groups every week or so. Ofthese participants, 13 graduated with

honors, as did only 5 oftheir matched controls. That the study had effectsin addition to that upon academic performance seems probable. (PerryJr., 1970, p. 29)

Participant Reflection

Journaling is a forum for making explicit the internal dialogue within our minds.

It involves self-analysis and reflection on events or discussions. This reflection can be

structured or free flowing, individual or shared, and can often deal with metacognition, a

goal in this research. Andrusyszyn and Davie (1997) pointed outthat:

Throughthis deliberate cognitive activity, learners have the potential tointentionally connect thoughts, feelings, and experiences related to thelearning activity in which they are engaged. This connection is

particularly important in a CC [computer-conference] environment,wheresifting through the volume ofinformation generated may preclude

73

meaningful reflection and construction ofknowledge. (n.p.)

Connell (2000) classified three mediumsfor reflection: learning diaries, logs, and

journals. Diaries were free flowing and contained emotion, logs were used moreto

reflect about curriculum content, whereas journals were used to record events and ideas

for future work. She examined two graduate courses in which learning logs were used.

Despite the small numberofparticipants, she found that students’ overall experiences

were positive. In addition students reported that learning logs promoted a deeperlevel of

understanding of both content and themselves. This deeper level has also been observed

in journals used in Computer Mediated Communication (CMC) environments (Heflich &

Putney, 2001). This was important in my research, as deep understandingis one of the

objectives of knowledge building.

“Journals and diaries have a long history as a meansofself-expression. Several

themesprevalent in adult learning - coming to voice, developing the capacity forcritical

reflection, and making meaning- are reflected in the way journals can be usedin adult

education” (Kerka, 1996, n.p.). Kerka believed that journaling could “provide tangible

evidence of mental processes” (n.p.). They aided in the ‘making of meaning’ through

allowing connection of past and current experiences. Because journalingis a

combination of writing and reflection, it is a tool that is ideally suited to online learning

environments.

There have been a numberof successful research studies using online journaling

in various forms (Admiraalet al., 1999; Andrusyszyn & Davie, 1997; Graybeal, 1987).

Graybeal (1987) investigated the use of team journals and reported that “team journals

makepossible an 'exchange' of energy andideasthatis virtually impossible in a journal

written ostensibly for the student herself or himself”. Graybeal observed that students

began to makesense of themselves and the world around them through a co-operative

shared venture. Andrusyszyn & Davie (1997) noted that individual dialogue with the

instructor on a regular basis provided reinforcement, validation, and support as well as

the stimulation of prompting and probing. In this case, the instructor, rather than a team

of participants, served as an audience to receive and respondto participants’ thoughts and

impressions. In this research, participants’ learning logs individual but accessibletoall

within the conference, and students were encouraged to read and commenton each

74

other’s logs in a constructive way.

Whatdoesthe researchtell us aboutthis journaling? Andrusyszyn & Davie

(1997) studied five graduate students and one instructor in a study about writing online

interactive journals. Most students were inexperienced using the computer as an

educational medium. The data collected contained 161 journal entries by students and 42

responses from the instructor. Students found that the journals were useful to learn about

oneself and to reflect upon deeper, critical issues. The researchers stressed that the

instructor should be well versed using this strategy and that a high level of trust was

required. I believe that the instructor involved with my research was highly skilled in

online teaching and that sheinstilled great trust from participants.

Heflich & Putney (2001) studied a group of 22 student teachers over an 11-week

practicum seminarandfelt that “the knowledge that cohort members had of one another

added to the depth of discussion that occurred in this seminar” (p. 5). However,

“Participant’s knowledge of one another, though,hadlittle effect on the ideas expressed

or the thought that was apparent within their responses”(p. 5). In my research,

participants were likely to havelittle prior personal knowledge of one another unless they

had been in previous classes together, and even then, as they were mostly off campus

students, this knowledge wasfurther limited.

Admiraal, Veen, Korthagen, Lockhorst, Wubbels, Hernandez,et al, (1999)

examined student-teacher case studies spread over four countries (Netherlands, Spain,

United Kingdom, and Norway). Groups were madeupof2, 4, 7, and 30 volunteer

student teachers and 1, 1, 1, and 4 supervisors respectively. The lengths of the studies

varied in time but were generally conducted over 3-5 months. Again deeper, more

thoughtful learning was reported. However, students tended to concentrate on perceived

problemsrather than successes. Further, messages contained simple summariesofdaily

events “without showinga rational analysis of their actions and thoughts whichis

commonin reflection” (p. 85). Hence, the authors recommend providing more guidance

and coaching in journaling. Accordingly, I provided scaffolding questions that

participants were askedto use in their learning logs.

To be successful in an online environment, journaling must be adequately planned

and monitored. The studies reviewed provide some guidance on whatinstructors could

75

do to help make an online journaling experience a productively educational one.

Andrusyszyn & Davie (1997) identified the important factors for success as being: 1) the

expertise of the instructor with the strategy; 2) the degree of trust and rapport that exists

betweeninstructor and students; and 3) the importance of grading the process, as opposed

to the substance, of the entries. Admiraalet al. (1999) similarly recommended guidance

and coaching from the instructor as important factors for the success of the journaling

pedagogyin an online environment.

While there are many formsofreflection, important to this research was

metacognitive reflection. Hence both the learning logs and theories of learning essays

wereusedto identify examples of metacognition. As was discussed in Chapter1,

metacognition hasclose ties with both dialectical reasoning and epistemic agency. These

three concepts came together to show participant developmentthrough monitoring

changesin participants’ levels of reasoning and metacognitive activity.

Moshman’s Levels ofJustification

In simple terms, dialectical reasoning is recognizing that you have gapsor a lack

of coherence in your understanding through a process of active metacognition, and then

doing something (using epistemic agency and metacognition) to increase coherence and

understanding. Moshman (1998) highlighted the parallels between epistemological

development, reasoning, metacognition, and cognitive development. While the

developmentof inference and reflective processes can be considered as a natural

progression ofintellectual development, reasoning, especially dialectical reasoning,

requires a consciouseffort. Moshmanprovided an argument for development occurring

within, as well as between, types of reasoning. For instance, analogical inferences move

to self-conscious analogical reasoning, and in certain domains, to binding precedent and

finally, in the domain of law, to legal reasoning. With logical reasoning,“although

children routinely make inferences in accord with the rules of logic, only later in

developmentdo individuals increasingly think about such rules and understandtheir

epistemicrole in justifying connections among propositions” (p. 957). For law-based

reasoning, “available evidence is thus consistent with a general trend from the use of

implicit principles to the deliberate application of explicit principles (Moshman,1998).

76

One maythusposit a developmental trend from (a) undifferentiated law-based inferences

toward (b) rule-based and principled reasoning” (p. 960). Table 11 below shows

Moshman’stypes of reasoningin detail.

From an examination of Table 11, we can see that dialectical reasoning is

different from case-based and law-based reasoningin that it is used when there exist

conflicts between the latter two. Such cases are prevalent in education whereoften rules,

principles, and precedents seem inconsistent with experience and analogies. Hence

Moshman(1998)states, “dialectical reflection may be defined as a deliberate effort to

makeconceptual progress though active metacognition” (p.961). Now for suchreflection

to become reasoning, Moshman contends, “it must involve a deliberate effort to apply

somecriterion of progress. As already suggested, a common and important suchcriterion

is increasing coherence”(p. 961)

Table 11

Moshman’s (1998) Levels of Justification

Type of Description Sub-forms DescriptionReasoning

Case-based “Thinkingconstrained Analogical “a situation or issue is considered onby attention to concrete reasoning the basis of othersituations ormanifestations (cases) exemplars.” (p. 954)

that are deemed “Similarities and differences arerelevant to achieving a deliberately assessed and coordinated”justifiable cognitive (p. 954)

outcomein the case athand”(p. 954) Precedent-based Similar to analogical reasoning

reasoning however“fidelity to precedentis

considered mandatory; apparent

deviations from precedent require

specific justification.”’(p. 955)

Legal reasoning “Legal thinking may be defined as(special case; thinking aimed at determining what themainly specific law requires or forbids”(p. 955).to the domain ofLAW)

Law-based

Dialectical

Rule-basedreasoning

“thinking by thedeliberate application

of abstract laws that areconstrued by the

individual as justifying

his or her beliefs and/or

actions” (p. 956).

Principled

reasoning

Dialectical

reflection

“We may definedialectical thinking asthe deliberate

coordination ofinferences for the

purpose of making

cognitive progress.Such thinking may be

designated asdialectical reasoning to

the extent that it restson explicit knowledgeaboutcriteria forassessing suchprogress. Thus the

developmentofdialectical reasoning

involved increasingly

explicit knowledgeabout the nature of

cognitive developmentand increasinglydeliberate efforts to

further that process”(p.

961).

Argumentation

77

The comprehension,application,justification, and coordination of

logical, mathematical, social or moral

rules in justifying connections among

propositions.“Rules are algorithmsthat yield adeterminate answer”(p. 959).

Logical reasoning: The use ofthe rulesof logic in justifying connections

among propositions. This is a form ofdeductive reasoning.

“principles... are general guidelines

whoseapplication involved heuristicjudgments”(p. 959).“Principled reasoning derives from

commitmentto someset ofprinciples

on the basis of a general metacognitive

understanding about the nature and use

of principles” (p. 959).

“Deliberate efforts to achievecoherence by reconstructing one’sintuitions, and/or conceptions of

precedent”“ dialectical reflection may

be defined as a deliberate effort to

make conceptual progress through

active metacognition”(p. 961).

This also includes coherence-based

reasoning: “reasoning aimed at

achieving the temporal coherence of anarrative that unfolds across time, andreasoning aimed at achieving moreabstract forms of structural coherence”

(p. 961).

“Kuhn provides a picture of reasoningas a collaborative process in which

people formulate, communicate,

criticize, justify, and revise their

various ideas. Argumentation is

usefully construed as a process ofdialectical reasoning in which two or

more individuals coordinate multiple

cases and lawsin sharedeffort to

make conceptual progress”(p. 961).

78

Participants’ types of reasoningin their learning logs and theories of learning

essays were coded. The types found were then comparedto the types in Table 11 and

progress towards dialectical reasoning was determined.

Participants

Changesthat occurred in epistemological viewsof students participating in an

online course were explored. Students taking graduate level courses in the Computer

Applications Department at OISE/UT wereaskedto participate in the study. Twenty-two

students started the Constructivist Learning and Design course (description below), 6 did

not finish the course, and 1 student did not complete the participation consent form.

Hencethe participants consisted of 15 master and doctoral students (n=15). While I

lookedatall levels of relevant experience, I expected to see the greatest changes in

participants new to the knowledge building pedagogy, online learning, and graduate level

courses. Based on the significant amountof attention being paid to online learning in the

journals, it would seem that there is something different to learning online compared to

face to face. From my own experience,thereis certainly a difference in expectations and

professor involvement between undergraduate and graduate work. Further, as discussed

in chapter 1, knowledge buildingis a radically different pedagogy comparedto traditional

schooling. Notwithstanding, learning logs questions were designed to promote change in

all participants regardless of experience.

Students at the graduate level were chosen for this research because I considered

them to be fully capable of the type of deep self-reflection required by the methods used

in this research. Further, the instructor had a deep personalinterest in the improvement of

online learning environments and integration of technology into teacher education. In

addition, graduate students are expected to be at the leading edge of the research and

hence should be capable of design-mode thinking. However, as indicated in the

introduction, most school pedagogies providelittle experience in this mode. This

combination of graduate school expectations and new learning experience provided very

complex andinteresting data.

79

Course Description

The course chosen for this research was fully online. Members did not meet face-

to-face as part of the course requirements. Interactions took place mainly through a web-

based learning environment called Knowledge Forum. While there were most certainly

personal emails between participants, participants and the instructor, I did not ask to use

them in this research. I believed that there would be more than enoughdata to analyze

without emails and this proved to be the case. The Knowledge Forum environment

allowed for multiple views and threaded discussion. Indeed, this environment generated

a massive amount of data. The course wasalso chosen because the course instructor was

the sameas for the pilot research described in Chapter 2.

The course wasoffered for the first time and wasentitled Constructive Learning

& Design. Its description read,

It will begin with an examination ofthe theory and research that underlies

constructivist learning andits historical andphilosophical roots.

Concepts like situated cognition, distributed cognition and constructivist

learning theory will be examined. The educational applications that havedeveloped out ofthese ideas, like problem based learning, collaborativelearning and knowledge building will be explored in regards to how suchconcepts can inform and enhancethe design ofonline environments andmethods ofteaching. We will look at different learning environments,both research projects and applications currentin thefield that instantiatevarious elements ofthese ideas. (Brett, 2003)

Asthis course covered many ofthe theoretical notions that formed the basis for

this research, and was aimedat improving online teaching, I expected to see some

excellent reflections from the participants in this course and was not disappointed.

Data Sources and Collection Methods

Asa design research study continual changes and perceived improvements

occurred throughout. Many of these changes werereported as part of the pilot study

results in Chapter 2. For the main study Table 12 provides an overall view of when data

were collected, what those data were, how the data were collected and for what the data

80

were used. The overall structure consisted of data collection during an introductory,

graduate level course in the Curriculum, Teaching, and Learning Departmentat

OISE/UT. It seemed reasonable to assumethat participants in introductory courses

would likely be unfamiliar with both the programme and online Knowledge Building.

However, biographical data was collected to confirm participants’ background

knowledge and experiencein these areas.

The data collection period occurred in the Fall 2003 session and had the benefit of

improved versions of the instruments and analysis methods based onthe pilot. Again, the

data sources were: 1) Web-based survey; 2) Learning logs; 3) Pre and post-essay; and 4)

The analytical tool kit (ATK). This section describes the data sourcesin detail.

Table 12

Chronological Overview of Data Collection for Main Study

When Data Collection Whatused for

Collected Method

Sept 03 Permission to use data Online survey Ethics reviewSept 03 Biographical

Sept 03 Epistemological views Theories of Determineinitial level

Justification learning pre-essay of epistemologicalunderstanding

Measureofdialectical

reflectionSept-Dec03 Metacognition Learning Logs Measure ofdialectical

Justification reflectionSept- Dec 03 ATK measures Automatically Indication of

recorded by knowledge buildingdatabase software Participation and

interaction patternsDec 03 Post-epistemological Theories of Determinepost level of

views learning pre-essay belief and design modeepistemologicalunderstanding

Measureofdialecticalreflection and

knowledge buildingJun 05 Codedpassagesfor Inter-rater reliability

metacognition anddialectical reflection

81

Learning Logs

Learning Logs were special notes, within the online conference, in which

participants kept an ongoingreflective journal of thoughts. These thoughts could have

been about course content, or process, or they could have been metacognitive in nature.

The learning logs were one of the course assignments and wereself-assessed. The

participants added to them weekly, from weeks 2 - 13, during the course. The

instructions provided were:

This is an ongoingjournal ofwhat you have learned in the course. Youare expected to add one entry per week. It involves reflection on yourchanging understanding ofideas in the course particularly in relation to

ideasfrom other courses you are taking or have taken. The idea behind

the learning logs is to have a location to deliberately reflect and try to

integrate ideas into higher order syntheses. NOTE: The learning logs

should take theform ofa single note that you add to weekly, and theseshould be put in the Learning Logs View. Each entry should be about 1-3paragraphs. It is more about insight than length! (Brett, 2003)

Learning logs were analyzed for knowledge building, metacognition, and

epistemological content. It was hoped that epistemological developmentover the

duration of the course would be seen in participants writing. The idea of development

was discussed in Chapter 1. Further, the belief-mode epistemological models reviewed in

Chapter 1 demonstrated that, while participants are in one stage or another, they could

also be at an in-between stage of development. Hence,participants could displaytraits in

several stages simultaneously. In orderto better clarify epistemological levels, the use of

reflection questions was established. For this second iteration of the research, the use of

reflection questions was made mandatory, as otherwise students might fail to use them

adequately, as happened duringthe pilot study.

Each week,a different set of questions was posed for the participants’ learning

logs. They were designedto elicit epistemological, knowledge building, or

metacognitive reflection depending on the content and toneofthat particular week’s

discussion. Before being posted, questions were agreed upon byresearcher and

instructor. Design research encouragesthis type ofiterative tinkering.

Participants were asked to reflect upon and answerall questions provided. They were

82

also encouraged to continue afterwards with their own reflections. Each week, by Friday

night at the latest, an update to the note containing the questions was made. The

following are those questions:

1. Week 2:

a. What are your goals for this course?

b. How do you react when someonecriticizes or counters your posting?

2. Week 3:

a. Over the past two weeks we have been exposed to many different theories.

How do you judgethe validity of one theory over another?

b. How do you decide whento contribute to the discussion?

3. Week 4:

a. In light of this week’s discussion on the social nature of words and

meaning, what effect do you think this has on assessment (student

testing)?

b. Explain how you evaluate your own learning throughout a course.

4. Week 5:

a. After several week of theory we are now getting into more practical

issues. How do you derive coherence between theory and practice?

b. How is new knowledge created?

5. Week 6:

A slight change for this week: Reread your “Theory of Learning” (view

1.8). Pick and discuss one idea that has evolved.

6. Week 7:

a. I have noticed an increased use of “outside” references in many notes.

What doesthis represent with respect to your view on evidence?

b. Are some references “better evidence” than others?

7. Week 8:

a. Using the current form ofthe rubric, assess your learning logs. (You do

not need to put a mark, but perhaps highlight your strengths and areas for

improvement.)

83

b. Is there a dichotomyusing a collectively designed rubric for individual

assessment?

8. Week 9:

Assumingthat knowledge building is a worthwhile pedagogy, what

one thing would makethe greatest difference to its widespread use within our

society?

9. Week 10:

a. Reread your learning log. How,if at all, has this log helped you?

b. Could it help you more?

10. Week 11:

Please design your own question this week and makeit evident for

others.

11, Weeks 12 & 13:

You are on your own. Thanksfor letting me be part of your learning.

The last 3 weeks were designedto see if participants would continue to ask

themselves questions thus showinga level of epistemic agency. Further, the questions

asked by individual participants were analysed to determine their nature (content oriented

or metacognitive). All learning logs were analysed and coded using NVivo.

Pre andpost-essay

Based uponthe findings ofthe pilot study, the original survey format was changed to

a pre- and post-essay. These were designedto elicit baseline and endstate

epistemological views concerningassertions, reality, knowledge,critical thinking and

design-mode thinking. The four questions used were designedto help identify the level

of participants’ epistemological understanding using Kuhn’s model (D. Kuhnetal.,

2000). The questions, which were developed in conjunction with my supervisors, were:

1. What do you currently understand learning to be - for yourself as a learner and for

your students if you teach?

84

2. Why(on whatbasis) do you hold those views, both for yourself and for your

students? (If you are not a teacher think ofa situation where you have taught

somebody something.)

3. What role does knowledgeplay in learning?

4, Whatrole do others play in your learning (e.g. peers, teachers etc)?

The essays were given as part of one course assignment and were graded (12%). The

pre-essays were downloaded upon completion and savedto ensure that the original

thoughts were preserved, as participants were able to revise any of their notes at any time

throughout the course. The pre- and post-essays were also analysed using NVivo.

Participation and Interaction Patterns

Participation and interaction patterns were assessed through use of The Analytical

Tool Kit, a specially designed set of scripts underlying the Knowledge Forum database

that automatically records participants’ actions to allow detailed statistical analysis.

Individual and group measures were used and overall indications of collaboration were

calculated. The Analytical Tool Kit measures were built by the research group at

OISE/UTandhavebeen used extensively in their research (see www.ikit.org). Basic

participation and interaction measures were designedto assess the participants’

contributions to the database. The questions asked were,“Is the user working with other

users, or in isolation? To what extent is the user aware of other work in the database?

Doesa user’s work span a variety of different views and problems,oris it concentrated in

one area?”(Burtis, 2002, n.p.) The specific measures used in this research were: number

of notes contributed, percentage of notes that are linked to other notes, number of views

worked in, percentage of notes in the database that the user has read, and the number of

times the user has revised a note. Overall, the following events were tracked:

1. Numberofindividual notes created;

Numberof group notes created;

Numberof notes worked on;

Numberof group notesinitiated;

AwPB

YON

Percentage of notes read.

85

. Numberofbuild-ons created;6

7. Numberofedits;

8. Numberofrevisions (this counts multiple edits per note);

9, Numberof views workedin;

10. Percentage of notes linked; and,

11. Numberofnotes created, of revisions, and of views worked on,per participant per

week;

Using various combinations of measures for individuals and groups enables analysis

of the degree of individual and collaborative work within the database. While measures

cannotbe directly linked to knowledge building, various patterns have been shown to

provide indication of knowledge building dynamics. In previous research, for example,

Zhanget.al (in press), found a relationship between extensive writing and reading,

combined with build on, rise above - summaries and higher-order syntheses - referencing,

use of scaffolds - and knowledge building, as measured by independentanalyses of

knowledge advancement. However, rise above and scaffolds were not available in the

version of Knowledge Forum used during the course so the analytic tools were used for a

more general purposein the current study.

Knowledge building is about a community collectively improving ideas. Hence,

there was strong face validity for measuring the participants’ actions listed above. In an

online community, one would expect that reading, writing and editing notes would be an

important indication that participants were interacting. Further, building on and linking

other participants’ notes seems an important precursorto the possibility of idea

improvement. So whilst these indications of participation within the database do not

directly equate to knowledge building they do provide an indication. Qualitative

assessments of continual idea improvementand collective responsibility for improving

ideas represent more direct measures.

Chan and van Aalst (2003) have studied knowledge building portfolios where

students identified collective knowledge advances that showed the community’s best

work and progress. They found that measuresofparticipation (numberofnotes read,

written, linked, revised) correlated to portfolio scores and conceptual understanding. Ina

related study (Lee, Chan, & van Aalst, 2005), three groups of students (n=1 19)

86

participated in online knowledge building, using the Knowledge Forum software, under

different conditions and there was one comparison group that did not engagein online

knowledge building. Of the three groups that used Knowledge Forum,one used only the

software with no special conditions, the second group used Knowledge Forum combined

with portfolios, and the third group used Knowledge Forum with knowledge building

portfolios. For this third group, the knowledge building principles were used as scaffolds

to creating the portfolio. Results showedthat this third group had a higher Inquiry index

(read, write, scaffold) than the other groups. Student notes within the databases were also

scored for depth of inquiry and depth of explanation. Results showeda significantly

higher mean score for the third group (the group that used the knowledge building

principles to scaffold their portfolios). Finally, conceptual understanding (an essay coded

using rubrics) scores also showedsignificant differences for the third group overthe

other groups. The Analytical Tool Kit permits analysis of groups’ or individuals’ notes

and many different combinations andattributes of these notes. The tool kit also allows

for user names to be replaced by codes, which helps assure confidentiality.

Howdid notes link? Were ideas being built upon? Were there changes in the

reading/responding patterns? These were someofthe questions explored using the

Analytical Tool Kit. For example, the Analytic Tool Kit was used to measure the

percentage of notes read, the numberof build-ons, numberofnotes linked, and the

numberof revisions, amongst others. Using various combinations of measures for

individuals and groups, it was possible to acquire a good understanding of the degree of

collaboration within the database. For instance, a measure ofparticipative activity was

developed from the numberofnotes created and numberofnotes revised. Interactivity

was calculated using the numberofnotes linked and numberofnotes read. Patterns

found using this tool provided a quantitative analysis of notes within the conference. The

Analytical Tool Kit also allowed for isolation of any time period within the database.

This feature was used to analyse changeovertime, particularly the first and last three

weeks of the course.

87

Qualitative Analysis

This section presents the process for coding of the pre- and post-essays (Theories

of Learning essays). The analysis of notes produced by this community of learners

producedvast and rich amounts of data. Due to the nature of an online course, data was

limited to the written word and to the interactions between learners as measured using the

Analytical Tool Kit. The pre- and post-essays, and weekly learning log questions were

the central body of data, which wasstill a substantial amount. As Brown (1992)states,“it

is clear that we must select a very small sample from our large data base, and that

selection is obviously going to buttress our theoretical stance. This selection is

nontrivial. The problem is how to avoid misrepresenting the data, however

unintentionally” (p. 162). To deal with this problem, Brown asked herparticipants to

note “whenan interesting interaction occurs, so that we can transcribe ‘just those events

999of interest’” (Brown, 1992, p. 162). In a similar way, I read notes throughout the course,

recording items that stood out, and was able to mold learning log questionsiteratively

and in a relevant way, based on the “hot topic” during any particular week’s discussions.

The unit of analysis, or what was coded asan instance, varied from one sentence

to several sentences and can be considered a “thematic unit” (Garrison & Anderson,

2003, p.144). The unit of analysis had to be flexible because contextual meaning varied

from instance to instance. An effort was madeto capture the right amount of content to

represent the classification for the epistemological level or metacognitive thought.

The narrative that occurred in the course conference wasrich in context. Having

personally participated in several online conferences, I believe that the majority of

participants’ contributions are extremely well thought out. Anderson & Kanuka (2003)

state, “Both theorists (Freenberg, 1989) and researchers (Anderson & Kanuka, 1997)

have argued that forums of communication that occur asynchronously allow respondents

to reflect on and time shift their responses, thereby increasing the quality of those

responses”(p. 149).

The disadvantage of using the narrative from the online notes wasthat it was not

possible to probe interesting areas in an individual’s commentary asis possible in an

interview. Other data sources were therefore included in combination with note analysis

88

in orderto triangulate results.

Participants’ experiences were often described in their notes. Hence, this rich

narrative, andits actors, figured largely in the reporting of results. “[T]o understand

phenomenasuchasstudent learning and to document how this develops during the

course of a design study,it is necessary to take into accountthe desires, beliefs, goals,

reasoning processes, and so forth of the students over time, and that is best done in the

form of a narrative” (Shavelsonet al., 2003, p. 27). In this way narrative played a

significant role in both the analysis and the reporting ofthis research.

I looked both for events and trends within the data set. Using all data collected on

individuals, I hoped to see epistemological changes. As a precursorto his main study of

intellectual and ethical development, Perry (1970) administered “‘a Checklist of

Educational Views” to 313 freshmen. The surveys wereinitially administered in the fall

of 1954 and again in the spring of 1955. He saw “somethat had changed scores

markedly from fall to spring” (p.8). Still, I believed the time frame for my research

would only allow a mere snapshotof overall development.

Coding

This section describes the process used to code the thesis frameworks and capture

emergent themes. All notes (essays and learning logs) were coded using NVivo. As

participants could edit their pre- and post-essays at anytime, pre-essay data was captured

immediately following the course deadline. This turned out to be prudent as some

participants modified their original post throughout the course. Learning logs were

captured after the course wasfinished.

In the codingofall relevant notes, a first read-through was conducted to identify

any interesting themes as well as some manifest data (references to outside sources).

This provided a chanceto re-read participants’ notes after the course was finished (I had

read them previously during the course) and get an (second)initial impression. A second

pass was madeto look for examples of knowledge building principles. A third pass was

made to code for Moshman’slevels ofjustification as outlined in Table 11. During this

reading, I coded many examplesofjustification that did notfall into Moshman’s

89

framework. Thesejustifications mainly fell into either a practical-based or an

experience-based categorization. Further, immediately after this third reading of each

participant’s essays, I summarized myoverall impressions of changesin the types of

justifications used. For example I noted whohadstarted with an unjustified or

experienced-based justification for their theory in the pre-essay and then moved to a more

coherent, dialectical justification in the post-essay (ie one based on consideration and

judgmentof reasons from multiple sources such as researchers, peers, personal

experience, and/orpractical justification). Five more passes were made through the data

to code for examples of metacognition and levels of epistemological understanding. The

first three passes were used to code examplesand to build a coding protocol. There was

muchrereading of the applicable theories and models before working coding protocols

were produced, Appendixes C and D. final two passes through the data occurred after

the coding protocols had beenfinalized. For design mode, passes through the data were

madeafter belief-mode analysis.

During the early passes, I was not sure how to code manysections that piqued my

interest, so these were coded to a “how to code” node. However, by the fourth time

through,all the data from this node found a home. This showed myskill at coding to my

framework had reacheda proficient level. I had rated participants’ levels of

epistemological understanding several times, each time refining the coding protocol. The

final protocols represent mybesteffort at rating these sometimes-elusive concepts and

they were the ones used by the secondrater.

Inter-Rater Reliability

Inter-rater reliability was assessed for levels of epistemological understanding in

the pre- and post-essays, and for metacognitive activity in the learning logs. Coding

protocols can be found at Appendixes C and D. The secondrater wasblind to

participants’ names and rated approximately 40% ofpre- and post-essays and learning

logs. The data chosenfor inter-rater reliability represented a wide spectrum that included

all age categories, both male and female participants, and a mix of participants rated high

and low in participation and interaction. The second rater was provided with an

90

introduction to the theory and the protocols for scoring responses. In addition, a training

session washeld to ensure that belief-mode meaningsand definitions were understood.

Finally, a coding debrief session washeld to discuss differences in coding. Differences

that could not be reconciled were discarded for both the inter-rater Cohen’s Kappa and

tables of results where applicable. For design mode, raters earmarked all ideas identified

by a participant for improvement. Onceidentified, the work surrounding that idea was

further rated on a 5-point scale from (1) need for improvementnoted, but no actual effort

to improvethe idea; (3) moderate effort at idea improvement; (5) clear cut and important

improvementofan idea.

Summary

The methods described above have been carefully chosen. They provided a broad

description of context and a quantitative analysis. Further, they provided the meansto

determine if change had occurred within individual participants. The multiple data

sources allowedfor triangulation of results, which in turn allows a greater degree of

confidence in the interpretation of these data.

CHAPTER 4: RESULTS AND DISCUSSION

This chapter starts with a general description of the database, the biographical

information of participants, and a brief discussion of inter-rater reliability. The advance

organizer, Table 13, provides details on how the data sources wererelated to the thesis

questions. Theresults will be presented in this same order.

Table 13

Thesis Questions with Related Data Sources

1) Are changesin belief- or design-mode epistemology evident over the duration of

the courses?- belief- or design-mode epistemological understanding determined from pre- and post-

essays on theories of learning- Explicit examples of changes in epistemological thinking

- Use ofjustification within the pre- and post-essays

Whatare the characteristics of student work in learning logs ?- Use of metacognition in the learning logs- Preference of directed questions vs non-directed questions within learning logs

- Use of self-directed questions in learning logs during last three weeks

- Preference for private vs public learning logs

Howdoes epistemological understandingrelate to participation and interactionpatterns in online discourse?- Activity, interactivity, and participation rating- Overall collaboration

- Overall edits (an indication of metacognition and idea improvement)- Use of authoritative sources

2) If there are signs of epistemological change within individuals, how do these

changes manifest themselves in the interactions amongst learners in the onlinelearning conference as measured through changes in the reading and respondingpatterns of users? Is there a change in the reading/responding patterns of users

over the course?- Who’s read whose notes- Whobuilt on to whom- Week-by-week analysis of metacognitive activity- First half second half comparison- Case analysis for participants who showed epistemological change:

1. Whatepistemological change occurred?2. Wasthere a changein type ofjustifications used?3. Whose notes did they read the most?

91

92

Percentage of other participants’ notes read;Percentage of professor’s notes read;

On whom did they build?Week-by-week analysis for revisions;

Week-by-week analysis for views worked on;Week-by-weekanalysis for notes created;Metacognitive activity over time; and

Metacognitive activity directly following course facilitation.=SoeAAS

—©

Description of Database

A Web Knowledge Forum database, used for a 13-week graduate-level course,

generates a wealth of complex data. There were 2554 notes created during 12 weeksthat

were used for the main discussions. One week (week 12) the participants used another

database to experiment with the concept of anonymous contribution in an online

discussion. Hencethat discussionis not reflected in the word count. Once non-

participant notes were eliminated there were 2162 notes that created approximately

500,000 words. The numberofwordscreated per view can be seen in Figure 1. While

all the notes were read, only the following notes were used for detailed analysis: Learning

Logs (LL - 210 Pages) and Theories of Learning (TOL - 157 pages).

The course topics were broken down per week and each weeka different group of

participants were responsible. They were required to organize the week’s activities and

to facilitate the discussion. The topics each week were:

Week3: History: Developmentof constructivist theory - philosophical and

psychological antecedents

Week 4: Social and cultural influences on the developmentof constructivist

approaches

Week5: Situated learning

Week 6: Moresituated learning

Week7: Distributed cognition

Week8: Moredistributed cognition

Week9: Knowledge building versus learning and the implications for design

Week10: Theoretic perspectives on online community and how these connectto the

93

various approaches we havebeen considering

Week 11: Evaluation and assessmentin constructivist learning environments

Week12: Design principles for online environments

Week13: Future challenges and directions for learning online

Number of Words Created per View

100000

90000

80000

70000

60000

50000

40000

30000

20000

10000 0

Figure 1. Number of Words Created per View.

Description of Participants

Twenty-two students started the CLD course but six did not finish and only one

student did not complete the participation consent form. Hencethe participants consisted

of 15 students (n=15). Upon completion of the consent form, participants were asked to

94

complete a small survey that gathered biographical data. Results are show in Table 14.

Participants for the main study are consistently identified using the following code:

Participation RatingSex_Age. So the following participant, 3_F_51-55, was ranked 3

using the participation rating (discussed below), female, aged between 51 and 55 years

old. For participant quotes, a “High” or “Low” was added to emphasize the participation

rating. Further, in mosttables, the post-essay level of epistemological understanding

““(E, M, A/E, or A/M)” was added for easy comparisonofdata.

Table 14

Participant Biographical data

Participant ID* KF Use? Online Courses® Degree1_F_36-40:(E) 1 >4 MEd(2nd course)

2_F36-40:(E) >4 3 MEd(4th course)

3_F51-55:(E) 2 2 MEd(2ndcourse)

4M41-45:(E) 3 >4 MEd(last course)

5_F31-35:(E) 2 1 PhD (7th course)

6_F_26-30:(E) 1 3 MA(Sth course)

7_F_36-40:(M) 2 2 MEd(3rd course)

8F_31-35:(E) 4 4 MEd(Sth course)

9M41-45:(E) 1 4 PhD (last course)

10_M_41-45:(A/M) >4 >4 MEd(7th course)

11_M_26-30:(A/E) >4 >4 MEd(Sth course)

12_F_26-30:(E) 3 2 MA(6th course)

13_M_41-45:(M) 2 3 MEd(3rd course)

14F26-30:(E) 0 0 MEd(last course)

15_F_31-35:(M) 0 0 MEd(4th course)

Notes: “Table is in order of participation ranking. "How many times have you usedKnowledge Forum (any version) before this course? “How manyonline courses have you

taken before this course?

Inter-Rater Reliability

Inter-rater reliability was assessed for levels of epistemological understanding in

the pre- and post-essays, and for metacognitive activity in the learning logs. Theinter-

rater reliability using Cohen’s Kappa was .71 for levels of epistemological understanding

and was .89 for metacognitive content. During the coding debrief, instances of code for

the levels of epistemological understanding in the pre- and post-essays that could not be

95

reconciled were discarded. Discarded instances did not make a difference to the overall

assessed levels of epistemological understanding. Only a slight difference in the total

numberofmetacognitive passages coded for 2 of the participants existed (rater #1: 21

and 22 versus rater #2: 17 and 21). These differences were reconciled.

For design mode,inter-rater reliability was close to 0. Raters used the same scale

but had twodifferent interpretations of what constituted the scope of idea improvement.

Rater 1 interpreted idea improvementas existing within the individual- ideas or theories

could be used, for instance, to help improvethe participants’ teaching practice. Forrater

2 - idea improvement meantliterally improving an existing theory or idea, similar to

making a new contribution to a body of knowledge. At this point, several other raters

were engaged and cameup with yet anotherinterpretation of idea improvement — they

looked at how an idea or understanding of an idea changed for each participant between

the pre- and post-essays. In the end, these differences did not matter as very few

noteworthy examples of design-mode epistemology were identified regardless of which

interpretation of idea improvement wasused.

Are There Changesin Participants’ Level of Epistemological Understanding Observableover the Duration of the Courses?

Pre- and Post-Essays - Theories ofLearning

The main thesis question asked, “Are there changesin participant’s level of

epistemological understanding?” To operationalise this question, pre- and post-essays

wereusedthat elicited participants’ views of knowledgein a learning context.

Specifically, questions were designedto elicit baseline and end state epistemological

views concemingassertions,reality, knowledge,andcritical thinking. Questions did not

point participants directly to the need for continual idea improvement, but they provided

plenty of scope to engage in such efforts. Assigning a weighting of 12% of participant’s

overall grade for the course emphasized the importance of the essays. The questions used

for both pre- and post-essays were:

1. What do you currently understand learning to be—for yourself as a learner and

for your students if you teach?

96

2. Why (on whatbasis) do you hold those views, both for yourself and for your

students? (If you are not a teacher think of a situation where you have taught

somebody something.)

3. What role does knowledgeplay in learning?

4. Whatrole do others play in your learning (e.g. peers, teachers etc)?

Epistemology concernsitself with beliefs pertaining to the nature of knowledge.

While there were some direct statements from participants that clearly commented upon

changesin beliefs, most examples were subtler. Hence,participants’ beliefs were

inferred from the content of the essays. Appendix C showsthe coding protocolfor levels

of belief-mode epistemological understanding.

Levels ofEpistemological Understanding Determinedfrom Pre- and Post-Essays onTheories ofLearning

Assigning a level of belief-mode epistemological understanding wasdifficult.

Manyiterations of coding protocols weretried before a final protocol was produced.

Appendix E showsthe results of an early coding. Part of the difficulty in coding was due

to the fact that students often displayed beliefs that spanned twoor eventhree levels of

intellectual development (D. Kuhn etal., 2000; Meacham, 2003)all within one essay.

Design-mode ratings were mucheasier because raters simply needed to judge if there was

intention - followed by effort - to improve an idea. In the end, a typical or overall level

for each participant was determined for both the pre- and post-essay which reflected the

most prevalent or typical belief-modelevel.

The protocol, Appendix C, wasfirst and foremost, based on D. Kuhn et al’s.

(2000) model, Table 2. In addition, several differentiators, that were based on the

epistemological models described in chapter 1, were used:

1. Did they show signs of “charting their own course”, making their own

goals (commitmentto learning)?

2. How confident were they in their answers and whatwerethe justifications

used (confidence in answering)?

3. What werethe sourcesoftheir justifications (justification for answer)?

97

and,

4. What value wasplaced on others (peers, teachers, scholars, researchers) in

the creation of knowledge and in learning?

Participants’ essays were scored based on where the majority oftheir

epistemologicalbeliefs fell within the belief-mode model. While no participants were

rated uniquely at the absolutist level, there were several that showed,at least residual,

signs of the certainty of knowledge,

I think that I saw knowledge as something absolute...like a definition or a

fact that could not be disputed. I’m starting to really see what is meant bythe social/cultural context ofmeaning and that knowledge andunderstanding is something that is highly contextual and dynamic.Clearly, my initial response was geared towards ONE TRUEKNOWLEDGE...whatever I thought that meant.

Okay...I’m embarrassed. (8F_31_35:Low)

And anotherhinted at the certainty ofreality,

Ina theoretical course such as this one, is there really a true right or

wrong answer? (10_M_41-45:Low)

implying that for some courses, or domains of knowledge, there could be a “true right or

wrong answer”and hence, an absolutist’s view point.

Aswell, there were manyparticipants whostarted the course with a cursory view

of the social constructive nature of knowledge,

My theory oflearning, however, is still extremely teacher-centred.

Although I have opened upto the idea ofdistributed cognition and theimportance oflearningfrom others (peers, older/younger students,

parents, volunteers, janitor, secretary, the guy on the street) not only the

teacher, Iam still veryfarfrom the concept ofteacherasfacilitator. Ifeelthat at the elementary panel, a goodfoundation needsto be established (alittle like building a house) and a more structured approach to learningneeds to be present. Iam stillfearful that iflearning is not teacher-centred, students can set erroneousfoundations(alittle like setting acrookedfoundation which will inevitably crack with time) on whichitisdifficult, dangerous, impossible to build at the secondary panel, the post-

secondary and the world ofwork. (7_F36-40:High)

98

However, most participants quickly realized the importance of others to learning and

knowledge, as the readings for week 4 introduced the concepts of social constructivism.

The lingering absolutist viewpoint of some did prove thorny and,as a result, several

participants were rated at multiple levels. Nevertheless, this was consistent with Baxter-

Magolda’s model (1992),a level that she classified as transitional knowing.

Someparticipants seemed to follow Kuhn etal’s (2000) developmental model in a

very explicit way as shown herein this excellent example of the typical movement from

absolutist to multipilist to evaluativist,

Finally, I am interested in how my understanding oflearning will evolve

throughthis course as I learn about different constructivist learning

theories. I have been moving awayfrom myessentialist background in

biopsychology, “the study ofbehavior as a result ofthe interaction of

biology and environment,” through my sojourn in English Literature,which stressed individual agency. Yet, Iam back again to my

psychologicalroots, butthis time in the context ofeducation. I intend todwell in the tension between the approachesthis time, to dialecticallyreflect on what I see learning to be and how this connects to goodteaching. (5_F_31-35:High)

This participant started her undergraduate work in a “pure science” degree where

arguably the belief of absolute knowledge and objectivist reality (as elaborated by Baxter

Magolda)is generally accepted (at least for most undergraduate students). Next she

movedinto the typically multiplist’s domain of English Literature where interpretation

and subjective viewsflourish. Finally she entered the domain of education where she

hopedto reconcile the previous opposing epistemological views. This was very similar

to Kuhn’s model.

From a design mode perspective, we see shifts in conceptual understanding,that

might be characterized as idea improvement, but the improvementis reflected in an

historical analysis - a recounting of what’s been learned and a statementof intention to

chart a new course of learning or a new educational mission. However, with respect to

our scoring scheme,an idea to be improved has not been identified. More generally,

there is not a sense from this student that she sees herself as an agent of educational

change. Hergoalis to reconcile different views--a belief-mode advance--not develop

new knowledge, which is a marker of design-mode and knowledge building. There is

99

metacognition, but no evidencethatit is linked to design-modeepistemology.

Asstated in chapter 1, most belief-mode epistemological theorists acknowledge

that people can exhibit signs of being at different levels at the same time, depending on

the context. This phenomenon wasevidentin this research as well. Participants were

rated at multiple levels within the same essay. Further, only a few participants provided

views that spanned the coding protocol categories. Hence,participants’ levels were

based on where the majority of indicators present, fell. Table 15 provides the rated level

of epistemological understanding for each participant.

As can be seen abovethere were 7 participants who wererated as evaluativist in

the pre-essay and they did not change during the post-essay. That left 8 participants who

could have shown some change. Of the remaining 8, 5 showed movementin their level

of epistemological understanding. This movement was from within the multipilist level

(and there wasthe full range within this level) to the evaluativist level. The remaining 3

showed no movement, staying within the multiplist level as determined by the pre- and

post-essays.

Table 15

Levels of Belief-Mode Epistemological Understanding and TypeofJustification Used for

the Pre- and Post Essays

Pre Level of Post Level of

Participant ID* EpistemologicalEpistemologicalPre Justification Post Justification

Understanding ’Understanding

Experience (4)1_F36-40 E E DR(5) DR(7)

Experience (5)2_F36-40 M E DR(3) DR (2)

- Experience (7) Experience(1)3.F51-55 E E DR(1) DR(4)

_ Experience (4) Experience (1)4M41-45 E E DR(2) DR(5)

5_F31-35 E E DR (3) DR(4)

Experience (3)6F26-30 M/E E DR(1) DR(2)

7_F_36-40 M M Experience (2) Experience (2)

100

Experience (2)8F31-35 E E DR(2) DR(2)

Experience (2)9M41-45 E E DR(1) DR (1)

10M41-45 A/M A/M Experience (1) Experience (4)11_M_26-30 A/M A/E Experience (2) DR (2)

Experience (4) .12F26-30 E E DR(1) Nil

Experience (5) Experience (3)13_M_41-45 A/M M DR(1) DR(1)

14F26-30 M E Experience (3) Experience (4)Experience (5) Experience (2)

15F31-35 M M DR(1) DR(3)

Notes: “Table is in order by participation ranking. "Based on Kuhn’s (2000) levels of

epistemological understanding, Table 2 (E = Evaluativist, M = Multiplist, A=Absolutist).“Justifications were coded to participant’s personal experience and Moshman’s (1998)levels ofjustification, Table 11.

Explicit Examples ofChanges in Belief-Mode Epistemological Thinking

A tangible benefit of the learning logs was the degree to which they supported

dialectical reflection (a deliberate effort to make conceptual progress throughactive

metacognition) consistent with the kind of rational constructivist thinking necessary at

the graduate school level. In the examples below, we canseethe use ofdialectical

reflection to highlight changes in epistemological views. The majority of participants

made such comments.

With respect to the relative nature of knowledge, a participant discoversthat,

Betsy didn't need whatI did, she didjustfine with gray. I think there are

advantages in both ways ofthinking but my pointis that I didn’t realize

until now that I can only think in black and white. So I have some learningto do. (15_F_31-35:Low)

Clearly this participant had some absolutist views and was discovering that a

world of multiple interpretations of knowledge exists. She hesitantly acknowledged that

she must grow to accept this “new world”. From a design perspective, we see

recognition of the need for learning, but not identification of ideas to be improved. There

101

is no effort to create objective knowledge in the Popperian sense of ideas out in the world

and on a path of continual improvement, independent of any specific knowerofthose

ideas. Rather, the perspective is a World 2 - knowledge in the head perspective. Another

participant recognized that knowledge wasnotfinite,

I think one ofthe biggest personal revelations during the course sofar isthat there is no “endpoint” so to speak regarding knowledge. It is always

growing, changing, evolving, and old concepts are continually revisited

andtested. It is truly dynamic, and I think that's why the pursuit of

knowledgeis so exciting. (13_M_41-45:Low)

Again, from a design-mode perspective there is recognition of continual change,

but there is no suggestion that the author sees himself as an agent in that change. Change

is, in this sense, a “primitive” - something that occurs without presumed need of

explanation. Ohlsson (1991) noted that this is how students wholack an understanding

of evolutiontreat it - they report that it occurs andthat it is important, but indicate no

need to understand or explain it and that can be quite exciting, but with no clear

indication of taking charge of that process oneself, in a direct way.

Finally, this participant cameto realize the socially constructed nature of

knowledge and hence the importance ofothers to her learning,

This week I began to develop more ofan appreciationfor the social

aspects oflearning. This occurred on a personalas well as theoretical

level. I began to see the impactofsociety and the environmenton learningin “general” and also began to appreciate the effects ofthesefactors onmy ownlearning. I realized that what I would take awayfrom thiscourse/experience wasdirectly related to the others in the class, myinteractions with them andtheir online contributions. As someone who

generally saw my “textbook”as my greatest resource, this was a big leap!(1_F_36-40:High)

It is interesting to note here that while she believed that the “textbook” was her

absolute, and hence “greatest” resource for learning (repository of knowledge), it was

indeed just an artifact that was written by someone. From a design perspective, however,

there is no attempt to deal with the textbook-peer issue as an improvable idea in its own

right. There is also no suggestion that the shift from textbook to peer has any component

102

that is in any way problematic or provides a context for new design challenges. Reading

of the textbook can in fact be interpreted as a form of social interaction and hence she

wasalready “socially interacting”. Indeed, later in the course shestates,

Previously, I hadn’t recognized thatthe artifacts which I used, even whileworking independently, were representations ofdistributedintelligence...therefore linking me with this web and rendering myindependence merely relative and, ultimately, an illusion. I have begun torecognize the subtle and often invisible contributions ofothers to my socalled independent workandlearning. (1_F_36-40:High)

There were many such cases where participants made explicit comments

concerning changesto their epistemological views. The areas of epistemological change,

and the number of passages codedas such, are presented in Table 16. Worthy ofnote is

that in 6 of the 11 cases, participants were rated as having no changein their level of

belief-mode epistemology, yet they themselves believed that they experienced some

significant change. However, only the pre- and post-essays were used to determined the

level of epistemological understanding. In Table 16, examples of explicit

epistemological change were taken from both the essays as well as the learning logs.

Hence, while the pre- and post-essays showed no changein epistemological levels, the

participants’ learning logs did seem to show change. It could be that, because the logs

are personal reflections, changes might be expected to show upherefirst and only later

show upin theories of learning. This would be similar to using technologyfirst

personally before understanding how it can be incorporated meaningfully into one’s

teaching. Howeverneither resulted in ideas set forth as improvable objects, as would be

expected if they held a design-mode epistemology.

It is interesting to note that 9 of the 11 participants who displayed explicit

examples of belief-mode epistemological development did so by recognizing the

importance of others to their own learning. Perhaps this was because ofthe subject

matter of the course, social constructivism. However, this marked increase could have

been due to the nature of the knowledge building pedagogy andthe skills of the professor

in facilitating epistemological development.

103

Table 16

Descriptions of Areas of Belief-Mode Epistemological Change and Overall Numberof

Passages Coded

Participant ID Areasof belief-mode epistemological change Number

ofpassagescoded

1_F36-40 social aspect to learning as apposedto textbook; 20importance of unanswered questions, debate andconflicting philosophies; importance of self evaluation;changing nature of knowledge; need to beintentional inlearning; knowledgeis notcertain.

2_F36-40 ability to disagree with the mainstream thinking; 5importanceofcritical thinking; relativity of knowledgevice absolute.

_31-35 importanceofcritical thinking; importance ofpeers. 226-30 relative and changing nature of knowledge; importance 3

of others.7_F_36-40 importance of others to learning; relative and ever- 3

changing nature of knowledge.

8F31-35 relative nature of knowledge; social nature of knowledge; 5knowledge changesovertime; “To get back to thequestion then, about what would makethe greatestdifference to the widespread use of knowledge buildingwithin our society...a change in perception aboutthenature of knowledge and our approach to learning ingeneral. Even as a student in a graduate course about

constructivism,I actually demonstrated the mostsignificant barrier ~ a lack of understanding ofthe

social/cultural context of meaning and a misconception

of the absolute nature of knowledge. People can'tembrace knowledge building given common or

traditional views of learning and knowledge...it doesn'tmake any sense within that context.”

9M41-45 importance of others; need to “bounceideasoff... clarify 1points, to see things from a different point of view.”

11_M_26-30 critical thinking is important; importanceofotherto 4learning; knowledge generated by human minds.

13_M_41-45 importance ofothers in learning; multiple correct 1

answers.14F_26-30 importance ofothers. 315_F_31-35 importance of others; importanceofcritical thinking; 7

5F6F

104

relative nature of knowledge; “However, this formatis

encouraging meto recognize that weall have different

levels of understanding, and we each comeat the course

from a different perspective. If someone does disagree or

counter my position, I am trying lookat it as anopportunity for learning, either to see the point fromsomeoneelse's vantage point, or, to confirm more

strongly my own beliefs.” As wellas, “I think one of thebiggest personal revelations during the course so far is

that there is no ‘end point' so to speak regardingknowledge. It is always growing, changing, evolving,and old concepts are continually revisited and tested.It istruly dynamic, and I think that's why the pursuit of

knowledgeis so exciting.”

It is also worth note that while there was epistemological changein belief mode

there was not a corresponding advancein operating in design mode. While several

participants noted that knowledge keeps advancing,this is, as suggested above - a simple

knowledge advance- in a waythatis not explained or understood, and something that

theorists or scientists do - seemingly through the scientific method. But there is no

suggestion that they see themselves as agents in this process or that their professionalrole

is that of designing education for the future to address the problemsidentified.

Use ofJustification Within the Pre- and Post-Essays

In addition to levels of epistemological change, each participant essay was coded

for the types ofjustifications they used. Passages in whichjustification was used were

compared and classified using Moshman’slevels ofjustification, Table 11. An additional

form ofjustification, experience based (justification given wasattributed to direct

experiences from their past) seemed to dominate muchofthe pre-essay and hence, was

also coded. This experience basedjustification seemedto fit better with the first three

belief-mode epistemological models (Perry, Belenky et al. and Baxter Magolda)

discussed in chapter 1, as they focused on how individuals interpret their educational

experiences (formal or informal).

Here are two examples of passages that were codedasdialectical reflection. In

the first example the participant was asked whysheheld a particular view on learning,

105

Comingfrom a science background I sometimes tendto get a bit

reductionistic in my points ofview regarding learning. Ifind myself

thinking about neuropsychology/physiology, synapses, cells,

neurotransmitters, feedback loops and the like. When I read Bruner, I wasaware that I also believe in the interpersonal, social and affectiveperspectives on learning. (1_F_36-40:High)

It was evident that she was using her personal experience, expert sources

(Bruner), and her affective perspectives to justify her view on learning. She used

metacognitive reflection to try to achieve temporal coherence. A second example asked

why(on whatbasis) do you hold those views, both for yourself and for your students?

This questionis really excitingfor me because as I read through my

earlier entry, I realized that much ofwhat I had experiencedin thefieldwhile delivering sales training was validated by the readings. Earlier onin the course, I did not have the theoretical knowledge to understand whythings I had observedin thefield actually worked, and why others didn't.However, in light ofthe readings, and understanding the work ofVygotsky, Pea, Lave and others, the theories I had started toformulateactually began to mature. .

I see this pastfew months as a true experience ofconstructivist learning;usingprior knowledge and experiences, and building on the contributions

oftheorists and classmates to scaffold my understanding, I have pushed

the boundaries and created new levels ofunderstandingfor myself.(15_F_31-35:Low)

Again, several sources ofjustification were used as well as metacognition. In this

latter case, the participant explicitly stated that she has “created new levels of

understanding”thus bringing a coherence to previous experiences and newly learned

theories.

Here is another example of how previous experience wasusedto justify how a

participant’s learning occurred,

I also can learnfrom pain. I hope I never again will open a beerbottlewith wet hands after I have been swimmingfor a while. It appears that mySkin gets soft and cuts into the bottle cap. An important, painful lesson, butlearning nonetheless. (9_M_41-45:Low)

Coding for justification was based on a statementorassertion andthe related

106

justification surrounding that statement. Passages coded asdialectical reflection had to

demonstrate efforts of achieving coherence between past understandings and new

learning through active metacognition. However, the form of the justification was more

important than the content, as expressed by Foucault (Dreyfus & Rabinow, 1982), one

“does not need to share the beliefs of those who take these serious speech acts seriously

in order to locate them amongall the things that are said and written” (p. 52). Hence, a

passage wascoded regardless of the coder’s beliefs. Table 15 shows the numberof

justifications that were rated as experience based as well as those rated dialectically

reflective. It was hypothesized that participants who showed epistemological

developmentover the course would also show increased sophistication in their

justifications, moving towardsdialectic reflection. This indeed wasthe case for 2

participants who showed changein their level of epistemological understanding between

pre- and post-essays. Twoothers, who showedchangein epistemological level, did not

show any increased sophistication in their justifications. In general, 10 of the 15

participants showed anincreasein justification between essays and 3 others stayed

approximately the same. Only 2 participants seemed to decrease the sophistication of

justifications used. Oneofthese participants had 3 dialectic reflections coded for the pre-

essay and only 2 for the post-essay. Howeverher experience-basedjustifications

decreased from 5 to 0. One could arguethat this was actually not a decrease but an

increase in sophistication, as she no longer solely used experienceto justify her

assertions.

The above sections showedthat there were indeed several changesin participants’

levels of epistemological understanding during this research. The next sectionswill

examine some of the manifest characteristics of the epistemological levels displayed by

participants. Following that, sections will explore how epistemological levels relate to

knowledge building and howthe epistemological levels relate to interactions between

participants.

Use ofMetacognition in the Learning Logs

Flavell (1979; 1987), a pioneerin this construct, contended that metacognition

wasbest understood as metacognitive knowledge (knowledge of person variables, task

107

variables and strategy variables) and metacognitive regulation (overseeing, planning,

monitoring, and measuring outcomesof cognitive activities). Examples of metacognition

were fairly easy to find within the learning logs. To code passages, one of the questions

asked was “Doesthis show an aspect of self management and/orself appraisal of

cognition?” This first example displayed participant knowledge of his cognitive process;

in other words, metacognitive knowledge,

Ifoundinpast courses that I have learned vicariously through other’sjournals. (9_M_4145:Low)

And this example demonstrated monitoring of cognitive and affective states;

metacognitive regulation,

Reading the assigned articles and developing questions which wouldallow us to explore the topic, wasinitially a “tad”stressfulfor me. I was

a bit self-conscious at times... How would the questions be received...would I be able to keep up with the comments? In the end it was a very

rewarding experience. (1_F_36-40:High)

Initially both types of metacognitive activity were lumped together. However, a

difference between the metacognitive knowledge and metacognitive regulation does

exist. Brown (1987) stated that metacognition “has been usedto refer to two distinct

areas of research: knowledge about cognition and regulation of cognition. The two

forms of metacognition are indeed closely related, each feeding on the other recursively;

attempts to separate them lead to oversimplification. However, they are readily

distinguishable, and they do havedifferent historical roots” (p. 67). So, the general node

on metacognition was further coded to separate the two types. This proved to be a more

difficult task. Often participants would discuss their cognitive strategies while at the

sametimeself-evaluating these strategies. In this case, the passage was coded to the

more prevalent node. For example, when asked, “How do you decide whento contribute

to the discussion?”this participant displayed both types of metacognition;first she

discussed her process (metacognitive knowledge) that led to a plan or goal

(metacognitive regulation),

108

First, I like toformulate an answerto the questions posed without readinganyone else’s posting so I tend to do this as early in the week as possible.Then I start reading other’s contributions and looking back at my own.Once I run into something that I agree with I usually post a comment, to

that effect, perhaps adding a question or a related idea.As I am writing this I have discovered that I don’t tend to post comments

when I disagree. I guess I’m comfortable elaborating and thinking out

loud and Iam comfortable asking questions.. but not questioning andcertainly not challenging. Ifsomeone posts a question to me, or challengesme, I alwaystry to answer, as non-defensively as possible, but I’m not

sure I have always achievedthis. :-)One "aha" moment came when I answered question two, above. I had

never really looked at the “why and when”ofmypostings. Now, I have anew learning goal. I am going to start responding to postings that I don’tagree with :-) (1_F_36-40:High)

And this example talks about a strategy (knowledge) to confirm learning

(regulation),

I think that one ofthe ways that I evaluate my ownlearning in a courseisthrough communicating new ideas with individuals outside ofthe course. Ifeel that ifl can communicate ideas to other areas outside ofthe confinesofthe course then there is a clear demonstration oflearning. IfI can bring

this new knowledge to my work environment and apply the theories andconcepts to this environment then not only am I understanding, but I'm

replicating the knowledge to possibly create something new. (12_F_26-30:Low)

It is interesting, from a design-mode perspective that the participant suggests a

process for generating new knowledge—butagain, doesnot identify an idea to be

improved.

Ascan be seen in Table 17, those participants who had a high participation rating

tended to produce more metacognitive activity. However, there was no pattern that

emerged as to which type of metacognition dominated for either group. Further, there did

not seem to be any patterns that emerged when metacognitive activity was compared to

the participants’ biographical data. Metacognitive activity was further analyzed below as

to how metacognitive passages wererelated to the types of weekly questions asked

within the learning logs.

109

Table 17 provides the total number of passages coded as metacognitive activity

then, the breakdown into metacognitive knowledge and regulation.

Table 17

Metacognitive Activity: Total, Knowledge, and Regulation

Total MetacognitiveMetacognitiveParticipantID Metacognitive Knowledge Regulation

Activity Activity Activity

1F36-40:(E) 102 64 382_F36-40:(E) 40 16 243F51-55:(E) 25 15 104M_41-45:(E) 31 14 175F31-35:(E) 34 15 196F26-30:(E) 36 14 227 F36-40:(M) 46 27 198F31-35:(E) 42 18 249M_41-45:(E) 16 6 1010_M_41-45:(A/M) 17 12 511_M_26-30:(A/E) 26 6 2012_F_26-30:(E) 15 8 713_M_41-45:(M) 12 2 1014F_26-30:(E) 2] 14 715 F 31-35:(M) 21 9 12

Preference ofDirected Questions Vs Non-Directed Questions Within Learning Logs

There was much discussion within the course database about whetherthe logs

should be scaffolded using questionsor left open. A participant who seemedto

championthis discussionreflects first on having no questions provided,

In rereading this log, Ifound that the content was more validfor me whenI took an open-reflection approach. As I reread thejournal, however, Ifound that this open approach tendedto be more content driven, and notas personal, or reflective as when I took a more holistic approach to thislog. Despite the benefits ofcontent recap and synopsis, I did notice a lackofintrospection with this approach. (4_M_41-45:High)

Hethen reflects on having questions provided,

110

The guidance that was provided throughoutthis course in our learninglogs concentratedprimarily on “reflection”, and as such, has hugebenefits that I would not have otherwise reaped. When tookthis

introspective approach, however, I did miss the content-synopsis that

would come with a more open andfreeflowing approachto thisjournal.(4_M41-45:High)

Here is a commentfrom anotherparticipant,

I do enjoy using the learning log as a place to store my thoughts about theweek and to encourage reflection. The questions posted help me to move

in directions that I might not have thought of Without the questions, I

wouldprobably tend to do more ofa summary ofthe readings rather thanuse itfor application and understanding. (15_F_31-35:Low)

Data about participant preference concerning the directed versus non-directed

nature of the learning logs was gathered from week 10 of the logs. That week the

directed question asked, “Reread your learning log. How,ifat all, has this log helped

you? Could it help you more?” Notall participants mentioned the issue of directed

versus non-directed logs, but 11 out of 15 did and the results are presented in Table 18.

Table 18

Directed Versus Non-directed Learning Logs

Metacognitive

Participant ID Preference RegulationActivity

1_F_36-40:(E) Directed 382_F_36-40:(E) Mix of Both 24

3_F_51-55:(E) Non-Directed 104M41-45.) Mix of Both 17

5_F_31-35:(E) No Comment 196_F_26-30:(E) Non-Directed 22

7_F_36-40:(M) Directed 198F31-35:(E) No Comment 249_M41-45:(E) No Comment 1010_M_41-45:(A/M) Non-Directed 511_M_26-30:(A/E) Non-Directed 2012_F_26-30:(E) Directed 7

13_M_41-45:(M) No Comment 914F_26-30:(E) Directed 715_F 31-35:(M) Directed 12

11]

The low participation group slightly preferred (3:2 of those who commented)

directed questions. This could have been dueto a generally low metacognitive regulation

count. Membersofthe high group liked the questions but also liked the freedom to do

whatever they wanted within the logs. There did not seem to be a preference of directed

versus non-directed and metacognitive regulation activity.

Use ofSelf-Directed Questions in Learning Logs During Last Three Weeks

Directed questions were provided to participants for weeks 2 to 10. At week 11,

participants were asked to “Please design your own question this week and makeit

evident for others.” For weeks 12 and 13, they weretold, “You are on your own. Thanks

for letting me be part of your learning.” This was done in conjunction with the instructor

and specifically to see how participants would react. At the high end ofthe belief-mode

epistemological models discussed in chapter 1, commitment (Perry, 1970), constructed

knowing (Belenky, Clinchy, Goldberger, & Tarule, 1986), contextual knowing (Baxter

Magolda, 1992), reflective thinking (King & Kitchener, 1994), and evaluativist (D. Kuhn

et al., 2000), there were some commonthreads. One commonidea wasthat of continued

commitmentto learning, of which critical thinking and evaluation wasintegral. At these

higherlevels of epistemological understanding, there was an inherent commitmentto

continuously andcritically reflect on one’s learning and knowledge. Moshman (1998)

would argue that these levels of epistemological understanding require dialectical

reflection; “Deliberate efforts to achieve coherence by reconstructing one’s intuitions,

and/or conceptions of precedent” (p.961). Remember“dialectical reflection may be

defined as a deliberate effort to make conceptual progress through active

metacognition”(p. 961). Hence, it was hypothesized that those participants who were at

the evaluativist level were more likely to ask themselves metacognitive questions and/or

show metacognitive activity within their learning logs during the final three weeksofthe

course oncethe scaffolds (ie the directed questions) were removed. Table 19 shows

participants’ post-essay levels of belief-mode epistemological understanding and

comparesthis to the final three weeks of metacognitive activity and the typesofself-

questions asked.

112

Table 19

Comparison of Post Level of Epistemological Understanding with Metacognitive

Activity and Self-Questions for the Final Three Weeks

Post Level of Metacognitive activity last 3UserID Epistemological e y las Questions type last 3 weeks

. weeksUnderstanding

1 F_36-40 E 18 1 metacognitive question

8F31-35 E 12 1 metacognitive question11-M_26-30 A/E 8 Content-based questions

2F36-40 E 7 No questions

3_F51-55 E 7 2 metacognitive questions4M41-45 E 7 1 metacognitive question

14-F26-30 E 6 Content-based questions

5_F31-35 E 5 1 metacognitive question6-F_26-30 E 4 1 metacognitive question9M41-45 E 4 1 metacognitive question

15_F_31-35 M 3 Content-based questions7_F 36-40 M 3 1 metacognitive question12-F_26-30 E 2 2 metacognitive questions

13_M_41-45 M 1 Content-based questions

10_M41-45 A/M 0 Content-based questions

It is interesting to note that the majority (except for 1) of participants rated as

evaluativist in their post-essays (post level of epistemological understanding) showed

higher metacognitive activity during the final three weeks than the others. Further they

asked themselves questions that were metacognitive in nature without prompting. In

contrast, those participants who were notrated evaluativist in their post-essays showed

low metacognitive activity and tended to ask content-based questions whenleft to fend

for themselves.

Preferencefor Private Versus Public Learning Logs

Perhaps the most contentious issue that arose within the course database was

whetherthe logs should be public, for all participants in the courseto see, orprivate, to

be shared just with the instructor and TA. There seemedto bea fairly evidentsplit

between participants who had a high versus low participation rating. Participants with

113

high level of participation were eagerto share their logs and seemed to learn much from

reading others’ logs. One participant stated,

Reading other people'sjournals has also been a learning experience. Eventhough weall read the samearticles, answer the same questions and usethe same "folders"...our experiences are so unique and varied. (2_F36-

30:High)

Participants who hada relatively low participation rating typically preferred more

privacy. For instance,a participantsaid,

An issue that I'vefaced with theformatfor thejournals is that they arepublic, I'd prefer them to be private or maybejust shared with [theinstructor] and to have the option ofposting to a publicforum ifonewishes. (14_F_26-30:Low)

Andfinally,

Ideally, I think I would like to know that my learningjournal was a more

private thing (only accessible to the instructor), with some sort ofregular

feedback or dialogue. Perhaps with the understanding that some ideas or

excerpts would be requested by the instructor to be shared with the groupand then “tossed around”. (11_M_26-30:Low)

Oneofthe advantages of allowing the logs to be made publicly available was to

provide models for participants who wereless inclined, or less experienced at expressing

themselvesin this context. There were some anecdotal commentsto the instructor to

suggest this was effective for some.

How Does Belief-Mode Epistemology Relate to Participants’ Online Discourse?

Participation and Interaction Measures

Asdiscussed in chapter 3, Knowledge Forum providesa public and collaborative

space the makes knowledgebuilding morelikely to occur. In orderto get a simplified

picture of the large amountof data produced by the Analytical Tool Kit, several measures

114

were combined. Asstated by Burtis (2002) these measures

are meantto be usefulfor assessing a user's contributions to the databasefrom a knowledge building perspective. Is the user working with other

users, or in isolation? To what extent is the user aware ofother work inthe database? Does a user's work span a variety ofdifferent views andproblems, or is it concentrated in one area? The measures are: numberof

notes contributed, percentage ofnotes that are linked to other notes,percentage ofnotes that have been keyworded, numberofviews workedin, numberofproblems worked on, percentage ofnotes in the databasethat the user has read, and the numberoftimes the user has revised anote. (p.2)

Activity Rating

The activity rating was a measure of individual note creation and revision; how

many individual notes did the participants create and how manyofthese were revised?

To derive the activity rating, raw counts of note creation and note revision were

individually ordered from most to least. A ranking of 1 (most) to 15 (least) was then

assigned to each participant. Ties were assigned the same ranking. The two rank scores

(one from the numberof individual notes created and one from the numberofindividual

notes revised) were added together. The seven highest rated scores were simply rated as

high and the remaining eight as low, Table 20. For example, 1_F36-40 created the most

notes (ranked Ist) and made the mostrevisions (ranked Ist). This gave hera total of 2

points and rating of high.

Interactivity Rating

Interactivity was a measure of howparticipants interacted with other participants’

notes. The rating looked at the percentage of notes read and the percentage of notes

linked, both these behaviours indicated interaction within the database. As with the

activity rating, the raw percentages were ordered from highest to lowest. A ranking of 1

(highest) to 15 (lowest) was then assignedto each participant. The two rank scores (one

from the percentage of notes read and one from the numberofnotes linked) were added

together. The seven highest rated scores were simply rated as high and the remaining

115

eight as low, Table 20. For example, 1_F_36-40 read the most notes (ranked Ist) and

linked the sixth highest percentage of notes (ranked 6th). This gave her a total of 7 points

and a rating of high.

Participation Rating

The participation rating was the combination of the activity and interactivity

rating. Participants who had been ranked high in each of the sub-ratings were rated as

high for the participation rating. The seven highest rated scores were simply rated as

high and the remaining eight as low, Table 20. For example, |_F_36-40 rankedfirst in

three categories and sixth in one. This gave hera total of 9 points and a participation

rating of high.

There were two participants whorated both a high and low in the sub-ratings. In

this case the overall score was used to determinethe participation rating. As can be seen

in Table 20, the overall score dropped by 7 points from (27 to 34) between the two

participants, a difference of approximately .5 SD. Hence, the participant who scored 27

wasrated high on the participation rating and the other participant low.

Activity and Interactivity ratings were used in order to compareparticipants’

contributions in greater detail. However, as can be seen in Table 20, there were not a

significant numberof participants who were very active or interactive in only one

measure. Those participants who werevery active in creating and revising notes tended

also to be very interactive in linking and reading notesrelative to the other participants.

Hence,the participation rating was the base rating used to compare with other measures.

Those with a high participation level were morelikely to be at the evaluativist

level. As far as changein level of belief-mode epistemology, there were 2 participants

rated high comparedto 3 rated low who showed change from pre- to post-essay. Given

the overall lack of design-mode epistemology, it seems reasonable to infer that

collaboration and interaction are prerequisites for establishing knowledge building

community dynamics, but these factors are by no meanssufficient. Participants’ previous

use of Knowledge Forum, numberofprevious online courses, and numberof courses

completed, did not seem to affect participation rating. The exceptions were the 2

116

participants who wererated the lowest. Neither had used Knowledge Forum nor

completed any prior online course. In fact they were the only two whohadthis history.

Table 20

Activity, Interactivity, and Participation Ratings with Their Corresponding Scores

# of # of Activity % of % of Interactivity Overal Participation

Notes Revisions Rating Notes Notes Rating 1 Score RatingParticipantID Created Read Linked1_F36-40:(E) 270 407 High 95 84 High 9 High2F36-40(E) 191 243 High 91 85 High 15 High3.F_51-55(E) 235 247 High 71 86 High 15 High4M41-45:(E) 159 116 High 92 87 High 16 High5_F_31-35”E) 159 173 High 88 86 High 18 High6_F26-30“E) 117 189 High 92 82 High 24 High7F_36-40:(M) 123 82 Low 87 84 High 27 High8F_31-35:(E) 86 346 High 71 72 Low 34 Low9M41-45:(E) 105 20 Low 63 85 Low 35 Low

isial) 7962 Low 83 76 Low jo bow

;oy‘alk 104 61 Low 63 74 Low 4] Low

12_F26-30.E) 81 224 Low 63 68 Low 42 Low13_M_41-45:(M) 94 13 Low 48 83 Low 46 Low14F_26-30:(E) 53 20 Low 18 74 Low 53 Low15_F 31-35:(M) 69 19 Low 36 ~=—70 Low 56 Low

MEAN 128.33 148.13 70.73 79.73 31.40SD 62.90 125.69 22.84 6.61 14.91MEDIAN 105 116 71-83 34

Overall Collaboration

While the interactivity rating discussed above was a basic measure of

collaboration, a more detailed measure waspossible by adding measures provided

through the “Note Creation — Details report” from the Analytic Tool Kit. This report

provided greater details concerning the types of notes to which each participant

117

contributed. Within the course database, there were two typesofnotes: a single-author

note and a multi-author note. This report provided details on multi-user notes of which

there were tworelevanttallies. For instance, participants could initiate a note with

multiple authors. They would then be recorded as having beenthefirst to contribute to a

multi-author note (numberof groupnotesinitiated). Or, participants could contribute to a

multi-authored note that someoneelse had created (numberofgroup notes created). To

calculate the overall collaboration score, the numberof group notesinitiated and created

were ranked and then addedto the interactivity score.

Anoverall collaboration score, Table 21, was calculated by summingthe ranking

score of the following: numberof group notesinitiated, number of group notescreated,

and the interactivity score. Participants’ overall collaboration scores did not follow the

sameorderoftheir ratings. However, except for 8F_31-34 and 2_F36-40,all

participants who had a high participation rating ranked higherin their overall

collaboration score than those whohada low participation rating (rememberthat a lower

overall collaboration score meant a higher ranking). This meant that those higher ranked

participants tended to initiate and to contribute more to multi-authored notes as well as

reading and linking moreto all types of notes. Overall collaboration did not seem related

to epistemology or participant biographical data. The only exceptions were the 2

participants who had noprior exposure to Knowledge Forum nor had completed any

prior online course.

118

Table 21

Overall Collaboration Score

Overall Ranked score for Ranked score for

. Interactivitycollaboration group notes group notes

Participant ID score initiated created score4M41-45:(E) 11 4 4 31_F_36-40:(E) 13 2 4 76-F_26-30:(E) 16 2 3 117_F_36-40:(M) 20 6 2 123_F_51-55.(E) 21 7 4 105_F31-35:(E) 23 9 7 78F_31-35:(E) 23 1 1 212_F_36-40:(E) 27 9 10 810_M_41-45:(A/M) 33 7 9 1712-F_26-30:(E) 36 4 7 259M41-45:(E) 38 12 12 1413_M_41-45:(M) 44 12 11 2111-M_26-30:(A/E) 46 11 14 2114-F_26-30:(E) 50 12 12 2615 _F 31-35:(M) 55 12 15 28

Overall Edits

As wasarguedin the literature review, an indication of metacognitive activity was

regulation of cognition. In the case of online discourse, notes within the databaseclearly

represent the product of cognition. Assumingthat participants have put thoughtinto the

writing of their notes, and consequently, edit based on rereading and rethinking with the

goal ofmaking better arguments, these edits may be considered forms of metacognition.

Granted, most likely, not all edits were metacognitive in nature. Correcting simple

spelling and grammarmistakes would not be considered metacognitive.

Notwithstanding, an analysis of the notes edited and numberofedits was conducted.

The two measures used for overall edits were the numberofnotes edited, and

numberof edits. The difference between these types of edits is clarified in Figure 2.

119

, — created but nevercditedNote 1

Note 2 ~ created and reopened 3 timesfor editing

Note 3 — created and reopened 6 timesfor editing

Note 4 .— created but never edited

Numberofnoted edited = 2

Numberofedits = 9

Figure 2, Explanation of How Edits Were Counted.

The overall edits ranking was calculated by sorting each measure from mostto

least and ranking each from 1 (highest) to 15 (lowest). Ranked scores for each were

added to obtain an overall edit score. Higher ranked participants (lower score) meant a

greater numberofedits and notesedited.

Results are shown in Table 22. For the mostpart, participants with a higher

participation rating tended to edit more oftheir notes and tended to edit those notes more

often. Once again, overall edits did not seem toberelated to level of epistemological

understanding or participant biographical data.

Use ofAuthoritative Sources

Table 23 showsuse of authoritative sources per participant, as reflected in a simple count

of the numberofreferences to authors within their notes. For the most part, those with a

high participation rating referred to authoritative sources more often than thoserated low.

120

Table 22

Overall Edits

Participant ID Overall edits Notes edited score Numberof edits score

1_F_36-40:(E) 3 2 1

3_F_51-55:(E) 4 1 32_F36-40:(E) 7 3 4

8F31-35.(E) 8 6 2

5_F_31-35:(E) 11 4 76-F_26-30:(E) 11 5 6

12-F_26-30:(M) 13 8 5

4M41-45:(E) 15 7 87_F_36-40:(E) 18 9 9

11-M_26-30:(A/M) 21 10 1]10_M_41-45:(A/E) 21 11 109M_41-45:(E) 25 12 13

14-F_26-30:(M) 26 14 1215_F_31-35:(E) 27 13 14

13_M_41-45:(M) 30 15 15

Table 23

Use of Authoritative Sources

Participant ID Use of Authoritative Sources Count

1_F_36-40:(E) 502_F_36-40:(E) 313_F_51-55:(E) 194_M_41-45:(E) 485_F_31-35:(E) 456_F_26-30:(E) 57_F_36-40:(M) 28F_31-35:(E) 59M41-45:(E) 14

10_M_41-45:(A/M) 511_M_26-30:(A/E) 2012_F_26-30:(E) 713_M_41-45:(M) 2414_F_26-30:(E) 015_F_ 31-35:(M) 4

121

Who’s Read Whose Notes

The Analytical Tool Kit provided a break down of who had read whosenotes.

For each participant, the report gave the total numberofother participants’ notes read.

For analysis, these totals were converted to a percentage of the total numberof notes

written by that other participant. For example, say Bill wrote 100 notes and Sue read 80

of those notes, Sue would have read 80% of Bill’s notes. This allowed for a better

relative measure in order to compare between participants as how notes read were

relative to notes created. These percentages were calculated for each participant. As

could easily be predicted, these results were very similar to the interactivity rating. A

more meaningful measure was needed. Whowasreading whosenotesand potentially

modeling behaviours? How did the reading activity break down with respect to high

versus low participation rates?

To try to get answers to these questions, the percentages, as calculated for Sue

above, were split into two groups (those rated high and those rated low). Each

participants’ percentages were summedfor each of the two groups. These summed

percentages were then divided by the numberofparticipants in each of the groups (7 for

high and 8 for low). For example, for the high group, Sue read 80% ofparticipant 1’s

notes, 90% of participant 2’s notes, ... 70% of participant 6’s notes, and 80% of

participant 7’s notes. Participants | to 7 wereall rated high. The sum ofthese

percentages were then divided by 7 for an average of 80% of the notes read by Sue for

the high group ofparticipants. This same procedure was repeated for participants rated

low. This was doneto give an indication if participants rated high were "scaffolding"

participants rated low. It is given that this is a rough measure but the logic is as follows:

If students read more from their high rated students they would have a better chance of

benefiting from the modeling of the high rated students. This produced some very

interesting results shown in Table 24.

Participants whoseparticipation rating was high read more notes of those who

were also rated high than those rated low. Equally, participants whose participation

rating was low read more notes of participants who werealso rated low than those who

122

were rated high, without exception! This possibly meant that those rated high were the

models for each other more so than those rated low. The same seemed to hold true for

those rated low.

The pattern aboveis interestingly different than the most productive classroom

environments in which students were taking responsibility for the knowledge advances of

the classroom as a whole (Lax, Taylor, Wilson-Pauwels, & Scardamalia, 2004; Zhang et

al., in press). In these contexts students are working together on a complex, common

problem.

Table 24

Who’s Read Whose Notes by Percentage High vs Low Participation Rating and

Percentage of Professor’s Notes Read

% of notes read of

participants with a % ofProf’sparticipation rating of notes read

% of notes read of

Participant ID participants with aparticipation ratingof High

Low

1_F_36-40:(E) 96 92 97

2F36-40:(E) 91 88 94

3_F51-55:(E) 73 64 84

4M41-45) 93 90 95

5_F_31-35:(E) 90 87 93

6_F_26-30:(E) 92 89 97

7_F_36-40:(M) 88 86 95

8F31-35:(E) 71 77 83

9M41-45:(E) 60 60 64

10_M_41-45:(A/M) 81 83 90

11_M_26-30:(A/E) 61 63 63

12_F_26-30:(E) 62 63 67

13_M_41-45:(M) 44 55 60

14F26-30:(E) 16 29 27

15_F 31-35:(M) 32 44 46

The percentage of the professor’s notes read wasalso calculated for each

participant andis included in Table 24. This measure also provedto be telling. The

participants rated high in participation (with the exception of 3_F_51-55) read over 90%

of the professor’s notes contrasted to those rated low (with the exception of 10_M_41-45)

whoread less. This meant that those participants with a low participation rating likely

123

missed out on the modeling of online knowledge-building behaviour provided by the

professor. These measures did not seem to berelated to belief-mode epistemological

understandingor participant biographical data with the exception of the 2 participants

who hadno prior exposure to Knowledge Forum and whohad not completed any prior

online course. They both seemedto read significantly fewer notes than the others.

Who Built on to Whom?

In addition to data on who read whosenotes, the participant and interaction data

indicated who built on to someoneelse’s notes. In a similar manner to who read whose

notes, the high versus low participation rating, plus the professor, categories were

produced for analysis. The numbersofbuild-ons were simply added together for each

participant, based on the high/low split. As might be expected, more ofthe participants’

notes from the high overall participation group were built upon than those notes from the

low group. This was probably due to the waythat the participation rating was calculated;

those who wrote more notes tended to receive higher overall scores. Results can be seen

in the “uncorrected” columnsin Table 25.

To level the playing field, a recalculation was made that accounted for thetotal

numberofnotes created by each participant. When this factor was accounted for a

reversal occurred. In the “corrected” columns, numbers were calculated by dividing the

numberofbuild-ons by the total numberofnotes created by each participant (times 100

for a percentage), then summingeither the highs or lows and dividing by the respective

number(7 or 8) of each. As can be seen in Table 25, the notes of the participants who

showeda low participation rating were built onto more “per capita” than thosein the high

group.

124

Table 25

WhoBuilt on to Whom by Category of High Versus Low Participation, Build-onsto the

Professor, and Ratio of Build-ons to Notes

Uncorrected Corrected Build-on Build-onsParticipant ID to the to notes

High Low High Low professor ratio

1_F36-40:(E) 189 127 111 147 26 .682_F36-40:(E) 108 68 61 74 25 1.003_F_51-55:(E) 133 69 77 77 28 164M41-45“E) 88 60 52 66 20 645_F_31-35:(E) 100 51 58 57 16 916_F_26-30:(E) 77 48 47 57 20 1.167_F_36-40°M) 64 54 39 64 12 1.488F31-35:(E) 47 48 32 54 6 1.799M41-45:(E) 66 43 39 51 6 1.0310_M41- 7745:(A/M) 39 38 25 46 1311_M_26-30:(A/E) 62 29 34 30 5 1.1012_F26-30:(E) 27 22 16 24 8 1.0113_M_41-45:(M) 60 44 35 51 8 7614F26-30(E) 27 26 16 29 4 1.2315_F 31-35:(M) 29 19 19 23 2 1.04

The last two columnsin Table 25 provide the numberofbuild-ons to the

professor’s notes and the ratio of build-onsto total notes created. In the case of build-ons

to the professor’s notes, it was evident that the participants with a high participation

rating built on to a much greater extent. To assess “build on”, the numberof build-ons per

participant was divided by total numberofnotes created. In general, it can be said that

participants with a high participation rating tended to start more threads of discussion,

where as those with a low participation rating tended to build-on moreoften.

Week-By-Week Metacognitive Activity

Metacognitive activity was analysed week-by-week, Table 26. Special attention

was given to weeks where metacognitive directed questions were asked. As well,

metacognitive activity for the week whenparticipants were acting as coursefacilitator

125

was noted. In general, participants who wererated high in participation showed high

metacognitive activity with or without the questions, although they tended to show

increased metacognitive activity on the weeks where questions were asked that were

designed to specifically elicit metacognition. A participant rated high on participation

reflects,

In discussing metacognitive experiences I realized that this learningjournal encouraged metacognitive activity. The thinking about learningabout thinking about learning....is helping me learn about thinking about

learning :-) Thejournal exercise has also led me to some goalsettingfor

the week to come. (1_F_36-40:High)

In contrast, those participants rated low in participation showed a definite increase

in metacognitive activity during the weeks where metacognitve questions were asked.

However,these participants seemedless self-reflective about their learning (many fewer

metacognitive examples were coded for these students) overall. It seemsthat the

scaffolded questions helped participants to be more metacognitive, a position supported

by commentssuchasthe following,

I do enjoy using the learning log as a place to store my thoughts aboutthe

week andto encouragereflection. The questions posted help me to movein directions that I might not have thought of Without the questions, I

wouldprobably tend to do more ofa summary ofthe readings rather thanuse itfor application and understanding. (14_F31-34:Low)

Participants were left to develop their own questionsfor the last three weeks of

the course. Again, participants who were rated low in participation tended to display

greater metacognitive activity when prompted bythe directed questions (weeks2,3, 4, 6,

8, & 10). It was evident that there wasoften little to no metacognitive activity during

weeks where the provided questions were not metacognitive. A participant who was

rated as having a low participation stated,

Myfirst experience with these kinds ofreflectivejournalsfor assessment

was in the previous course I took with Clare. I must admit they are hard to

get used to and, althoughthey are a great tool to engage in metacognition,they are hardfor me to do on a regular basis. I know many others in the

126

course love working on them andfind them very useful, but myfeelingsare a little more mixed. (11_M_26-30:Low)

Howeverfor another participant, who received very high participation rating, the purpose

and value seemedcrystal clear,

Ifeel that I did a lot ofmetacognitive thinking about my own learning and

this was one ofthe main points ofwriting a learningjournal (wasn't it?) -to monitor our ownprogress andto clearly identify what issues we could

clear up and which ones we couldn't. Also to "rise above"by trying tointegrate our learningfrom various weeks. These are my impressions of

why weshould be writing learningjournals. (2_F_36-40:High)

Analysis was also conducted to see if metacognitive activity changedafter the

weekin which participants were acting as the course facilitator. The shadedcells in

Table 26 represent the weeks where participants were facilitators. There did not appear

to be any significant changes caused byfacilitation.

Table 26

Week-by-Week Analysis of Metacognitive Activity Within the Learning Logs

Numberof instances of metacognitive thought coded per week

Participant ID 73 4 5 67 8 9 «0 iM 2 21F36-40.) 1015 6 8 10.952F36-40:(E) 4 5 6 4 6 33_F_51-55:(E) 4 2 2 3 1 14M_41-45:(E) 7 4 45F31-35:(E) 27 2 26F26-30:(E) 5 4 77F_36-40:(M) 5 1 5 18F31-35:(E) 2 1 1 39M41-45:(E) 3 2 110M41-45:(A/M) 5 111M_26-30(A/E) 4 412F26-30:(E) 2 313M41-45:(M) 3 214F26-30:(E) 315, F 31-35(M) 33

Notes: “Bolded week numbers represent weeks where a metacognitive question was

asked. * This participant combinedher learning log responses for weeks 10 to 13.

First Half Versus Second HalfComparison

127

To enable the comparisonover time for changesin revisions, views worked on,

percentage of notes read, and notes created, the participants’ totals were added for the

first half of the course (weeks 1-6) as were their totals for the second half (weeks 7-13

excluding week 12 as this week was conducted in another database). An additional time

frame (finer grained) of weeks 1-3 and weeks 10, 11, & 13 was calculated for percentage

of notes read and notes created. Results are displayed in Table 27.

Table 27

First Half Versus Second Half Comparison of Revisions, Views Worked On, Notes Read,

and Notes Created

Views %notes %notes Notes

Number worked read read Notes created

of on I‘half weeks created weeks

revisions 1“half -—2"4 14+24+3- “half 14+2+3-Participant ID I“half- -—2" half weeks ~2" weeks

2 half half 10+11+13 half 10+11+1394-95 94-94 134- 63 — 79

|_F_36-40:(E) 194-193 31-19 10969-75 67-71 67 -— 30-59

2_F_36-40:(E) 65-147 13-18 10664-71 64-72 97 - 52 — 66

3_F_51-55:(E) 96-110 18-19 1094M41-45:(E) 34 — 54 14-15 81-67 78-56 75-65 35-42

5_F_31-35:(E) 82 — 65 21-20 70-76 81-77 76-61 35-41

6_F26-30:(E) 113-51 17-15 83-86 82-82 60-47 25-21

7_F_36-40:(M) 29 — 42 11-12 81-76 77-67 58-54 32-33

8F_31-35:(E) 88-167 13-11 70-42 67-67 34-46 20-39

9M41-45:(E) 7-8 14-13 54-46 61-47 57-40 42-35

10_M41- 78-75 77-69 24 — 2045:(A/M) 31-20 16-16 42 —2811_M_26-30:(A/E) 34-20 12-13 51-67 32-63 54-39 23-24

12_F_26-30:(E) 72 —60 15-13 32-30 35-20 42-27 31-2113_M_41-45:(M) 6-5 10-17 35-39 14-14 33-50 18~3614_F26-30:(E) 8-6 10-13 12-17 33-11 18-27 11-20

15_F31-35:(M) 7-1 11-11 33-25 33-25 33-22 20-7

128

Case Analysis for Participants Who Showed — Change in Rated Belief-ModeEpistemology

The second of my main thesis questionsstated, “If there are signs of

epistemological change within individuals, how do these changes manifest themselvesin

the individual actions and interactions amongstlearnersin the online learning

conference?” To help answerthis question the actions and interactions of the 5

participants who showedindications of belief-mode epistemological change were

analysed. Twoofthese participants had a high participation rating (2_F_36-40 &

6_F_26-30) while the other 3 had a low rating (11_M_26-30, 13_M_41-45 & 14F26-

30). They represented an interesting range of characteristics: male and female; three

different age groups; and, a wide range of previous Knowledge Forum use, online

courses completed, and courses completed towardstheir degrees.

The following analysis was conducted:

. Whatepistemological changes occurred?

. Was there a changein type ofjustifications used?

. Whose notesdid they read the most?

. Percentage of other participants’ notes read;

1

2

3

4

5. Percentage of professor’s notes read;

6. On whomdid they build?

7. Week-by-week analysis for revisions;

8. Week-by-weekanalysis for views worked on;

9. Week-by-week analysis for notes created;

10. Metacognitive activity over time; and

11. Metacognitive activity directly following course facilitation.

An overall comparison of the above analysis is presented in Table 28.

2_F36-40:High

This participant showedthe clearest indication of belief-mode epistemological

change. Her pre-essay level was rated as multiplist whereas her post-essay wasrated as

129

evaluativist. She had five explicit examples of belief-mode epistemological change, the

second highest for this study. She showed the greatest movementin the areasofcritical

thinking, reality, and the influence of others. For example, she used anotherparticipant

to help scaffold her thinking about the grey areas (she wasa self-proclaimed black and

white thinker),

So thankyou my goodfriend Betsy, not onlyfor your answers andthe long

time it must have taken you to put them together, butfor showing me a gapin my gameplan. Thefirst step in overcoming a weaknessis to recognize

it (no I’m not an alcoholic :) so that is why this week was monumentaltome. I’ll have to put in some quality metacognitive thinking about how toovercomeit... (2_F36-40:High)

Asfor reality, she clearly showed movementfrom a one-reality world to a world with

multiple realities in this explicit example of belief-mode epistemological change,

Ifthere is a body ofknowledge existing outside ofus in the world, we will

never know it as it exists. We will only know what we have learnedofit,coloured by our own experiences andperception, and biased by our

misconceptions. Everyone’s knowledgeis relative. Ifind thisfascinating.I used to be something ofan objectivist. (2_F_36-40:High)

Further, as this statementindicates, there is a clear understanding that things need to

change, and that she needs to do more,but no indication of an idea that is improvable and

that she will work to advance.

This participant’s justifications decreased from 3 to 2 examples ofdialectic

reflection, however, there were no experience-basedjustifications in the post whereas

there were 5 such justifications in the pre-essay. While on the surface this lookslike a

decreasein justifications, the results could be interpreted as an increase, as she only

presented dialectically reflective justifications in her post-essay and not the purely

experience-basedjustification that were abundantin her pre-essay.

This participant had used Knowledge Forum on morethan four previous courses,

had completed three online courses, and had completed three courses towards her MEd,

prior to this course. Forthefirst three weeks of the course she created 30 notes and for

the last three weeks she created 59 notes. Notes created the first half (67) versus the

130

second half (106) of the course revealed a large difference as well. She also showed an

increase in notes read duringthefirst half (69%) comparedto the second half (75%).

This might have been dueto her being a coursefacilitator during week 10.

This participant read 91% of otherparticipants’ notes, slightly favouring high

participants. She read 94% of the professor’s notes. While these overall percentages

might seem contradictory to the first versus second half percentages of notes read, which

were lower, it was dueto the fact that this participant kept reading notes and went back in

the course discussion to read, after the course wasfinished. The final summary of-

participation and interaction data was produced about three months after the end of the

course. So while she did not get to read all the notes during the course, she did continue

reading after, which showed excellent perseverance and commitmentto learning. She

built most upon the professor (25), 3.F_51-55:High (23), then 1_F36-40:High (21),

finally herself (16). She built more upon high than lowrated participants and there were

examples whereshe definitely learned from others (scaffolding). During the week she

wasfacilitator (week 10) she had almost double the numberofrevisions comparedto her

average but there was no difference from her average in the numberofviews worked on.

Metacognitive activity wasfairly consistent from week to week with the week

where she wascourse facilitator being the most active. There were no significant

changes for the weeks where metacognitive learning log questions were asked. Overall,

this participant had the fifth highest metacognitive activity with over half being

regulation activity.

6_F_26-30:High

This participant showed some movementepistemologically but only slightly. In

belief-mode she wasinitially rated as bordering between multiplist and evaluativist

ending up clearly evaluativist in the post-essay. She had three explicit examples of

belief-mode epistemological change. Her development was evidentin the confidence of

her answers andthe level ofjustifications she provided, as well as the influenceofothers.

For example, she comesto realize the importance of others for her learning,

Throughout the course, but more so toward the end, I began to realize that

131

the learning situation — especially on-line — is completely dependent on

the individuals that establish the community. The learning situations arecreated by the different types oflearners with whom I communicate.

(6_F26-30:High)

This participant showed a definite increase in the sophistication of her

justifications between pre- and post-essays for belief mode. She doubled the numberof

dialectic reflections and decreased the numberof experienced-basedjustifications from 3

to 0. However, for design mode there wasnot a parallel shift. As suggested in the note

above, there was understandingthat “the learning situation - especially on-line - is

completely dependent on the individuals that establish the community.” This seems

clearly an improvable idea, although not recognized as such. Is the learning environment

completely dependent on the individuals? Whatis the role of the teacher? Can some

unpromising communities - from the perspective of individuals involved - be

transformed? If so, how’? There are many and fascinating design issues here, butthis

participant doesnot tackle them.

This participant had used Knowledge Forum for one previous course and had

completed three online courses prior to this course. This was her fourth course towards

her MA.Forthefirst three weeks of the course she created 25 notes and for thelast three

weeks (week 10, 11 & 13, as week 12 was completed in another database) she created 21

notes with a slightly greater difference whenthefirst half (60) and second half (47) were

compared. Shedid a great job of reading otherparticipants’ notes—readingat least 92%

of others’ notes, slightly favouring high over low rated participants, althoughit is hard to

interpret this finding clearly when she read such a high percentage of the database. Her

reading rate was the sameforthe first and last three weeks (82%) but wasslightly higher

for the second half (86%) when comparedto thefirst half (83%). She read 97% of the

professor’s notes. She built upon the professor’s notes the most (20) then 1_F_36-

40:High (13) and 7_F_36-40:High (13) and then herself (11). So, she built more on high

than low rated participants.

There wasalmosta four-fold increase from her average in the numberofrevisions

made for the week when she wasa facilitator. While, she participated in slightly fewer

views that week, she did create slightly more notes than her average. This participant

was more metacognitive during the first half of the course and was even more so when

132

prompted by the learning log questions. She also showed the most metacognitive activity

the week when she facilitated.

11_M_26-30:Low

This participant was very difficult to rate for level of belief-mode epistemological

understanding. While he showed overall movement from multiplist to evaluativist he

maintained someabsolutist characteristics throughout. His absolutist views were around

the nature of reality and the essence of knowledge. For instance, he believed that the

teacher should “play the role of guide and facilitator of knowledge” but at the same time

that there exists some form of absolute knowledge,“knowledge is very important to

learning, but it is important to recognize that not all knowledgeis absolute andstatic”.

Andfurther that “while the truth about reality may be difficult to achieve, it can be made

easier...”. This time he was seemed unsure and wasstill clinging to some form of

absolute reality. These contrasting beliefs placed him in the turbulenttransition stage.

However,his use ofjustifications between the pre-and post-essays showed a marked

increase in sophistication. He used purely experience-based justifications in the pre-

essay but progressedto dialectical reflection in the post-essay.

In the following example,this participant highlighted the need for a more

constructivist pedagogy and what might be a benefit. He suggests that individual and

collective goals should be combined, but does not go beyond that simple suggestion.

How would teachers go about making this happen? How would combining goals account

for the goals with in mandatory curriculum? While a good suggestion was made,there

wasnot follow up with design-mode considerations, nor was there any sense that this

should be done.

Teachers who encourage students to engage in questioning and learningfrom each other, and who do so in a way which ensures that students movein the direction offulfilling knowledge objectives, will more likely end up

with more students going beyondtask goals or even learning goals. I maybe offthe mark here, but I would think that when individual learning goals

are combinedto develop collective learning goals, students will be on the

path towards knowledge building. (11_M_26-30:Low)

133

This participant had used Knowledge Forum on more than four previous courses

and had completed more than four online courses. This washisfifth course towardshis

MEd.Forthefirst three weeks of the course he created 23 notes and forthe last three

weekshe created 24 notes. However,thefirst half (54) compared to the second half (39)

told a different story. He read about double the percentage of notes the last three weeks

(63%) as comparedto thefirst three weeks (32%). There wasa slightly smaller

difference for the second half (67%) comparedto the first half (51%) of the course. He

tendedto read just slightly more of the lowest 4 participants and did not read any more

than 70% of anyone’s notes. He read only 63% ofthe teacher’s notes. He built the most

on 1_F36-40:High and 2_F_36-40:High (16 each) then 3_F_51-55:High and 7_F36-

40:High (9 each).

There wasno difference from his average in the number of views worked on, nor

numberof revisions made for the week when hewasa facilitator. Further, his

metacognitive activity did not significantly change from thefirst six weeksto the last six,

nor whenhe wasfacilitator. However, his metacognitive activity did increase the week

when metacognitive questions were askedin the learninglogs.

13_M_41-45:Low

This participant showed movementaway from an absolutist perspective that was

present during his pre-essay towards a solid multiplist view on his post-essay. In his pre-

essay he stated that “Learning [knowledge] can be transferred...” and “I gave them

knowledge...” inferring that knowledge was an entity on its own that can be easily passed

from one individual to another. By the post-essay, his view of knowledge had changed,

As a result, you may have believedyour reply to a question was the only

correct answerbut only tofind out there were other answers to the samequestion because we were evaluating the questionfrom differentknowledge backgrounds. (13_M_41-45:Low)

This participant’s essays showed 1 exampleofdialectical reflection each when

coded for justification. However, there wasalso a decrease in experienced-based

justification from pre- to post-essay. As argued above, this can represent an increase in

134

sophistication in the use ofjustification.

Concerning design-mode epistemology,this participant showed a good

ability to critique the theories and ideas often noting areas for improvement, as

demonstrated below,

Accordingly, I believe it is dangerous to think ofthese student goalorientations as student “dispositions” towards education. Labeling

students is dangerous and can be very harmfulto a student’s selfesteemandself-confidence. (13_M_41-45:Low)

Heidentified an improvable idea and describes the surrounding problem.

However, he did not offer any solutions nordid he feel that it was his responsibility to

ensure that labels were not given to students. What are waysthat student labeling can be

prevented? How could labeling be designed so as to minimize harm to student’s self-

esteem andself-confidence?

This course was the third time that this participant had used Knowledge Forum,

washis fourth online course, and washis fourth course towards his MEd. Forthefirst

three weeks of the course he created 18 notes and for the last three weeks he created 36

notes. His note creation showed the sametrendfor the first half (33) compared to the

secondhalf (50) of the course.

He wasvery consistent in his reading of notes for the first three weeks andlast

three weeks at 14% each. He read slightly more notes the second half (39%) compared to

the first half (35%). However, he read only 48% ofparticipants’ notes and only 60% of

the professor’s notes. He read moreofthe notes from low than from high participation

rated participants and built most on 1_F_36-40:High and 8F_31-35:Low (10), and then

7_F_36-40:High. While this participant wasfacilitator, a separate database wasused,

and hence,his data for revisions and views worked on during this time is not an accurate

accountofhis activity and was not reported here. Howeveras the learning logs were part

of the main database,this data is a good reflection of metacognitive activity. He was

more metacognitive during the first six weeks, showed about the sameactivity for the

week he wasfacilitator, and definitely showed increased activity during the weeksthat

metacognitive questions were asked.

135

14_F_26-30:Low

This participant showed slight movement from a multiplist to an evaluativist level

over the course. However, her pre- and post-essays were very short and henceI wasless

confident in this rating. The rating was based on her expanded view ofthe influence of

other in her learning,

My original definition oflearning was based on individual learning - therewas no mention oflearning that takes place in communities or distributedcognition. As mentioned in my most recent learningjournalentry, this

piece was what Ifelt I was missing in my thinkingfor my MRP topic. Theacknowledgement and understanding that learning and knowledge

construction takes place as the organizational, community in addition to

the individuallevelis key to articulating, understanding, describing andanalyzing the health messages putforward by the news media. (14_F_26-40:Low)

This participant did not use any dialectical reflection in either essay. She

increased the numberof coded experienced-basedjustifications by 1 from pre- to post-

essay. This seemsa contradiction to what one might expect from the growth in

epistemological understanding showedbythis participant and was not explainable by

Kuhn’s (2000) model.

Concerning design-mode epistemology,this participant showed somelevel 2

activity,

Assuming that knowledge building is a worthwhile pedagogy, what onething would makethe greatest difference to its widespread use within oursociety?

I think that the one thing that would make the greatest difference to thewidespread use ofknowledge building within our society would be to de-emphasize the individual, competitive nature oflearning that is so highlyvalued in our society.

Too often, group workis assignedat all levels offormal education

(elementary, high school, post-secondaryinstitutions) without instruction.

It is assumed that students will understand the value and importance ofworking together. Skills to help students work together are not introduced,

136

taught, or developed. The very reasonfor students working together is notshared, besides “You will need to work with other people in the

workplace.” Everything students learn in school is geared towardsindividuality and competition.

Skills to help students work together and understand the value of

collaboration should be taught beginning in elementary school. Unitsshould include an investigation ofthe history ofthe knowledge in question,

and an understanding the role that many people over time played in the

building ofknowledge; that discoveries are never made by one person.(14_F_26-40:Low)

The “teach skills” idea seemsless like design and morelike a “teach primitive.” But the

participant was also contemplating various waysofteaching that were suggestive of

design mode. For example, the idea of engaging students in the “history of knowledge”

suggests a re-conceptualization of the problem in design terms, as the intervention

addresses the problem in a new way. However,the original question was provided as

part of the weekly learning log questions. It is unclear if this participant would have

discussed this problem had she not been prompted.

This course wasthefirst time that this participant had used Knowledge Forum,

washerfirst online course, and washer last course towards her MEd. Forthefirst three

weeksof the course she created 11 notes and for the last three weeks she created 20

notes, and 18 notesthe first half compared to 27 notes the last half of the course. She

created only slightly more notes during the week she wasfacilitator (5 notes created

comparedto her average of3.5).

Herreading rate was 18% for the first three weeks and 27% forthe last three

weeks. This was the opposite trend whenthefirst half (33%) was comparedto thelast

half (11%) of the course. She read only18% of the course notes and only 27% of the

teacher’s notes. She built most on 7_F_36-40:High(8), then 8F_31-35:Low (6) and

1_F_36-40:High (6). She revised her notes once (average of 1.71) and worked only in

one view during the week whenshe wasa facilitator. This meansthat she did not

contribute to her learning log that week. Her metacognitive activity was about the same

for the two halves of the course and she showedincreased activity during the weeks that

metacognitive questions were provided.

Summary ofCase Analyses

137

There were few noticeable patterns that emerged amongthe participants that

showed belief-mode epistemological change. Reading patterns, revisions, and note

creation all showed wide ranges. This wasalso true for the coding of the metacognitive

activities. There were only two similarities that emerged acrossall participants. First,

they all built onto the notes of participant 1_F36-40. This participant was one ofthe top

3 people upon whosenotes they built and of these top three personsat least 2 of them had

high participation ratings. Second, these participantsall explicitly noted that their notion

of the importance or influence of others, especially peers, had changed overthe course.

Specifically, that their peers had now becomelegitimate sources of knowledge.

Table 28

Comparison of Participants Who ShowedSignsofbelief-mode Epistemological Change

2F36- 6_F26- 11_M_26- 13_M41- 14F_26-40:High 30:High 30:Low 45:Low 30:Low

Whatbelief- M-E; M -E; A/M — A/E; A/M - M; M-E;

mode 5 explicit 3 explicit 4 explicit | explicit 1 explicitepistemologic examples; examples; examples; examples; examples;al change Critical Confidence of Critical Multiple Influence of

occurred? thinking, answers, thinking, how correct others

reality, and justifications, knowledgeis answers and

influence of and influence generated, and influence of

others of others influence of others

others

Wasthere a Decreasein Double the Moved away Decreasein Increase in

change intype experience- number of DR from experience- experience-

of based and moved experience- based based

justifications away from based and

used? experience- ended with

based DR

Whosenotes 91% high 92% high 61% high 44% high 16% high

did they read 88% low 89% low 63% low 55% low 29% low

the most?

Percentage of 91 92 63 48 18

other

participants’

notes read

Percentage of 94 97 63 60 27

professor’s

notes read;On whom did Prof, Prof, 1_F_36- 1_F36- 7F36-

they build? 3 F 51- 1 F 36- 40:High, 40:High, 40:High,

138

Revisions (perhalf)

Views worked

on (perhalf)

Notes created

(3-week) &

(per half)

Metacognitive

activity over

time

Metacognitive

activity

directlyfollowingcoursefacilitation

55:High,|_F_36-40:High

65-147

13-18

30-59

67-106

- Constant

- Not

dependenton

questions

asked- Not available

40:High,

7F36-

40:High,

113-51

17-15

25-21

60-47

- Decreased

over time

- Dependent

on questions

asked

- Increased

activity

$5:High and

7F_36-40:High34-20

12-13

23-24

54-39

- Constant

- Dependent

on questions

asked

- Constant

8F31-

35:Low, and

7‘F36-

40:High

6-5

10-17

18-36

33-50

- Decreased

over time

- Dependent

on questions

asked

- Increased

activity

8F31-

35:Low, and

1F36-

40:High

8-66

10-13

11-20

18-27

- Constant

- Dependent

on questionsasked

- None

CHAPTER5: CONCLUSIONS, IMPLICATIONS, AND FUTURE RESEARCH

The problem motivating this research on epistemological developmentis the

sustainability of online knowledge building communities. A knowledge building

community is a special kind of community - one in which memberstake collective

responsibility for sustained idea improvement. According to the Merriam-Webster online

dictionary, sustain means“to give support to; to supply with sustenance: nourish; to

support by adequate proof.” This definition, when applied in the context of online

communities and knowledgebuilding, involves supported discussion and feedback

between membersin order to support and nourish each other’s ideas. It also brings with

it a commitmentto the public life of information, where ideas belong to the world, not

just to the individual mind. It entails a commitmentto explore the ideas of the local

community and beyond - engaging authoritative sources constructively rather than

settling in on personalbeliefs - or even the belief system of the local community - but

ignoring the world of ideas beyond that community. Instead, participants anticipate

limitations in their current views, monitor writings of others to gain some sense of what

those beyond the community are saying and identifying as problematic, etc. More

generally, they mine available resources so they can workat the cutting edgeofthe field,

as they understand it. They appreciate that today’s accounts will be superseded.

Understanding that helps them, for example, to attend to what a concept(e.g., social

constructivism) can’t help them explain or do, as well as whatit offers. Working with

current understanding and anticipating advancesis part of the knowledge building design

process.

So, how are the ideas within the community sustained? I believe community

members must be open to new ideas and open to improvementof their own ideas. Hence,

one must be accepting of others’ opinions and feedback — consider other community

membersas valid sources of knowledge and feedback, and have some way of

determining whatis valid. Further members must be comfortable working in design-

mode. Someindicators in this research that implied sustainability were the number of

edits of an original idea (note), the links back to other notes and discussionsthat build

upon an idea or help bolster another, and most importantly, the increase in level of belief-

139

140

mode epistemology. Specifically, this increase in epistemological understanding

involves justification and acceptance of others as valid sources of knowledge. In fact, the

realization of the importance of others becomesa reliance on others for furthering of

ideas, in a true social constructivist sense of knowledge. Further, metacognition is a

revisiting of some content or mental processat a higher level, which again could imply

“giving support to” an idea. This is a complex and multi-faceted problem yet onethatis

at least partly dependent upon individual epistemology. As oneparticipant expressedit,

To get back to the question then, about what would make the greatest

difference to the widespread use ofknowledge building within oursociety... a change in perception about the nature ofknowledge and our

approachto learning in general. (8_F_31-35:Low)

If one accepts the arguments for the link between level of epistemological

understanding and sustainability, it follows that epistemic agencyis linked to

sustainability, as community memberswill need to know howto “negotiate a fit between

personal ideas and ideas of others” through the understanding of what constitutes

justification and “using contrasts to spark and sustain knowledge advancement” by

acceptance ofothers as valid sources of knowledge, and constructive use of authoritative

sources. Hence, another important factor, as stated in chapter one, was howindividuals

become their own ‘educators’, understanding their own epistemic assumptions.

Moshman(2003) advocated that learners must develop an “explicit understanding of the

fundamental nature andjustifiability of knowledge and reasoning”(n.p.). This was what

he called metacognitive development. Metacognition was coded using the learning logs.

Closely related was dialectic reflection that was investigated via the participants’

justifications used in describing their theories of learning. These two factors were also

used for comparison to the participation and interaction indicators. Lastly, this research

looked at someofthe interactions between participants and howthis related to

justifications used and metacognition.

This research succeeded in exploring these areas, albeit with a very limited

population and sample size, and yielded productive indicators that provide direction for

further research. However, no causal conclusions can be made about the influence of the

learning logs on development, nor can webecertain about the durability of the effects

141

seen within this research. Further, the nature of the course content for the main research,

that is constructivism,likely influenced participants to some extent. Different course

content may have produceda very different developmentaltrajectory. Providentially,

this research generated numerousquestions, which will be explored in the following

sections of this chapter. Conclusions specific to the research questions are presented

below,along with their significance and further limitations.

Are Changesin Belief- or Design-Mode Epistemology Evident over the Duration of the

Courses?

Differences between pre- and post-essays indicated belief-mode epistemological

developmentfor 5 participants. This is an important finding given that 4 of these

participants had completed at least 4 courses towards a graduate degree. One might

expect that students, more than half way though their graduate level courses, would have

already developed a more sophisticated view of knowledge,especially in the field of

education, where discussion of curriculum, learning and knowledge are frequent. In fact,

of the 7 participants rated as evaluativist during the pre-essay, 5 of them had completed 4

or more courses. Hence, any movementwas seen as noteworthy. For those who showed

movement, could it have been explained bythe effect that Knefelkamp (1999)called

functional regression? — “the phenomenon seen when adult learners undertake new

learning in a new learning environmentand ‘functionally’ regress to multiplistic thinking

until they feel comfortable in the new environment(wefirst observed this with graduate

students)” (p. xviii). As described earlier in this thesis, multiplistic thinking is whereall

are considered to have an equalright to an opinion and where no judgments are made

between opinions. In this way, starting graduate school might be likened to freshman

year of university. This could explain why somestudents showed belief-mode

epistemological change, especially for the 2 older participants. The 3 participants who

were aged 26-30 might not have been out of school long enough to have regressed

(assuming that they had reached the evaluativist level by the end oftheir last

programme). However, from these data, such interpretations must remain only tentative.

It is not possible to identify what caused the changes in epistemological views and it was

142

not within the scope ofthis research to try to explain why change occurred. Simply put,

it can be said that a changein level of belief-mode epistemological understanding,as

described by Kuhn etal’s (2000) model, was determined for 33% ofparticipants ofthis

study. Eighty percent of this movement was from multiplist to evaluativist. Further,

whenthe 7 participants who werealready at the most sophisticated level are excluded

(i.e. they were rated as evaluativist during both the pre- and post-essays), the change was

63% of participants. Overall then, change wasevident.

Ofinterest too, was the finding that belief-mode epistemological change occurred

across a wide spectrum of age, Knowledge Forum usage,prior online experience, and

programme completion. It would take a much larger population to make any substantive

generalizations. However, based on the limited numberofparticipants it can be said that

there seemsto be a broad range of belief-mode epistemological levels even in graduate

school.

The tension between supporting, sharing, and evaluating beliefs and taking steps

to address weaknesses and move beyondcurrentbeliefs is, in many ways, the tension

between working in belief mode and working in design mode. Thelatter requires a

commitment to public and shared work with ideas. I provided several accounts of a shift

in epistemology from an objectivist conception of knowledge(a static, objective, “true”

state of knowledge) to a view of knowledgeas continually evolving. But there was not

evidence of corresponding movementto a “World 3” accountof “objective knowledge”

which,in the Popperian sense, involves a world of ideas that are explicit, public, and “out

in the world” where they can be worked on by others and continually improved. In

Popper’s World 3, ideas have a life beyond the mind of the person whoset them in

motion; placing them there helps increase the chancesoftheir being taken up by others

and continually improved. Knowledge builders frequently try to assess the “out in the

world”nature of ideas, attempting to ascertain the cutting edge of their community

knowledge as well as the cutting edge of knowledgein the field as a whole. Treating

“social constructivism” as a newfound “objective truth” rather than an idea to be

improvedrepresents the difference between objectivist in the “truth” or “static sense” of

the word, and “objective knowledge”in the dynamic, public, continually improvable

sense. Thelatter, design mode conception brings with it a commitment to advancethe

143

profession, not simply one’s work with in the profession. I saw less movementalongthis

design-modedimensionthat along the belief-mode dimension. However,as indicated in

Chapter 1, this was to be expected. The “new knowledge challenge” that underlies

knowledge building and design modeisitself a new challenge, as suggested by

Scardamalia, 2004:

Historically learning has constituted an adequate objectivefor educationbecause knowledge has not been thought ofas growing, but rather indangerofbeing lost. With the information revolution we have

unprecedented growth in information and a corresponding need to keep

students abreast ofrapid changes. But even this challenge pales bycomparison to the ‘new knowledge’ challenge that requires staying up todate as a prerequisite to contributing in your ownrightto the cultural

wealth ofsociety. Preparing studentsfor knowledge generation represents

a radically different challengefor education—differentfrom both theancient challenge ofcultural transmission and the more recent challenge

oflifelong learning. (n.p.)

Implications ofLevel ofEpistemological Understandingfor Practice

Faculty and instructors might wantto take this range of epistemological levels

into account in the design of course activities. As suggested in the data reported,

collaborative work, where students rely on each other for knowledge, may be experienced

as problematic by students whoare absolutist in their thinking. Further, muditipilists, will

need very specific guidance about evaluating and judging others’ opinions and shared

knowledge. This research seemed to confirm that one cannot expect students with an

absolutist epistemology to respond well to certain teaching techniques prescribed by

constructivist pedagogies nor to cope well with the “grey areas” found at the leading edge

of research. However, carefully designed pedagogies and the use ofdirected learning

logs can support students’ in their epistemological development.

Meacham (2003) stated

manyprofessors overestimate the level ofintellectual development oftheirstudents. They teach at too high a level and then are disappointed withthe students’ oral and written performance. Kuhn, Schwartz and Fisher,and Moshmanagree that we can do a betterjob offacilitating student

144

learning ifwe attend less to rearranging course content and more toadjusting our teaching methods to betterfit students’ current intellectual

levels. (p.6)

Understanding students’ intellectual levels is not an easy task, making effective

assessmentofthese levels an important research objective. Additionally, I believe that

understanding howto guide learners to the evaluativist level of epistemological

understanding is important to sustained lifelong learning. Moreover, understanding how

to guide learners to adopt a design-modeattitude is important to sustained improvement

of ideas. Once learners can effectively judge others’ justifications, they can more

effectively navigate the enormous amounts of information and opinions available. This

ability provides a sense ofpower and motivation. Kuhn (2003) makes the argumentthat,

Iffacts can be ascertained with certainty and are readily available toanyone who seeks them, as the absolutist understands, or ifany claim is asvalid as any other, as the multiplist understands, there is little point toexpending the mental effort that the evaluation ofclaims entails. Only atthe evaluativist level are thinking and reason recognized as essential

supportfor beliefs and actions. Thinking is the process that enables us to

make informed choices between conflicting claims. Understanding this

leads one to value thinking and to be willing to expendthe effort that itentails. (n.p.)

This effort could be interpreted as the pursuit of higher education or some form of

technical or artistic apprenticeship. But perhaps this effort may be extended to a broader

range of life experiences in the form of continued questioning and investigation even

outside formallearning situations. Regardless, at the evaluativist level of epistemological

understanding, one realizes that a community’s progressive discussion is fundamental to

the advancement of ideas. Hence, the involvement with some kind of community

becomes fundamental to sustainability ofideas.

Implications ofLevel ofEpistemological Understandingfor Theory

The participants themselves, through explicit self-admission of epistemological

change, seemedto validate the changes that were found. Each ofthe 5 participants who

showedchange from their pre- to post-essay was also coded as having explicit examples

145

of epistemological change as shown within their learning log data. However, there were

an additional 6 participants who also provided explicit examples within their learning

logs but were not rated as having shown a difference in levels between pre- and post-

essays. This disparity points to the problem ofusing explicit self-admission of

epistemological changeas a sole indicator. Overall, it is fair to say that coding for levels

of belief-mode epistemological understanding wasdifficult. While Kuhn’s model was

theoretically very simple and seemedto fit very well to the population usedin this study,

it did not facilitate the easy differentiation of participants at the more sophisticated

multiplist and evaluativist levels. Hence, the coding protocol included several additional

characteristics derived from the other models reviewed in chapter 1 in orderto classify

participants. Notwithstanding, Kuhn’s model did proveeasier to use than Baxter-

Magolda’s model that was usedfor thepilot research.

Oneaspect that neither model adequately addressed was ‘how and when’

epistemological change takes place. What role does metacognition play in change? Is

attainmentofthe highest level of epistemological understanding common between

knowledge domains? Kuhn et al. (2002) has noted differences between the orders in

which epistemological development occurs between knowledge domains. Are some

domains more conducive to epistemological developmentor is development more

dependent on pedagogy? Answering these questions could lead to the creation ofa

pedagogical tool to aid students in epistemological development.

In this research, the secondrater (for the belief-mode analysis) had a level of

epistemological understanding somewhere between absolutist and multipilist (this level

wasbased on extendeddiscussion with the rater during training). During the training

session, it was sometimesdifficult to agree on multiple realities, as the second rater

strongly believed that there was one ultimate reality. Despite this difference, a high

interrater reliability was obtained. However, this highlights potential problems when

researching epistemological development. Dothe raters need to be at the highest level of

Kuhn’s (2000) model? Logic would seem to suggest that unless one was functioning at

the higher levels, one might not be able to rate those levels adequately.

This research also led to the question:Is it useful to study a social phenomenon,

such as knowledge building, by using an individually oriented model of epistemology?

146

Using an individually based model to explore a social phenomenonis akin to using just

the stock marketor just the inflation rate to measure a country’s economy. While it gives

someindication, it cannot do properjustice to the complex nature of social phenomenon.

Certainly a combination of these individually based models increases the validity of

conclusions, and that was the method usedin this research. The addition of measuring

design-mode epistemology strengthens the case. Unfortunately, social epistemological

models are not well defined andthereis little agreement between what constitutes a

community/social epistemology or even what should be the emphasis/purpose of such a

philosophical domain (Goldman, 1999, 2001). Will such a social epistemological model

be applicable to all social circumstancesorwill they be different depending on the

community? I believe that much more needs to be explored before such a model could be

used for actual research.

The above argumentleads to the questions of how individual models of

epistemology have becomeso dominantin the literature. Does the movement between

levels, as described by Kuhn,represent the primary shift in epistemology? Could there

not be multiple epistemological streams that develop in parallel? Idea improvementor

design mode seemsto be another viable form of epistemological development. However,

participants in this research seemed to reach a new “absolute” beyond whichthey could

not move — the ability to improve uponthe ideas andtheories from theliterature. Why

were design mode examples virtually absent from the data analyzed in this research?

What Were Some Manifest Characteristics of the Epistemological Levels?

In general, evaluativists showed the highest levels of total metacognitive activity.

This observation was consistent with Moshman’s (1998) theory of cognitive development

where active metacognition waspervasive at the higher levels of reasoning. Overall,

coded metacognitive activity showed a greater relation to participation rating than to level

of belief-mode epistemological understanding. Hacker (n.d.) discussed the possibility

that regulation could become automatic through frequentuse or over learning.

Regulation behaviours may well accountfor differences in participants’ metacognition.

Analternative explanation is that those with fewer metacognitive passages coded did not

147

record their thoughts, even though offline they may have been more metacognitive. In

support ofthe latter interpretation, one participant did mention that she kept a private

journal where she felt more comfortable in divulging her innermost thoughts. However,

in general the use of directed questions would seem to negate the effect of these alternate

explanations. Directed questions explicitly asked participants to record their thinking or

regulation of thinking. Evaluativists who stated that their preference was for non-

directed learning logs, showed higher metacognitive activity in the final three weeks of

the course (when no directing questions were asked). This is consistent with the

characteristic behaviours in the epistemological models discussed in chapter 1, where at

the higher levels of epistemological development, learners become committed and active

in charting their own learning course.

Throughthe explicit examples of belief-mode epistemological change, 9 of the 11

participants reflected on their newfound understanding of the importance of others’ in

their learning. Moreinterestingly, the importance of others was the single common

thread amongthe participants who wererated as having changed epistemologically. All

of these participants were rated as having changedinthis area and additionally, 4 of 5

self-identified this characteristic. It seems that acceptance ofpeers as valid contributors

to knowledge wasdifficult, even for those who wererated as evaluativist using Kuhn’s

model. This is a significant finding because ofits potential impact on how faculty might

support the developmentofstudents’ epistemology. It suggests that pedagogical design

might profitably emphasize the importance of peer contributions to the construction of

their knowledge. Specific questions to support considerations of the importance of peer

interactivity could also be included in learning logs.

From a design mode perspective, arguably the mostinteresting finding was the

relative absence of work in design mode, regardless of level of belief-mode

understanding. This suggests that design modeis not simply a new facet of belief-mode

epistemology, but a different, albeit complementary dimension. The conceptof “new

knowledge” and the design-mode epistemology underlying it, appears to be new to

students, even at advancedlevels. As reflected in the comments from a number of

students, the creation of new knowledgeis thoughtto be for experts, artists, and geniuses

alone - through a processthat is a mystery to them.

148

Implications ofCharacteristics ofthe Epistemological Levelsfor Practice

This research, particularly in the pilot data, showed that participants were not

likely to metacognitively reflect without prompting. This agreed with research discussed

in chapter 3 by Admiraal, Veen, Korthagen, Lockhorst, Wubbels, Hernandez,etal,

(1999) where students’ messages contained simple summaries of daily events “without

showing a rational analysis of their actions and thoughts which is commoninreflection”

(p. 85). These authors recommend more guidance and coaching in journaling. To

overcomethis problem,this study used directed questions that were designed to

encourage epistemic agency andgreater self-awareness of learning. As suggestedearlier,

these questions provided plenty of scope for work in design-mode,although they did not

encourageit directly. It seemsclear that future efforts will need to engage students in

design mode moredirectly.

There has been much successreported about the adoption of online journaling

(Admiraal et al., 1999; Andrusyszyn & Davie, 1997; Graybeal, 1987), and more recently,

online “blogs” showa lot of potential similarities with an alternative technical structure

for such journaling. In the current research, participants’ learning logs were accessible to

all within the conference, and students were encouraged to read and comment on each

other’s entries. It was clear that the learning logs encouraged explicit recording of

participants’ conceptual and metacognitive growth. While there was not much discussion

by participants concerning the self-assessment of their learning logs within the logs

themselves, other communications with the instructor (mainly email) suggested to the

instructor that the range of self assigned grades was both broader andlinkedto their

actual reflective performance in a way that was probably more accurate, and more useful

from a learning perspective than if they had been instructor-assessed.

Participant commentaries indicated that the purpose of the logs need to be well

publicized and need to be explained in a variety of waysto help those, unfamiliar with

such agency, to become comfortable with the process. Further, while not required by

someparticipants, questions that scaffold metacognition should be available to support

development. Oneparticipant summedupthe use of the logs to encouragea rationalist

149

epistemologynicely:

Perhaps one ofthe biggest changes in understanding I have had regardingthe logs is changing my mindset that a reflection style project can be used

in the overall evaluation process, and can be used effectively. Prior tostudying constructivist learning theories, this is not a position I wouldhave taken. Instead, I would have viewed them as a “nice to have”tool,but not something that could accurately be used in the evaluation process.This has been a big shift in my thinking. (1_F_36-40:High)

Six of the nine scaffold questions encouraged participants to self-assess. Self-

assessmentis a valuable skill required byall students who desire to become continuous

learners. It has also been associated with the higher levels of epistemological

development(Baxter Magolda, 1999; Perry Jr., 1970). A participant shows her

astonishment concerningthe self-assessed nature of the learning logs,

We even have a chanceat self-assessmentin this course through ourLearning Logs... what amazing powerin the hands ofstudents! What achangefrom a traditional classroom. (1_F_36-30:High)

The learning logs were madea self-assessment assignmentbytheinstructor. It

waspointed out by course participants in earlier offerings of the course that in spite of

course discussion about constructivism and focus on student frames of learning, the

faculty almost exclusively did all graduate course grading. Adding a self-assessment

componentto the learning log assignment seemedto bea logical and efficacious location

in which to experiment with the effectiveness of self-evaluation. Participants seemed to

respond positively to this opportunity. One student expresses this cogently,

The learning log is a good knowledge building device/exercise that tracksreflections and a good way to assess my learning and theoretical growthto date. The inclusion ofthis assessment tool hasforced me to sit downand record my thoughts, which I believe to be paramountfor complex

individual learning to occur. The log is a good record ofdiscoveries and aplace to return to during and after the course to reflect on what I said, Ihave learned, and confirm the validity ofmy reflections as the courseprogresses and thereafter. (13_M_41-45:Low)

150

They were required to send an emailto the instructor at course end self-assigning a grade

from a small rubric, Appendix F, and a briefjustification.

In summary,learning logs encouraged participants to write and engage with ideas

and their own understanding in an explicit way. There was more than double the number

of words within the learning logs view than in any other week’s view,figure 1.

Assigning a grade and using mandatory questions greatly improved the use of logs, and

their reflective nature, as seen by the results of the second course compared to the first.

Finally, as a pedagogical tool, learning logs with directed questions (scaffolding)

encouraged epistemic agency and greater self-awareness of learning. An important next

step in the research will be use of the learning log to scaffold knowledge work in design

mode.

Implications ofCharacteristics ofthe Epistemological Levelsfor Theory

This study centered on students at the graduate level because they were

considered to be fully capable of the type of deep self-reflection required by the methods

used in this research. Further, the instructor had a deep personalinterest in the

improvementofonline learning environments and integration of technology into teacher

education. Would similar results be seen with studentsat different levels of study or does

it depend on age, context, and subject matter? How is metacognition best used to

influence epistemological growth?

Future research using learning logs could perhaps benefit from being more

explicit about the goals of using scaffolding questions to scaffold their metacognitive

thought. Meacham (2003) stated “It is during the transition from multiplist to evaluativist

understanding that professors can be most helpful, particularly by involving them

[students] in activities in which the value oftheir intellectual activities and efforts can be

self-evident” (p. 6). As one participant stated,

Although, I do not think that my logs were particularly profound or

exemplary ofdeep reflection we can’t all be great theorists, they have leadme to make immediate changes in my practice in my work environment

and have pushed meto reflect on my teaching practice. The practice ofweekly reflections did set me into a reflection mode not only with the

151

prescribed weekly questions but with all other new knowledgefrom thearticles and the ensuing “discussion”. The simplefact ofhaving to reflecton a prescribed weekly question led me to habitually approach newknowledgefrom that perspective questioning the use, relating it to mypresent andpossiblefuture contexts. (7_F_36-40:High)

Whatare the implications for such metacognitive research with young children?

Brown (1987) argues that knowledge about cognitionis

late developing;it requires that learners step back and consider their owncognitive processes as objects ofthought and reflection. For Piaget,

reflected abstraction requires hypothesis testing and evaluation, and theability to imagine possible worlds and their outcomes; thereforeit

demandsformal operational though (Piaget, 1976). For others, earliersigns ofemergance are possible; however, reflection is rarely attributed tothe very young child or novice, regardless ofhow precocious they mightbe (Brown & DeLoache, 1978). (p. 68)

The implication then is that children under the age of 11 (according to Piaget) are

not likely to show signs of metacognition. However, Brown goesonto argue, “young

children monitor and regulate their own activities. Indeed, a case can be madethatall

active learning involves self regulation” (p.68). Hence, it seems that self-regulation

might be effectively researched with young children. Part of the difficulty is the

confusion when using the term metacognition to describe two closely related, yet separate

forms of cognitive activity. Whether such self-regulation would actually support

metacognitive development in youngchildren is not clear from the research. However,

Moshman (2003) claimed that “The developmental literature challenges, however, to be

clear about just how advanced cognition differs from childish cognition, which

apparently is not as childish as we thought. My responseto this challenge, in a word,is

metacognition”(n.p.). If metacognition is possible in young children, how wouldthis

influence their learning? Could metacognition be scaffolded and used in a knowledge

building environment? There is evidence that elementary school children can work in

design mode, and even provide impressive accounts ofit, thus demonstrating amazing

metacognitive abilities (Scardamalia, 2002). Clearly, social cultural issues are important.

If we can move design modeinto the elementary schools, on a broader scale, we should

see very different accounts of design modeattertiary levels. It may actually be harder to

152

change epistemologies if we wait until advancedlevels.

How does Epistemological Understanding Relate to Participation and Interaction Patterns

in Online Discourse?

As mentionedearlier in this chapter, the use of a framework that measures

epistemology from an individualistic perspective may not be appropriate for examining a

social phenomenon. However, as argued in previous chapters, metacognition, especially

regulation, is related to epistemic agency, and many examples of metacognition were

coded. As well, constructive use of authoritative sources, a principle within the

knowledge building framework, was identified within the participants’ notes, on the basis

of resource material that they brought into their notes. There were various other

indicators of knowledge building principles, but there were also counter-indicators. For

example, an important principle is “democratization of knowledge.” But there was a

tendency toward the opposite - toward the ‘rich get richer’ within the who’s read whose

notes, as those rated high read more ofthose rated high (plus the prof’s notes) and those

rated low read more of those rated low and muchlessofthe prof’s notes. The knowledge

building principles can also be viewed as design parameters for knowledge building.

Several teachers, working directly on this principle, have demonstrated moredistributed

interactions than were in evidencein this result. And, following from a design-mode

epistemology, Scardamalia and Bereiter (2006) makeclear that the principles of

knowledge building are improvable objects in their own right.

The use of the Analytical Tool Kit allowed for general indications of the

conditions that would allow knowledge building to occur. Hence, the following

inferences can be made concerningthe relation of epistemological level and participation

and interaction measuresfor this research:

I, The greatest indication of a relationship between a sophisticated

epistemology and potential for knowledge building wasthat of the top 9

participants, in ALL ofthe participation and interaction ratings, 8 were rated as

evaluativists.

2. Except for one, participants rated as evaluativist were also rated high in

participation, while participants with a low participation rating showed a mix of

153

epistemologicallevels.

3. Evaluativists used authoritative sources more than multipilists. Further,

participants who were codedas having used dialectical reflection tended to have a

higher authoritative source count.

4, Edits and overall collaboration followed participation rating for the most

part. One would expect that evaluativists would be inclined to edit more and this

wasthe caseofthe 6 top evaluativists. Multipilists revised the least.

5. Evaluativists, who were high in participation rating, showed greater

epistemic agency as they used more metacognitive questions and showed greater

metacognitive activity during the last three weeks.

6. Prior use of Knowledge Forum did not seem to play a determiningrole in

who demonstrated a high participation rating. Howeverall participants who

scored high in participation rating had previously used Knowledge Forum atleast

once, while the two lowestin participation rating did not have prior Knowledge

Forum experience.

Implications ofthe Relationship ofHow Epistemological Understanding Relatesto Participation and Interaction Patterns in Online Discoursefor Practice

Based on the data above wesee that evaluativists were coded as having more

examples of metacognition, revision, and use of authoritative sources. Would then the

encouragementofrevisions, authoritative sources, and metacognition support

epistemological growth? If this is the case,it may be possible to design a pedagogythat

would promote the developmentofa more sophisticated epistemology. As was discussed

above, scaffolds, self-assessment, and learning logs could be used to promote these

activities. The knowledge building principles that are manifested in the Knowledge

Forum software, provide a powerful pedagogy that allows for complex interaction and

referencing of others’ notes. Faculty and instructors could emphasize the “importance of

others”to the learning process leading to higher levels of epistemological sophistication.

Theresults of this research seemed to show relations among epistemological

level, metacognition, and epistemic agency. While this research was not designed to

establish how a more sophisticated epistemological view would influence knowledge

154

building, it can be concluded,at least for the participants in this research, that

evaluativists participate and collaborate to a greater extent in a knowledge building

environmentthan do multiplists and absolutists. However, this does not lead directly to

work in design mode. Determining how to promote epistemological growth could lead to

a more effective use of collaborative pedagogies such as knowledge building.

Implications ofthe Relationship ofHow Epistemological Understanding Relates toParticipation and Interaction Patternsin Online Discoursefor Theory

Despite the promising results above, the methodsusedin this research could not

identify directly if a change in level of epistemological understanding affected knowledge

building activity. It was not evident whenparticipants’ views changed which might have

been important to determine factors surrounding the change. A comparisonofactivity

during the first half of the course and the second half provided no conclusive trends.

Perhaps if one could accurately identify the point where epistemological change

occurred, one might see corresponding local changes in knowledge building behaviour.

Based on the epistemological models reviewed in Chapter1, it was not clear if a change

in level of epistemological understanding occurred based on one event or was the

collective result of many events. Additionally, many researchers, including in the current

research, found examples in which participants were seemingly at multiple levels at the

same time. Is there an event that causes a ‘permanent’ change in epistemological

understanding or does change occur gradually and cumulatively?

Were There ChangesIn TheInteractions Between Individuals?

There were not manystriking similarities or differences between the participants

showing belief-mode epistemological change. Instead, there was a wide range of

individual reading patterns, revisions, and note creation over the duration of the research

study. There wasalso a wide range of metacognitive activities coded, however, no

discernible overall patterns emerged for this group. As predicted by Moshman’s (1998)

theory, 4 of 5 participants decreased their reliance on experience-basedjustification from

155

pre- to post-essay. However, there were two similarities that emerged acrossall

participants. First, they all had participant 1_F_36-40 as one of the top 3 people upon

whomtheybuilt and, of the top three personsthat they built upon, at least 2 were rated

high in participation rating. Second, these participantsall explicitly noted that their

notion of the importance of others, especially peers, had changed over the course. That

is, their peers had now becomelegitimate sources of knowledge.

Implications ofthe Changes In The Interactions Between Individualsfor Practice

Kuhn (2003; 2004) advocated that the role of the professor should be to develop

activities for intellectual developmentto the evaluativist level, wherein, their intrinsic

value becomesevident. In this case, the professor should model and scaffold to show

personal commitmentandbelief in the value of both the discussion and learning logs. In

this research, there was modeling of notes within the weekly discussion by both the

professor and researcher. However, neither the professor nor the researcher engaged in

modeling a learning log, apart from the directed questions. Could a professor’s learning

log be beneficial for students and how would it differ from the log of a student? My

guessis that it would show muchgreater evidence of design mode. In fact, comparing

the pilot to the main research, there was a noticeable difference between the quality and

sophistication of the learning logs. The use of mandatory learning log directed questions

and scaffolding lead to increased metacognitve thought in the second course as compared

to the pilot, where modest use of the questions was evident, and hence, the logs

concentrated mainly on content. Could a professor’s learning log increase the usefulness

of the logs even further? There was evidence of confusion as to the purpose ofthe logs.

However,as stated by oneparticipant,

For meit is now perfectly clear why we have learning logs and I thinkthey should be an integral part ofany graduate level course. Here is ourplace to “make sense ofthings” to investigate our own learning and seewherethe holes lie. In my early confused weeks it helped lot to writedownthe various things I couldn't understand and why I couldn'tunderstand them. It also helped to write down my own metacognitive

thoughts because once they are down in words, it was a way oftrackingthem. (2_F36-40:High)

156

Could learning logs only be used at the graduate level or could younger students

benefit from their use to promote metacognition? This is an important area for future

research.

The main benefit of the logs was to promote metacognition. Hacker(n.d.)

advocated the training of regulation as an effective strategy to improveintellectual

performance. Hecited several studies where better performance ona task and the choice

of the more effective strategy was a direct result of monitoring training. It was evident

that even though manyparticipants admitted that the logs were difficult, they did believe

that the logs were worthwhile,

For myselfas an adult learner, I am applying metacognition more

consistently and effectively after the thorough discussions we hadonline.

I think I understand the promise ofreflecting upon one’s thinking in orderto affect one’s learning. Consciously “thinking about my thinking”, Ifeel,

enables me to take morerisks in learning. I know I'll make mistakes but

reflecting upon my thinking and learning those mistakes are positive indue process. (6_F_26-30:High)

Implications ofthe Changes In The Interactions Between Individualsfor Theory

It was anticipated that greater belief-mode epistemological sophistication would

lead to a numberof changesin participant behavior; specifically an increase in activities

that would lead to increased sustainability of online learning communities. While

limited, the results do indeed show such behavior. First, participants who showed a

changein sophistication of epistemological view, tended to build more upon participants

whowererated high in participation rating. This could mean that others’ entries

provided somescaffolding, similar to Vygotsky’s zone of proximal developmentto those

participants. The participants who wererated as high, demonstrated evidence of some

epistemic agency and an ‘indication of knowledge building’ using the Analytical Tool

Kit. As stated in Chapter 1, the Socio-Cognitive Dynamics of Knowledge Building are

built around the explicit goal of continual improvement of ideas. This implies that

sustained work by a community is needed to produce continual improvement and work in

design mode.

157

The second important indicator that might support sustainability is the change in

howpeers were viewed. Changed participantsall explicitly noted that the importance of

peers to their learning had increased; peers had now becomelegitimate sources of

knowledge. This was important in increasing the likelihood of sustainability becauseit

validated the time spent in peer discussion online. Specifically, as time is often sited as

the greatest hindrance to the sustainability of online learning communities, it becomes

evident that the more time that can be spent with whom one considers a “valid source of

knowledge” becomescritical. Traditionally, only professors and experts were considered

such sources. Getting access to experts’ time is often difficult. When peers are

considered valid sources of knowledge, students can now feel that being online with

peers is valuable and rewarding time spent. This increasesthe likelihood of returning to

their online communities. Of course, an important design challenge remains- to increase

peer interaction around the constructive use of authoritative sources, and thus support

students engaging in effective knowledge creation in broader cultural contexts beyond

mere collaboration.

Participants could have changedat any point between the pre- and post-essay. If

participants did not change until the last week of the course, for example, then the

changes would not have been evident in the measures used by this research. Without

knowing the how or whenof epistemological change, it is hard to say specifically what

could be done to encourage suchintellectual growth. If metacognitive reflection does

affect intellectual growth, then this study provides somepreliminary indicators as to what

could encourage metacognitive activity. What wasof interest in this study was the

degree to which learning logs supported reflection consistent with the kind of mature

scientific thinking necessary at the graduate schoollevel, described by Kuhn andPearsall

(2000) as “the coordination of theory and evidencein a consciously controlled manner”

(p 114).

Closing Remarks

This research did have some important implications for both theory and practice.

The use of learning logs as a research tool clearly impacted participant’s learning to a

158

much greater extent than expected. Could such a simple pedagogical tool be employedto

increase epistemological sophistication through encouragementofmetacognition? It is

very possible, but not without overcoming someofthe significant challenges in the

measurement of epistemological changes that were uncoveredin this research. “The

study of more comprehensive kinds of cognitive change, however, especially those

involving change in knowledge acquisition strategies themselves, poses serious

methodological challenges” (Kuhnetal., 1995, p. 8). However, the possible benefit to

students is great. Brown (1987) stated,“It is true that metacognitive-like concepts are

fraught with some of the mostdifficult and enduring epistemological problems of

psychology. However, [she] argues that metacognitive-like entities lie at the very roots

of the learning process; and ... is an exciting revival of interest in mechanismsof change

and development”(, p. 66). |

If epistemology seeks to uncover the procedures and conditions that make

knowledge possible, then it seems reasonable that teachers take more responsibility in

helping students with the use of learning logs. To assign them is not enough. My

experience seemsto indicate that they need to be studied during and after a course and

then revisions made to teaching based on the learning thus derived.

There is much value in finding waysto increase epistemological sophistication

and this maydirectly link to sustainability of online communities. Kuhn (2003)stated,

The value ofan intrinsically valued activity, in contrast, lies in the activityitself. The benefits ofthe activity emanate directlyfrom it. One engagesin it becauseit is experienced as valuable in its own right. The advantageis clear: Continued commitment to the activity is ensured. It is notdependent on external maintenance ofa relation between the activity andsome independently valued outcome. (n.p.)

Perhapshelping learners find the intrinsic value, through the developmentofa

sophisticated epistemology,is the key to ensuring sustainable participation in that world

where knowledgeis an infinite resource.

REFERENCES

Admiraal, W., Veen, W., Korthagen, F., Lockhorst, D., Wubbels, T., Hernandez,F., et al.

(1999). Tele-guidance to develop reflective practice: experiences in four teacher

education programmesacross Europe. Journal ofInformation TechnologyforTeacher Education, 8(1), 71 — 88.

Anderson, T., & Kanuka, H. (1997). On-line forums: new platformsfor professionaldevelopmentand group collaboration. Journal ofComputer MediatedConferencing, 3(3).

Anderson, T., & Kanuka, H. (2003). e-Research methods, strategies, and issues. Boston:

Allyn and Bacon.

Andriessen, J., Baker, M., & Suthers, D. D. (Eds.). (2003). Arguing to learn: Confrontingcognitions in computer-supported collaborative learning environments: KluwerAcademic Publishers.

Andrusyszyn, M.-A., & Davie, L. (1997). Facilitating reflection through interactivejournal writing in an online graduate course: A qualitative study. Journal of

Distance Education/Revue de l'enseignement a distance, 12.1(2).

Austin, Z. (2002). Creating contextfor community knowledge building. Unpublished

(Doctoral dissertation, University of Toronto, 2001). Digital Dissertations,

Toronto.

Baxter Magolda, M. B. (1992). Knowing and reasoningin college: gender-related

patterns in student's intellectual development(1 ed.). San Francisco, California:Jossey-Bass.

Baxter Magolda, M.B. (1999). Creating contextsfor learning and self-authorship :

constructive-developmental pedagogy (1st ed.). Nashville, TN: VanderbiltUniversity Press.

Baxter Magolda, M.B. (2001). A constructivist revision of the measure ofepistemological reflection. Journal ofCollege Student Development, 42(6), 520-534.

Belenky, M., Clinchy, B., Goldberger, N., & Tarule, J. (1986). Women's ways of

knowing: The developmentofself, voice, and mind. New York: Basic Books.

Bell, P. (2002). Science is argument: Developing sociocognitive supports for disciplinaryargumentation. In T. Koschmann,R. Hall & N. Miyake (Eds.), CSCL 2: Carrying

forwardthe conversation (pp. 499-505). Mahwah, NJ: Lawrence Erlbaum

Associates.Bereiter, C. (2002). Design research for sustained innovation. Cognitive Studies, Bulletin

ofthe Japanese Cognitive Science Society, 9(3), 321-327.

Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: LawrenceErlbaum Associates, Publishers.

Bereiter, C. (2003, 28 November 2003). Bringing classroomsinto the knowledge age.Paper presented at the Reform Initiatives in Teaching and Learning, University ofMacau.

Bereiter, C., & Scardamalia, M. (2006). Education for the knowledge age: Design-centered models of teaching andinstruction. In A. Alexander & P. H. Winne

(Eds.), Handbook ofeducational psychology (2nd ed.). Mahwah, NJ: LawrenceErlbaum Associates.

159

160

Brett, C. (2002, February 18, 2003). Course outlinefrom knowledgeforum database note#10. Retrieved February 26, 2003

Brett, C. (2003). Course description. In email received by Bruce C Forrester (Ed.).Ottawa.

Brown,A.L. (1987). Metacognition, executive control, self-regulation, and other more

mysterious mechanisms. In F. E. Weinert & R. H. Kluwe (Eds.), Metacognition,motivation, and understanding (pp. 65-116). Hillside, New Jersey: LawrenceErlbaum Associates.

Brown,A.L. (1992). Design experiments: Theoretical and methodological challenges increating complex interventionsin classroom settings. The Journal ofthe LearningSciences, 2(2), 141-178.

Bruffee, K. A. (1999). Collaborative learning : higher education, interdependence, andthe authority ofknowledge (2nd ed.). Baltimore, Md.: John Hopkins UniversityPress.

Burtis, J. (2002). Analytic Toolkitfor Knowledge Forum (July 16, 2002 -- v4.0 ).Retrieved 15 Jannuary, 2003, from http://analysis.ikit.org/atk/atkdoc.html

Chan, C. K. K., & van Aalst, J. (2003). Assessing and scaffolding knowledge building:

Pedagogical knowledge building principles and electronic'portfolios. Paperpresented at the International Conference on Computer Support for Collaborative

Learning, Dordrecht, the Netherlands.

Collins, A. (1988). Toward a design science of education. In E. Scanlon & T. O'Shea(Eds.), New directions in educational technology (Vol. 96, pp. 15-22). New York:

Springer-Verlag.

Collins, A. (1999). The changing infrastructure of education research. In E. C. Lagemann

& L. S. Shulman (Eds.), /ssues in education research: Problems andpossibilities(pp. 289-298). San Francisco: Jossey-Bass.

Connell, J. (2000). An evaluation ofstudent diaries as toolsfor learning and assessment.Retrieved February 3, 2002, from

http://www.newcastle.edu.au/oldsite/services/iesd/publications/eunexus/articles/te

achingreview/connell.htmDewey, J. (1933). How we think: A restatement ofthe relation ofreflective thinking to the

educative process. Lexington, MA:Heath.

Dewey, J. (1938). Experience and Education. New York: Macmillan.Dewey,J. (1998). Experience and education (60th anniversary ed.). West Lafayette, Ind.:

Kappa Delta Pi.

Dillman, D. A., & Bowker, D. K. (2001). The web questionnaire challenge to surveymethodologists. In U.-D. R. M. Bosnjak (Ed.), Dimensions ofinternet science.Lengerich, Germany: Pabst Science Publishers.

Dillman, D. A., Sinclair, M. D., & Clark, J. R. (1993). Effects of questionnaire length,

respondent-friendly design, and a difficult question on responserates foroccupant-addressed census mail surveys. Public Opinion Quarterly, 57(3(Autumn, 1993)), 289-304.

Dillman, D. A., Tortora, R. D., Conradt, J., & Bowker, D. K. (1998, August). Influence ofplain vs. fancy design on responce ratesfor web surveys. Paper presented at theJoint Statistical Meetings, Dallas Texas.

Dreyfus, H. L., & Rabinow,P. (1982). Michel Foucault Beyond structuralism and

161

hermeneutics. Chicago: The University of Chicago Press.Driver, R., Asoko, H., Leach, J., Mortimer, E., & Scott, P., 23, 5-12. (1994). Constructing

scientific knowledge in the classroom. Educational Researcher, 23, 5-12.

Fisher, K. W., & Pipp, S. L. (1984). Progress of cognitive development: Optimallevels

and skill acquisition. In R. J. Sternberg (Ed.), Mechanismsofcognitivedevelopment. In P.M. King & K.S. Kitchener (1994) Developing reflectivejudgment: Understanding andpromoting intellectual growth andcritical thinkingin adolescents and adults. San Francisco: Jossey-Bass. New York: Freeman.

Flavell, J. H. (1971). First discussant's comments: What is memory developmentthedevelopmentof? Human Development, 14, 272-278.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitivedevelopmentinquiry. American Psychologist, 34, 906-911.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-

developmental inquiry. American Psychologist, 34, 906-911.Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In

F. E. Weinert & J. H. Flavell (Eds.), Metacognition, motivation, and

understanding (pp. 21-29). Hillside, New Jersey: Lawrence Erlbaum Associates.Garrison, D. R., & Anderson, T. (2003). E-learning in the 21st century: A frameworkfor

research andpractice. London: RoutledgeFalmer.Glesne, C. (1999). Becoming qualitative researchers (second ed.). New York: Longman.

Goldman,A. (1999). Knowledge in a Social World. New York, NY: Oxford UniversityPress.

Goldman,A. (2001). Social Epistemology. Retrieved 15 September, 2005, fromhttp://plato.stanford.edu/entries/epistemology-social/

Graybeal, J. (1987). The team journal. In T. Fulwiler (Ed.), The Journal Book(pp. 306-311). Portsmouth, NH: Boynton/Cook Publishers.

Hacker, D.J. (1998). Definitions and empirical foundations. In D. J. Hacker, J. Dunlosky& A. C. Greesser (Eds.), Metacognition in educational theory andpractice (pp. 1-23). Mahwah, NJ: Erlbaum.

Hacker, D. J. (n.d.). Metacognition: Definitions and empiricalfoundations. Retrieved 23April, 2004, from http://www.psyc.memphis.edu/trg/meta.htm

Heflich, D. A., & Putney, L. G. (2001). Intimacy and reflection: Online conversation in apracticum seminar. The Journal ofComputing in Teacher Education, 17(3), 10-17.

Hofer, B. K., & Pintrich, P. R. (1997). The development of epistemological theories:

Beliefs about knowledge and knowingandtheir relation to learning. Review ofEducational Research, 67(1 (Spring 1997)), 88-140.

Kember, D. (2000). Action learning and action research. London: Kogan Page Ltd.Kerka,S. (1996). Journal writing and adult learning. Retrieved January 31, 2002, from

http://ericacve.org/docs/digi74.htm

King, P. M., & Kitchener, K. S. (1994). Developing reflectivejudgment: Understandingandpromoting intellectual growth andcritical thinking in adolescents and adults.

San Francisco: Jossey-Bass.Kluwe, R. H.(1987). Executive decisions and regulation of problem solving behavior.In

R. H. Kluwe (Ed.), Metacognition, motivation, and understanding (pp. 31 - 64).Hillside, New Jersey: Lawrence Erlbaum Associates.

162

Knefelkamp,L. L. (1999). Jn the introduction of: Forms ofintellectual and ethicaldevelopmentin the college years (2 ed.). New York: Jossey-Bass.

Kuhn,D. (1991). The skill ofargument. Cambridge, England: Cambridge UniversityPress.

Kuhn,D. (2003). Understanding and valuing knowledge as developmental goals -Featured Topic. Liberal Education(Summer).

Kuhn,D., Cheney, R., & Weinstock, M. (2000). The development of epistemological

understanding. Cognitive Development(15 (2000)), 309-328.Kuhn,D., & Dean, D. J. (2004). Metacognition: a bridge between cognitive psychology

and educational practice. Theory into Practice(Autumn).Kuhn,D., & Pearsall, S. (2000). Developmental origins ofscientific thinking. Journal of

Cognition and Development, 1, 113-129.

Kuhn,D., White, S. H., Carver, S. M., & Klahr, D. (1995). Strategies ofknowledge

acquisition. Chicago,IIl.: Society for Research in Child Development.Kuhn,T. S. (1970). The structure ofscientific revolutions (second ed. Vol. 2). Chicago:

University of Chicago Press.Lax, L. R., Taylor, I., Wilson-Pauwels, L., & Scardamalia, M. (2004). Dynamic

curriculum design in biomedical communications: Integrating a knowledge

building approach and a Knowledge Forum® learning environment in a medicallegal visualization course. journal ofBiocommunications, 30(1).

Lee, E. Y. C., Chan, C. K. K., & van Aalst, J. (2005). Student assessing their ownknowledge advances in a knowledge building environment. Paper presented at theInternational Conference on Computer Supported Collaborative Learning 2005,

Taipei, Taiwan.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions, and

emerging confluences. In N. Denzin & Y. Lincon (Eds.), Handbook ofQualitative

Research (2nd ed., pp. 163-188): Sage.Lipton, P. (2004). Inference to the best explanation (2 ed.). London: Routledge.McLoughlin, C., & Marshall, L. (2000, 2-4 February). Scaffolding: A modelfor learner

support in an online teaching environment. Paper presented at the 9th AnnualTeaching Learning Forum,Perth: Curtin University of Technology.

Meacham,J. (2003). Student intellectual development. Liberal Education, 89(Summer),6.

Moshman,D. (1995). Reasoningas self-constrained thinking. Human Development(38),53-64.

Moshman,D.(1998). Cognitive development beyond childhood. In D. Kuhn & R.S.Siegler (Eds.), Handbook ofchildpsychology: Vol 2. Cognition, perception, and

language (pp. 947-978). New York: Wiley.

Moshman,D.(2003). Intellectual freedom for intellectual development - Feature Topic.Liberal Education(Summer). ,

Neumann,A., Pallas, A. M., & Perterson, P. L. (1999). Preparing education practitionersto practice education research. In E. C. Lagemann & L. S. Shulman (Eds.), /ssuesin education research: Problems andpossibilities (pp. 247-288). San Francisco:Jossey-Bass.

Ohlsson, S. (1991). Young adults’ understanding ofevolutionary explanations:Preliminary observations. Pittsburgh: Learning Research and Development

163

Center, University of Pittsburgh.Olson, D. R.(in press). Metacognition and normativity. Prepublication draft.Perry Jr., W. G. (1970). Formsofintellectual and ethical developmentin the college

years. New York: Holt, Rinehart and Winston,Inc.

Piaget, J. (1972). Judgment and reasoningin the child. Totowa, NJ: Littlefield, Adams.Popper, K. R. (1994). Knowledge and the mind-body problem. In defence ofinteraction.

New York: Routledge.Popper, K. R. (1999). All life is problem solving. New York: Routledge.Preece, J. (2000). Online Communities: designing usability, supporting sociability. New

York: John Wiley & Sons,Inc.

Scardamalia, M. (1991). Higher levels of agency for children in knowledge building: A

challenge for the design of new knowledge media. The Journal ofthe Learning

Sciences, 3(3), 265-283.

Scardamalia, M. (2002). Collective cognitive responsibility for the advancement ofknowledge. In B. Smith (Ed.), Liberal Education in a Knowledge Society (pp. 67-98). Chicago: Open Court.

Scardamalia, M. (2003). Knowledge Building. Journal ofDistance Education, 17(Suppl.3, Learning Technology Innovation in Canada), 10-14.

Scardamalia, M. (2004). Instruction, learning, and knowledge building: Harnessing

theory, design, and innovation dynamics. Educational Technology, 44(3), 30-33.Scardamalia, M., & Bereiter, C. (Eds.). (2003). Knowledge building. New York:

Macmillan Reference, USA.

Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, andtechnology. In K. Sawyer (Ed.), Cambridge Handbook ofthe Learning Sciences

(pp. 97-118). New York: Cambridge University Press.Shavelson,R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of

educational design studies. Educational Researcher, 32(1), 25-28.Solomon, D. J. (2001). Conducting web-based surveys. Practical Assessment Research &

Evaluation, 7(19).

Takao, A. Y., Prothero, W., & Kelly, G. J. (2002). Applying argumentation analysis toassess the quality of university oceanographystudents’ scientific writing. JournalofGeoscience Education, 50(1), 40-48.

Thagard, P., & Millgram, E. (1997). Inference to the Best Plan: A Coherence Theory ofDecision. In A. Ram & D. B. Leake (Eds.), Goal-driven learning (pp. 439-454).Cambridge, MA:MIT Press.

The Design-Based Research Collective. (2003). Design-based research: An emerging

paradigm for educational inquiry. Educational Researcher, 32(1), 5-9.

University of California Berkeley. (1997). Field guide to design experiments ineducation. Retrieved 2002, November5, from

http://www.soe.berkeley.edu/sandhtdocs/guide.htm]Vygotsky, L. S. (1978). Mind in society: the developmentofhigher psychological

processes. Cambridge, MA:Harvard University Press.

Woodruff, E. (1999). Concerning the cohesive nature ofCSCL communities. Paperpresented at the Computer Support for Collaborative Learning (CSCL) 1999Conference, Stanford University, Palo Alto, California.

Woodruff, E. (2001). CSCL communities in post-secondary education and cross-cultural

164

settings. In T. Koschmann,R. Hall & N. Miyake (Eds.), CSCL II: Carrying

forward the conversation (pp. 157-168). Mahwah, NJ: Erlbaum.

Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R.(in press). Socio-

cognitive dynamics underlying knowledge building. Educational TechnologyResearch & Development.

APPENDIX A - PERL SCRIPT FOR PILOT POST-SURVEY

#This PERL script logs on students, verifies that they have not already

submitted survey data, and then presents the survey. Upon completion, thesurvey is sent to a Mysql database. Students are only allowed to submit the

survey once. Overall commentsare allowed at the very end and are appended toa textfile.

#Two timestampsare taken to measure time required to complete survey.

#This programme was written and designed by Bruce Forrester between Dec 2002 and

Jan 2003. I can be reached at beforrester@sympatico.ca

#use CGI::Debug; #uncomment for debugging

use DBI;

use CGI qw(:all);uselib;

$q = new CGI;

HHtHHHHH define the database string variables ######HHH

my $database = "survey_bf";my $db_server = "localhost";my $user = "XXX"; # real usernamenot included for security

my $password = "XXX"; # real password not included for security

#open the connection to the database server

my $dbh = DBI->connect("DBI:mysql:$database:$db_server", "$user","$password") or die "Can't connect to database: ". $DBI::errstr,"\n";

my $choice = $q-> param ("choice"); # get user choice

print $q->header; # start HTML

PEPEAEEEEHEREEEEHEEEEAEEEAEEEEEEEPEEEEEPEREHEHEHEEAStart of logic for Script 4AHHHPEAEEEEEETEERHEHEEEREHEHEHEETEEEAEEEEAEAEEEEE

if ($choice eq "") #send login form if new person

{print $q->start_html(-title=>'CTL1602 Post-Course Survey Login',-bgcolor=>aqua);print $q->center($q->h2("Login Page for CTL1602 Winter 2003 Post-Course Survey,

")P3print $q->center($q->h2('Please Login:')),p;

165

166

login ();print $q->br;print $q->center($q->submit(-name=>'choice’, -value=>'Login’)),p;

print $q->endform;print $q->hr;

print $q->b(tt(blockquote( "If you cannot login to this survey, please email:", $q->a({href=>'mailto:bcforrester@sympatico.ca'},"Bruce Forrester"),)));

}if (Schoice eq "Login") # login and verify no survey submitted

print $q->start_html(-title=>'CTL1602 Post-Course Survey',-bgcolor=>aqua);

my @userdata = verify_login();no_data_entered(@userdata);my$id =shift(@userdata);my $firstname = shift (@userdata);

print $q->h2("Welcome $firstname,"),p;print $q->h2('Please complete the following Survey. There are a total of 29 questionsand should take around 20 minutes to complete.’),p;

initial_timestamp($id);display_entry_form ();print $q->submit(-name=>'choice'’, -value=>'Submit’);print $q->submit(-name=>'choice’, -value=>'Cancel'),p;

printhr;

print $q->b(tt(blockquote( "If you have comments about this survey, please send themto:", $q->a({href=>'mailto:bcforrester@sympatico.ca’},"Bruce Forrester"),)));

print $q->hidden(-name=>'firstname'’, -default=>$firstname);print $q->hidden(-name=>'id', -default=>$id);print $q->endform;

}elsif ($choice eq "Submit") # write submited data to the database. Send

a thankyounote and ask for further comments

{print $q->start_html(-title=>'Data Submitted',-bgcolor=>aqua);my$firstname = $q->param(‘firstname’);

my $id = $q->param(‘id’);my (@userdata);

push(@userdata, $id,$firstname);no_data_entered(@userdata);my $firstname = $q->param(‘firstname’);

print $q->h1("Thank you $firstname, for taking the time to complete this survey. ");print $q->br;print $q->img({-src=>"jumpsmiley.gif"});

print $q->img({-src=>"skipitrip.gif"});print $q->br;

167

print $q->start_form(-method=>'get', -action=> self_url();print $q->h4("Please make any overall commentshere: "),p;

print $q->textarea(-name=>'comments’, -rows=>5, -columns=>50),p;

print $q->hidden(-name=>'firstname’, -default=>$firstname);

print $q->submit(-name=>'choice', -value=>'Submit Comments’);

print $q->endform;insert_data ($dbh);

}elsif ($choice eq "Submit Comments") #allows the user to submit comments

{comments();my $firstname = $q->param(‘firstname’);print $q->start_html(-title=>'Good Bye',-bgcolor=>aqua);

print $q->h3("Good Bye, $firstname"),p;print $q->h3("Thanks for your comments.");

}

elsif ($choice eq "Cancel") #allowsthe user to quit

{print $q->start_html(-title=>'Good Bye',-bgcolor=>aqua);my $firstname = $q->param(‘firstname’);print $q->h3("Good Bye, $firstname");

}

print $q->end_html;

$dbh->disconnect;

PEELEEEEEEEHEEHEHEAEEEAEEEEEEEHeeEnd of LogicsetteePEEEEETEERHEAEEHEEHEHEEEEE

sub initial_timestamp # this sub recordstheinitial start time of survey

{my$id = shift;

my $sth = $dbh->prepare ("UPDATE 1602_id SET accessed='y' WHERE id = $id") ordie "Can't prepare SQL statement:" . $dbh->errstr(),"\n";

$sth->execute () or die "Couldn't execute SQL statement:".$sth->errstr(), "\n";$sth->finish; #close off the statement handle

}

FREEAHEEE

sub comments #enter user comments into thefile

{my @comments = $q->param(‘comments'’);

168

my $firstname = $q->param(‘firstname’);#These lines open the post_comments.txt, lock it so that only one person can

#write at a time, prints the data to it, then closesthefile

open (NEEDS,">>post_comments.txt") || Error(openr’,'file');flock (NEEDS,2)|| Error(‘lock’,'file’);

print NEEDS "\n\nTHIS IS A NEW RECORD by$firstname\n";

print NEEDS "@comments\n";close (NEEDS)|| Error(‘close'’,'file’);

}

HHHHHHHHHAHAHAHAHAEEE

sub login # Subroutine to build the login form

{print $q->start_form( -method=>'get', -action=> self_url());

print $q->center("Username:",$q->textfield(-name=>'name’,-size=>20,-maxlength=>30)),br;

print $q->center("Password: ", $q->password_field(-name=>'pass', -size=>20,-maxlength=>20));

}

PEERTEEREEREAEHEHETEEAEEETEPEPEPEEEEEEEE

sub verify_login #Subroutineto verify that the user exists

{my $name = $q->param(‘name’);my $pass = $q->param(‘pass’);

my @passcheck = $dbh->selectrow_array ("SELECTfirstname FROM 1602_id WHEREname = ‘$name’ ANDpass= ‘$pass'");

if (!defined (@passcheck)) #no matching record found

print h2( “Incorect login nameor password",p "Please click back.");

die;

}else

{my @userdata = $dbh->selectrow_array("SELECTid, firstname FROM 1602_idWHERE name= '$name' ANDpass= '$pass'"");

}}

PERERATEETHAEEAEEEEEAEEEEEEEEP

sub no_data_entered #This subroutine verifies that the user has not alreay submitted a

survey

{

169

my $id = shift;my $firstname = shift;

my $proceed = $dbh->selectrow_array("SELECT * FROM postdata WHERE id = '$id");if (defined ($proceed))

{print h2("Sorry $firstname, you have already submitted a survey."),p;

print $q->blockquote("If you need to redo this survey, please send mail to :", $q->a({href=>'mailto:bcforrester@sympatico.ca’},"Bruce Forrester"),);

die;

}}

PEREEEEEEEHEHEHEHEEHEHEHEEEHEHEPEEAEEEEEEE

sub display_entry_form # Subroutine to build the form

{print hr;

print $q->start_form(-method=>'post', -action=> self_url());print b("1) In class/online I prefer to explore problems provided by the:");print $q->radio_group(-name=>'RIAP_1’, -values=>['instructor','textbook'’,‘peers’,'myself’, 'other'],-rows=>2, -default=>"-"),p;

print "If other, please provide details: ";

print $q->textfield(-name=>'RIAP_1_1', -size=>60, -maxlength=>500),p;

print b(""2) I prefer to tackle problemsthat: ");print $q->radio_group(-name=>'RIAP_2',-rows=>2, -values=>["havea correct

answer','could have several answers','are more open-ended','other'], -default=>"-"),p;

print "If other, please provide details: ";

print $q->textfield(-name=>'RIAP_2_1', -size=>60, -maxlength=>500),p;print b("3) I prefer to explore: ");

print $q->radio_group(-name=>’RIAP_3',-rows=>2, -values=>['theory’,'casestudies','problems currently being faced by my peers/myself at work','Other'],-default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'RIAP_3_1', -size=>60, -maxlength=>500),p;print b("4) How would yourate the following relative to the importance to your

learning?"),p;print "<table border=1>";

print "<tr><th>For my learning, this is:</th><th>Essential</th><th>Important</th><th>Nice to have</th><th>Not Important</th></tr>";print "<tr><td>Instructors</td><td>", $q->radio_group(-name=>"IDCUAS_1", -value=>['1'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_1",-value=>['2'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_1",-value=>['3'], -default>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS1",-

value=>['4'], -default=>"-"),"</td></tr>";

print "<tr><td>Textbooks</td><td>", $q->radio_group(-name=>"IDCUAS_2",-

170

value=>['1'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_2",-value=>['2'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_2",-value=>['3'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS._2",-value=>['4'], -default=>"-"),"</td></tr>";print "<tr><td>Journal Articles</td><td>", $q->radio_group(-name=>"IDCUAS_3”,-value=>['1'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_3",-

value=>['2'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_3",-value=>['3'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_3",-value=>['4'], -default=>"-"),"</td></tr>";

print "<tr><td>Peers</td><td>", $q->radio_group(-name=>"IDCUAS_4", -value=>['1'],-default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_4", -value=>['2'],-

default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_4", -value=>['3'],-default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS4", -value=>['4'],-default=>"-"),"</td></tr>";

print "<tr><td>Classroom/Online Discussion</td><td>", $q->radio_group(-

name=>"IDCUAS_5", -value=>['1'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_5", -value=>['2'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_5", -value=>['3'], -default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_5", -value=>['4'], -default=>"-"),"</td></tr>";

print "<tr><td>Other</td><td>", $q->radio_group(-name=>"IDCUAS_6", -value=>['1'],-default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_6", -value=>['2'],-

default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_6", -value=>['3'],-default=>"-"),"</td><td>", $q->radio_group(-name=>"IDCUAS_6", -value=>['4'],-default=>"-"),"</td></tr>";

print "</table>",p;print "If other, please provide details: ";

print $q->textfield(-name=>'IDCUAS_6_1’, -size=>60, -maxlength=>500),p;print b("5) I feel that it is more important to explore: ");

print $q->radio_group(-name=>'IDCUAS_7',-rows=>3, -values=>['deeply into an area of

knowledge ',' across a wide range of knowledgeareas',' other'], -default=>"-"),p;print "If other, please provide details: ";

print $q->textfield(-name=>'IDCUAS_7_1', -size=>60, -maxlength=>500),p;

print b("6) When judging the importance of someone's opinion, what do I base mydecision on? "),p;print $q->textarea(-name=>'IDCUAS8’, -rows=>10, -columns=>70),p;

print b("7) I know that I have fully understood something when: "),p;print $q->textarea(-name=>'IDCUAS_9"" -rows=>10, -columns=>70),p;print b(""8) I feel comfortable expressing myideasin class/online, even if they are notfully developed: ");print $q->radio_group(-name=>'IP_1',-rows=>3, -values=>[' Yes ',' No _,'Other'],-default=>"-"),p;

print "If other, please provide details: ";

print $q->textfield(-name=>'IP_1_1', -size=>60, -maxlength=>500),p;print b("9) When I respond to someone'sideain class/online, I often: ");

print $q->radio_group(-name=>'IP_2',-rows=>4,-values=>[' do not respond ',' look forholes in their arguments ',' add myideasto the original ',' try to be supportive ',' try to

171

think of somethingto add ',' try to rephrase the idea ',' try to reference a reading ',' other"|, -default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'IP_2_1', -size=>60, -maxlength=>500),p;print b("10) At times when presented ideas are different from mine, I:");

print $q->radio_group(-name=>'EA_1',-rows=>4, -values=>[' adopt the new idea as myown. ',' look to see where there are differences ',’ allow myself to be influenced by the

idea ',' refuse to be influenced ',' judge the idea on its merits',' like to persist with an

idea until I really understandit ',’ look to see how the idea was supported ',’ other’],-

default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'EA_1_1', -size=>60, -maxlength=>500),p;print b("11) In courses,I feel that learning motivation, direction, and long-term planning

should beleft to: ");print $q->radio_group(-name=>'EA_2',-rows=>3, -values=>[' Instructor ',' Peers ','Myself ',' Combination ',’ Other '], -default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'EA_2_1', -size=>60, -maxlength=>500),p;

print b("12) I give the most credenceto ideas put forward by: ");

print $q->radio_group(-name=>'EA_3', -rows=>4, -values=>['the teacher’, 'my

peers','myself’, 'the researchliterature’,‘all of the above’, 'other'], -default=>"-"),p;print "If other, please provide details: ";

print $q->textfield(-name=>'EA_3_1', -size=>60, -maxlength=>500),p;print b("13) I knowit is time to move on to new areas of learning when... "),p;

print $q->textarea(-name=>'EA_4', -rows=>10, -columns=>70),p;print b("14) I feel that student assessment should be usedfor... "),p;print $q->textarea(-name=>'ETA_1', -rows=>10, -columns=>70),p;

print b("15) I most value assessment ofmy understanding by: ");print $q->radio_group(-name=>'ETA_2',-rows=>3,-

values=>['instructor’,'peers','standardizedtests','myself,'other'], -default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'ETA_2_1', -size=>60, -maxlength=>500),p;

print b("16) My preferred method by whichto be assessed is (and why): "),p;print $q->textarea(-name=>'ETA_3', -rows=>10, -columns=>70),p;print b("17) I know that I learned something in a course when... "),p;

print $q->textarea(-name=>'ETA_4', -rows=>10, -columns=>70),p;

print b("18) I feel that I can learn just as much from mypeersas myinstructor:");print $q->radio_group(-name=>'SKA_1', -rows=>3,-values=>["Yes','No’','Other'],-

default=>"-"),p;print "If other, please provide details: ";

print $q->textfield(-name=>'SKA_1_1', -size=>60, -maxlength=>500),p;print b("19) I feel that I am able to positively contribute to most course discussions : ");print $q->radio_group(-name=>'SKA_2', -rows=>3,-values=>['Yes',"No’,'Other’],-default=>"-"),p;

print "If other, please provide details: ";

print $q->textfield(-name=>'SKA_2_1', -size=>60, -maxlength=>500),p;

172

print b("20) For question 19, why do youfeel that way?"),p;print $q->textarea(-name=>'SKA_2_2', -rows=>10, -columns=>70),p;

print b("21) I feel that it is important for the instructor to summarize the important ideasat the end of the discussion : ");

print $q->radio_group(-name=>'SKA_3',-rows=>3, -values=>['Yes’,'No’,'Other'],-default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'SKA_3_1', -size=>60, -maxlength=>500),p;print b("22) Everyone in the course should be able to express his or her views and ideason a topic before moving on: ");

print $q->radio_group(-name=>'KBD_1', -rows=>3,-values=>['Yes','No’,'Other'],-default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'KBD_1_ 1’, -size=>60, -maxlength=>500),p;print b("23) I think that having class discussions are important only for certain subjects:

")3print $q->radio_group(-name=>'KBD_2’, -rows=>3,-values=>[Yes’,'No’,'Other'],-default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'KBD_2_1', -size=>60, -maxlength=>500),p;print b("24) Did you find that the format of this course caused you to think about ideasoutside of 'devoted class time’ ?"),p;print $q->radio_group(-name=>'PKB_1’, -rows=>3,-values=>['Yes','No’,'Other'],-

default=>"-"),p;

print "If other, please provide details: ";print $q->textfield(-name=>'PKB_1_1', -size=>60, -maxlength=>500),p;print b("25) How did youfind the knowledge building pedagogythat was usedin this

course?"),p;

print $q->textarea(-name=>'PKB_2', -rows=>10, -columns=>70),p;

print b("26) I would use the KF tool again."),p;print $q->radio_group(-name=>'PKB_3’, -rows=>3,-values=>['Yes',’'No','Other'], -default=>"-"),p;

print " If other, please provide details: ";print $q->textfield(-name=>'PKB3_1’, -size=>60, -maxlength=>500),p;print b("27) How manytimes have you used Knowledge Forum (any version) before this

course?"),p;

print $q->radio_group(-name=>'BD_1', -rows=>1,-values=>['0','1','2','3','4', '>4'],-default=>"-"),p;

print b("28) How manyonline courses have you taken before this course?"),p;print $q->radio_group(-name=>'BD_2’, -rows=>1,-values=>['0'"'1','2','3','4', '>4'], -

default=>"-"),p;

print b("29) How manycourses have you completed in your program beforethissession?"),p;

print $q->radio_group(-name=>'BD_3’, -rows=>1,-values=>['0','1','2','3','4', '>4'], -

default=>"-"),p;print br;

}

173

PEEEEEEEHEHETEAHEEAEEEEEHEHEAEHAEHEEAEE

sub insert_data # subroutine to insert data into mysql

{my $RIAP_1 = $q->param(‘RIAP_1");my $RIAP_1_1 = $q->param(‘RIAP_1_1’);my $RIAP_2 = $q->param(‘RIAP_2’);my $RIAP_2_1 = $q->param(‘RIAP_2_1');my $RIAP_3 = $q->param(‘RIAP_3’);my $RIAP_3_1 = $q->param('RIAP_3_1');my $IDCUAS_1 = $q->param(IDCUAS_1’);my $IDCUAS_2 = $q->param(‘IDCUAS2’);my $IDCUAS_3 = $q->param(‘IDCUAS._3’);my $IDCUAS_4 = $q->param(‘IDCUAS_4');my $IDCUAS_5 = $q->param(‘IDCUAS_5’);my $IDCUAS_6 = $q->param(‘IDCUAS_6');my $IDCUAS_6_1 = $q->param(IDCUAS._6_1°);my $IDCUAS_7 = $q->param(‘IDCUAS_7));my $IDCUAS_7_1 = $q->param(IDCUAS_7_1');my $IDCUAS_8 = $q->param(IDCUAS_8’);my $IDCUAS_9 = $q->param(‘IDCUAS_9');my $IP_1 = $q->param(‘IP_1’);my $IP_1_1 = $q->param(‘IP_1_1');my $IP_2 = $q->param(‘IP_2');my $IP_2_1 = $q->param(‘IP_2_ 1");my $EA_1 = $q->param(‘EA_1’);my $EA_1_1 = $q->param(‘EA_1_1');my $EA_2 = $q->param('EA_2’');my $EA_2_1 =$q->param('EA_2_1’);my $EA_3 = $q->param('EA_3’);my $EA_3_1 = $q->param('EA_3_1);my $EA_4 = $q->param('EA_4’);my $ETA_1 = $q->param(‘ETA_1");my $ETA_2 = $q->param('ETA_2');my $ETA_2_1 =$q->param(ETA_2_1');my $ETA_3 = $q->param(ETA_3');my $ETA_4 = $q->param(‘ETA_4');my $SKA_1 = $q->param(‘SKA_1');my $SKA_1_1 = $q->param(‘SKA_1_1');my $SKA_2 = $q->param('SKA_2’);my $SKA_2_1 =$q->param('SKA215);my $SKA_2_2=$q->param(‘SKA_22');my $SKA_3 = $q->param(‘SKA_3');

174

my $SKA_3_1 = $q->param(‘SKA_3_1’);my $KBD_1 = $q->param(KBD_1');my $KBD_1_1 = $q->param(‘KBD_1_1');my $KBD_2 = $q->param(‘KBD_2");my $KBD_2_1 = %$q->param(KBD_2_1°);my $PKB_1 = $q->param(‘PKB_1);my $PKB_1_1 = $q->param('PKB_1_ 1°);my $PKB_2 = $q->param(‘PKB_2');my $PKB3 = $q->param(‘PKB_3’);my $PKB_3_1 = $q->param(‘PKB_3_1’);my $BD_1 = $q->param('BD_1’);my $BD_2 = $q->param(‘BD_2’);my $BD_3 = $q->param(‘BD_3’);my $id = $q->param(‘id’);#prepare and execute the SQL statement

my $sth = $dbh->prepare ("INSERT INTO postdata (RIAP_1, RIAP_1_1, RIAP_2,RIAP_21, RIAP3, RIAP31, IDCUAS_1, IDCUAS_2, IDCUAS_3, IDCUAS4,IDCUAS_5, IDCUAS._6, IDCUAS_6_1, IDCUAS_7, IDCUAS_7_1, IDCUAS8,IDCUAS_9, IP_1, IP_1_1,IP_2,IP21,EA_1,EA_1_1,EA2,EA21,EA3,EA31,EA4,ETA_1,ETA2,ETA2_1,ETA3, ETA4, SKA_1,SKA_1_1,SKA_2,SKA21,SKA_22,SKA3,SKA_3_1,KBD_1,KBD_1_1, KBD 2,KBD21,PKB_1,PKB1_1, PKB2, PKB3, PKB31, BD_1, BD_2, BD3, id)VALUES(?,?, 2, 25 25°25 25 2525 25 25 25 Po Po 25 Po 25 Py Po Py Py Po Py Dy Py Py Py Po Py Py Py 25 2s2525 2 25 25 Po 25 25 25 Py Poly oo oy Py os Py Py 2» ?)") Or die "Can't prepare SQL statement:".$dbh->errstr(),"\n";$sth->execute ("$RIAP_1", "$RIAP_1_1", "SRIAP_2", "$RIAP_2_1", "$RIAP_3","SRIAP3_1", "$IDCUAS_1", "S$IDCUAS._2", "$IDCUAS.3", "$SIDCUAS_4","$IDCUAS_5", "$IDCUAS_6", "SIDCUAS_6_1", "SIDCUAS_7", "$IDCUAS_7_1","SIDCUAS_8", "$IDCUAS_9", "$IP_1", "$IP_1_1", "$IP_2", "SIP_2_1", "$EA_1","SEA11", "$EA2", "$EA21","SEA_3","$EA_3_1", "$EA_4", "SETA_1","SETA2", "$ETA_21", "$ETA_3", "$ETA_4", "$SKA_1","$SKA_1_1", "SSKA_2","$SKA21","$SKA_22","$SKA_3", "$SKA_31", "$KBD_1","$KBD_1_1","$KBD2", "$KBD_2_1","$PKB_1","$PKB1_1","$PKB2", "$PKB_3", "$PKB3_1","$BD_1", "$BD_2", "$BD_3", "$id") or die "Couldn't execute SQL statement:"$sth->errstr(), "\n";

$sth->finish; #close off the statement handle

}

APPENDIX B —- SAMPLEOF PRE-COURSEPILOT SURVEY

1. Inclass/online I prefer to explore problems provided by the:

a. instructor

b. peers

textbook

d. myself

e. other. If other, please provide details:

2. I prefer to tackle problemsthat:

a. have a correct answer

b. are more open-ended

could have several answers

a©

other. If other, please provide details:

3. I prefer to explore:

a. theory

b. problems currently being faced by my peers/myself at work

case studies

a.9

other. If other, please provide details:

4. How would yourate the following, relative to the importance to your learning?

(rank 1-6 using numbers only once. 1 is most important, 6 is least important)

instructora

b. textbook

c. journal articles

d. peers

e. classroom/online discussion

f. other. If other, please provide details

5. I feel that it is more important to explore:

175

176

a. deeply into an area of knowledge

b. across a wide range of knowledge areas

c. other. If other, please provide details:

6. When judging the importance of someone's opinion, upon what do I base my

decision?

7. [know that I have fully understood something when:

8. I feel comfortable expressing my ideasin class/online, even if they are not fully

developed:

a. yes

b. no

c. other. If other, please provide details:

9. When I respond to someone'sideain class/online,I often:

do not respond

s try to think of something to add

look for holes

a©

try to rephrase the idea

add myideasto theoriginal

try to reference a reading

just be supportive

rm

mo

other. If other, please provide details:

10. At times whenpresented ideas that are different from mine,I:

adopt the new idea as my owna

b. judge the idea on its merits

c. look to see wherethere are differences

d. like to persist with an idea until I really understandit

e. allow myself to be influenced by the idea

177

f. look to see how the idea was supported

g. refuse to be influenced

h. other. If other, please provide details:

11. In courses, I feel that learning motivation,direction, and long-term planning

should beleft to:

a. instructor

b. peers

myself

d. combination

e. other. If other, please provide details:

12. I give the most credence to ideas put forward by:

a. the teacher

b. my peers

c. myself

d. other. If other, please provide details:

13. I know it is time to move on to new areas of learning when...

14. I feel that course evaluation should be usedfor...

15. I most value assessmentby:

a. instructor

b. myself

c. peers

d. standardizedtests

e. other. If other, please provide details:

16. My preferred method by whichto be assessed is (and why):

17.

18.

19.

20.

21.

22.

23.

178

I know that I learned something in a course when...

I feel that I can learn just as much from mypeers as myinstructor:

a. yes

b. no

c. other. If other, please provide details:

I feel that I am able to positively contribute to most course discussions:

a. yes

b. no

c. other. If other, please provide details:

For question 19, why do youfeel that way?

I feel that it is important for the instructor to summarize the importantideasat the

end ofthe discussion:

a. yes

b. no

c. other. If other, please provide details:

Everyonein the course should be able to express his or her views and ideas on a

topic before moving on:

a. yes

b. no

c. other. If other, please provide details:

I think that having class discussions are important only for certain subjects:

a. yes

b. no

c. other. If other, please provide details::

179

24. Briefly describe your experience with computers, online learning, and

collaboration.

25. My age groupis:

20-25 36-40 51-55 66-70

26-30 41-45 56-60 71-75

31-35 46-50 61-65 76-80

APPENDIX C - PROTOCOL FOR SCORING LEVELSOF EPISTEMOLOGICALUNDERSTANDING

The pre- and post-essays, “theories of learning”, posed four questions:

1. What do you currently understand learning to be—for yourself as a learner and

for your students if you teach?

2. Why(on whatbasis) do you hold those views, both for yourself and for your

students? (If you are not a teacher think of a situation where you have taught

somebody something.)

3. What role does knowledgeplay in learning?

4. Whatrole do others play in your learning (e.g. peers, teachers etc)?

The “Assertion” wasthe participant’s current understanding of learning, question

1. Questions 2-4 were used to judge participants’ levels based on the characteristics in

the Table Al below.

A tick was awarded for segmentsof the essay that correspondedto the applicable

characteristic and level. These ticks were then added for each level and the participant

was scored to the Absolutist, Multiplist, or Evaluativist level depending into whichlevel

the majority ofticks fell. Basing the overall level on the most ticks was necessary as

participants often showedsigns of multiple levels within the same essay and sometimes

even for the same characteristic.

180

Table Al

181

Characteristics Based on Level of Epistemological Understanding

Characteristic Absolutist Multiplist EvaluativistAssertion Assertions are Assertions are Assertions are(A strong FACTSthat are OPINIONSfreely JUDGMENTSthat canstatement that correct or chosen by and be evaluated and

somethingistrue; Statingemphatically

that somethingis

true)Reality

(Actual being orexistence)

Knowledge(General

awareness or

possession ofinformation,

facts, ideas,

truths, or

principles)

Critical thinking(Thinkingcontaining orinvolvingcomments andopinionsthatanalyze or judgesomething,

especially in adetailed way)

Commitmentto

incorrect in theirrepresentation of

reality (possibilityof false belief).

Reality is directly

knowable.“While truthaboutreality may

be difficult toachieve,it can be

made easier whenwerecognize...”

Knowledge comesfrom an external

source and is

certain.

Critical thinking

is a vehicle forcomparing

assertions toreality anddetermining theirtruth or falsehood.

Needed much

accountable only totheir owners.

compared according tocriteria of argument and

evidence.

Reality is not directly knowable“It is these realities ofothers along with ourownthat westrive to understand, yet we cannever take any ofthese realities asfixed.”

Knowledge is generated by human mindsandisuncertain.

“T think it is very important to recognize that

knowledge is not carved in stone. When we comeinto contact with people who hold very different

opinions (based on different knowledge, different“worldviews”)from our own - even radically

different and surprising opinions, beliefsystems,and waysoflife - we musttreat them withrespect.”Critical thinking is Critical thinking is

irrelevant. valued as a vehicle thatThis level of this promotes sound

characteristic is seen assertions and enhancesthrough the lack its understanding.discussion. “Learners need to

constantly question whatthey know and aim tobroaden their views

throughinteraction withnew ideas and with adiverse group of

people.”Needed some Show signsof “charting

182

Characteristic

learning

Confidencein

answering

Justification for

answer

Absolutist

guidance frominstructorto build

their

representations,relationships andunderstandings

Seemed very sureof their answers.

There are rightand wronganswers. Thereis

a definite answer.

Basedon facts.

Multiplistguidance frominstructor. Do not

yet believe that theyare capable of“charting their own

course”.“In learning the

instructor has a veryimportant role in mylearning becausethey set the toneforthe course and

provide directionforlearning to occur.”

Seemedsure oftheirsanswers becauseit

wastheir opinion.All answers are

equally correct orthere is no right

answer.

Based solely on

personal experience.“The views I hold are

basedprimarily onthe work I have donewith adults, and my

understanding ofadult learningmethodologies. I amnot a teacher, andwill limit mycommentsfor thisquestion to thecorporate

environment. ”

Evaluativist

their own course” andmaking their ownlearning goalsforlife

long learning. Able tobuild their own

representations,

relationships and

understandings.“IT recognize to be aneffective teacher, I need

to be a lifelong learnerwho continually seeks tobetter my teaching.”

Somewhat sure aboutanswers but very open to

changes based on new

information and

research.Some answers are morecorrect than others basedon justification.“I believe I have come

some way in my view onlearning. I also knowthat I still have some

way to go”

Based on a combination

of personal experience,

research, experience of

others, etc. (did they

use quotes from other tohelp justify their

answer?)“T hold these viewsbased on my many andvaried experiences as an

educator and onknowledge acquired

through study oftheresearch andtheoriesaround learning and inparticular, the

constructivist school ofthought, in this course.”

183

Characteristic

Whatvalue was

place on others

(peers, teachers,

scholars,researchers) inthe creation of

knowledge andin learning?

Absolutist

Morevalue placed

on teacher and

experts than peers

and self.

Multiplist

All have valid

opinions as no

differentiation can be

made between them.No real mention ofimportance of others

to help with criticalanalysis.“For quite some

time, I haveunderstood the

importance ofothersin my learning,

however, only nowdo I have a better

grasp ofthe extent towhich not onlyothers, but socio-cultural elementscontribute to orshape my learning”

EvaluativistAll are important with ajustified perspective.

Usedto help in critical

analysis.“Peers providealternative argumentsand counterarguments

that help a learnerprogress towards more

refined arguments orunderstanding and

knowledge, ifthey share

commitments and goalsfor progress.”

APPENDIX D - CODING CHART FOR METACOGNITION

The unit size for metacognition is not fixed to a specific number of words. It

could be as short as one sentence or be several paragraphs long. Coding should capture

the concept or complete thought.

Description (from D.J. Hacker):

Consciousand deliberate thoughts that have as their object other thoughts. They

fit into two areas:

1. Knowledge of one's knowledge, process, and cognitive and affective states; and

2. The ability to consciously and deliberately monitor and regulate one's knowledge,

process and cognitive and affective states.

Sometimes these two areas are both evident in the same thought. I have provided

examples of this case in #3 below.

Examples

1. These examples show knowledge of one’s knowledge, process, and cognitive and

affective states:

a. This shows knowledge of one’s cognitive process-

Ijudge the validity ofone theory over another based on my personalexperiences that relate to the thrust ofthe theory. I compare these thrustswith what I know andhowthey relate to my culture, my society thatincludes, myfamily, my work, my community, and my individualbeliefs.

b. This shows knowledge of one’s cognitive process and affectivestate -

Interestingly, Ifound the exercise to be inhibiting. I believe that I did notpost asfreely as I hadpreviously in other weeks. I think Ifound myselftobe uncomfortable in anonymous guise.

C. This shows knowledge of one’s cognitive process-

184

2.

185

When I mentally re-construct my existing knowledge to accommodate a

newlearning, it is an indicator that I have learned.

d. This showsan understanding of cognitive and affective state and process-

It has been an incredible 2 weeks sofar. Thefirst week was amazing to meas the ice was so quickly broken and classmates were already discussingtheir own learning! I’ve never seen such “metacognitive” discussionsevolving so quickly in an online course! I tried to explain the reasonsforithappeninglike this in Note 230. For me, that was an insight into how self-revelation promotes afeeling ofcomfort in online classes

e. This showsan understanding of cognitive process-

So basically when I a) get really motivated or b) can’t understandsomething or c) can’t explain something, I try to get outside resources.99% ofthe time they arefrom the internet...

These examples show the ability to consciously and deliberately monitor and

regulate one's knowledge, process and cognitive and affectivestates:

a. This showsa deliberate effort to regulate cognitive states -

My goalsfor the course are to establish a greater understanding of

learning in general and constructivist learning and design in particular

b. This showsan ability to monitor one’s cognitive and affective state -

This was a confusing timefor me yet again.

c. This showsan ability to monitor knowledge andprocess-

I think the discussions and reflectivejournals online require more selfandpeer evaluation because I am more slowly and deliberately thinking about

186

writing, editing and reading.

d. This shows knowledgeof one’s affective state and monitoring of

process-

I think I generally react quite poorly tofeedback andcriticism, perhapsjust in comparison with how I'dlike to react. I think that I am overlydefensive, but I am working onit :) and have noticed an improvement in

this area as I’ve become more comfortable in the online environment.

e. This showsthe consciouseffort to monitor and regulate one’s affective

state -

1 amfinding the inability to have a proper, full discussion(i.e.: face toface) about a topic somewhatdifficult. I imagine I will become morecomfortable with on-line conversations throughout the course ofthesemester, and eventually be able to both express myselfand respond toothers in a more thorough and effective manner.

3. These examples show a combination of knowledgeof, conscious monitoring of or

deliberate regulation of one’s knowledge, process and cognitive and affectivestates:

a. This showsan understanding of cognitive state and deliberate effort to

regulate one’s process-

I was struck by Bruner’s statement that we all have deep abstract thoughtsthat often are not discussed or contextualized. PerhapsifI keep this in the

back ofmy mind when I read a critic ofmyposting I will be amenable towhat they say because I believe we all have something to offer based onour experiences that we bring to the table.

b. This shows both an ability to monitor knowledge andaffective

state, and knowledge of one’s process-

Iflamfeeling confused, I tend to refrainrather than post questions to

187

help me understand becauseIgetfrustrated ifno one responds. Thisiswhere synchronous chats would help online. 1am much morelikely toblurt out what I consider dumb questions in F2F, maybe becauseit's notwritten and recorded?

C. This showsboth an understanding of knowledge and conscious monitoring

of knowledge and process-

Throughout a postgraduate course, I will usually pause after each week to

take stock ofwhat I have learnt. Ifat the end ofeach week I canfirstidentify afew conceptsfrom the assigned readings and explain them,ithas been a good week. IfI can organize the concepts and link them to one

another and Ifeel I have grasped the major “jist” ofthe intended goalfor

the week, it has been a very good week. However, ifI can take these

concepts, those ofprevious weeks and ofprevious courses and reflectupon them to link them to my everyday life, work and experiences, it has

been an extremely good week. This is the major benefit ofjournal writing:itforces me to explicitly state these links and establish relationshipsbetween them

APPENDIX E - AN EARLY DETERMINATION OF EPISTEMOLOGICAL LEVELS

The following are my recorded comments immediately following the reading of

each participant’s pre- and post-essays. In some cases I madea rating of level of

epistemological understanding immediately, while in other cases I needed to combine my

comments with other data. In these cases, the comments, plus consideration of data from

the learning logs, were used in the assignmentofa level.

1_F_36-40:High

PRE — even though she wrote her theory during week 2, one could see already

that the course reading havestarted to make her questionsher previous thinking. She

showed a great use of outside authorities to help her make her points. She used

experienced-basedjustification 4 times. She had 5 cases where she displayed dialectical

reasoning.

POST- she used only 1 experience-based justification. There were 7 sections

codedat dialectical reasoning (DR). While she was not sure of her answersin the pre,

her post posed even more questions than answers, yet also provided a muchricherset of

justifications for possible answers. She showed howherthinking had changed and how

or what/who had helped/caused that change. While there were not definite answers there

were directions towards where she was leaning. Overall, she showed movement towards

greater dialectical reasoning.

2_F_36-40:High

PRE — she presented somevery interesting experience-based (5) justifications for

her views. In addition, she used other sourcesthat shecritically analysed (3 DR). She

did seem very sure about her answers andis clearly a multipilist.

POST — lots of change noted in her learning theory. She held onto the

fundamentals from heroriginal essay but was now able to justify her beliefs agreeing or

disagreeing with other sources (2 DR). While the numberofjustifications decreased, the

188

189

complexity greatly increased as she incorporated a muchgreater range ofideas in the

search for coherence. She believed that others make a hugecontribution to her learning,

but it was herself who decided what to accept and what to reject. She also clearly

identified many areas where she has changed in epistemology, but remains a multipilist,

“Tf there is a body of knowledge existing outside of us in the world, we will never know

it as it exists. We will only know what wehavelearned ofit, coloured by our own

experiences and perception, and biased by our misconceptions. Everyone’s knowledgeis

relative. I find this fascinating. I used to be something of an objectivist.”

3_F_51-55:High

PRE — she did a good job at answering the questions mainly based onreflecting

on herpractice as a teacher (7 experience-based, 1 DR). She seemed very confidentin

her answers until at the end where she summarized andreflected on the answersshe gave.

Hereshe realized that she taught in the same manneras she best learned. She went on to

critically analyze this and asked herself lots of questions asto its merits.

POST-— the post section was ~75% shorter than the pre. She used 0 Experience-

based, | practical-based, and 4 DRjustifications. She was confident in her answers but

also left room for things to change. She extensively used authoritative sources to help

justify her answers mixed with her own personal experience andcritical analysis and

reflection (thus DR).

4M_41-45:High

PRE ~ he wasclearly a multiplist as he states, “I subscribe to a belief that there

are multiple socially constructed realities, and not a single correct view of knowledge and

reality.” Howeverdid not talk about some being more valid than others. He had a very

good understanding of where he stands on the questions andjustified using experience-

based (4) reasoning even if they conflict with his beliefs. He did consider other sources -

2DR. Heleft room to changehis views.

POST-— it was very evidentthat his justifications were much broaderand included

190

a good mix of personal experience backed up by external sources. He had one

justification that was solely experience-based and 5 DR. While his answers seem

confident, however, he did mentionin thetitle “Post-Course Theory - Still Evolving My

Theory of Learning Revised”that his theory wasstill evolving.

5_F_31-35:High

PRE shestarted by saying that this was her “current view” of learning, knowing

that it was fluid and will change. She used many viewpointsin her justifications (3 DR).

She almost presented a classic example of Kuhn et als., model of transitions from obj-

>sub->harmony.

POST- incredible post discussion. It was the most coherent argumentout of the

class. It brought together many sourcesand lines of reasoning to reach conclusions, but

still suggested that there could be better ways to be found. She directly addressed

argumentation (she has read Kuhn and Moshman- very obviousin her discussions). She

used 5 DRjustifications; in fact I might have just coded the entire submission as DR.

6F_26-30:High

PRE — she used mainly experienced based (3) reasoning on one DR. “I

do not believe that knowledgeis an absolute truth of the way things are nor do I

think knowledgeis objective. Instead I think that knowledge is a common understanding

and agreement of facts amonga sector of society or cultural group.” This seemed to

mean that there must be some way to agree on knowledge. I would put her as early

evaluativist.

POST — “I think I understand the promise of reflecting upon one’s thinking in

orderto affect one’s learning” she was muchless sure if her answers. She had a complex

understanding of how others affected her learning and knowledge. She had more DR (2)

and lots of metacognition. I would put her as an evaluativist

191

7_F36-40:High

PRE — she was very sure of her answers and backed them up using exclusively

experienced-basedjustification (2). She used one loosely translated quote to end her

ToL, “The work of a teacheris not to teach ratherit is to see to it that students learn.

Develay (liberal translation)”. It seemed very black and white.

POST -— she wasstill very sure of her answer. The entire response to the four

questions was onebig analogy with 1 practical-based and 2 experienced-based

justifications thrown in. There were hints of the theories learnt in class used twice.

8F31-35:Low

PRE — she was very cautious about her answers. She started the explanation and

justification of her theory using experience-based (2) reasoning and also questioned

herself metacognitively. She then carefully moved into a more coherentjustification (2

DR)that is again peppered with metacognitive thoughts specifically goals and

commitments to greater learning (and application) to come. She showeda real

commitmentby stating somelearning goals. Evaluativist.

POST- again she stressed the transitory state of her theory of learning, however

this time she felt much more confident and justified by explaining her changesin

epistemology (2) using DR (2) and using examples of metacognitive regulation (2). “I

canstill live with myinitial thoughts on learning...essentially, it’s the depth and increased

saturation of sense, meaning and context that constitute the changes in my thinking.” She

also included manyauthoritative sources. This lead to a very coherent and well thought

out explanation of her theory butstill left room for changes and improvements with a

stated commitmentto look for these ways to improve. Again Evaluativist.

9M_41-45:Low

PRE — he introduced and finished his essay by saying that it was in a hugestate of

change even as he was writing it. He was aware of manytheories but choose his mainly

192

based on his own personal experience (3). He seemedto rely on the teacher for

motivation and direction “Learning is therefore often linked somehow to our emotional

state. I think I learn more whenI find a teacheris energetic and engaging.” Near the end,

he did try to bring his previous thoughts andjustifications together in a coherent manner

(1 DR). I would classify as early multipilist.

POST -— it seemed that he had a change in epistemological thinking by realizing

the importancethat other play in his learning. In fact, his theory comes togetherat the end

through reference to 5 classmates andtheir input to his theory (1 DR). However,he is

still believes knowledge to be very subjective, dependent on one’s point of view. I would

still rate him as a multipilist, as he did not justify why he felt the way he did about others’

theories and comments.

10_M_41-45:Low

PRE — he wasvery confident in answering the questions and used experienced

based (1) and analogical based reasoning (1). I rated him as an early multiplist (and not

a absolutist) mainly becauseas stated that “others could havea different view of

knowledge”.

POST -— very little change from pre. He does use more experienced based

justifications (4) and only used one source from the course. Heis still very sure ofhis

answers. I would say no changein his epistemological level based on the essays.

11_M_26-30:Low

PRE ~ he used only experience-basedjustifications (2). He seemedfairly

confident in the answers he gave, but did mention at the beginning and end that“I do

recognize that my own understanding about learningarestill incomplete”. I would

classify as multipilist.

POST— he showedgreat progressin the sophistication ofhis justifications (2 DR)

by including authoritative sources. “I think it is not so much a matter of my theory of

193

learning changingas it is seeing more justifications for my belief in student centered

learning.” He showed movementin goalsetting, “The concept of Zone of proximal

developmentis an important concept to me because it demonstrates the need to have

students workin a challenging, collaborative environment wheretheyare assisted in their

learning, provided feedback and gradually given the freedom to take responsibility for

their own learning”. He also showed movementin epistemological thinking concerning

the distribution of knowledge. However he made one commentaboutthe nature of

reality that seemed somewhatabsolutist, “while truth about reality may be difficult to

achieve, it can be madeeasier...”. I believe that he was close to movingto the

evaluativist level?

12_F_26-30:Low

PRE — she wassure of her answers and used experienced (4), practical (1) and one

DR (somewhat weak). Multipilist.

POST still sure of her answers though they have changed. She used one source

from the course but did not justify her answers. She saw the importanceof others but

only to the extent of sharing their experiences. Very Multiplist.

13_M_41-45:Low

PRE — he seemedfairly confident in his answers. He used 5 experience-based

justifications, 4 practical-based and 1 DR. Hetalked about the importance ofothers but

did so in such an encompassing wayit washard to tell what was important to him.

Multipilist.

POST — he has included many sourcesasjustification for his beliefs. He used 2

experience-basedjustifications, 3 practical-based, 2 analogical, and 1 DR. Hedid not

expand the sophistication of types ofjustification. His still seemed confident in his

answers. He did show movementin whyhe thought that others were importantto his

learning, “they provide a different perspective that can be used to challenge your own

thoughts on a subject”. Multipilist.

194

14F_26-30:Low

PRE — she seemedfairly confident in her answers. Provided lots of experience-

based justification (3 in total and 1 practical-based) for her answers. She did realize the

importance of others for learning and is open to reconstructing here views on learning

and teaching (1 DR)

POST — expansion of learning to include the importance ofthe social (others

influence). She wasstill fairly confident in her answers. Still very experience-based (4)

in her justifications, however did seem to go througha significant change in one aspect of

epistemology. She saw the value of others in her learning and shestated, “I have

definitely come a long way!”

15_F_31-35:Low

PRE — seemedfairly confident in her answers. She used 5 experience-based

justifications and 1 DR. Shedid realize the importance of others in her learning as they

challenge her do her best and sharedtheir passion for certain fields.

POST still seemed fairly confident in her answersbut she self-identified several

significant changes to her previous conceptions. She had 3 experience-based

justifications and 3 DR. Sherealized the importance of others not only in their ability to

motivate her to learn but also in their ability to produce artifacts that she was able to use

to better understand and confirm her own experiencesas valuable knowledge. I think

there was movement here towards the evaluativist level.

APPENDIX F - RUBERIC USED FOR SELF-EVALUATION OF LEARNING LOGS

Learning Journal: (due weekly, starting in Week 3 (a reflection on week 2, and so

on). This assignment involves an ongoing reflection on your changing understanding of

ideas in the course particularly in relation to ideas from other courses you are taking or

have taken. (23%*) Tofacilitate this reflection, each week a set of questions will be

provided for you to choose a focus for your reflection.

The idea behind the learning logs is to have a location to deliberately reflect and

try to integrate ideas into higher order syntheses. Minimally, weekly contributions are

required. The learning logs can take the form of a single note that you add to weekly, and

these should be put in the Learning Logs View.Each entry should be about 1-3

paragraphs,it is more about insight than length! You can include any numberofthings,

for instance you could note parallels in concepts from different articles or approaches to

research;identify contradictions or confusions you notice; come up with interesting

questions unansweredin the literature. You should include justification and evidence for

your ideas. You are welcome to commenton each other’s logs, but rememberthat these

represent each person’s unique journey, so comments should focus on developing ideas

or noting parallel concerns.

Belowis starting point rubric for evaluating learning logs.

*This element of the course assessment is fundamentally a self-assessment, each

of you will be asked to submit a mark you think is fair. To move us toward that end I

wantus, as a class to considerthis rubric during the course, and consider how it might be

modified to support deep learning through applying some ofthe ideas we investigate in

the course.

Beginning Rubric for Learning Log Reflections(4 is highest, 1 is lowest)

4. Entries were regular and characterized primarily as very reflective (i.e.,

reacting to ideas from the course materials, elaborating ideas from readings and from

others’ contributions, and contemplating or extending ideas from readings and from

195

196

others’ contributions. Value-added contributions). As well, entries monitor the learner’s

own growth in understanding, and monitor learner understanding by providing

appropriate evidenceandjustification.

3. Entries mainly characterized asreflective (i.e., reacting to ideas from the course

materials, elaborating ideas from readings and from others’ contributions,).

2. Entries mainly characterized by description and limited depth ofreflection (i.e.,

reacting to ideas from the course materials).

1. Entries were infrequent and characterized primarily by description and limited

depth of reflection(i.e., reacting to ideas from the course materials).