Communication Patterns in Massively Open Online Courses

This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institution

and sharing with colleagues.

Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to personal, institutional or third party

websites are prohibited.

In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website orinstitutional repository. Authors requiring further information

regarding Elsevier’s archiving and manuscript policies areencouraged to visit:

http://www.elsevier.com/authorsrights

http://www.elsevier.com/authorsrights

Author's personal copy

Communication patterns in massively open online courses

Nabeel Gillani a, Rebecca Eynon b,c,1

a University of Oxford, Department of Engineering Scienceb University of Oxford, Oxford Internet Institutec University of Oxford, Oxford Internet Institute, 1 St Giles, Oxford, OX1 3JS, UK

a b s t r a c ta r t i c l e i n f o

Article history:Accepted 19 May 2014Available online 28 May 2014

Keywords:Massively open online coursesSocial network analysisDiscussion forumsCommunities of practiceSocial constructivismhigher education

Despite the hype and speculation about the role massively open online courses (MOOCs) may play in higher ed-ucation, empirical research that explores the realities of interacting and learning in MOOCs is in its infancy.MOOCs have evolved from previous incarnations of online learning but are distinguished in their global reachand semi-synchronicity. Thus, it is important to understand the ways that learners from around the world inter-act in these settings. In this paper, we ask three questions: (1) What are the demographic characteristics of stu-dents that participate in MOOC discussion forums? (2) What are the discussion patterns that characterize theirinteractions? And (3) How does participation in discussion forums relate to students' final scores? Analysis ofnearly 87,000 individuals from one MOOC reveals three key trends. First, forum participants tend to be youngadults from theWestern world. Secondly, these participants assemble and disperse as crowds, not communities,of learners. Finally, those that engage explicitly in the discussion forums are often higher-performing than thosethat do not, although the vast majority of forum participants receive “failing” marks. These findings have impli-cations for the design and implementation of future MOOCs, and how they are conceptualised as part of highereducation.

© 2014 Elsevier Inc. All rights reserved.

1. Introduction

Since 2011, massively open online courses (MOOCs) have been asubject of both hope and concern. Advocates like Thomas Friedmanhave enthusiastically labelled MOOCs as the revolutionary forces thatwill disrupt traditional practices in higher education and others havebegun to investigate how they may help people in the developingworld gain access to education (Ryan, 2013). Others, like philosophyprofessors at San Jose State University, have expressed public concernand frustration over the displacement of diverse, personalized pedago-gy with one-size-fits-all courseware, reminiscent of David Noble'sprophecy of online learning as a harbinger of “Digital Diploma Mills”(Noble, 1998).

If the evolution of MOOCs follows Gartner's hype cycle (Fenn,Raskino, & Gammage, 2009), then the polarised response to suchcourses is to be anticipated. Media coverage of MOOCs has fuelledboth hype and qualified disillusionment as members of the media, aca-demia and industry have offered their perspectives on the future ofhigher education.

However, similar to previous online initiatives in higher educationbefore the introduction of MOOCs, many of these discussions have notbeen grounded in educational theory or a systematic, rigorous

investigation of this courseware's potential to enhance learning. Yet,as history has told us, empirical investigation of the actual realities ofsuch initiatives is essential to move the debate forward (Selwyn, 2011).

In this paper we aim to contribute to this goal, through focusing onone central aspect of MOOCs whichwe suggest are different to previousincarnations of online open education. Prior to the introduction ofMOOCs, learning via open education initiatives tended to be a relativelyindividual experience, where students navigated content primarily ontheir own.MOOCs are different from preceding open online learning ef-forts because they potentially enable a global student body to learnthrough online videos and assessments, as well as forum discussionswith other students and course staff while a course is “in session”. Wepropose that this structure encourages semi-synchronous interactionsbetween course participants that may ultimately affect the ways inwhich they learn. While a number of theoretical educational perspec-tives shed light on the role of discussion in learning, this notion hasnot been thoroughly investigated in the context of global onlineeducation.

2. Background

MOOCs can be viewed as the newest incarnations of earlier attemptsat open education at a distance, which has previously included MITOpenCourseware (OCW), iTunesU, and Carnegie's Open Learning Initia-tive (Abelson, 2008; Johnstone, 2005; McKinney, Dyck, & Luber, 2008).MOOCs are hybrids of previous attempts at online distance education:

Internet and Higher Education 23 (2014) 18–26

E-mail addresses: [email protected] (N. Gillani), [email protected](R. Eynon).

1 Tel.: +44 1865 287206.

http://dx.doi.org/10.1016/j.iheduc.2014.05.0041096-7516/© 2014 Elsevier Inc. All rights reserved.

Contents lists available at ScienceDirect

Internet and Higher Education


they bring together early approaches to online learning and the scaleand potential reach of open courseware efforts.Many online learning ef-forts started off as being synchronous, and therefore, engendered groupdiscussions, but the scale of these discussions was small, and the partic-ipants often geographically homogeneous. The rise of global-scale openeducational resource repositories like OCW and iTunesU offered accessto content “to the masses” but failed to create a space, at least at theironset, where learners could engage with new content together.

While most MOOCs so far have focused on conveying university-level content, we suggest that their potential perhaps lies in enablingthe creation of new knowledge and insights through online learner in-teractions. It is perhaps these interactions in MOOCs—conversationsthat are framed by course content, between thousands of people fromaround the world, each with unique life experiences and socioculturalcontexts—that separate them from previous distance learning effortsand thus deserve significant research attention. While MOOCs are notentirely synchronous in comparison to most offline settings since theyenable learners to watch lectures and participate in discussions accord-ing to their own schedules, they do have a defined start and end date,and many encourage students to partake in weekly assignments asthe course progresses (Kizilcec, Piece, & Schneider, 2013). To begin tounderstand what MOOCs mean for education then, it is important tofirst learn howparticipants in these settings come together, discuss con-cepts, and form communities. Indeed, some students in MOOCs spendas much or more time on using the discussion forums as they do view-ing lectures or doing homework (Seaton, Bergner, Chuang, Mitros, &Pritchard, 2013), suggesting that the discussion component of thesecourses is a particularly important area of study.

There are a number of theoretical frameworks that highlight thevalue of social interactions to support learning (Leont'ev, 1978;Vygotsky, 1978) that can and have been used to understand how indi-viduals interact and learn in online discussion forums. For example, inStahl's work on computer-supported collaborative learning (CSCL) hedemonstrates how learners can collaboratively create knowledge asthey interact online (Stahl, Koschmann, & Suthers, 2006). Stahl is partic-ularly explicit in highlighting the differences between cooperation inthese learning settings—where individuals are largely responsiblefor working on their own within a group setting in order to gainknowledge—and collaboration, which, as Roschelle and Teasley (1995)explain, involves group-directed negotiation and the construction ofshared goals and meaning. These notions of interaction are especiallyimportant when analysing how learners in MOOCs engage with theirpeers. It is unclear if and when collaboration or cooperation takesplace among thousands of learners, nor is it obvious underwhich condi-tions each type of interaction occurs.

Another important interaction framework to consider is one of“communities of practice” (CoP), or groups of individuals thatshare a common profession or trade and engage in collaboration toachieve a shared set of objectives (Lave & Wenger, 1991). The con-cept of CoP in recent years has been broadly interpreted, adapted,and applied to better understand how people learn with each otheronline (Thomas, 2005). An outstanding question, however, is if thenotion of a CoP—a structure that is characterized by increased en-gagement and collaboration among its participants over time—canapply to discussions in a massively open online learning setting. Inthe past decade, some scholars have noticed the potential conflictbetween previously-held frameworks (like CoP) for describing andunderstanding learning in light of today's society—a society defined,in part, by interconnected webs of knowledge and information,many of which can be accessed instantly, on-demand. Connectivismhas emerged as an alternative learning theory that emphasizes therole of the individual as an actor in a larger, imperfect, uncertain,and sometimes chaotic network of knowledge sharing (Siemens,2005). Each of these theories then, focuses on relationships betweenlearners and how these relationships affect various participation andinteraction modalities. Together, they highlight the need to examine

the nature of learner interactions in MOOC forums and to understandthe learning implications of such activities.

Methodologically, there are a number of approaches to analysingonline interactions. Content or discourse analysis has been used in pre-vious higher education research in order to understand learner interac-tions that take place online (De Weaver, Schellens, Valcke, & Van Keer,2006; Gunawardena, Lowe, & Anderson, 1997). While valuable, suchtechniques would be practically impossible in MOOCs due to the thou-sands of participants.MOOCs offer researchersfine-grained data collect-ed on learners' participation andmutual interactions (e.g. Breslow et al.,2013; Kizilcec et al., 2013) that have never been available at such scalesbefore. These “digital traces” that were once virtually impossible to cap-ture in campus-based contexts—such as the number of students thatopened a textbook before the final exam, or which students spoke towhich other students about the final problem set—are now availableat unprecedented levels of detail within the MOOC setting.

While these digital traces could be analysed in a variety of differentways, herewe employ network analysis. Network analysis has explodedin recent years as a method of investigating how individual actors—in-cluding those in educational contexts—interact with one another(Easley & Kleinberg, 2010). Social network analysis (SNA) in particularhelpsmodel the spatially and temporally influenced social relationships(edges) between individuals (nodes). From such analysis it is possibleto understand who is talking to whom in a MOOC and how these inter-actions develop and change over time. Previous studies in educationhave leveraged these techniques, albeit with small-scale datasets(Cho, Gay, Davidson, & Ingraffea, 2007; Palonen & Hakkarainen, 2000).The rise of “big data” (boyd, and Crawford, 2012) in recent years andthe tools that enable its analysis have encouraged more recent large-scale investigations that leverage the theory and practice of SNA inlearning contexts (Kossinets & Watts, 2006; Vaquero & Cebrian, 2013).Therefore, MOOCs, and the discussions they harbour, are well-suitedto benefit from the methodological advances that have fuelled thegrowth of big data research and network analysis in recent years.

2.1. Research questions

This study examines data from a MOOC with an emphasis on real-world learning in order to generate initial insights intowhouses the dis-cussion forums, how, and forwhat purpose. The investigation is primar-ily exploratory in nature; it employs observations, surveys, and socialnetwork analysis to gain insights into the structural attributes of com-munication between learners. It also explores the demographic charac-teristics of forum participants. Finally, because many learners appear totakeMOOCs for different reasons andwith different levels of formal en-gagement (Belanger & Thornton, 2013; Breslow et al., 2013), we inves-tigate how forumparticipation relates to the students' finalmarks in thecourse. Our research questions are as follows:

1. What are the demographic and educational characteristics of stu-dents that participate in MOOC discussion forums?

2. What are the discussion patterns that characterize students' partici-pation in the forums?

3. How does participation in discussion forums relate to students' finalscores in the course?

3. Methodology

This research takes a case study approach in order to address thethree research questions outlined above. Such an approach is particular-ly appropriate given the relative newness of the topic (Yin, 2009). Thesubject of the case study is on a purposively selected Coursera-basedMOOC on Business Strategy that aimed to promote interaction andlearning around real world business problems; using both weekly dis-cussions of business cases in course forums and a peer-assessed finalproject that enabled students to conduct a strategic analysis of any

19N. Gillani, R. Eynon / Internet and Higher Education 23 (2014) 18–26


existing company. It is important to stress that it was this focus on thepromotion of interaction in the forums that was the reason the casewas selected for this study. The six-week long course began in March2013. Over 87,000 participants signed up for the course. As will bediscussed further below, precise numbers of participation changethroughout the course, as learners join late or drop out over the course.

Fromprevious research in this area, it seems thatMOOC participantstend to be young but still old enough to have had one or two full-timejobs; many are lifelong learners that already possess undergraduate oradvanced degrees; and some are looking at the MOOC as a way to en-hance their professional credentials and skill sets (Belanger &Thornton, 2013; Deboer, Breslow, Stump, & Seaton, 2013). While thisdoes not support the widening participation argument that manyhave made about MOOCs (Marr, 2013), it does indicate their potentialto be powerful educational resources. Because MOOC participantshave unique life experiences and motivations that they carry withthem to the course, interactions between diverse groups of learnersmay enable students to share their ideas and insights with their peersaround topics that have the capacity to impact their “real”, offline lives.

In this particular course, students were responsible for watching aseries of short, 5–10 minute lecture segments (totalling approximately1–2 hours per week) and reading segments from the course's underly-ing text, The Strategist's Toolkit eachweek. Thereweremandatoryweek-ly quizzes that aimed to help reinforce lecture content. Students werealso encouraged to read a business case study each week and to discussthe cases in the discussion forums by answering a series of related ques-tions. Students were required to complete a final project, which in-volved undertaking a strategic analysis of an organization of theirchoosing.In this case study we used two main methods: pre and post-course surveys of learners and analysis of learners' “digital traces”.

3.1. Pre and post-course surveys

In order to gain insights into course participants' backgrounds,moti-vations, and overall experiences with the course, both pre and post-surveys were administered through the MOOC platform. The pre-course survey had seven closed response questions and asked studentsto indicate their previous educational experiences, current professions,and motivations for joining the course.

In addition to repeating the questions on the pre-course survey, thepost-course questionnaire had additional questions that gauged howmuch time participants spent on the course and their overall experi-ences. The post-survey had twenty-six questions and was distributedthrough the online course interface. Using the course platform to man-age survey creation and distribution enabled survey responses to belinked to the digital traces collected for each participant.

A total of 7337 students responded to the pre-course survey,representing approximately 9% of all enrolled students at the start ofthe course, and almost 15% of those that logged in at least once. Therewere 1000 studentswho completed the post-course survey. It is difficultto calculate precise response rates due to the transitory nature of someparticipants. Nevertheless, the response rate is low and findings need tobe caveated with the likely existence of response bias (Armstrong and

Overton, 1977; Couper, 2000), since most respondents to the surveywere those that completed the course and a large proportion of regis-trants never once logged in. Tables 1 and 2 provide responses to samplequestions from the pre and post-course surveys.

3.2. Learners' digital traces

The cornerstone of this study's methodology is the analysis of gran-ular learner data. These data were recorded throughout the course andincluded full-text records of forum participation and students' finalmarks in the course. Each assignment submission, lecture view/down-load, and forum post/comment/thread creation was time-stamped forover 87,000 participants. These records were kept anonymous byusing obfuscated IDs for each learner.

As Haythornthwaite suggests, empiricism is a key part of social net-work analysis: conclusions about communication between social actorscan only be made after observing their interaction patterns, and not apriori based on previous classifications. In this way, a collection of actorscan only be described as belonging to a “group” once the network under-lying their interactions is derived and interpreted (Haythornthwaite,1996). Observing and interpreting the network are a precursor, and fun-damental, to understanding the existence of communities in socialsettings.

Informal observations of forum activity were recorded through ap-proximately 2–3 hours of weekly analysis during the course's progres-sion. The objective of these observations was to examine specificinteractions between students in the different sub-forums in order tobegin to understand the type of content that was being generated—i.e., to add a qualitative layer to the data-driven network visualizationaspect of the study. While in future research a more sustained and in-depth ethnographic approach would be valuable, these observationshelped explore the research questions by confirming the demographictrends observed through the surveys, as well as the patterns of commu-nication interpreted through the social network analysis component ofthe investigation.

4. Results

4.1. Background information on course participants

Approximately 143 countries were represented from six differentcontinents. Over 2500 students received a nonzero score in the course,implying that they submitted at least one quiz or the final project.Over 47,000 students were considered “active” (i.e. logged into thecourse's website at least once) and more than 39,000 of them viewedor downloaded at least one of the lecture segments. These patterns re-flect the different interpretations of what it means to “participate” inthese settings (Kizilcec et al., 2013) and will be discussed more below.

Based on the responses to the pre-course survey, participants wereprimarily well-educated young adults from developed countries withprior experience in business. Nearly 40% of respondents were betweenthe ages of 25 and 34, over 80% had at least a bachelor's degree, and ap-proximately 40% had a master's or professional degree (e.g. MBA, J.D.).

Table 1Sample pre-course survey questions and responses.

Question Responses

Please indicate your highest level of education. Some high school (1.5%); completed high school (4.2%); some college (12.2%); bachelor's degree (42.1%); master's degree(36.3%); doctorate (3.5%)

Please indicate your age. Under 25 (28%); 25–34 (39.9%); 35–44 (19%); 45–54 (8.8%); 55–64 (3.6%); 65 and over (0.7%)Which of the following best describes you? Hobbyist (4.6%); academic/professor (2.7%); research scientist (1.8%); industry professional (54.6%); student (25.6%)Whydid you enroll in this course? (Please check allthat apply.)

Subject is relevant to my occupation or field of study (57%); I don't have access to other educational resources (5.1%); course isoffered by a prestigious institution (28%); professional development (80.9%);Want to enhancemy resume (37.8%); curious aboutonline courses (24.7%); think the course will be fun and enjoyable (37.5%)

How many hours per week will you dedicate tothis course on average?

1–3 (23.7%); 4–6 (53.3%); 7–9 (16.2%); 10–12 (5.2%); 13–15 (1%); N15 (0.7%)

20 N. Gillani, R. Eynon / Internet and Higher Education 23 (2014) 18–26


Moreover, approximately 54% of pre-course and 58% of post-course sur-vey respondents identified themselves as “industry professionals”. Themost popular motivation for taking the course was “professional devel-opment” (see Table 1).

4.2. Background attributes of forum users

At the time of the data export, 4337 participants had created at leastone post or comment in the online discussion forums. This number was60% higher than the aforementioned 2500 individuals who received anonzero final score, suggesting that a large number of forum users par-ticipated in discussions without submitting any of the required assign-ments. Forum participants accounted for just over 4% of all courseregistrants. Still, while comprising a small proportion of all users,forum participants were still a large and highly interesting group. Theremainder of this analysis will focus on these individuals.

Understanding who used the MOOC forums was fundamental togauging how this participation occurred. Geographic informationcould be identified for 3631 out of 4337 total forum participants. Fifty-seven percent of identifiable forum participants were from eitherEurope or North America and 24% were from Asia. Table 3 shows theproportion of identifiable forum participants by continent. For compar-ison purposes, the course-wide proportions for those with known loca-tion information are also shown.

A total of 1984 forum participants indicated their highest level ofprior education on the pre-course surveys. Eighty-three percent ofthese individuals had a bachelor's or master's/professional degree. Ap-proximately 10% (204) of forum participants who completed the pre-course survey indicated “some college” as their highest level of educa-tional attainment. Table 4 shows the educational attainment for forumparticipants and course registrants.

Based on the 1964 forum participants that indicated their motiva-tion for joining the course on the pre-course survey, nearly 93% indicat-ed professional development as their primary motivation; 6%participated mainly out of curiosity and a desire for enjoyment; 1% par-ticipated primarily because they did not have access to other education-al resources, and less than 1% because the course was offered by aprestigious institution. These percentages were virtually identical forthe broader set of survey respondents (i.e., irrespective of forum partic-ipation). This suggests that all students taking the course, includingforum participants, were primarily interested in gaining skills andknowledge that they could then apply to their professional lives—andthat the forum participants, in terms of motivation, were indistinguish-able from other course participants.

4.3. Social network analysis: visualizing virtual discussions

The course's discussion forum comprised the following sub-forums:technical feedback, where students suggested technological compo-nents of the course; course material feedback, where students sug-gested how to improve content and the communication of keyinformation; study groups, where learners introduced themselves andarranged meet-ups in their respective localities or other online venues

(e.g. Facebook); readings, where students could discuss concepts cov-ered in the text; lectures,which facilitated discussion around topics cov-ered in lecture videos; cases, which engaged students in weeklydiscussions about the strategic decisions and market positions of com-panies such as Google, Apple, and Disney; and Final projects, which en-abled students to ask questions and engage others on the final caseanalysis assignment. A social network analysis approach was employedto visualize and interpret forum participation data. Using the visualiza-tion capabilities of the Python-based NetworkX library, social networkswere computed and illustrated to depict how discussions emergedaround various topics.

4.3.1. Different network topologies for learners' forum discussionsWe used social network analysis to capture broad trends in commu-

nication patterns in the forums. A number of key decisions need to bemade in building these networks which have important implicationsfor the network structure. Here, we analysed the dynamics of eachsub-forum separately for a more granular understanding of learner be-haviours. Network nodes represented learners that created at least onepost or comment in a discussion thread; an edge connected twolearners simply if they co-posted in at least one discussion thread. Ourconnections between learners were undirected as there was no obviousway to discern whichway information was transmitted between actorsin this relationship.

In this study, participation was defined as posting text to the discus-sion forum.We recognize that this is a relatively simple definition, but isappropriate in the context of this large-scale quantitative study that isfocusing on patterns of interaction. Simply viewing posts withoutadding to the discussion explicitly did not qualify as participation, al-though previous literature has focused on investigating this type offorum-lurking behaviour more explicitly (Preece, Nonnecke, &Andrews, 2004).

The network visualizations were analysed in terms different learnerattributes, including geographical location and final course perfor-mance. The colours used for nodes in the networks represented a partic-ular attribute of forum participants (see Fig. 1). To preserveconfidentiality of participants, the network diagrams are discussedonly in aggregate terms.

4.3.2. Key structural trends observed via thread networksFor large time-intervals (e.g., the entirety of the course), the “thread

network” visualizations tended to be dense because the criteria forconnecting two nodes were relatively weak. Therefore, we focused onweekly visualisations of each sub-forum.While some of these visualiza-tions were still dense, they offered more detail and insights into the nu-ances of forum discussion patterns than longer time windows, and/oranalysis of all interactions across all the forums.

Below, we explore the thread-networks for two sub-forums to illus-trate similarities and differences in interaction patterns.

4.3.2.1. Readings sub-forum. The readings sub-forum enabled students todiscuss the weekly textbook readings. Fig. 1 captures the dramaticchange in this sub-forum's communication trends from week 1 to

Table 2Sample post-course survey questions and responses.

Question Responses

I enjoyed this course. Strongly disagree (0.9%); disagree (0.4%); neutral (3.5%); agree (36.3%); strongly agree (58.9%)My experience in this MOOC has motivated me to take more MOOCs. Strongly disagree (1.9%); disagree (0.8%); neutral (11%); agree (39%); strongly agree (48%)I am confident in my ability to strategically analyze businesses to helpthem address challenges.

Strongly disagree (1.8%); disagree (1.3%); neutral (11.9%); agree (65.6%); strongly agree (19.5%)

My favorite part of the course was ______. Lectures (52.1%); quizzes (9.3%); cases (11.2%); final project (25.9%); other (1.4%)I completed this course because (Please check all that apply): _________. I wanted formal acknowledgement (62.7%); I enjoyed discussions with other students (15.2%); I

wanted to help an organization in the course (13.6%); I enjoyed the course material (76.7%)How many hours/week did you dedicate on average to this course? 0 to 3 (32.3%); 4 to 6 (46.5%); 7 to 9 (13.5%); 10 to 12 (5.3%); 13 to 15 (1%); 16 to 18 (0.7%); N 18(0.9%)



week 2. Week 1 was characterized by a very dense network with a de-fined centre of communication.Most participants appeared to engage ina number of different threads; this yielded a network where manynodes are connected to many other nodes. In week 2, however, almostthe opposite was true. Participation levels dropped significantly andforum users appeared to bemore focused in their communications, par-ticipating in a small number of different discussions. The remainingweeks showed similarly declining participation and smaller group-focused discussions became more typical.

4.3.2.2. Study groups sub-forum. The study groups sub-forum enabledparticipants to connect with one another to form support networksaround the course's content. Observations revealed that most threadsin this sub-forum were created by students in order to form studygroups around particular time zones, languages, or geographiclocations. Even though the sub-forum was established to encouragethe formation of small groups, week 1 showed a dense centre of com-munication with just a few isolated node groups on the periphery.This suggests thatmost studentswere not exclusive in their interactionsand instead participated in many threads early on.Week 2 demonstrat-ed a shift towards smaller groups ofmore focused, exclusive interaction.Participation dwindled in subsequent weeks and, as a result, the num-ber of distinct study groups using the forums also decreased. Fig. 2shows how interactions in this particular network evolved.

4.4. Extending the analysis of MOOC communication networks

The network images illustrate a growing trend towards “fragmenteddiscussions” between learners in theMOOC. Thismodularity in commu-nicative tendencies across sub-forums suggests that learners gravitatetowards dispersed, focused discussions after an initial period of morecentralized interaction.

4.4.1. Persistence of participationTo understand consistency of participation in discussions, we

computed the Jaccard Index for different sub-forums. This is asimple measure of similarity between two sets. For any sets A andB, J(A,B) = |A∩B||A∪B|, i.e., the Jaccard similarity index between Aand B is equal to the ratio of items in both A and B to those itemsin A or B (Hamers et al., 1989).

The Jaccard indices for pairwise forum participation reveal that par-ticipation was highly inconsistent. The largest overlap in the number ofparticipants from any two sub-forums over the entire span of the coursewas 0.13, which occurred between the cases and final projects sub-forums. This means that these settings shared only 13% of the partici-pants that engaged in either sub-forum. The remaining overlaps weremuch lower, dropping to as low as 1.4%. The largest similarity betweenany twoweekly networks over all sub-forumswas .16 betweenweeks 5and 6, and the lowestwas .002betweenweeks 1 and 7.Weekly overlapswithin sub-forums demonstrated higher participation consistency insome instances: the participation overlap between weeks 4 and 6 and5 and 6 in the cases sub-forum was 0.26 and 0.25, respectively. Thisanalysis suggests that most forum participation was not due to thesame few individuals in the course—that disparate crowds, not cohesivecommunities, of learners utilized the discussion forums.

4.4.2. Forum participation and performance in the course

4.4.2.1. Forum participation by final score. Our final analysis investigatedforum activity with respect to students' final marks. Just over 20% offorum participants earned 90% or higher in the course, even thoughonly 2% of course participants overall finished with these scores.

To further understand the comparison of forum users' final scoreswith those of their other course counterparts, we computed a one-way analysis of variance using the Kruskal–Wallis test (Kruskal &Wallis, 1952) to compare the mean final score of forum users withthose that watched or downloaded at least one lecture; primarily be-cause comparing against the larger body of course registrants—nearly87,000 individuals—would have yieldedmean final scores that were se-verely skewed. The results indicated a strongly statistical and practicalsignificant difference between the mean final scores of forum partici-pants (24.4%) and those that participated in lectures (5.2%) (p= virtu-ally 0.0, H-score= 2191). Practical significance (e.g. the effect size)wascomputed using Cohen's d, yielding a value of 4.0 (d N .8 implies practi-cal significance). These values suggest that there is a statistically andpractically significant relationship between a student's participation inthe forums and his or her performance in the course, although it cannotbe concluded that forum participation caused higher final marks.

4.4.2.2. Conditional distributions of users' attributes and scores. Analysingthe final marks of course participants in the context of earlier demo-graphic data helps reveal relationships between these two forumpartic-ipant characteristics. Over 35% of those that earned 90% or higher in thecourse were based in Europe—the second highest cohort was approxi-mately 26% from North America, followed by 23% from Asia.Europeans also comprised the highest proportion of forum users withat least a master's degree. Unsurprisingly, then, those with at least amaster's degree also comprised the largest cohort (nearly 45%) ofusers receiving at least 90% in the class. Forum participants with abachelor's degree comprised the largest group of participants thatearned 50% or less in the course.

4.4.2.3. Assortativity by final score. Previous research has investigated thetendency of high-performing students to interact with other high-performing students in online learning settings (Vaquero & Cebrian,2013). Tomeasure the existence of similar trends in thisMOOC's discus-sion forums, we computed the assortativity of each network withrespect to participants' final marks. At best, there were weak correla-tions between participants' final scores and the final scores of thosethey tended to interact with. The highest final-score assortativity, orPearson r-value, was 0.23 in the final projects forum. The lowest was0.07 in the sub-forum dedicated to discussions about the video lectures.Table 5 shows these assortativity values for each sub-forum,with corre-sponding two-sided p-values. Note that due to the large sample sizes, allbut one of these correlations are statistically significant at the *p b 0.05level. In the context of Cohen's effect sizes for social science research(Cohen, 1992), most of these correlations indicated, at best, relation-ships between communication and final scores of small to mediumpractical significance.

Here, we note that a lack of strong assortativity does not implydisassortativity, or a tendency for certain grade earners to interactwith people from other grade-earning cohorts. Still, relatively weakassortativity values implies there is no compelling indication thatstudents who achieved final marks similar to others also favouredcommunication with those individuals.

5. Discussion

In this sectionwe provide an overview of the key findings from eachof our research questions.

Table 3Proportion of forum participants and total course participants from each continent.

Continent % Course participants (N) % Forum participants (N)

North America 32% (15,481) 32% (1,145)South America 7% (3496) 10% (372)Europe 28% (13,412) 25% (918)Asia 26% (12,723) 24% (879)Africa 5% (2453) 7% (234)Oceania 2% (1125) 2% (83)



5.1. Forum participants are well-educated, primarily from the Westernworld, and want to gain skills that will help them professionally

The results revealed that most forum participants (over 80%) had atleast a bachelor's degree and that two thirds were from the Westernworld—both trends that have characterized many MOOCs to date(Breslow et al., 2013; Kizilcec et al., 2013). Thus the arguments thatMOOCs widen participation need to be assessed more critically. Whileoutside the scope of this paper, it may well be that language was an im-portant factor here. The course was offered in English, thus those whosefirst language is English may well fare better in the forums. However,more work is required as we only have geographic data, not dataabout the first language of participants. Overwhelmingly, “professionaldevelopment” was the most popular reason for taking the course:over 93% of forum participants indicated this as their primary motiva-tion, while most others (6%) were motivated by curiosity about thecourse or the pursuit of enjoyment. Unlike the geographic and educa-tion data, the statistics for learners' motivations are very different thanthose reported for earlier MOOCs (Belanger & Thornton, 2013). There-fore, taking our findings with those of other research, there is no singlereason why people take MOOCs. Their motivations are likely to vary ac-cording to the type of course and their own unique backgrounds andexperiences.

Knowing forum participants' motivations for taking the course canhelp learning technologists and educators develop tools that arebetter-equipped to accommodate them. Forum technologies have hard-ly evolved in the past 15 years, and in some cases, their limited func-tionality has failed to accommodate the diverse needs of students(Thomas, 2002). Indeed, as the qualitative observations of interactionsin the forums revealed (Gillani, Eynon, Osborne, Hjorth, & Roberts,2014), the fact that a number of forum participants in theMOOC tended

to move away from the discussion forums to alternative technologiessuch as Facebook, Skype, and Email suggests a failure of the currenttools in meeting the needs of a diverse student body. There are thusnew opportunities to re-imagine how these online discussions can befacilitated within the course environments themselves.

These trends, of course, are observations from one particular MOOC,and are subject to the nuances of a particular pedagogical approach andset of learning objectives. However, this course was selected as a goodexample of a case where interactions among students were encouragedand thus we suggest has wider resonance (Tracy, 2010) to othercourses. Considering how forums may leverage, or perhaps even inte-grate with the technologies that MOOC participants use in their every-day lives is fundamental to understanding how discussions evolve inthese global classrooms.

5.2. MOOC discussion forums harbour crowds, not communities of learners

The preliminary network images presented here largely showdiminishing, and increasingly fragmented, forum participation as thecourse progressed. As demonstrated by the results, groups that formedaround particular discussion topics were generally short-lived and eventhose sub-forums that started off with highly centralized communica-tion and diverse user activity quickly became modularized.

These insights suggest that MOOC participants that engage in real-world discussions come together and disperse as crowds, not communi-ties, of learners (Haythornthwaite, 2009a,b). Lave and Wenger havesuggested the formation of “communities of practice” in situated learn-ing settings,where communitymembers are integrated from their earlystages of “legitimate peripheral participation” to high-volume participa-tion as they drive collaboration among other members (Lave, 1991). Inthe context of theMOOC's discussion forums, however, nearly the oppo-sitewas true: students started off with high-volume participation in on-line discussions, and over time, their apparent commitment to theseconversations tailed off.

The low proportion of students participating in the forums fromweek to week further illustrates the ephemeral nature of the crowdsthat formed in the course. The fact that only 0.2% of participants thatused the forums in the beginning of the course also participated duringthe peer-review week—even though all forum users constituted nearly5% of total course registrants—demonstrates inconsistent use. It can beargued that much of this inconsistency in forum participation is due tothe fact that participation in the course as a whole is difficult to define.Since participants take MOOCs of their own volition, they are free to

Table 4Proportion of forum users and total course participations (based on pre-course surveycompletion) and their highest level of educational attainment.

Highest level of educationalattainment

% Course participantswith this level (N)

% Forum participants (N)

Some high school 2% (103) 1% (26)Completed high school 4% (282) 3% (66)Some college 12% (805) 10% (204)Bachelor's degree 42% (3,050) 43% (849)Master's/professional degree 36% (2,832) 40% (781)Doctorate 4% (265) 3% (58)

Fig. 1. Thread participation in weeks 1 and 2 (left to right) of the readings sub-forum.



engage or disengage as they see fit. This highlights an important point:MOOCs are not like their more formal counterparts in online education,and therefore, theymust not be considered in the same light or evaluat-ed along the same dimensions. They certainly are not a replacement forexisting higher education provision, and instead may prove to be an in-teresting addition. Knowing that forum discussions harbour the forma-tion of loosely-structured crowds could aid in the development of newmetrics that help evaluate the contributions and achievements oflearners that participate in myriadways. Future research by the authorswill explore this phenomenon in greater depth.

5.3. Higher-performing students use the discussion forums, but they do notonly interact with other higher-performing students

In the case of this MOOC, students were evaluated based on weeklyquizzes (which they could resubmit any number of times for re-evaluation) and their final project analyses.

Noted in Section 4, this MOOC, likemany others, had a high attritionrate: only about 2500 of the 87,000 participants received a nonzero finalscore. However, it is important to acknowledge that “attrition”, or“dropping out”, are inherently ambiguous in these settings. SinceMOOCs require a low-commitment and since many are yet to offer for-mal accreditation, they have so far primarily attracted individuals thatare interested in continuing to learn and gain skills instead of thoseseeking formal recognition for their work (Kizilcec et al., 2013). Thereis no comprehensive or standardised metric, then, to assess whethereach student has learned what he or she initially took the course tolearn, nor canwe inferwhy a particular achievement outcomeoccurred.

Therefore, students' final scores in these courses only partially ex-plain enrolment, participation, and achievement. Still, our analysis ofthe final scores of forum participants in particular provides an impor-tant insight and demonstrates the complexity of trying to understandhow andwhy students conversedwith one another. The fact that the fo-rums had a statistically and practically significant proportion of highgrade earners in comparison to the rest of the course—as evidenced bythe fact that 20% of forum participants received a final score of at least90%—indicates that forum participation revealed something about stu-dents' performance. Indeed, the qualitative observations confirmedthat many students used the forums to ask specific questions about as-signment due dates and requirements (Gillani et al., 2014). A recentpaper by researchers atHarvard andMIT on data from anEdXMOOC re-vealed a similar tendency of course certificate earners (generally high-scoring individuals) to turn to the discussion forums for help in solvinghomework problems (Breslow et al., 2013). For this investigation, theconverse statistic was also compelling: 44% of those individuals that re-ceived at least a 90% in the course participated in the forums.

Butwhowere these high scoring forum participants? The results in-dicated that Europe-based forum participants were both the highestgrade-earners and held the highest proportion of master's/professionaldegrees—even though they comprised only 25% of total forum partici-pants. This confirms trends observed in earlier MOOCs: well-educatedpeople from the Western world tend to complete MOOCs and do wellin them. Further mixed method research is required to examine whythis may be the case.

However, there is more to this story, as high performers constituteda substantial proportion of forumparticipants, but certainly not amajor-ity; over 73% of forumparticipants received below a 50%, and these par-ticipants accounted for nearly 60% of all forum comments and posts.While there are different ways to interpret this finding, one possibilityis that most discussion forum participants are more interested inconnecting with others than they are in being formally recognized fortheir work. Another interpretation is that many were turning to the fo-rums for help. Indeed, previous investigations of online learning com-munities have revealed that those that are most active in forumdiscussions are not necessarily the highest performing students(Davies & Graff, 2005). Bymany conventional standards, it is revelatorythat such a large amount of forum activity was attributed to those that“failed” the course. However, the data suggest that final scores are nota sufficient measure of student ability or performance as they fail to re-flect why so many people do so poorly but are nevertheless so active indiscussions. Clearly, there is an opportunity to re-imagine evaluation in

Fig. 2. Thread participation in weeks 1 and 2 (left to right) of the study groups sub-forum.

Table 5Assortativity values, p-values, and effect sizes by learners' final marks, according to inter-actions in different sub-forums.

Sub-forum Pearson r-value(assortativity)

p-Value Cohen's effect size

Final project 0.23 b 0.001 Small to mediumQuestions for professor 0.16 b 0.05 Small to mediumReadings 0.12 b 0.001 Small to mediumTechnical feedback 0.17 b 0.05 Small to mediumLectures 0.07 b 0.1 SmallCases 0.08 b 0.01 SmallCourse material feedback 0.22 b 0.001 Small to mediumOrganizations seeking analyses 0.22 b 0.005 Small to mediumStudy Groups 0.14 b 0.001 Small to medium



MOOCs and incorporate more sophisticated methods that account forthe nuances of participation and contributed reflected in the use of thediscussion forums.

From these results, we can conclude that participation in the forumswas dominated by people who received very high or very low finalmarks. However in most cases, there was not a strong tendency forlearners to interact with other learners that received similar marks.The assortativity analysis demonstrates that there was at best a medi-um, and in general, relatively small correlation and corresponding effectsize in howmuch similarly-scoring participants tended to communicatewith one another. This finding is contrary to earlier literature indicatingthat high-performing students tend to interact with other high-performing students early on in online courses (Vaquero & Cebrian,2013).

It is reasonable to argue that MOOCs have the potential to not onlyenable communication between learners from different parts of theworld, but also learners with different motivations and skill-levels. Pre-vious literature on group formation and collaboration has suggestedhow a collection of individual participants and their unique characteris-tics/contributions can help yield a general “group intelligence” oftengenerally uncorrelated with the intelligence of each individual(Woolley, Chabris, Pentland, Hashmi, & Malone, 2010). The possibilityof observing a similar phenomenon in MOOCs will take time as partici-pants tend to be highly educated already, and each course is not neces-sarily tailored to encourage group discussions. However, understandinghow learners communicatewill lay the groundwork for further explora-tions into how MOOCs might serve as the democratizing forces manyhave named them to be.

6. Conclusion

This research has started to uncover which types of students inMOOCs tend to interact with one another, in order to add to empiricalevidence and research on the potential of MOOCs in higher education.The study comes at a time when MOOC research is in its infancy, withrecent papers that move beyond the hype exploring broad trends inlearner participation and achievement—without much focus, as yet, oncommunication trends. We believe that that the discussion forums—which bring together thousands of people from around the world in asemi-synchronous setting—are a key component of what distinguishesMOOCs from previous incarnations of online learning. Therefore, inves-tigating how interactions occur in the forums may offer new insightsinto the pedagogical value and potential of MOOCs.

Using surveys and users' digital traces from a business strategyMOOC focused on real-world, applied learning, we sought insightsinto how communication in the course evolved and the backgroundcharacteristics of those that participated in peer discussions. The datasetwas significant: digital traces for nearly 87,000 course participants, in-cluding information on their nuanced behaviours both within and be-yond the discussion forums. In order to better understand how theyused the forums, we employed social network analysis to define and vi-sualize communication networks. Social network analysis helped revealnot only the backgrounds of forum participants, but alsowhen and howthey interacted with one another, and the relationship between forumparticipation and learners' final marks in the course. This methodologyhelped yield insights into learner participation and communication inthis setting. Yet in future research the use of more qualitative methods,such as interviews and participant observation, will be important togain a more holistic understanding how people experience learning inthese spaces.

Overall, most forum participants were well-educated, from Europeor North America, andwere taking the course to gain professional skills.These learners formed crowds and not communities: some studentsengaged in rich discussions around topics of shared significance, butlargely, forum use was inconsistent and non-cohesive. A sizeable pro-portion of discussion participants comprised high-performing students,

although most others received what would have been considered as“failing”marks. While we focus here on only one case study of a courseoffered in business strategy we believe the approaches and findingshere are of value to other courses from other disciplines which encour-age dialogue around real world problems. While the motivations of thestudents to take the coursemaywell differ, the ways to understand andsupport communication in forumswhich are set up to discuss particularcases are likely to hold resonance (Tracy, 2010) for researchers and ac-ademics across a range of disciplines.

There is much further work to be done to gain fundamental insightsinto howMOOC participants use the discussion forums and both quali-tative and quantitative work are required to fully understand the nu-anced learning process that take place. It is time to shift the MOOCdebate from questions about how they compare to campus-based edu-cation or how they can replace existing higher education provision, tohow their potentially unique contributions—namely, their promotionof global-scale conversations—can be used to help improve learning.Understanding how lifelong learners from around the world connectwith their peers to ask questions and promote dialogue, often abouttopics that relate to their lives beyond the MOOC, is an important firststep in exploring how these courses can meet the high expectations ofthe higher education community. Perhaps this will be the added valueof MOOCs—bringing people with shared intellectual interests togetherto engage directly with one another around real-world issues.

References

Abelson, H. (2008). The creation of OpenCourseWare at MIT. Journal of Science Educationand Technology, 17(2), 164–174.

Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys.Journal of Marketing Research, 14(3), 396–402.

Belanger, Y., & Thornton, J. (2013). Biolectricity: A quantitative approach. Durham, NC:Duke Center for Instructional Technology.

Breslow, L., Pritchard, D., DeBoer, J., Stump, G., Ho, A., & Seaton, D. (2013). Studying learn-ing in the worldwide classroom: Research into edX's first MOOC. Research & Practicein Assessment, 8, 13–25.

boyd, d., & Crawford, K. (2012). Critical questions for big data: Provocations for a cultural,technological, and scholarly phenomenon. Information, Communication & Society,15(5), 662–679.

Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2007). Social networks, communicationstyles, and learning performance in a CSCL community. Computers & Education,309–329.

Cohen, J. (1992). Statistical power analysis. Current Directions in Pyschological Science,1(3), 98–101.

Couper, M. (2000). Web surveys: a review of issues and approaches. Public OpinionQuarterly, 64, 464–494.

Davies, J., & Graff, M. (2005). Performance in e-learning: Online participation and studentgrades. British Journal of Educational Technology, 36, 657–663.

De Weaver, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemesto analyze transcripts of online asynchronous discussion groups: A review. Computers& Education, 46(1), 6–28.

Deboer, J., Breslow, L., Stump, G., & Seaton, D. (2013). Diversity in MOOC students' back-grounds and behaviours in relationship to performance in 6.002x. Proceedings fromthe sixth conference of MIT's Learning International Networks Consortium (LINC). :MIT (Cambridge, MA).

Easley, D., & Kleinberg, J. (2010). Networks, crowds, markets: Reasoning about a highlyconnected world. Cambridge: Cambridge University Press.

Fenn, J., Raskino, M., & Gammage, B. (2009). Gartner's Hype Cycle Special Report for. :Gartner, Inc.

Gillani, N., Eynon, R., Osborne, M., Hjorth, I., & Roberts, S. (2014). Communicationcommunities in MOOCs. (arXiv:1403.4640v2).

Gunawardena, C., Lowe, C., & Anderson, T. (1997). Analysis of global on-line debate andthe development of an interaction analysis model for examining social constructionof knowledge in computer conferencing. Journal of Educational Computing Research,395–429.

Hamers, L., Hemeryck, Y., Herweyers, G., Janssen, M., Keters, H., Rousseau, R., et al. (1989).Similarity measures in scientometric research: The Jaccard index versus Salton'scosine formula. Information Processing & Management, 25(3), 315–318.

Haythornthwaite, C. (1996). Social network analysis: An approach and technique for thestudy of information exchange. Library and Information Science Research, 18(4),323–342.

Haythornthwaite, C. (2009a). Crowds and communities: Light and heavyweight modelsof peer production. Big Island, Hawaii: Hawaii International Conference on SystemSciences.

Haythornthwaite, C. (2009b). Online knowledge crowds and communities. InternationalConference on Knowledge Communities (pp. 1–16). University of Nevada: Center forBasque Studies (Reno).



Johnstone, S. (2005). Open educational resources serve the world. Educase Quarterly,28(3), 15.

Kizilcec, R., Piece, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzinglearner subpopulations in massive open online courses. Leuven, Belgium. The 3rdProceedings of the Learning Analytics & Knowledge Conference.

Kossinets, G., &Watts, D. (2006). Empirical analysis of an evolving social network. Science,311, 88–90.

Kruskal, W., & Wallis, A. (1952). Use of ranks in one-criterion variance analysis. Journal ofthe American Statistical Association, 47(260), 583–621.

Lave, J. (1991). Situated learning in communities of practice. In L. B. Resnick, J. M.Levine, S. D. Teasley, L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectiveson socially shared cognition (pp. 63–82). Washington, D.C.: American Psycholog-ical Association.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.Cambridge: Cambridge University Press.

Leont'ev, A. (1978). Activity, consciousness, and personality. Prentice-Hall.Marr, L. (2013). Widening participation, lifelong learning, and MOOCs: Challenging the

‘true nature’ of the university. Widening Participation and Lifelong Learning, 1–5.McKinney, D., Dyck, J. L., & Luber, E. S. (2008). iTunes University and the classroom: Can

podcasts replace professors? Computers & Education, 52, 617–623.Noble, D. F. (1998). Digital diploma mills: The automation of higher education. Science as

Culture, 7(3), 355–368.Palonen, T., & Hakkarainen, K. (2000). Patterns of interaction in computer-supported

learning: A social network analysis. In B. Fishman, & S. O'Conner-Divelbiss (Eds.),Fourth International Conference of the Learning Sciences (pp. 334–339). Mahwah, NJ:Erlbaum.

Preece, J., Nonnecke, B., & Andrews, D. (2004). The top five reasons for lurking: Improvingcommunity experiences for everyone. Computers in Human Behavior, 20(2), 201–223.

Roschelle, J., & Teasley, S. (1995). The construction of shared knowledge in collaborativeproblem solving. In C. O'Malley (Ed.), Computer supported collaborative learning(pp. 69–97). : Springer Berlin Heidelberg.

Ryan, L. (2013). MOOCs are on the move: A snapshot of the rapid growth of MOOCs. EFMD.Seaton, D., Bergner, Y., Chuang, I., Mitros, P., & Pritchard, D. (2013). Who does what in a

massive open online course? International journal of human-computer studies, 68,223–241.

Selwyn, N. (2011). Education and technology: Key issues and debates. London: Continuum.Siemens, G. (2005). Connectivism: A learning for a digital age. International Journal of

Instructional Technology and Distance Learning, 2(1), 3–10.Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative

learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of thelearning sciences (pp. 409–426). Cambridge, UK: Cambridge University Press.

Thomas, A. (2005). Children online: Learning in a virtual community of practice. E-learning, 2(1), 27–38.

Thomas, M. J. (2002). Learning within incoherent structures: The space of onlinediscussion forums. Journal of Computer Assisted Learning, 18, 351–366.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitativeresearch. Qualitative Inquiry, 16(10), 837–851.

Vaquero, L., & Cebrian, M. (2013). The rich club phenomenon in the classroom. Nature:Scientific reports (pp. 1–8).

Vygotsky, L. (1978).Mind in society. Cambridge, Massachusetts: Harvard University Press.Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence

for a collective intelligence factor in the performance of human groups. Science,686–688.

Yin, R. K. (2009). Case study research: Design and methods (5th ed.). London: Sage.


Communication Patterns in Massively Open Online Courses

Documents

Transcript of Communication Patterns in Massively Open Online Courses