1

Abstract This paper examines the qualitative findings from a mixed-methods comparison study of the use of an online multi-user virtual environment called Anytown which supplemented face-to-face writing instruction in a fourth grade classroom to determine implications for the design of such environments and the reported impact of this design on students and teacher. Taking a case-based approach, this study examined the experiences of 44 students in two classrooms in order to detect major differences between the participation of students in a class in which face-to-face instruction was their only means of practicing and receiving feedback on their writing and a second class which provided students with an additional eight hours of opportunity to practice descriptive writing within the Anytown multi-user virtual environment. The main findings suggested that several elements must be present in the design of digital learning environments stemming from literature on using games and learning as a means of encouraging on-task behaviors. Further, it was noted that with a social constructivist design, specific forms of scaffolding that emerge from game context should be developed within the system to encourage student peer cooperation and use of system affordances while reducing reliance on face-to-face direction-giving. (Keywords: online, MUVE, on-task behavior, games)

This research was supported in part by a CAREER Grant from the National Science Foundation, and directly by the National Science Foundation Grants #9980081 and #0092831.

2

Introduction

After years of theorizing that the use of video games and simulations should have

positive impacts on student learning (Aldrich, 2003; Cassell & Jenkins, 2000; Dickey,

2007; Jenkins, Squire, & Tan, 2003b; Prensky, 2001), research into their use in

elementary and middle school settings has begun to support these claims with data. The

use of Multi-user Virtual Environments (MUVEs) such as Quest Atlantis and River City

have been correlated with improved content area learning in science as measured by

items taken from state standardized tests. The results of studies examining game-like

learning environments such as Indiana University’s Quest Atlantis and Harvard’s River

City have correlated use of them with increased motivation to learn (Dede, Ketelhut, &

Ruess, 2006; Tuzun, 2004).

The Anytown MUVE was developed as a means to supplement existing

instruction and provide a space within which students could experience a choice-based,

multi-path narrative, interact with digital characters in a dynamic manner, and practice

descriptive and persuasive writing in the immersive manner most often found in either

the real world itself or video games, detailed simulations, and authentic problem-based,

constructivist learning environments. Anytown, modeled as a small town in a generic,

rural Midwest state, is one world among several within the Quest Atlantis (QA) system

which has the primary goal of engaging students in science inquiry learning. The main

difference between Anytown and much of the rest of the QA is that it focuses on student

descriptive writing skills. While elements of science inquiry are present within the

narrative context and interactive activities of the unit, students in Anytown role-played as

cub reporters at the small town newspaper and were required to engage in investigative

3

reporting. Situating learners within a writing role that they could conceivably engage in

as working professionals later in life was expected to increase student engagement.

Creating relevant learning tasks has been found to be effective with other forms of

anchored instruction (Bransford, Vye, Bateman, Brophy, & Roselli, 2003; Vanderbilt,

1990, 1993). Figure 1 presents Anytown’s Main Street, which acts as an interactive space

for locating details, evidence, and clues to be included in student writing.

Figure 1. Main Street in Anytown.

Events that learners investigated include arson at a local landmark building, vandalism of

historic locations, and strange occurrences in a local park.

4

While a more complete description of the design of Anytown and the quantitative

results measuring learning gains during its implementation are reported elsewhere

(Warren, Barab, & Dondlinger, In Press; Warren & Dondlinger, In Press), this article

focuses on reporting the findings of a qualitative, case study approach and seeks to

contextualize those results within the body of research on games for learning as well as

the full scope of research on the Anytown MUVE.

Research on games for learning

While the evidence is slowly building that complex digital environments such as

video games, digital simulations and MUVEs can positively impact student time on task

and attitude towards learning subject matter (Barab et al., 2006; Jenkins, Squire, & Tan,

2003a; Squire, 2006; Squire, Giovanetto, Devane, & Durga, 2005; Steinkuehler, 2004;

Tuzun, 2004), the question as to whether media and software is actually impacting

student learning remains (Clark, 1991; Kozma, 1991). Much of the research literature

related to using games for learning that has employed empirical methods has not shown

significant differences when using this form of environment in comparison with direct

instruction (Hays, 2005).

Further, the complexity of these digital systems makes it very difficult to parse

out what pedagogical element is actually responsible for learning gains, whether it be the

interaction among student, instructor, system, system agents, pedagogical tools, or the

epistemic stance that drove the development of the curriculum. A recent review of 160

mathematics and reading software products conducted with the support of the United

States Department of Education (2007) found no evidence that any digital product

5

impacted student learning at a statistically significant level. While the educational

technology community questions both the methodology and validity of the study, the cost

to benefit ratio involved with these complex systems given the myriad confounding

variables and varied research designs is valid and troubling (Nagel, 2007).

Nevertheless, what is at the heart of game systems that appears to be correlated

with a willingness to engage for long periods are activities that allow for play. Play has

been hypothesized to be one of the most fundamental forms of human activity and means

of learning (Baudrillard, 1994; Derrida, 1997; Wittgenstein, 1968). It allows for the

mind’s exploration of the rules and consequences of engaging with or breaking them. In

some instances, this play is subtle, with the learner testing their finger against the blade of

the rule and discovering it is unpleasant. In others, players examine the rules by

slamming into them physically, learning their harsh consequences concurrently.

Vygotsky (1978) observed that children at play encounter a number of the rules to which

they submit freely as part of the act of play, a theory supported by the research of Barab

and Jackson (2006). A graphic depiction of the view of the conception of play that guided

the design of Anytown can be seen in Figure 2.

6

NOTE: = moments of play or interplay

Figure 2. Conception of play as interaction between experience and internal rules From Warren & Jones (in press)

However, the design of games or game-like systems by educators and researchers

is largely unguided by professionals in the field of commercial game design, and

scientific analysis of these complex systems is yet incomplete. An exception is Squire’s

(2004; 2005; 2006) work with the real-time strategy game Civilization III, an analysis of

a curriculum designed around a commercially produced video game rather than the

design of an entirely new interactive system. Despite a number of books on the subject of

game design, commercial companies have a vested interest in keeping the secrets of what

makes a game successful held in proprietary patents, which limits the transparency of

their process for educators interested in developing similarly engaging products. As a

result, instructional designers seeking to leverage the purported benefits of the use of

games, simulations, and digital learning environments have an incomplete picture of

7

either the processes or costs associated with effectively designing these digital learning

spaces.

The Anytown MUVE was a successful design intervention in terms of its principal

design goals: reducing the amount of teacher time spent answering directional or

procedural questions, increasing the amount of voluntary student writing, and improving

student scores on a standardized writing assessment. The quantitative methods used to

measure these results are described more fully elsewhere (Warren et al., in press), but are

summarized in Table 1.

Table 1. Quantitative results from Anytown study two Treatment Comparison Differences Teacher time spent answering directional or procedural questions

M = 12.118, SD = 6.6951

M = 28.413, SD = 3.9033

t (15) =5.947, p = .043

Student voluntary writing

M = 1.0870, SD = .288

M = .000, SD = .000

t (40) = -16.410, p =.006

Student scores on standardized writing assessment

F (1, 40) = 4.32

However, because of the complexity of the variables introduced by the use of such a

designed digital environment, the results of these scores indicates that the Anytown

MUVE was correlated with improved writing and elements of it are likely to have played

a role in these differences. The quantitative data fails to indicate what elements within the

MUVE or within the classroom environments, both comparison and treatment, might

8

have facilitated these improvements. Moreover, these data do not suggest how the

designed environment could be improved to better serve both student and teacher

experience.

The purpose of this analysis is to apply qualitative, case study methods to

contextualize the quantitative findings and better reveal what worked well and what could

be improved. The principal research questions for this analysis follow:

1. Did embedded pedagogical elements reduce teacher time answering directional or

procedural questions?

2. Did the motivating elements (compelling narrative tied to game challenges and

rewards) increase student motivation, the factor expected to impact voluntary

writing?

3. Did the immersive context and/or more meaningful guidance and feedback from

the teacher impact student writing achievement?

The rationale for employing these qualitative methods follows Denzin & Lincoln’s

(2003) prescription for a method that “seeks answers to questions that stress how (sic)

social experience is created and given meaning” (p. 13). This case study was used to

examine “a phenomenon of some sort occurring in a bounded context” (Miles &

Huberman, 1994 , p.25). Jonassen and Hernandez-Serrano have described case based

reasoning as “entail(ing) the elicitation, analysis, and inclusion of stories as a primary

form of instructional support” (2002, p. 65). The studied phenomenon in question was

9

student and teacher experience with the Anytown digital learning environment as

compared to more traditional classroom instruction.

Research methods

Qualitative methods tend to look at qualities, characteristics, and attitudes

inherent in a system. The use of observations and interviews as primary methods of data

collection allows qualitative researchers to make sense of a situation or of what happened

in a system prior to an intervention, and what is happening since that intervention

(Denzin & Lincoln, 2003; Gall, Borg, & Gall, 1996). This examination of the Anytown

system took place as a formative review as well as a summative evaluation of the

interactions, benefits, and detriments stemming from the implementation of this

intervention.

Participants and setting

The research setting was an elementary school in a small, Midwestern city near a

large, land-grant research university. Participants included 44 students in two 4th grade

classrooms and their teachers. Both teachers reported being strong users of face-to-face

problem-based learning environments in their classroom instruction. One classroom

functioned as a comparison; the other classroom supplemented traditional instruction

with the Anytown technology-supported learning environment as a treatment condition.

Students were quasi-randomly selected by the school’s computer system for assignment

to their respective classes in two months prior to the study.

10

Data Collection Methods

Data collection occurred over a four-week period using observation, interviews, and

video recording methods. A total of 20 periods of instruction were documented: ten periods

in the comparison classroom and ten periods in the treatment condition. Five researchers

collected and coded the field notes, interview transcripts, and transcripts of recordings for

each class period involved in the study.

Field notes. At least three observers trained in qualitative methods took notes in

each classroom over the course of the collection period. In the treatment classroom,

observers focused on documenting student interaction with the in-system embedded

scaffolds, learner willingness to complete tasks that included goals nested within the

game narrative, and out-of-system face-to-face discussions related to the narrative

context or the explicit rules of the learning system. During laboratory and classroom

instruction periods, researchers observed the teachers in both the comparison and

treatment group and recorded instances of teacher procedural or directional questions

answering behaviors in their field notes. Researchers then developed a coding scheme to

classify utterances documented in the field notes. Codes were triangulated and member

checked across observers. The coding scheme agreed upon by the five observers is

detailed in Table 2.

11

Table 2. Field note coding scheme

Text Color Code Light blue Peer help (ph) Dark blue Voluntary activity (Quests, outside school work) (va) Red Frustration point, problem, or question (fp) Light green Informal assessment Pink Hypothesis (nascent or otherwise, includes claims) (h) Light Gray - (25%) Clarifying question (cq) Yellow Teacher redirect to system (tr) Violet Other tension (ot) ( ) = text tag

Analysis began with a large number of possible codes identified by each member, and

each was then folded into larger categories that were agreed upon by the group. Each

category is related to student and teacher discourse.

Informal and semi-structured interviews. Researchers also conducted semi-

structured and informal interviews with teachers and students in the comparison and

treatment both during the data collection period as events occurred that warranted

interview and at the conclusion of the collection period. These interviews provided a

means of documenting student and teacher learning and teaching experiences and to

verify codes generated by other data collection methods.

Audio and video recording. Within the naturalistic setting of the computer lab,

treatment classroom, and comparison classroom, MP3 audio recordings of student

activity were made and transcribed. Examining the treatment teacher’s normal classroom

writing instruction allowed a comparison between the teachers to determine the degree of

similarity between them. Both sets of teaching activities could then be compared with

learning activity and instruction in the Anytown supplemental treatment sessions. All

12

audio and video data was collected throughout the period as a means of capturing typical

student interaction outside the digital environment and during face-to-face, teacher-led

activities in the treatment condition.

Data Analysis Methods

Using case-based interpretive approach (Gall, Borg, & Gall, 1996; Robson, 2002),

field notes and transcripts were coded line by line and integrated with reflections and

reviewing comments. Another researcher on the team then reviewed the codes,

comments, and reflections to check for validity and agreement regarding each. Once the

validity of the codes was established, they were then folded into larger categories. A

selection of three full class periods of coded field notes, interview transcripts, and

transcripts of recordings for each class were analyzed. This selection represents seven

hours and thirty-two minutes of transcript text and field notes related to the sixteen hours

of observation. The study analyzed 1,982 lines of text, which included 23,968 words and

111,371 characters.

The codes that emerged surrounded instructional issues related to the teacher,

students, directions, researcher interactions, and power relationships among all

participants. These were represented by 22 codes that addressed the interactions of

participants, researchers, and system. A description of the codes used to produce these

results is presented in Table 3.

13

Table 3. Coding scheme for teacher and student speech acts within transcripts/field notes Indicating marker Coded Meaning Treatment

Number of lines/ (Percentage)

Comparison Number of lines/

(Percentage)

Black text Non-relevant speech act Light blue text Directional question asking

343 (33.6%) 537 (55.8%)

Sea green text Technical problems

104 (10%) 7 (.7%)

Plum text Within system directions 81 (8%) 2 (.2%) Gray text Clarifying question 222 (22%) 101 (11%) Dark red text Interface help 97 (10%) 4 (.4%) Orange text Rule-setting 203 (20%) 104 (11%) Dark yellow text Student or teacher frustration 71 (7%) 182 (19%) Blue-gray text School system tension 32 (3%) 29 (3%) Red text Direct teaching 337 (33%) 491 (51% Lavender text Student information offered 51 (5%) 149 (15%) Tan text Empathy expression 23 (2%) 12 (1%) Turquoise text Grammar question/instruction 141 (14%) 412 (43%) Rose text Teacher informational statement

281 (28%) 291 (30%)

Lime text Teacher informal assessment of work

318 (31%) 212 (22%)

Medium blue highlight

Teacher check for understanding

189 (19%) 229 (29%)

Red highlight Typical interchange 62 (6%) 67 (7%) Bright green highlight

Whole class direction

147 (14%) 287 (28%)

Pink highlight Teacher/researcher discussion 77 (8%) 0 (0%) Light blue highlight

Peer help 37 (4%) 117 (12%)

Yellow highlight Teacher redirect to system 492 (48%) 13 (1%) Gray highlight Teacher redirect from off-task 312 (31%) 368 (38%) *Reported percent is the number of lines as a percent of total lines

Findings

Analysis of this qualitative data revealed both successes and challenges with

using a learning game or simulation in a classroom. Despite the challenges, the

qualitative findings support the interpretation that the quantitative results can be

attributed to the design. However, contextualizing the results within the qualitative

responses of participants allows us to draw further conclusions for how to improve future

implementations of the design.

14

Embedded pedagogical elements

Intentionally designed pedagogical agents, both characters and objects, were

embedded in the 3D space and were meant to complement the ill-structured problems

posed by the narrative. Some of these agents created designed points of frustration that

forced student reliance on the system and their peers in order to solve these problems.

Others functioned as gate-keeping mechanisms to prevent learners from skipping vital

stages of the in-system writing instruction or critical pieces of supporting evidence

necessary to their investigation. Both types of pedagogical agents were intended to

empower students to direct their own learning and to free the teacher from answering

directional and procedural questions.

A review of the video, audio, and field notes for both classes indicated that

students in the Anytown treatment class worked on task-related activities longer and more

often than did those in the comparison classroom, and the comparison teacher, Ms.

Cedar, spent much more time giving directional guidance. Analysis of the interactions

between students and Ms. Cedar suggests that most of the power in the comparison

classroom was concentrated in the hands of the teacher. Feeling powerless to direct their

own learning to more motivating tasks because of strictures placed on them by the

teacher, students engaged in transgressive behaviors as they sought to take back power by

what means were available to them. These behaviors included talking or whispering to

peers, drawing unrelated pictures, kicking their feet, throwing objects, talking to the

researcher, asking irrelevant questions, and putting their heads on their desks. Although

they were in the minority, a few students remained engaged with their learning tasks

15

throughout each period they were observed. Notably, these few were the strongest

writers, most advanced readers, and highest achievers in the class. Review of student

interactions with the teacher and peers indicated that their power in the classroom was

connected to a positive rapport with the teacher, motivation stemming from achievement

and conforming to the teacher declared norms.

While the treatment teacher, Mrs. Teak, did spend time answering directional and

procedural questions early in the intervention, she eventually weaned students of their

reliance on her by redirecting students to each other, to the system affordances, and to

peers in order to solve the problem. For example:

Ms. Teak: Did you ask me something?

Eric: Yes. I wanted to know how to pick things up, but now I just want to know

how to pick up the gas can. How you get the gas can out, but somebody already

answered it. What do we have to do first?

Ms. Teak: Ask Robert or Isabelle.

In this instance, the teacher avoided directing the student to the correct answer and

intentionally redirected him to his peers instead. In this way, she did not assert her power

over the student in the learning context and instead empowered him to solve his own

problem through peer negotiation, which is a key feature of problem-based learning

environments (Jonassen, 1999). Had the teacher reclaimed her power to direct student

learning behaviors, she would have likely stymied the affordances of the learning

environment or reinforced his existing dependence on her as arbiter of information.

16

However, several students were stymied by collecting the Gas Can clue, a

designed frustration point preventing student advancement without successful completion

of a sequence of learning activities preceding this specific game task. Retrieving this

pedagogical object (as shown in Figure 3) was required in order gather the evidence

necessary to complete the Burning Cabin Mystery Quest (depicted earlier in Figure 1). A

secondary purpose of this designed frustration point was to compel student reliance on

their peers.

Figure 3. The Gas Can, a designed frustration point for enacting cognitive challenge

Student frustration with the object was revealed in the in-system chat feature. More than

half of the student dialogue in the chat board either requested help with getting the gas

can or offered responses to these requests. The following was a typical interchange:

Kenny: Can you help me with picking up the Gasoline Tank?

17

Isabelle: pick it up by clicking on it

Kenny: How do you pick it up?

Kenny: The Gasoline tank?

Jimmy: i dont know try aging (sic)

The frustration surrounding gaining the Gas Can item generated more discussion than all

other computer-mediated discourse within the system. The audio recordings during these

class periods revealed that student face-to-face discourse also revolved around solving

this specific problem. The designed frustration point appeared to function in the way that

the designer intended, forcing students to rely on each other and the system for

information.

Motivating challenges and rewards

One of the affordances of using digital media for instruction that has been

reported by Glazewski et al (2004), Brush and Saye (2003), and Goldberg et al, (2003) is

that students remain engaged longer than in more traditional forms of classroom

instruction. In digital games, the motivational elements are tied to the narrative plotline

and include challenges and rewards that players earn through their successful interactions

with the environment. As part of the design of the Anytown, we sought to include many

similar forms of extrinsic rewards to help motivate writing. During the study, we sought

to examine what emerged as motivating rewards in both the comparison and treatment

classrooms.

18

Motivation in two classrooms. In the comparison classroom, Ms. Cedar often

challenged students to engage with the subject matter through direct questions related to

the topic under discussion; however, it was often the same students who repeatedly

volunteered to answer and appeared to have high levels of intrinsic motivation to achieve

at classroom tasks. These six students had also completed reading the entire Quest

Atlantis book as part of a small circle of readers and were consistently the highest

achievers in class, an observation in keeping with the assertion that ties to narrative

structures motivate engagement. This group was rewarded by being allowed to read in a

small group and were provided with independent time with the teacher to discuss the

book. This extrinsic motivation was maintained in their future reading activities, which

were similarly remunerated. As a reward however, it appeared to mainly motivate this

small subsection of students and the remainder of the students did not participate.

In the Anytown treatment, students reported feeling unmotivated when the teacher

restricted their free choice activity in the 3-D environment that would have allowed them

to complete optional game tasks that encouraged exploration and note taking. This lack of

motivation was reported most during the third and fourth periods when the teacher

implemented a policy that required students to raise their hands and wait for her to review

their work on-screen prior to submitting it to the Anytown system for review by her later.

This policy was due to a miscommunication between the designer and teacher as to the

purpose and functions of the system. Ms. Teak reported that she felt that this further

oversight was necessary because she did not fully understand how the design was

intended to function or her role in it.

19

In order to overcome this obstacle, the designer/researcher was compelled to

intervene and ask the teacher to desist from requiring this review so that the system could

function as intended and the study could proceed. Once the teacher removed this

requirement, the system functioned as intended and students reported feeling empowered

to explore and choose their own writing and game activities in the space. Consequently,

the curricular, instructional, and motivational elements seemed to function as intended.

Students in the treatment class were motivated to complete a total of 26 voluntary writing

activities.

Transgressive behavior. The opportunities within Anytown for transgressive

student behavior are also of note; a few students excelled at exploiting these. For

example, students told Ms. Teak, peers, and researchers that they were intrigued by the

“Dark Lakes” and “Raintree Manse” thematic mini-games. These areas were intended as

rewards for completing academic tasks or to provide additional experiences that students

could write about in non-descriptive forms such as poems and fictional stories.

Participant students revealed that their intrigue was because of the unique ambiance of

these spaces resulting from their appearing out of place in the town and that they were

further engaged by obstacles intended to keep them out until they had completed pre-

requisite Quests. These locations took four male students--Stanley, Sawyer, Eric, and

Walter--away from their main task fairly often. This behavior most often took place in

locations that were important to later Quest learning activities, but had no relevance to

those that the boys were working on at the time. Their wish to engage in activities not yet

open to them by the built-in system of rewards pulled them off-task as they attempted to

exert their own power and “game” the system. The term “gaming the system” refers to

20

behaviors that are either in violation of the system’s intended constraints or that

undermine established game rules (Baker, et al, 2008). Although intriguing locations

intended to reward players prompted undesirable behavior from some players, this

behavior confirms the motivating quality of such design elements.

Immersive visual experience

Given their immersive visual richness, graphical virtual environments have

obvious applications for science inquiry. They can simulate practices such as observation

and experimentation without constant replenishment of costly supplies or learner

exposure to the hazards inherent in laboratory settings. The impact that such

environments may have on writing, however, is less intuitive. Nevertheless, the designers

speculated that the immersive quality of a 3-D MUVE could have a positive impact on a

learner’s ability to visualize their writing topics and therefore provide richer descriptive

detail in their essays and other writing assessments. Further, it was hypothesized that if

they engaged in a form of experiential learning about a mysterious environment that the

details of the events, places, and people would remain more firmly in their minds and

then would transfer to more vivid, detailed descriptive writing.

The comparison classroom teacher employed problem-based and constructivist

methods to engage students in generating solutions to ill-structured problems in a face-to-

face (F2F) setting rather than the immersive 3-D MUVE, Anytown. However, the F2F

environment was restricted mainly to the immediate classroom with a few opportunities

to visit the library, computer lab, and other institutional spaces. By necessity, this would

21

be more physically static and less expansive than the game world we would provide the

students in which they could practice their writing.

As previously noted, the more intriguing areas of Anytown enticed some students

too much, compelling them to “game the system” in order to gain access to areas

prohibited until prior Quests were completed. One student, Johnny, spent a lot of time

exploring areas that were designed to be off-limits until prerequisite tasks had been

completed. However, Ms. Teak asked questions of Johnny to involve him in verbally

describing his experience as a means of leveraging those details to complete his assigned

writing task, showing interest in his unique experiences and working to have him identify

relevant details. Johnny later used this discussion as he wrote about his explorations in

his Quest response:

“I liked this eccentric building that I was in. I can describe it for you. It looked

like some kind of really old building. It had some stairs. I went down and I came

to a tunnel, and there were red lights on the wall.”

Given that one of the goals of the use of this game system was to engage students in

descriptive writing, Johnny’s use of adjectives and visual descriptions of his environment

was a positive development. The teacher later reported that this student had not turned in

a single piece of writing during the first two months of school leading up to the use of the

Anytown system.

Eric was another student who found the immersive qualities of the game

compelling. He sought to enter the mine that serves as an entrance to the “Dark Lakes”

22

ghost story area by repeatedly bumping the walls that enclose this area. This behavior

ensured him little progress on his mandatory writing activities. However, Eric was able to

overcome the obstacle and gain access to the prohibited place, a feat that gave him a

sense of power over the system. Although this behavior also caused some friction

between Eric and his teacher, he was unwavering in his queries about the purpose of the

Dark Lakes mini-game area. It was only when Ms. Teak asked to see the extent of his

work on his assigned Quest that Eric relented and returned to working on that assignment

so that he could get to the Dark Lakes area later. While Eric’s behavior was not what the

designers intended, the immersive intrigue of the environment did entice Eric to finish his

work.

Implications

The large number of qualitative data sources led the researchers to several

implications stemming from the use of this interactive game environment. These related

both to the Anytown environment specifically and more generally to the areas of games,

simulations, and other interactive digital designs. Because the content area focus of the

unit was literacy practices of elementary schools students, several conclusions focus on

how best to leverage these digital environments to increase student engagement with

reading and writing. Further, the difficulties encountered with communicating the

designer’s intention for the system properly to the users and the consequences of that

challenge also influence these findings.

23

Innovation in classroom practices

Any new curriculum will require the development of new classroom practices and

the creation of training techniques at the university and in-service levels that encourage

teachers to engage with emerging technologies in ways that prepare them for meaningful

integration with content. Further research should examine if teachers recognize when

their administrative duties, such as answering repeated procedural and directional

questions, take them away from teaching. If teachers engage in increased instructional

discourse with students, do they spend more time providing specific feedback on student

writing? Do they increase their discourse with students in the computer lab related to the

characteristics of good writing such as proper spelling, punctuation and capitalization? If

the teacher does not leverage the increased time they have in the computer lab to improve

student learning as a result of a digital environment, then the innovation is not helpful. If

future research shows that teachers are engaging in significantly increased instruction,

feedback on student work, challenging poor knowledge constructions, and informal

assessment of student learning, then the environment allows the teacher to engage in

teaching behaviors associated with improvements in learning (Webb, Nemer, & Ing,

2006; Webb, Nemer, Kersting, Ing, & Forrest, 2004).

Teacher versus system control

As noted earlier, the perceived conflict between a teacher’s need for control and

the need for the learning environment to function as designed indicated the necessity for

clearer communication with the teacher prior to the implementation of such a game-like

environment. Although the treatment teacher was provided with a six-page manual that

24

outlined the Anytown environment, the activities, and the teacher role, accompanied by a

one hour, face-to-face training session with the researcher, this may not have been

enough. When the teacher asserted her control over students by preventing voluntary

activities and requiring teacher review of all quests before submission stymied the design

intentions of the researcher. Encouraging teachers to experience the student’s point of

view prior to implementation and to use technologies in an immersive way should result

in better technology integration.

However, such preparation goes beyond mere familiarization with a technology

product. Ensuring that teachers are comfortable with their role in problem-based or

constructivist learning environments is imperative. Preparation programs should provide

additional background about what the teacher might expect in terms of student cognitive

challenge, the student-centered design of the environment, and what kinds of teaching

behaviors would defeat the purposes of the system. This is in keeping with the results of

Dede’s work (2006) which recommended that additional teacher professional

development would make the River City science and history-based MUVE more

effective.

Technology and literacy

Both the findings of this study and those of Steinkuehler’s (2004) work with the

game Lineage II indicate relationships between student engagement and digital learning

environments as means of creating intrinsically motivating writing practices for K-12

students. Krashen’s (2004) review of research related to reading and writing noted that

increased reading practice is often correlated with improvements in student writing even

25

more so than increased writing practice. Future studies should examine whether this form

of digital environment can motivate students to engage in regular reading. Also of interest

is linking reading to the design of learning environments that can be prepared using those

principles of engagement already employed by game companies, digital and analog, and

by successful teachers to improve student willingness to read. Existing games such as

Blizzard’s World of Warcraft (Dickey, 2007) already provide paper books that

complement the game experience, providing additional information and stories about

characters that might be experienced in the game. In fact, pre-orders of the World of

Warcraft software were accompanied by copies of the first novel in a trilogy that could

be used by players to help provide a narrative frame for the world in which they would

play while linking the new game to the past story of Warcraft III, the previous title in the

series. This helped to draw players into a new story without risk because it complemented

their daily experience within the game.

Future Research

It is the hope of the researchers that several major lines will be explored through

future research: those involved with Anytown and those that focus on the use of games,

game-like learning environments in general, and student problem-solving within game-

infused problem-based learning environments. In terms of the Anytown environment, the

role of peer teaching and support appeared to be an important part of both student

successes on learning tasks within the space and in overcoming major difficulties with

the game elements of the design. While this is important, further research is needed to

fully understand the role that peer support plays in learning using such an environment.

26

Beyond peer support and the role of game incentives in general for motivation, it is

also important to understand how such learning objects can be designed effectively within

such an environment. Anytown contained over 150 such objects that students could earn or

otherwise receive; however, students used or referenced only a fraction of these during their

time interacting in the space. What still must be explored is which objects were motivating to

students and what the qualities were present that made them so. Also, questions must be

addressed related to which objects have high and low impact on learning or engagement with

a learning game system, so that future designers have a sense of what must be present in an

effective learning object that is contextualized by a game space.

References

Aldrich, C. (2003). Simulations and the future of learning. San Francisco: Pfeiffer.

Barab, S. A., & Jackson, C. (2006). From Plato's Republic to Quest Atlantis: The role of

the philosopher-king [Electronic Version]. Technology Humanities Education

Narrative. Retrieved 1/20/2006 from http://www.thenjournal.org/feature/78/.

Barab, S. A., Warren, S. J., Zuiker, S., Hickey, D., Ingram-Goble, A., & Dodge, T. (2006,

April 7-11, 2006). Transfer of Learning in Complex Learning Environments.

Paper presented at the American Educational Research Association Annual

Meeting, San Francisco, CA.

Baudrillard, J. (1994). Simulacra and Simulation (The Body, in Theory: Histories of

Cultural Materialism). Ann Arbor: University of Michigan Press.

Bransford, J., Vye, N., Bateman, H., Brophy, S., & Roselli, B. (2003). Vanderbilt's

AMIGO Project: Knowledge of How People Learn enters cyberspace. Retrieved

27

4/20/2004, 2004, from

http://www.vanth.org/mmedia/vanth0103/vanth0103cd/papers/AmigoWFig.pdf

Brush, T., & Saye, J. (2003). The Effects of Multimedia-Supported Problem-Based

Historical Inquiry on Student Engagement, Empathy, and Historical Reasoning.

Paper presented at the American Educational Research Association.

Cassell, J., & Jenkins, H. (Eds.). (2000). From Barbie to Mortal Kombat. Cambridge,

MA: The MIT Press.

Clark, R. E. (1991). When Researchers Swim Upstream: Reflections on an Unpopular

Argument About Learning from Media. Educational Technology(2).

Dede, C. (2006). MUVEEs Project. Cambridge, MA: Harvard University.

Dede, C., Ketelhut, D., & Ruess, K. (2006). Designing for motivation and usability in a

museum-based multi-user virtual environment. Retrieved 3/11/2006, 2006, from

http://www.gse.harvard.edu/~dedech/muvees/documents/AELppr.pdf

Derrida, J. (1997). Of grammatology (G. C. Spivak, Trans. Corrected ed.). Baltimore,

MD: The Johns Hopkins University Press.

Dickey, M. D. (2007). Game design and learning: A conjectural analysis of how

massively multiple online role-playing games (MMORPGs) foster intrinsic

motivation. Educational Technology Research & Development, 55(3), 253-273.

Education, U. S. D. o. (2007). Effectiveness of reading and mathematics software

products: Findings from the first students cohort. Retrieved. from.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational Research: An introduction

(6th ed. Vol. I). White Plains, NY: Longman Publishers.

28

Glazewski, K., & Clark, D. (2004). Supporting scientific inquiry in online learning

environments. Unpublished manuscript, West Lafayette, Indiana.

Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student

writing: A meta-analysis of studies from 1992 to 2002. The Journal of

Technology, Learning, and Assessment, 2(1).

Hays, R. T. (2005). The effectiveness of instructional games: A literature review and

discussion (Technical Report No. 2005-004). Orlando, FL: Naval Air Warfare

Center Training Systems Division.

Jenkins, H., Squire, K., & Tan, P. (2003a). Entering the education arcade. Computers in

Entertainment 1(1), 17.

Jenkins, H., Squire, K., & Tan, P. (2003b). Entering the education arcade. . Computers in

Entertainment 1(1), 17.

Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M.

Reigeluth (Ed.), Instructional Design Theories and Models: A new paradigm of

instructional theory (Vol. 2, pp. 215-239). Mahwah, NJ: Lawrence Erlbaum

Associates, Publishers.

Kozma, R. (1991). Learning with media. Review of Educational Research, 61(2), 179-

211.

Krashen, S. D. (2004). The power of reading : insights from the research (2nd ed.).

Westport, Conn.; Portsmouth, NH: Libraries Unlimited; Heinemann.

Miles, M., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand

Oaks, CA: Sage Publications, Inc.

29

Nagel, D. (2007). DOE software study: Are the data flawed? [Electronic Version].

Technological Horizons in Education (THE) Journal Retrieved April 19, 2007

from http://www.thejournal.com/articles/20463.

Prensky, M. (2001). Digital game-based learning. New York: McGraw-Hill.

Robson, C. (2002). Real world research. Malden, MA: Blackwell Publishing.

Squire, K. (2004). Replaying history. Unpublished dissertation, Indiana University-

Bloomington, Bloomington, IN.

Squire, K. (2006). From Content to Context: Videogames as Designed Experience.

Educational Researcher, 35(8), 19-29.

Squire, K., Giovanetto, L., Devane, B., & Durga, S. (2005). From users to designers:

Building a self-organizing game-based learning environment. Tech Trends, 49(5),

34-42, 74.

Steinkuehler, C. (2004). The literacy practices of massively multiplayer online gaming.

Paper presented at the American Educational Research Association, San Diego,

CA.

Tuzun, H. (2004). Motivating learners in educational computer games. Unpublished

Dissertation, Indiana University, Bloomington, IN.

Vanderbilt, C. a. T. G. a. (1990). Anchored instruction and its relationship to situated

cognition. Educational Researcher, 19(6), 2-10.

Vanderbilt, C. a. T. G. a. (1993). Anchored instruction and situated cognition revisited.

Educational Technology, 33(3), 52-70.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological

processes. Cambridge, MA: Harvard University Press.

30

Warren, S., Barab, S., & Dondlinger, M. (in press). A MUVE towards PBL writing:

Effects of a digital learning environment designed to improve elementary student

writing. Journal of Research on Technology in Education.

Warren, S. J., & Dondlinger, M. J. (in press). Designing Games for Learning. In R. Fertig

(Ed.), Handbook of Research on Effective Electronic Gaming in Education.

Hershey, PA: Idea Group Reference.

Warren, S. J., & Jones, J. G. (in press). Supporting teachers, researchers, and emerging

technologies: Evaluating games and simulations intended for learning

Connexions.

Webb, N., Nemer, K., & Ing, M. (2006). Small-group reflections: Parallels between

teacher discourse and student behavior in peer-directed groups. The Journal of the

Learning Sciences, 15(1), 63-119.

Webb, N., Nemer, K., Kersting, N., Ing, M., & Forrest, J. (2004). The effects of teacher

discourse on student behavior and learning in peer-directed groups (Spencer

Foundation Grant Report). Los Angeles: CSE/CRESST/University of California,

Los Angeles.

Wittgenstein, L. (1968). Philosophical investigations (G. E. M. Anscombe, Trans. 2nd

ed.). Oxford: Blackwell.

31

Acknowledgements:

The authors would like to thank several people for their hard work in support of the

design of Anytown and many hours spent collecting data for this study. These include:

Heather Warren, Dan Hickey, Steve Zuiker, Adam Ingram-Goble, Inna Kouper, Susan

Herring, Eun-ju Kwon, Robert Appelman, and Mitzi Lewison.