An Animated Pedagogical Agent in the Learning of The Present Perfect Tense for Brazilian Students

AN ANIMATED PEDAGOGICAL AGENT IN THE LEARNING OF THE PRESENTPERFECT TENSE FOR BRAZILIAN STUDENTS

Talvany Carlotto1

Patrícia Jaques 2

Abstract: The use of Pedagogical Animated Agents is considered a very effective way of im-proving learning, and they can be evaluated in a computer system in terms of what authorshave been calling ’Image Effect’, ’Embodied Agent Effect’ and ’Modality Effect’. However,no research has been found on how these agent effects affect students’ learning in a languagelearning system. This paper describes the design, implementation and analysis of a web-basedlanguage learning system and its animated pedagogical agent. Sixty-four Brazilian universitystudents with intermediate level of English, divided into three groups, participated in the exper-iment. The first group used the system without the agent, another group used the system witha static version of the agent and the last group used an embodied version of the agent. Stu-dents’ performance in each group was compared in order to evaluate the image effect and theembodied effect of the agent. Results were collected using pretests and posttests and evaluatedusing t-tests. No significant improvement on performance was found for the first group. Thesecond and third groups, however, had statistically better results in the posttest. The differencebetween posttest and pretest results for each student was used in unpaired t-tests, in order toevaluate the agent effects. Results revealed the presence of the image affect in the experiment,but the absense of the embodied agent effect.

Keywords: Animated Pedagogical Agents. English as a Foreign Language. Agent Effect.

1 INTRODUCTION

The process of learning a foreign language is different from the process of learning one’smother tongue. Acquiring a second language usually involves dealing with natural barriers thathinder the development of the students’ language skills. One of the common problems foreignlanguage students usually face, for example, is that they are not exposed to the language veryoften.

The use of the English language has such a history of expansion that English is consideredthe most spoken language in the whole world (MYDANS, 2007) and the official language ofseveral important world organizations (e.g. European Union). According to Doughtly andLong (2005), about 300 to 400 million people speak English as their native language. Thesenumbers seem less impressive when we take a look at the number of people for whom theEnglish language is understood and used as a second or foreign language: almost 2 billion(DOUGHTLY; LONG, 2005).

According to Sklar (2011), teaching the English present perfect tense to native speakers of

1Computer Science Undergraduate Student. UNISINOS, Av. Unisinos, 950 Bairro Cristo Rei, CEP 93.022-000São Leopoldo, Brazil. Email: [email protected]

2Advisor, PhD, Programa Interdisciplinar de Pós-Graduação em Computação Aplicada (PIPCA) - UNISINOS,Av. Unisinos, 950 Bairro Cristo Rei, CEP 93.022-000 São Leopoldo, Brazil. Email:[email protected]

2

Brazilian Portuguese is one of the biggest challenges for teachers. The Portuguese languagedoes not have an equivalent verb tense for the English present perfect, so students tend to use,for example, the simple past in situations where the present perfect would be more suitable.Another problem occurs when Brazilian students try to make a literal translation: the resultoften leads to a wrong meaning conveyed.

Computer-assisted language learning (CALL) is a field that relies on language theories inaddition to computer science principles (LEVY, 1997). In short, research in CALL is the studyof how computers can help language teaching. One strong issue that has to be considered in thedevelopment of CALL systems is how to expose the student to social interactions. The use ofAnimated Pedagogical Agents (APAs), specially in a pedagogical role, can make this exposureto the language happen (DOWLING, 2000).

One way to evaluate APAs is by their image effect, embodied agent effect and modalityeffect (see section 3.1). However, no research has been found on evaluating these effects onlanguage learning systems. Having that in mind, the project this article reports involved theresearch, development, execution and analysis of an experiment involving an APA in a web-based CALL system. The system focuses its instruction on the correct use of the present perfecttense in English, as presented in Raymond Murphy’s book English Grammar in Use (MUR-PHY, 2004). The APA created for the system is responsible for guiding and helping the studentin his or her learning process.

2 THE ENGLISH LANGUAGE

Modern language teachers no longer limit instruction to making students memorize a set ofrules that will somehow make them learn the language properly. Teachers now know that givingthe student the capacity to speak a language involves more than making them recite memorizedsyntagmas. Students have to be able to spontaneously communicate with others, and to do sothey must understand meaning instead of learning by drilling (WARSCHAUER; KERN, 2000).

"Learning a second language is a long and complex undertaking. Your wholeperson is affected as you struggle to reach beyond the confines of your first lan-guage and into a new language, a new culture, a new way of thinking, feeling,and acting. Total commitment, total involvement, a total physical, intellectual,and emotional response are necessary to successfully send and receive mes-sages in a second language. Many variables are involved in the acquisitionprocess. Language learning is not a set of easy steps that can be programmedin a quick do-it-yourself kit. So much is at stake that courses in foreign lan-guages are often inadequate training grounds, in and of themselves, for thesuccessful learning of a second language. Few if any people achieve fluencyin a foreign language solely within the confines of the classroom." (BROWN,2000, p. 1).

3

Brown (2000) reinforces the importance of research in foreign language teaching. Accord-ing to him, teaching is not simply applying a set of methods that will make the teaching processsuccessful, but to guide and facilitate learning, enabling the learner to learn, setting the condi-tions required to the acquisition of the new knowledge. What usually happens is that the teacherdevelops a personal teaching philosophy, which includes teaching style, methods and generalapproach in classroom. This philosophy reflects the personal perceptions of the teacher, as wellas his or her previous experiences on observing how the students learn (BROWN, 2000).

2.1 The present perfect tense

In order to talk about the present perfect tense, it becomes necessary to distinguish tensefrom aspect. The distinction between the two terms can be especially confusing for non-native speakers (SKLAR, 2011). Tense is the "grammaticalised expression of location in time."(COMRIE, 1985). Comrie means that tense is a category that places a situation, such as anevent or an action, relative to some other point in time. Other situations occur before, after orduring it. Languages usually have past, present and future tenses. Concerning aspects, Comrie(1976, p. 3) says that they are "different ways of viewing the internal temporal constituency ofa situation". That is, aspect speaks of how a situation is distributed within time.

We could also consider Schmit (2001)’s definition for tense and aspect, which states that"tense is concerned with the relation between the location time of an event and the time ofspeech (tense properties) and aspect is concerned with the temporal properties of an eventualitydescription and how these temporal properties are related to some reference time" (SCHMIT,2001, pg 403).

According to Comrie (1976), languages that use grammatical aspects (as the English lan-guage) may split aspect into perfective and imperfective. The perfective aspect presents anaction as a whole, emphasizing the completeness of an event. The imperfective aspect, how-ever, expresses the incompleteness of an event at a specific point.

In the Portuguese language there is not a grammatical structure capable of, by itself, com-pletely expressing the content of a statement built with the English present perfect. When tryingto express in Portuguese the meaning of English tenses that use the present perfect tense, Brazil-ian students tend to use other Portuguese verbal structures, such as the Presente Simples, thePretérito Perfeito Simples or the Pretérito Perfeito Composto (Table 1). In the example of thetable, the statement I’ve been here does not exactly correspond to any of the verb tenses of thePortuguese language.

3 ANIMATED PEDAGOGICAL AGENTS

Reeves and Nass (1996) have proved that humans treat media, including computers, in asocial manner. They presented the media equation (REEVES; NASS, 1996), explaining that

4

Table 1 – Verb tenses examplesVerb Tenses ExamplesPresente Simples Eu estou aqui.Pretérito Perfeito Simples Eu estive aqui.Pretérito Perfeito Composto Eu tenho estado aqui.Present Perfect I’ve been here.

Source: Created by the author

users ascribe social rules to their interactions with media. They could attest, for instance, thathumans like computers better when they are polite, even though computers are not meant to beor not to be polite. It seems that this human perception when interacting with computers couldbe used during the building of APAs.

Jennings and Wooldridge define intelligent agents as computer programs that present anadaptable, independent, and goal-oriented behaviour (JENNINGS; WOOLDRIDGE, 1998).Intelligent agents that take an educational or pedagogical role to facilitate or to improve thelearning of the student are called Pedagogical Agents (GURER, 1998). They take the formof visible characters on the screen in multimedia learning environments designed to facilitatelearning (MORENO, 2005; SCHROEDER; ADESOPE, 2012). They can exhibit various typesof life-like behaviours, such as speech, emotions, gestures and eye, head and body movements(DEHN; MULKEN, 2000). Such characters can serve numerous pedagogical roles, such asinstructors, coaches, tutors, and learning companions (CHOU; CHAN; LIN, 2003; BAYLOR;KIM, 2005; HAAKE; GULZ, 2009).

According to Dehn and Mulken (2000), as APAs’ behaviours somehow mimic the social ex-change present in human interactions, animated agents with these characteristics are considereda powerful addition to multimedia learning environments, since they allow these environmentsto exploit verbal as well as nonverbal forms of communication (ATKINSON, 2002; ATKIN-SON; MAYER; MERRILL, 2005). Research (REATEGUI et al., 2006; JEUNG H.AND CHAN-DLER, 1997; MOUSAVI; LOW; SWELLER, 1995) has indicated that the agents can be usedduring the learning process of the student as an interesting resource to relieve the cognitive loadof the student.

Generally speaking, the insertion of a virtual character in an interactive learning environ-ment improves students’ learning (LESTER; STONE, 1997). This notion has been called per-

sona effect."The simple question as to whether an animated interface agent improves human-computer interaction does not appear to be the appropriate question to ask. Rather the questionto ask is: what kind of animated agent used, in what kind of domain influences what aspects ofthe user’s attitudes or performance?" (DEHN; MULKEN, 2000).

One has to be careful, though, when using APAs. One of the negative effects of their use maybe, for instance, the so-called split-attention effect. The split-attention effect occurs when theagent and its presence are not intellectually integrated with the content to be taught. When that

5

happens, the image of the agent represents an additional, non-desirable cognitive load, probablybecause the students might chose to look at the agent more than at the content that shouldbe integrated with the spoken text (DUNSWORTH; ATKINSON, 2007; CRAIG; GHOLSON;DRISCOLL, 2002). One interesting alternative would be to direct the learner’s visual attentionexplicitly to the content (ALIBALI; DIRUSSO, 1999).

3.1 Effects

Moreno and Mayer (2000) have proposed a way to analyze the several questions relatedto the cognitive and affective impact of the agents. What they did was to measure the agents’impact on learning by what the authors called embodied agent effect, image effect, and modalityeffect (MORENO; MAYER, 2000) .

The embodied agent effect makes use of a fully embodied agent that delivers aural in-structions. The effect compares results of instructions delivered by the agent and instructionsdelivered simply by text (DUNSWORTH; ATKINSON, 2007). The agent may direct a learner’sattention by moving around the screen, using gaze and gestures, providing nonverbal feedback,and conveying emotions (ATKINSON, 2002). According to Atkinson (2002), an agent withthose characteristics of instruction delivery is capable of simulating the student-instructor rela-tionship, and thus might facilitate learning (ATKINSON, 2002). That was demonstrated alsoin Craig and his colleagues’ work (CRAIG; GHOLSON; DRISCOLL, 2002). They have de-scribed how much learning had a student that went through interactions with animated agents,in comparison with students whose interactions were with static agents. Comparing the resultsof students that have interacted with two different versions of the same agent, one version staticand another version fully embodied, presents itself as a good way to demonstrate the embodiedagent effect.

The image effect explores the visual presence of the agent during learning. It measures theimpact of the presence of the agent on learner performance and attitudes. To do so, it comparestwo aurally delivered instructions, one of them with the presence of the agent on the screenand the other one without the agent (DUNSWORTH; ATKINSON, 2007). Lester and Stone(1997) have affirmed that even if the agent is not expressive, the mere presence of the agentcan positively affect the learning experience of the student. According to them, students havesaid that learning with the animated agent is more enjoyable than without it. Having said that, agood way to demonstrate the agent effect is to compare two aurally delivered instructions, oneof them having the agent delivering the instruction and the other one not having the agent at all.

According to the modality principle, the textual information should be presented to the stu-dents orally instead of by written words (MAYER, 2001). According to Ginns (2005), this prin-ciple states that learning will increase if the textual information is presented by audio and withrelated visual information. Ginns proves this principle in his work (GINNS, 2005), but someother authors as Schueler et al. (2008) affirm that, under certain circumstances, the modality

6

principle does not apply.

Mayer (2005) created the cognitive theory of multimedia learning (CTML), which explainsthe modality effect. The CTML is based on three premisses about human information process-ing when learning: dual channels, limited capacity and active processing. The dual channelspremisse is that humans have separate channels for the visual and auricular processing of infor-mation: the visual information is processed in the visual channel and the auricular informationin the audio channel. The second premisse regards the limited capacity that each channel hasof processing information at a specific moment. The active processing premisse is the one thatsays that students must actively process the information. To achieve that, they must use theprocessing channels to organize the information received (MAUTONE; MAYER, 2007).

4 RELATED WORK

Research on the use of APAs is not new. In the medical field, an example of that kind ofresearch can be found in Johnson and his colleagues’ (JOHNSON; SHAW; GANESHAN, 1998)work, in which they presented the agent Adele. The agent Kurrupako has taught algorythms(REATEGUI et al., 2006) and Lester e Stone (1997) used the agent Herman in the field ofbotanics and phisyology. The use of animated agents in learning systems have been exploredover the years .

Research was also done in the field of language learning in order to use pedagogical agentsto improve learning in CALL systems. Choi and Clark, for instance, studied a learning envi-ronment for ESL students to learn relative clauses. It was a very simple system, composed ofpowerpoint slides and the Microsoft agent, and it was applied to 74 college students (CHOI;CLARK, 2006). They tried to evaluate the learning differences between a group that receiveda pedagogical agent and a group that received an electronic arrow and voice narration as com-panions in the learning process. They found no difference in the overall results of each group,thus their study did not support the agent effect. However, if only the data of students withlow level of prior knowledge were considered, there would be gain using the agent. The dataof students with intermediate to high level of prior knowledge show no benefit when using theagent. The study results indicate more significant gain with the students that scored the lowestin the pretest. If the student already knows the content, or is already familiar with it, maybe theagent does not really help.

Galluccio (2008) designed an experiment to investigate the effect of agent accent (Anglo orHispanic), appearance (Anglo or Hispanic), and type of instructional activity (Grammar-based,communication-oriented) on student’s performance, motivation, and perception of agent. Theusers of the system were college students enrolled in basic Spanish classes. Results indicatedno differences in performance, motivation or perception between the groups using the differentagents with accent and appearance typically Anglo or Hispanic. Also, no difference in perfor-mance or motivation was found in the grammar-based activity or the communication-oriented

7

activity. However, when the results of both kinds of activities – grammar based activity andcommunication-oriented activity - were merged, the Hispanic tutor version presented better re-sults. Besides that, students usually preferred the Hispanic agent to receive feedback from,while the agent perception of the students varied – some preferred the Hispanic version whileothers preferred the Anglo version.

Theodoridou (2009) created Laura, a pedagogical agent to help students on Spanish LexicalAcquisition. In her system, the students were presented with new vocabulary (orally and textu-ally) and exercises for practicing the vocabulary learned in context. During the experiment, thestudents could chat with Laura via keyboard about the material presented. The results of thestudy did not differ between the two versions regarding vocabulary acquisition.

Ko (2010) studied listening anxiety and listening comprehension skills in Korean collegestudents . Three groups were analyzed: one with an American agent, another one with a Koreanagent and another one without an agent. No statistical differences in listening anxiety levelsand listening comprehension skills were found between students who worked with the agentand students who worked without the agent. The American agent and the Korean agent groupsalso did not present different results.

Sabet and Shalmani (2010) compared the effects of anthropomorphic (resembling humans)and non-anthropomorphic (little or no resemblance to humans) animated characters in multi-media programs with environments with no characters at all. The environment used evaluatedEnglish vocabulary retention of Iranian learners of English as a Foreigh Language (EFL) . Thestudy showed that the groups with the agent outperformed the group without the agent, andthe group who used anthropomorphic animated agents outperformed the other group (SABET;SHALMANI, 2010).

Regarding the agent effects studied on this article, there is, for instance, Atkinson’s work(ATKINSON, 2002) in the field. Atkinson used an animated pedagogical agent to optimize acomputer-based learning environment designed to teach learners how to solve word problems.The agent of his system could deliver instruction either textually or orally, using gaze and ges-ture to direct the focus of the students on the relevant part of the example. Final results wereconsistent with the embodied agent effect and the image effect. The modality effect, however,was not found in the experiment, since participants that had instruction delivered orally andwithout the agent did not outperform participants who had instruction delivered textually andwithout the agent.

Dunsworth and Atkinson also studied the agent effects (DUNSWORTH; ATKINSON, 2007).Their work had the purpose of evaluating the modality effect, the embodied agent effect, and theimage effect in a system that lectured about the human circulatory system. Results of the studycould not attest the modality effect in the agent, but the embodied agent effect and the imageeffect were both present. The conclusion of the study was that an agent programmed to coordi-nate narration with gaze and pointing incorporated into a science-focused multimedia-learningenvironment can foster learning.

8

The latter two works described here are a very significant contribution to the field of theagent effects this article proposes to study. However, they evaluated the effect in very specificfields - Math and Medicine. No significant work was found that evaluated the agent effect inthe language learning field. A pedagogical agent used in a CALL system could help students intheir understanding of meaning during learning (see section 2).

5 THE SYSTEM

The system was developed as a website, to facilitate its access in a computer laboratory.The technology used for the development was Java3 with Servlets and JSPs, using Model ViewControl (MVC)4 model and MySQL5 for database. For the APA, it was used DIVAlite (detailsabout DIVAlite are in the next section).

The goal of the system was to evaluate the image effect and the embodied agent effectrepresented by the agent.

The embodied agent effect says that an agent able to simulate the student-instructor rela-tion by moving, gazing, and delivering auditory messages, can facilitate learning (ATKINSON,2002). In this project, the embodied agent effect can be evaluated by comparing the results ofthe students that used the static version of the agent with the results of the students that used theembodied version of the agent. It was predicted that the learning of the participants assigned tothe embodied version would be significantly bigger than their peers from the static version ofthe agent.

The image effect can be evaluated by comparing the results of the students that used theversion without the agent with the results of the students that used the static version of theagent. It was predicted that the learning of the participants assigned to the static version wouldbe significant bigger than their peers from the raw, no-agent version of the agent.

5.1 The agent

The DIVAlite framework is a software tool dedicated to the integration of embodied con-versational agents into the internet (SANSONNET et al., 2012). It enables the integration of ananimated agent into a webpage interacting with Document Object Model (DOM) elements ofthe page. There are a few standart agents in DIVAlite that can be used, and they all have basicmovements already incorporated. It is also possible to create new agents with DIVAlite. To dothat, it is sufficient to provide the framework with the images correspondent to each movementof the agent.

Patti is an animated agent created specifically for this project. She was created using DI-VAlite and she is the representation of a wise, polite, disciplined, responsible and trustworthy

3http://www.oracle.com/technetwork/java/index.html4http://www.oracle.com/technetwork/articles/javase/index-142890.html5http://www.mysql.com/

9

Figure 1 – Patti standing


teacher. She is blonde, has blue eyes and wears black clothes (see Figure 1). Patti appears inthe second and third versions of the system. She greets the student and introduces herself on thefirst screen (see Figure 2), explains the material on the content screens and reacts to the resultsof the students on the exercises.

Patti’s voice was recorded using an ipod. The voice recorded belonged to a Brazilian teacherof EFL that acquired American accent after studying American pronunciation and living in theUSA for a while. Some noise reduction was applied to the audio later using simple computersoftware. The drawings of Patti and of her movements were made in pencil, and later digitalizedusing photoshop (see Figure 3).

5.2 The versions

In order to evaluate the agent effect and the embodied agent effect, the system was built inthree different versions: (i) the first version is a raw version, without the presence of the agentor of the highlights of the content; (ii) the second version contains the agent standing on thescreen right and speaking - as she speaks, the content that she is talking about is highlighted,i.e. the color of the area of content changes its color from white to light blue ; (iii) the thirdversion contains the agent in its complete form: speaking, moving and pointing to the content.

In the first version, the interaction with the system begins with a login screen, where theuser information is validated. Once logged in, the user is presented with the welcoming screen

10

Figure 2 – Patti on the first screen


Figure 3 – One of the drafts for Patti


11

Figure 4 – Content in the first version of the system


of the system, with simple instructions of use, inviting him or her to click on the correspondentbuttons to initiate his or her activity.

The activity consists of four content sections and four exercises sections. The material forthe content and for the exercises was adapted from the seventh chapter of Murphy’s book En-

glish Grammar in Use (MURPHY, 2004). The sections of content and the sections of exercisesare presented in a sequence of screens. The student goes to the next screen by clicking on thebutton Next. The content screens contain text, tables and images altogether to present the ma-terial (see Figure 4). The exercises screens contain the exercises and a Verify button. The usercan fill in the answers of the exercise and click on Verify in order to check if the answer pro-vided is correct (see Figure 5). The first exercise involves structuring simple statements usingthe present perfect. The second exercise requires structuring present perfect statements usingjust, already or yet - it is identified for each statement which adverb should be used. The thirdexercise evaluates the composition of statements using the same three adverbs from exercisetwo, but this time the student has to decide which adverb to use. Finally, the fourth exerciseasks the student to choose between been and gone for simple statements. After the completionof the four exercises, a closure message is displayed.

This version of the system does not contain any interaction with the APA. The student doesnot see the agent, neither hears it.

The second version is composed of the raw version of the system (first version) and the APAin its simpler form. In each content screen, Patti appears and orally gives the explanation aboutthe material. While she speaks about a specific area of the content, this area gets highlighted(see Figure 6). However, Patti stands the whole time on the screen right, with her arms on herback. She does not point to the content neither moves around the screen in the second version.In the exercises, Patti is also standing on the screen right: if the student answers more than half

12

Figure 5 – Exercise in the first version of the system


of the questions properly, she makes a positive gesture. Otherwise, she displays a somewhat sadface (see Figure 7).

The third version has all of the features of the second version with the addition of Pattimoving an pointing. Whenever she is talking about an area of the content, she comes closer tothat area and points to it (see Figure 8). As in the second version, this area is also highlighted.

In order to analyze the agent effect present in the interaction of the students with the system,the results of the students that used the first version of the system are compared with the resultsof the students that used the second version of the system. To analyze the embodied agent effect,the results of the second version are compared with the results of the third version.

6 THE EXPERIMENT

In the experiment, participants were submitted first to a pretest, followed by the web-system,followed by a posttest and a questionnaire.

6.1 Participants

The participants were Brazilian university students from different courses of a SouthernBrazilian university. They were all enrolled in classes that require pre-intermediate level ofEnglish as prerequisite and participated in the study as a regular class activity. In total, sixty-nine students participated in the experiment, but only the data of 64 students were consideredvalid (see section 7 to know about the data validation used).

13

Figure 6 – Patti on the second version of the system


Figure 7 – Patti reaction to wrong answers


14

Figure 8 – Patti pointing to the content


6.2 Procedure

The experiment was conducted in computer laboratories at the university. Instructions aboutthe experiment were given to the students in Portuguese, their native language. The system,however, including explanations, audio content and menus, was all in English. So were the testsand the questionnaire. The conductor of the experiment would give the group of students someinformation about the experiment. He would tell them that the experiment consisted of a websystem built for helping the user to learn the present perfect. Also, he would explain how theexperiment would be conducted: first the pretest, followed by using the system, followed by thepostest, followed by the questionnaire. It was emphasized that all information was anonymous,that the tests should be answered only based on what they knew. Each student received anindividual username to access the system. The username and password were received in a pieceof paper. Moreover, each student was randomly assigned to one of the three groups. For that, itwas used an envelop containing pieces of paper with the corresponding numbers of the groups– 1, 2 or 3. The student would draw a piece of paper from the envelope, and the number writtenon it would correspond to the group that the student would be assigned to. The first part of theexperiment was the pretest. Students received it printed on a sheet of paper and filled it in in penor pencil. The only information they should provide with the tests, with the system, or with thequestionnaire, besides the answers, was the username they had received to access the system.After returning the pretest to the conductor, the student would access the system in the computerwearing headphones. In the system, the student received the instruction – with or without thepresence of the agent – and then answered the posttest. After that, the students that had usedversions of the system that contained the agent received a questionnaire to evaluate the agent.

15

6.3 Pretest and postest

Students were submitted to a prettest and a posttest, before and after (respectively) interact-ing with the system, both based on Murphy’s book. The tests had four exercises, each exercisehaving a group of questions aiming at evaluating a specific grammar point developed in theactivity. The first exercise assessed the capacity of the student of structuring the present perfecttense in simple sentences, i.e. using have / has + past participle. The second and third exercisesassessed the use of the adverbs just, already and yet in present perfect statements. While in thesecond exercise the correct adverb was informed, in the third exercise the student should choosethe apropriate adverb for each statement. The fourth exercise assessed the correct use of been

and gone in present perfect sentences.

6.4 The questionnaire

After using the system, the students that interacted with the agent received a questionnaireto evaluate their experience. This questionnaire is closely based on Galluccio’s questionnaire(GALLUCCIO, 2008), and uses Likert scale questions. It evaluates students’ motivation ac-cording to the four model categories of the ARCS model6: Attention, Relevance, Confidenceand Satisfaction. In the questionnaire, there were 4 questions to evaluate the ARCS’s categories,one question to evaluate willingness to repeat the experience, and one to determine overall mo-tivation. In each question, the student is asked how much he thinks that the agent helped himin each category (attention, relevance, confidence and satisfaction). Besides, the questionnairepresents open-ended questions so that the student can comment about each of the ARCS cate-gories.

7 DATA ANALYSIS

There were 2 students that, intentionally or not, did not return the pretest. Furthermore, 3students did not answer the posttest. These 5 students (without pretest or posttest) were notconsidered in the analysis. The data of 64 students were considered valid according to thesecriteria. 22 students were in the first group, 21 students were in the second group, and 21students were in the third group.

The correction of pretest and posttest followed these criteria: in the first exercise, the focusof the correction should be on the structure of the present perfet (have/has + past participle). Forthird-person pronouns, has should be used; for the rest of the pronouns, have should be used.The past participle was required, but as the acquisition of the past participle was not the goalof the system, statements with wrong past partiples were considered correct, as long as it waspossible to notice that the student had attempted to use the past participle. So, for instance, past

6http://www.arcsmodel.com/home.htm

16

Table 2 – Students final gradesGroup 1 Group 2 Group 3

Pretest Posttest Pretest Posttest Pretest Posttest7.50 6.58 6.88 9.08 2.50 6.084.38 5.25 7.50 5.58 6.88 9.580.00 2.25 0.00 0.00 4.38 7.674.38 6.92 5.63 10.00 1.88 7.335.63 6.17 3.13 3.42 5.00 4.835.63 6.58 7.50 9.58 5.00 5.757.50 9.08 3.75 7.67 8.75 8.176.88 6.17 2.50 4.75 7.50 7.177.50 6.50 6.88 6.25 9.38 8.179.38 10.00 8.75 10.50 9.38 9.507.50 4.33 6.88 7.50 6.88 7.005.63 6.50 5.63 8.33 6.25 10.085.63 4.75 9.38 8.17 6.88 4.755.00 3.50 1.25 4.00 4.38 8.584.38 9.00 6.25 9.50 3.13 8.170.00 4.33 5.00 5.50 3.13 9.175.63 3.33 4.38 3.17 1.25 1.331.88 4.00 6.88 10.00 6.88 10.008.13 9.58 5.63 8.67 5.63 5.586.25 6.58 5.00 7.00 0.63 6.678.75 9.00 6.25 5.75 5.63 6.006.25 10.00


participles such as falled instead of fallen, or improven instead of improved, were consideredcorrect in the evaluation.

Each test was analyzed and given a grade. The maximum grade was 10.0, and each of thefour exercises was responsible for 25% of the total grade. The final grades of each of the threegroups of students can be found in Table 2.

The data of each group in the pretest and in the posttest had its normality tested accordingto Shapiro-Wilk normality test (Table 3). As all p-values are bigger than the alpha=.05, we canattest that the null hypothesis (that the population is normally distributed) is accepted and thedata are from a normally distributed population.

After attested the normality of the data, it was applied the paired t-test in each group, to seeif the true difference in means of each group was not equal to zero, i.e., if there was learninggain in each group (Table 4).

The paired t-test applied to the first group (the group without the agent) gives p-value 0.5845.That value is not lower than the significance level of 0.05, so the null hypothesis is not rejected,meaning that the mean is equal to zero and there was statistically no significant gain in perfor-mance in group 1.

17

Table 3 – Normality in groups by Shapiro-Wilk normality TestW p-value

G1 Pretest 0.9529 0.3608Posttest 0.9445 0.2444




Table 4 – Paired t-test resultst df p-value mean

Group 1 0.5553 21 0.5845 0.1859091Group 2 3.5003 20 0.002253 1.398571Group 3 3.4319 20 0.00264 1.917143


The fact that the first group did not present significant learning could be explained at leastby two possible reasons. One reason could be the lack of interesting features on the screen -students were asked simply to read the text, with very little visual aids or motivational featureson the screen. Students might not have put much effort into studying, and that could explainthe lack of significant gain in performance. Another reason could be that students might havebeen expecting for the agent to appear eventually. Even though they had been warned thatsome versions of the system did not have agents, some expectation might have been created onwhether their version of the system would have the agent or not. That expectation may haveled them not to pay attention to the content, and going straight to the exercises. Having notpaid attention to the content, they did not learn much, and the results in the posttest were notsignificantly better than the ones in the pretest.

The paired t-test applied to the second group (the group with the simpler version of theagent) and to the third group (the group with the complete version of the agent) gives p-values0.002253 and 0.00264, respectively. Those values are lower than the significance level of 0.05,so the null hypothesis is rejected in both cases. The mean is not equal to zero and there wasstatistically significant gain in performance in group 2 and in group 3.

Table 5 describes the performance gain per student in each group, i.e. the result of theposttest of the student minus the result of the pretest of the student. The normality of theperformance gain data was also tested by Shapiro-Wilk normality Test (Table 6) and the datawas from a normally distributed population (p-values were bigger than the alpha=.05).

The image effect was evaluated comparing the results of the first and the second versionsof the system. An unpaired t-test was used to make that comparison, results can be found onTable 7. It is possible to see that p-value in this case (0.02545) is lower than the significancelevel of 0.05, so the null hypothesis is rejected. The true difference in means is not equal to

18

Table 5 – Performance gain in each groupGroup 1 Group 2 Group 3-0.92 2.21 3.580.88 -1.92 2.712.25 0.00 3.29-2.46 4.38 5.460.54 0.29 -0.173.46 2.08 0.75-0.92 3.92 -0.581.79 2.25 -0.33-1.00 -0.63 -1.210.63 1.75 0.13-0.67 0.63 0.130.88 2.71 3.83-0.88 -1.21 -2.131.00 2.75 4.21-0.38 3.25 5.04-0.67 0.50 6.04-2.29 -1.21 0.082.13 3.13 3.131.46 3.04 -0.040.33 2.00 6.04-2.25 -0.50 0.381.25


Table 6 – Normality in Performance Gain by Shapiro-Wilk normality TestW p-value

G1 0.9686 0.6791G2 0.9519 0.3702G3 0.9141 0.06608


19

Table 7 – Performance Gain Unpaired t-testt df p-value

Group 1 with Group 2 2.3225 39.434 0.02545Group 2 with Group 3 0.7569 36.234 0.454


Figure 9 – Questionnaire data


zero and group 2 had statistically more performance gain than group 1, thus demonstrating theimage effect.

The embodied agent effect was evaluated comparing the results of the second and the thirdversions of the system. An unpaired t-test was applied and it presented p-value 0.454 (seeTable 7). That value is not lower than the significance level of 0.05, so the null hypothesis is notrejected, meaning that the true difference in means is equal to zero and there was statisticallyno significant difference between the performance gain in groups 2 and 3, thus we cannot saythat the agent demonstrated the embodied agent effect.

The questionnaire applied to test students’ motivation had its data compiled into one graph(see Figure 9). For the graph, the answers Totally disagree and Partially disagree were com-bined into Negative, and the answers Totally agree and Partially agree were combined intoPositive. For each of the five categories (Attention, Relevance, Confidence and Satisfaction), itis possible to see in the graph how many student answered the question positively, how manyanswered negatively, and how many were neutral.

It is possible to see that, in general, students’ perception of the contribution of the agent foreach category was positive. Most students would like to have Patti again as a pedagogical tutor.

20

For the questions about the agent helping with confidence and satisfaction with the result, therewere more neutral than positive answers. That can be justified by the fact that there was verylittle interaction between the agent and the students during the exercises.

Analysis of the open-ended questions show that students, in general, liked the image andthe voice of the agent. However, some students complained that the pace of the explanationwas very fast. On the other hand, some students complained that the pace was too slow. Itbecomes clear that it is very hard to find a proper pace that is suitable for every student. A goodsuggestion for a future work is to let the student determine the pace of the explanation. Onesuggestion is to divide the explanation into small parts, and let the student decide whenever heor she wants to go to the next part.

Also in the questionnaire, some students from the second group pointed out that it would benice if the agent pointed to the content as she spoke. That was, of course, on of the features ofthe third version of the system, but the students of the second version did not know that.

8 FINAL REMARKS

The goal of this study was to investigate the image effect and the embodied agent effect ofthe APA in the learning of the English present perfect tense. A total of 64 university studentsparticipated in the experiment, split into three groups - one group for each version of the system.Analyzing the results of each group, it was possible to attest that the agent demonstrated theimage effect and did not demonstrate the embodied agent effect.

The results of this experiment partially support Dunsworth’s and Atkinson’s results (seesection 4), since also here the image effect was found, but the embodied effect was not. Themodality effect, however, could not have its presence tested here, since to do that it would benecessary to have a version of the system with aurally-delivered instruction, but without theimage of the agent on the screen.

The embodied version of the agent did not have better results than the static version. Oneexplanation for that is that the agent did not present many features that would make it embodied.It only moved to the content it was explaining and pointed to it. It is possible to suppose that, ifmore characteristics of a real human teacher were incorporated to agents, maybe results in thethird group could be better. Research for more methods of making the agent as close to a realhuman-teacher as possible would be very useful.

UM AGENTE PEDAGÓGICO ANIMADO NA APRENDIZAGEM DO PRESENTPERFECT TENSE PARA ALUNOS BRASILEIROS

Resumo: Agentes Pedagógicos Animados são considerados uma forma eficaz de melhorar aaprendizagem, e podem ser avaliados em termos do que os autores vem chamando de Efeitoda Imagem (Image Effect), Efeito do Agente Corporificado (Embodied Agent Effect) e Efeitoda Modalidade (Modality Effect). Entretanto, não foram encontrados projetos de pesquisa so-bre esses efeitos em um sistema de aprendizado de idiomas. Esse artigo descreve o projeto, aimplementação e a análise de um sistema web de aprendizagem e seu agente pedagógico ani-mado. Sessenta e quatro alunos universitários brasileiros com nível intermediário de inglês,divididos em 3 grupos, participaram do experimento. O primeiro grupo usou o sistema semo agente, outro grupo usou uma versão estática do agente e o último grupo usou uma versãocorporificada do agente. A performance dos alunos em cada grupo foi comparada para que sepudesse avaliar o Efeito da Imagem e o Efeito do Agente Corporificado. Os resultados foramcoletados com o uso de pré e pós-testes e avaliados usando testes t-pareados. Não houve ganhode performance no primeiro grupo. Os segundo e terceiro grupos, entretanto, tiveram resultadosestatisticamente melhores no pós-teste. Testes t-não-pareados foram executados usando a dife-rença entre os resultados do pós e pré-testes para cada aluno, para avaliar os efeitos do agente.Os resultados mostraram o Efeito de Imagem no experimento, mas não mostraram o Efeito doAgente Corporificado.

Palavras-chave: Agentes Pedagógicos Animados. Inglês como Língua Estrangeira . Efeito deAgente.

22

REFERENCES

ALIBALI, M. W.; DIRUSSO, A. A. The function of gesture in learning to count: more thankeeping track. Cognitive development, [S.l.], v. 14, n. 1, p. 37–56, 1999.

ATKINSON, R. Optimizing learning from examples using animated pedagogical agents.Journal of Educational Psychology, [S.l.], v. 94, p. 416–427, 2002.

ATKINSON, R.; MAYER, R.; MERRILL, M. Fostering social agency in multimedia learning:examining the impact of an animated agent’s voice. Contemporary Educational Psychology,[S.l.], v. 30, p. 117–139, 2005.

BAYLOR, A. L.; KIM, Y. Simulating Instructional Roles through Pedagogical Agents. Int. J.Artif. Intell. Ed., Amsterdam, The Netherlands, The Netherlands, v. 15, n. 2, p. 95–115,Apr. 2005.

BROWN, D. Principles of Language Learning and Teaching. 4th ed.. ed. New York:Addison Wesley Longman, 2000.

CHOI, S.; CLARK, R. E. Cognitive and Affective Benefits of Animated Pedagogical Agentsfor Learning a Second Language. Journal of Educational Computing Research, [S.l.], v. 34,p. 441–466, 2006.

CHOU, C. Y.; CHAN, T. W.; LIN, C. J. Redefining the learning companion: the past, present,and future of educational agents. Computers and Education, [S.l.], v. 40, p. 255–269, 2003.

COMRIE, B. Aspect: an introduction to the study of verbal aspect and related problems.[S.l.]: Cambridge University Press, 1976. (Cambridge Textbooks in Linguistics).

COMRIE, B. Tense. [S.l.]: Cambridge University Press, 1985. (Cambridge Textbooks inLinguistics).

CRAIG, S. D.; GHOLSON, B.; DRISCOLL, D. M. Animated Pedagogical Agents inMultimedia Educational Environments. Journal of Educational Psychology, [S.l.], v. 94,p. 428–434, 2002.

DEHN, D. M.; MULKEN, S. van. The impact of animated interface agents: a review ofempirical research. International Journal of Human-Computer Studies, [S.l.], v. 52,p. 1–22, 2000.

DOUGHTLY, C. J.; LONG, M. H. The Handbook of Second Language Acquisition.Malden: Blackwell Publishing, 2005.

DOWLING, C. Intelligent Pedagogical Agents in Online Learning Environments. In:INTERNATIONAL CONFERENCE ON EDUCATIONAL USES OF COMMUNICATIONAND INFORMATION TECHNOLOGIES, 2000. Anais. . . [S.l.: s.n.], 2000.

DUNSWORTH, Q.; ATKINSON, R. K. Fostering multimedia learning of science: exploringthe role of an animated agent’s image. Computers & Education, [S.l.], v. 49, n. 3,p. 677–690, Nov. 2007.

23

GALLUCCIO, R. Animated Pedagogical Agents as Spanish Language Instructors: effectof accent, appearance, and type of activity on student performance, motivation, and perceptionof agent. [S.l.]: Florida State University, 2008.

GINNS, P. Meta-analysis of the modality effect. Learning and Instruction, [S.l.], v. 15, n. 4,p. 313–331, Aug. 2005.

GURER, D. The Use of Distributed Agents in Intelligent Tutoring. In: ITS WORKSHOP ONPEDAGOGICAL AGENTS, 1998, San Antonio, Texas. Anais. . . [S.l.: s.n.], 1998. p. 20–25.

HAAKE, M.; GULZ, A. A Look at the Roles of Look & Roles in Embodied PedagogicalAgents - A User Preference Perspective. Int. J. Artif. Intell. Ed., Amsterdam, TheNetherlands, The Netherlands, v. 19, n. 1, p. 39–71, Jan. 2009.

JENNINGS, N.; WOOLDRIDGE, M. Agent Technology: foundations, applications, andmarkets. [S.l.]: Springer, 1998.

JEUNG H.AND CHANDLER, P. S. J. The role of visual indicators in dual sensory modelinstruction. Journal of Educational Psychology, [S.l.], v. 17, p. 329–433, 1997.

JOHNSON, L.; SHAW, E.; GANESHAN, R. Pedagogical Agents on the Web. In: ITSworkshop on pedagogical agents. San Antonio, Texas: [s.n.], 1998. p. 2–7.

KO, Y.-A. The effects of pedagogical agents on listening anxiety and listeningcomprehension in an english as a foreign language context. 2010. Tese (Doutorado emCiência da Computação) — Utah State University, Logan, UT, USA, 2010. AAI3433403.

LESTER, J. C.; STONE, B. A. Increasing believability in animated pedagogical agents. In:THE FIRST INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS, 1997, NewYork. Anais. . . ACM Press, 1997. p. 16–21.

LEVY, M. Computer-Assisted Language Learning: context and conceptualization. NewYork: Oxford University Press, 1997.

MAUTONE, P. D.; MAYER, R. E. Cognitive aids for guiding graph comprehension. In:JOURNAL OF EDUCATIONAL PSYCHOLOGY, 2007. Anais. . . [S.l.: s.n.], 2007.p. 640–652.

MAYER, R. The Cambridge Handbook of Multimedia Learning. [S.l.]: CambridgeUniversity Press, 2005. (Cambridge Handbooks in Psychology).

MAYER, R. E. Multimedia learning. New York: Cambridge University Press, 2001.

MORENO, R. Multimedia learning with animated pedagogical agents. In: MAYER, R. (Ed.).The Cambridge Handbook of Multimedia Learning. [S.l.: s.n.], 2005. p. 507–523.

MORENO, R.; MAYER, R. E. Life-Like Pedagogical Agents in Constructivist MultimediaEnvironments: cognitive consequences of their interaction. In: CONFERENCEPROCEEDINGS OF THE WORLD CONFERENCE ON EDUCATIONAL MULTIMEDIAHYPERMEDIA, AND TELECOMMUNICATIONS (ED-MEDIA, 2000. Anais. . . [S.l.: s.n.],2000. p. 741–746.

24

MOUSAVI, S.; LOW, R.; SWELLER, J. Reducing cognitive load by mixing auditory andvisual presentation modes. Journal of Educational Psychology, [S.l.], v. 87, p. 319–334,1995.

MURPHY, R. English Grammar In Use with Answers: a self-study reference and practicebook for intermediate students of english. [S.l.]: Cambridge University Press, 2004. (Grammarin Use Series).

MYDANS, S. Across cultures, English is the word. 2007.

REATEGUI, E. B.; CERON, R. F.; BOFF, E.; VICARI, R. M. Kurrupako : um agenteanimado sócio-afetivo para ambientes de aprendizagem. 2006.

REEVES, B.; NASS, C. The Media Equation: how people treat computers, television, andnew media like real people and places. Stanford, CA: CCSLI Publications, 1996.

SABET, M. K.; SHALMANI, H. B. Visual and Spoken Texts in MCALL Courseware: theeffects of text modalities on the vocabulary retention of efl learners. English LanguageTeaching, [S.l.], v. 3, p. 30, 2010.

SANSONNET, J. P.; CORREA, D. W.; JAQUES, P.; BRAFFORT, A.; VERRECCHIA, C.Developing web fully-integrated conversational assistant agents. In: PROCEEDINGS OFTHE 2012 ACM RESEARCH IN APPLIED COMPUTATION SYMPOSIUM, 2012, NewYork, NY, USA. Anais. . . ACM, 2012. p. 14–19. (RACS ’12).

SCHMIT, C. Cross-Linguistic Variation And The Present Perfect:the case of portuguese.Natural Language and Linguistic Theory, [S.l.], v. 19, n. 2, p. 403–453, 2001.

SCHROEDER, N. L.; ADESOPE, O. O. A Case for the use of Pedagogical Agents in OnlineLearning Environments. Journal of Teaching and Learning with Technology, [S.l.], v. 1,p. 43–47, 2012.

SCHUELER, A.; SCHEITER, K.; GERJETS, P.; RUMMER, R. The modality effect inmultimedia learning: theoretical and empirical limitations. In: PROCEEDINGS OF THE 8THINTERNATIONAL CONFERENCE ON INTERNATIONAL CONFERENCE FOR THELEARNING SCIENCES - VOLUME 2, 2008. Anais. . . International Society of the LearningSciences, 2008. p. 311–318. (ICLS’08).

SKLAR, F. A. Present but not perfect: a study of problems brazilian students encounterwhen learning the english present perfect tense. 2011. Dissertação (Mestrado em Ciência daComputação) — Karlstads Universitet, Karlstad, Sweden, 2011.

THEODORIDOU, K. D. Learning with Laura: investigating the effects of a pedagogicalagent on spanish lexical acquisition. 2009. Tese (Doutorado em Ciência da Computação) —The University of Texas at Austin, Texas, USA, 2009.

WARSCHAUER, M.; KERN, R. Network-based Language Teaching: concepts and practice.[S.l.]: Cambridge University Press, 2000. (Cambridge Applied Linguistics).

An Animated Pedagogical Agent in the Learning of The Present Perfect Tense for Brazilian Students

Documents

Transcript of An Animated Pedagogical Agent in the Learning of The Present Perfect Tense for Brazilian Students