Compurers in Human Behavioc Vol. 10, pp. 107-125.1994 Printed in the U.S.A. All rights reserved.

0747-5632/94 $6.00 + .OO Copyright 0 1993 Pergamon Press Ltd.

Cooperation in Problem Solving and Educational Computer Programs

Gijsbert Erkens and Jerry E. 9. Anclriessen

Utrecht University

Abstract-In this article, the central issue revolves around the way cooperation between pairs of IO-12-year-old students is carried out in three problem solving contexts, one of which involves the computer We found important diflerences in cooperation within these three contexts. Analyses of cooperation dialogues in these settings show that during cooperation problem solving has to tyke place on a content and a communication level. Cooperation requires that the cooperating subjects acquire a common frame of reference in order to be able to negotiate and communicate their individual viewpoints and inferences. These results are relevant for cooperative learning and intelligent tutoring systems. On the basis of the conclusions of this analysis a prototype of an “intelligent cooperative educational system” has been developed. Some preliminary results on cooperation with this experimental computer program are presented.

INTRODUCTION

Research within the field of intelligent computer-assisted instruction has focused mainly on domain-specific questions of content representation, student modelling, and didactic intervention by the program, acting as a tutor. Tutoring, however, is interactive by nature. Its effect will greatly depend on the coordination and fine-tun- ing of communication between the tutor and the student. This process does not only concern the conceptual aspects of information exchange, but also involves knowl- edge about the communicative aspects of the specific problem solving situation.

A handicap is that there still exists a lack of insight in the way students commu- nicate and thereby coordinate their information processing while cooperating in different problem solving contexts. When students cooperate and communicate in natural language, information is exchanged, not only concerning the problem itself but also about metacognitive aspects such as the plausibility of the information and beliefs about the state of the information of the other.

Requests for reprints should be addressed to Gijsbert Erkens, Department of Education, Heidelberglaan 2, 3584 CS Utrecht, The Netherlands.

107

108 Erkem and Atdriessetl

In this article, the central issue revolves around the way cooperation between IO-12-year-old students is carried out in three problem solving contexts, one of which involves the computer. The differences in cooperation within these three contexts are analyzed and related to cooperative learning and intelligent tutoring systems. The main goals of this article involve, first, showing the importance of the analysis of the processes of cooperation during problem solving and, second, showing how different problem solving contexts may affect these processes.

First, we will discuss the results of the analysis of protocols of dialogues between students cooperating on a problem solving task. These protocols have been analyzed in depth with regard to the relation between problem solving pro- cesses and communication. Then, the results of a second task will be discussed, in which pairs of students cooperated to produce a coherent text. This discussion serves as an introduction to the comparison of problem solving and interaction by the same subjects in different domains.

On the basis of the results of these two studies a first version of a prototype of a “dialogue monitor” for an “intelligent” cooperative system for the first task has been implemented. This monitor is the central part of a computer-assisted educa- tional program that can “think along” with the student and that cooperates in solv- ing a problem task jointly. Student and system interact in a mixed-initiative dia- logue that is argumentative in nature. The program has been experimentally tested with students (lo-12 years) of two elementary schools. Some preliminary results will be described in this article. It ends with a discussion on the implications of our research for the construction of intelligent cooperative systems.

COOPERATION IN TWO PROBLEM SOLVING ASSIGNMENTS

Research Project 1: Dialogues of Cooperative Problem Solving

In the DSA project (analysis of dialogue structure in interactive problem solving), the relationship between the cognitive aspects of information processing and the communicative process of information exchange during cooperative problem solv- ing is studied.

The Camp Puzzle. The task that is used to study the relation between information exchange and information processing during cooperative problem solving is called the “Camp Puzzle.” It is meant for students from the highest grades of elementary school (10-l 2 years old).

The Camp Puzzle is similar to so-called “Smith, Jones, and Robinson” prob- lems (Wickelgren, 1974). In this kind of logical problem one has to combine dif- ferent statements of information in order to derive some characteristics of a speci- fied group of individuals. However, in the Camp Puzzle this task information has been split and distributed among the two cooperating partners. By this splitting of information cooperation becomes necessary in order to complete the task. The cooperation partners have to exchange the relevant information, explain their rea- soning, and negotiate about their inferences and task strategies. The solution process of the task will partly be accomplished at the communicative level (i.e., by the task dialogue).

In the instruction to the Camp Puzzle the following situation is described: There is a group of six children who have gone for a week’s camping holiday. Two of them separately wrote a letter about the children in their group. The two students

Cooperation in problem solving 109

that work on the task are each given one of these letters. The information in each separate letter is insufficient to answer all the questions.

For example, one letter contains the information “The friend of Jill comes from Haarlem.” In the other letter the sentence “Ann comes from Haarlem” can be found. The students may infer that Ann is the friend of Jill, thereby ignoring the possibility that more children may come from the same city.

The students have to infer six characteristics of the six children. The collectively found solutions for the 36 subproblems can be written down in a (6 x 6) solution matrix (see Figure 1). In the student-computer version the puzzle has been reduced to a total of 24 subproblems (4 x 6 matrix). In both versions, the students are allowed 45 min to work on the task. The number of correctly solved subproblems can be taken as an indication of overall task performance. Although the task is per- ceived as difficult, the motivation and the task orientation of the students on both versions was remarkably high.

Verbal observation system. Protocols of the task dialogues were obtained with the aid of a semiautomatic transcription system on the basis of video recordings. This system is called the verbal observation system (VOS). The VOS system is a com- prehensive and fine-grained coding system containing some 300 communicative and semantic coding categories. It was developed to transcribe propositional con- tent as well as pragmatic and communicative characteristics of utterances. The VOS system uses clue-words in the utterances to encode the communicative func- tion and content. In the VOS system utterances are transcribed along three main characteristics: (a) dialogue act, (b) propositional content, and (c) illocution.

The propositional content is encoded in a predicate form in which the arguments can be embedded. For example, “The friend of Jan comes from Haarlem” is repre- sented in a form something like [city, [friend, Jan, X], Haarlem]. The following types of propositions are distinguished: direct assignments, indirect references, equalities, set distributions, and axioms.

The dialogue act represents the communicative action of an utterance. Utterances like “Does the friend of Jan come from Haarlem?,” “But from Haarlem comes Jan’s friend,” or “No, the friend of Jan comes from Haarlem!” all have the same propositional content but differ in dialogue act (respectively, question,

2. grade:

3. city:

4. sports:

5. likes:

6. dislike:

Figure 1. Solution matrix of the Camp Puzzle.

110 Erkens and Andriessen

Example: -__----

“Hait !” “Eh. eh,...”

function: -__------_-- --_-___-

“Here it says:..” “Ann from Haarlem”

reading aloud from lette

‘Let’s write Haarlem.” pmposa 1 “Where does Ann live 1’ open question elicitative “Ann lives in Haarlem 1” check question (yes/no)

“Haarlem.’ “Ann lives in Haarlem” ‘Oh. I didn’t know that” ‘Yes. all right!” “No, .--not Ann.”

f&r1 responsive

“Because Ann does” “But Jill does not” “Then she comes from...” “If Ann is Jill’s friend.” “And Ann does” ‘So Ann has to live there”

!!Zri arg-ntation

cwrites in solution-imtrixp writing action

Figure 2. Main dialogue-acts and communicative functions in the VOS system.

counter, and denial). In the VOS system 6.5 dialogue acts are distinguished, in 19 main categories representing five communicative functions. In Figure 2 the 19 main categories are given, together with their communicative function.

The illocution category represents stated illocutionary force as described by Searle (1969). The illocutionary part of an utterance provides the listener extra information on how to interpret the information transferred. The category system only considers explicitly stated illocution. In the Camp Puzzle the illocution refers in most cases to the certainty of the information (e.g., “I am not sure that . . .”

Results. The main results of protocol and statistical analyses of the cooperative task dialogues can be summarized as follows. First, on the task-content level, sev- eral logical inference procedures and task strategies have been distinguished. The task is rather complex for the students. Unlike normal Smith, Jones, and Robinson puzzles, the solutions in the Camp Puzzle are not all unique. That is, more than one child can come from a certain city, engage in a certain sport, and so forth.

Essentially, the Camp Puzzle can be represented as a constraint satisfaction problem, in which forward reasoning (finding positive instantiations) and back- ward reasoning (finding negative instantiations) are required. For example, the indirect referent type of information “The friend of Jan comes from Haarlem” can be used for the subproblem “Who is the friend of Jan?” in two different ways:

1. Making a list of possibilities by looking for children from Haarlem; searching for information on the frequency of Haarlem and trying to reduce this list by further constraints.

2. Making a list of impossibilities by searching for children known to come from another city and trying to expand this list by other constraints.

However, many students handle the implicit implication in this type of state- ments as a bi-implication: They tend to consider the first person mentioned as com-

Cooperation in problem solving 111

ing from Haarlem as the friend of Jan. Faulty and correct inferences by the students have been identified for each type of proposition.

Second, regarding the task dialogues, the topical structure in the dialogues coincides with the subproblem structure of the task, similar to that found by Grosz (1978). The sequence of subproblems is not rigid, and a solution path has to be found. For this purpose, topics have to be initiated, tried, and evaluated in the ongoing dialogue. Remarkably, topics are seldom explicitly proposed (“Let’s search for the friend of Jan”), but are initiated implicitly by exchanging relevant information concerning a topic.

Most of the dialogue acts in the Camp Puzzle are informative, responsive, or argumentative (see the next section). Contrary to what one would expect, the dia- logues contain very few open questions (e.g., “In which city does Jan live?‘). The students seem to hold on to another cooperative principle (cf. Grice, 1975): “If my partner has found something interesting, he will tell me, I don’t have to ask for it!” Yes/no questions are found more frequently (e.g., “Does Jan live in Haarlem?“). These questions function mostly to check information exchanged by the partner.

Furthermore, the students are very concerned with the plausibility or certainty of the propositions transferred or inferred by themselves or by their partners (25% of the utterances have an explicit illocution part). Several plausibility levels (five in our model) can be distinguished, depending on the source of information and on the depth and complexity of the inference procedure.

On the basis of statistical sequential analyses, different topic structures and dif- ferent argumentation or reasoning sequences have been identified in the protocols. The most common pattern for the topic structure is as follows:

1. Attention signalling to the partner, 2. Or: exchange of information,

Or: eliciting information exchange, 3. Or: conclusion (of a solution),

Or: support of information by the partner, Or: check of information by the partner, followed by: Or: confirmation, Or: discussion with counterarguments, denials, explanation, and reasons,

4. Or: responding the question for information, followed by: Or: support of the reply, Or: checking of the reply,

5. Or: writing a solution in the matrix and continuing, Or: continuing with new information transfer.

Based on these and other findings in the dialogue protocols the prototype of a dialogue monitor was developed.

Research Project 2: Dialogues of Cooperative Story Construction

The second domain of problem solving we studied is that of writing a coherent text. The purpose of this research was to examine the strategies for producing a coherent text and to relate these results to the possibilities of intelligent tutoring for writing.

The final aim is a specification of the decisions in the form of executable pro- duction rules that can be used to simulate text production behavior. A computer simulation could be constructed that functions as a test of the coherence of the interpretations. Furthermore, the results of such a simulation can be used to con- struct a student module to be implemented in a computer program that coaches text

112 Erkens md Anrlriessen

production. Such a module has not yet been constructed, as the analysis of the results proved to be a complicated matter.

The ‘story Construction Task.” In this assignment, 144 subjects worked in pairs to produce a text of 20 sentences in 10 rounds. Each round each subject received a set of four cards with alternative sentences, and the task was to negotiate to select a sentence from each set to put in the text.

In addition, the names of one or two main characters (or a pronoun) had to be inserted in each sentence. Subjects were seated behind a computer screen and had to type the numbers of the selected sentences and the first letters of the names of the preferred characters. The resulting text immediately appeared on the screen. The subjects were able to correct errors. All sessions were videotaped. The task took between 35 and 45 min.

The sentences used in the task are parts of four texts about two children who experience some trouble with one of their bicycles. The original texts each contain 20 lines, and the sentences with the same ranking in every text are presented as the four alternatives at every round during the game process. The possible number of texts resulting from this is very large.

As construction progresses, possible choices become constrained by previous choices. These constraints determine the potential continuations from a certain point, and the awareness of them by a writer influences the coherence of the text he is constructing. The main purpose of the study was to find out which constraints the subjects are aware of and which are used for sentence selection.

To illustrate the task, and to show the problems the subjects are expected to solve, consider Text A in Figure 3. Text A starts with two characters having a prob- lem and ends with an inquiry into the cause of this problem. There are several pos- sible continuations, but there are quite strict constraints. The next sentence should involve a reaction to the question. It should be a relevant reaction, by the right per- son, with John as the more likely character to react. All four alternatives seem a probable candidate for the next sentence, but their implications for the text are dif- ferent. Comparisons of the plausibility of the causes might reveal that someone sit- ting at the back of a bicycle (Alternative b) is not a very probable cause of a flat tire. Furthermore, this utterance can be interpreted as an insult regarding the weight of the person sitting on the backseat! However, with a different first sentence (e.g., Text B in Figure 3), Alternative b would be the most plausible reaction, because it is the only one with an element of uncertainty.

Exaaples ‘Story Construction Task”, round 3. sentence 6

Text Te*t

1: John sat at the back of gills” bicycle. 1: John arat gill at the bicycle-shelter. 2: You have a flat tire !, John said. 2: You have a flat tire !. John said. 3: A flat tire, that’s not possible. Bill said. 3: A flat tire. that’s not possible, Bill said. 4: Well. look for yourself 1 John said. 4: Well, look for yourself ! John said. 5: How can that be ? Bill said. 5: How can that be ? gill said.

alternatives presented: a: # waited. and thought about it for a marant. b: Iiaybe. because soarsone sat at the back, said #. c: A hole in the inner tube, # said. d: I drove it against a stone, said #.

Figure 3. Illustration of the Story Construction Task (Round 3).

Cooperation in problem solving 113

Analysis. For the analysis of content, an elaborate scoring system was developed that involved the themes, the conversation, and the rhetorical functions of chosen sentences in comparison with the alternatives (see Andriessen, 1991). These aspects were treated as possible viewpoints on text coherence that a subject could be aware of. The notion of viewpoint was borrowed from the domain of artificial intelligence and instruction (Wenger, 1987), and denotes a limited focus of atten- tion at a certain level of depth. The limited content domain that the task addresses and the structured nature of the assignment allow a specification of the decisions involved in the selection of sentences by comparing selected and not-selected sen- tences in each round. This procedure has been carried out for all sentences in the texts in order to determine the subjects’ strategies for sentence selection.

Thinking-aloud protocols were used. The purpose of the analysis of the proto- cols was to examine the arguments used during negotiations by the two cooperat- ing subjects. A secondary purpose was to examine whether cooperation would lead to better - or different - arguments and better - or different - texts than indi- vidual story production. We will first present an example of a protocol and then discuss the general results.

Protocol example. In Figure 4 a protocol fragment is shown of two girls (11 years old) cooperating on the task. The fragment concerns the selection of the Sentences 5 and 6 (Round 3). Where a sentence is read aloud, it appears in italics. Reasons are underlined.

This fragment is characterized by frequent rereading and by discussion and inspection of each other’s alternatives with both subjects taking decisions. The pro- tocol shows attention to the situation described in the text and to the moods of the two characters. Rereading has an important function in determining sentences and sentence subjects. Besides that, these subjects do not explicitly state many reasons to select or reject sentences.

However, the resulting text had average quality and did not show the reasoning the subjects displayed in their verbalizations. The problem here, and in fact in many other protocols, is that subjects sometimes appeared to show good insights which did not always result in the appropriate selections. An important reason for this is that in this assignment the process of collaboration itself involves extra constraints: Subjects not only have to attend to the task of producing a coherent text, but they also have to negotiate with their partners. Those partners can have different insights, other viewpoints, and different skills. As viewpoints and insights are often unclear, implicit, and hard to explain by subjects of this age, cooperation does not lead to improved texts. The most important vehicle for com- munication of viewpoints in collaborative text production probably is reading sen- tences aloud.

Results. Generally speaking, there were no differences in rated quality between the texts produced in cooperation and those produced individually under comparable conditions, but more texts produced in cooperation were of average quality. These texts were characterized by a sequential approach, in which themes of sentences were continued and selections that led to the solution of the flat tire problem were preferred. The viewpoints on text coherence were often limited to the current and the previous sentences. This resulted in an alternation of the two story characters as subjects of consecutive sentences. As a consequence, the roles of the two charac- ters (e.g., the one with the flat tire and the one that is the helper) were not properly developed, and the two were sometimes confused.

I14 Erkens and Atuiriesserl

Protocol M : Pair 24. SOL-, level ratings: 3-3-3 (unfinished) Round 3:

Text sofar: lc. Janet net Karen on the bicycle. 2b. You”ve got a flat tire, Janet said. 3a. That’s inpossible, Karen said, a flat tire. 4a. Look for yourself, over therel, Janet said.

options (1): 5a. How is that possible?. # looked at # in surprise. 5b. How did that come about?, # asked. 5c. Well, how did that happen?. # asked angrily. 6d. What shall ye do now?, # asked.

options (2): 6a. # waited, and thought for a while. 6b. I drove it against a stone, # whispered. 6c. A hole in the inner tube, # said. 6d. Haybe, because soarsone was sitting at the back, # said.

[both subjects read text aloud] Yes. How is that possible, how did that coax? about. Here, “what shall we do now, ? asked”, or “how is that possible”. Eh. _ How is that possible?, Karen looked at Janet in surprise. Yes. Eh... i drove it... no. At the back-seat. No, we don’t have sawbody at the back-seat. A hole in the inner tube. Waited and thought. Well, let’s take this one. shall we? I drove it against a stone. # whispered. Wait a minute... What shall we do, or how did it happen... How is it possible, Janet asked. No. here it says: Lwk for uaurself, over there! Then what’s next? Look for yourself, over there. To look. How did it happen... What sha 1 I mp do now?. Karen asked. Then, it should be Janet.... or Karen? Karen waited, and thought for a while. Yes. Wait a minute, What shall llle do now?. # asked. Who asked? Karen. Karen. Better say: What shall he do now?. Janet asked. Jamt uaited, and thcmgh t for a wlr i le. Janet wa i ted, and thought for a wlr i le. Yes. Yes? Yes.

Figure 4. Protocol fragment from the Story Construction Task (Round 3, two girls, 11 years old).

A probable cause was that the manner of text production by individual subjects of this age relies almost uniquely on their mental representation of events, and does not concern actual text very much. This kind of behavior in the production of writ- ten texts has been described as knowledge-telling by Bereiter and Scardamalia (1987). In the cooperation version of the task, the contents of two mental represen- tations, one by each subject, have to be communicated. Although the situation should, in principle, lead to the same type of dependency as for the Camp Puzzle, the multiple constraints and the relatively open-ended nature of the assignment resulted in compromising rather than finding the best solutions.

It seems that, in addition to all the constraints already involved in text produc- tion, subjects working in pairs on this task have trouble with finding a common frame of reference. Our research on individual story construction showed that sub- jects rely to a great extent on their mental representations of the text. It appeared that communication of these representations was very difficult. As a result, in con- trast with the cooperation in the Camp Puzzle, differences in viewpoints were not always recognized.

This conclusion is reinforced by an observed trade-off between global represen- tation-based explanations and local thematic ones when multiple constraints had to be fulfilled. Because representations are not always adequate for the integration of

Cooperation in problem solving 11.5

multiple constraints involved in sentence selection, communication suffers from this. The problems with the extra burden of communicating evaluations indirectly result in local text.

Reading aloud sentences or text fragments almost completely replaced explana- tions, and functioned to communicate viewpoints. The most common pattern of negotiation can be described as follows:

1. Reading aloud the last sentences of the text, 2. Or: First subject reads aloud one or more sentences,

Or: First subject proposes complete sentence, 3. Or: Second subject reads aloud one or more sentences,

Or: Second subject proposes complete sentence, Or: Subjects inspect sentences of first subject,

4. In case of problems, rereading and checking of text, 5. Reading aloud sentences and alternating the sentence subject, 6. Inserting sentences in text and rereading of result.

Finally, motivation was not an important cause of less optimal selections. Cooperating subjects appeared to enjoy the assignment very much.

Comparing Cooperation Dialogues From the Two Tasks

To get a better insight into the skill quality of cooperation, we are interested in comparing task dialogues from the two projects. If cooperation is a separate com- municative or metacognitive skill, then the task dialogues of the same couples on both tasks should be similar. On the other hand, if the nature of cooperation is situ- ation-dependent, differences between the tasks are expected.

Both tasks have been developed in an earlier research project (Erkens, Kanselaar, & Van der Linden, 1983) and were given to the same sample of 10-12- year-old students. The protocols of a small set of 12 couples have been analyzed exhaustively in both research projects. This set of protocols was selected for this comparative analysis. The dialogue acts of the utterances from the verbatim proto- cols of the Story Construction Task have been transcribed with the aid of the VOS system for the purpose of this analysis. Although the set is rather small, some indi- cations can be obtained with regard to the similarity of cooperative problem solv- ing on both tasks. In the next section, we will focus on structural characteristics of the dialogues on both tasks. Note that the two assignments have the same coopera- tive task structure: The two subjects each receive different task-information (letter or sentence alternatives) to be used for a joint product (solution matrix or story).

Results. In Table 1 the means, standard deviations, and range of some structural characteristics, percentages of communicative acts, and scores of dialogue proto- cols on both tasks are shown. The variable LENGTH reflects the number of utter- ances on both tasks. Although both tasks take about 45 min, the number of dia- logue acts in the Camp Puzzle is two to three times as high as those in the Story Construction Task. Most of the time the students in the Story Task engage in non- verbal activity (i.e., the silent reading of story and alternatives).

Asymmetry in speech (ASYM), measured as the difference in percentage of speech between the two partners, is meant to give an indication of verbal domi- nance in a couple. The couples themselves differ strongly in this aspect, as can be seen from the range and standard deviations. However, no difference in asymmetry is found between the two tasks.

116 Erkens and Atzdriessen

Table 1. Means, Standard Deviations, and Range of Percentages of Communicative Acts and Scores on Two Cooperation Tasks (A! = 12)

Camp Puzzle Story Construction

Description of Variables Variable M SD Range M SD Range

Length of protocol LENGTH 643.0 155.94 462-968 160.0 56.62 87-278 Asymmetry in speech ASYM 14.7 7.95 2-25 9.8 7.41 2-28 Length of turns TURN 2.3 0.33 2-3 2.0 .17 2-2 Attention calls ATTENT 7.6 2.48 3-11 6.4 4.27 2-l 4 Informative utterances INFORM 42.9 5.83 34-53 45.9 6.04 38-57 Eliciting utterances ELICIT 11.5 1.87 a14 14.9 2.75 lo-20 Responsive utterances RESPO 19.5 3.68 14-26 26.1 3.73 21-33 Argumentative utterances ARGUM 18.5 5.51 11-31 7.0 4.79 2-l 6 Score of task product SCORE 22.3 4.67 14-33 11.3 3.05 6-16 Faults in task product FAULTS 8.3 4.23 3-16 3.1 1.83 l-7

The mean number of utterances in a turn (TURN, the sequence of utterances of one person without interruption) is similar on both tasks (2.3 and 2.0). Dialogue acts with the same communicative function (for an overview, see Figure 2) were taken together. The distribution of percentages is very similar in the task dialogues for three of these communicative acts.

The percentages of attention-calling acts (ATTENT), informative acts (INFORM), and eliciting acts (ELICIT) are rather the same in both tasks. As found earlier in the Camp Puzzle, informing (via statements and reading aloud) is far more frequent (46%) in the Story Task than eliciting information exchange from the partner via questions and proposals (15%).

A structural difference between the two tasks seems to be in the relative amount of responsive acts (RESPO) and argumentative acts (ARGUM). In the Story Construction Task more dialogue acts function to respond to, or support, utterances of the partner (26% vs. 19%). The RESPO category generally indicates mutual sup- port; the majority of the acts in this category are confirmations. In contrast, the Camp Puzzle dialogues contain far more arguments and reasoning (ARGUM, 18% vs. 7%). This result agrees with the impressions and observations in the analysis of the separate tasks. It is possible that the constraints are more open and visible in the Camp Puzzle; faulty conclusions can be directly contrasted with other information in the letters. Furthermore, the current focus of discussion is less ambivalent by the topical subproblem structure of the Camp Puzzle. In the Story Task the partners can focus much easier and more implicitly on different viewpoints and perspectives (i.e., rhetorical, subject, conversational, etc.) to solve the current subproblem. This cir- cumstance makes it harder to accomplish coordinated argumentation and discussion.

The “scores” on the Camp Puzzle concern the number of correctly solved sub- problems (SCORE) and the number of false solutions filled in the solution matrix (FAULTS). The score on the Story Task (SCORE) reflects a cumulative rating based on several aspects of coherency and quality in a text by means of the afore- mentioned scoring system. FAULTS are the number of incoherences in the text.

To sum up, in the Camp Puzzle we observed a higher total number of utter- ances, which were more often of a responsive and argumentative nature, than in the Story Construction Task. No differences between the two tasks were found in the asymmetry of participation in the dialogue and in the mean number of utter- ances in a turn.

Correlations were computed in order to compare these variables on both tasks. The correlation matrix is shown in Table 2. None of the correlations obtained one-

Cooperation in problem solving 117

tailed significance at the p c .Ol level. Of course, the small sample size requires rather high correlations to obtain significant results (about r =.65 for p < .Ol at N = 12). Still, we want to point at some tendencies of covariation that can possibly be expected in a larger sample. Note that these tendencies have to be interpreted with care and should be confirmed in further research.

Probably some sort of positive relation exists in the verbosity of the couples during cooperation on different assignments (57, LENGTH). Asymmetry in speech (ASYM) and average number of utterances per turn (TURN) do not seem to be related across the tasks. As for the communicative acts, no covariation was found, except for the relative number of arguments (59, ARGUM). Reasoning and argumentation may be real characteristic features of cooperative problem solving of couples. Scores and faults on both tasks do not seem to be related (SCORE & FAULTS).

In conclusion, it can be said that the domain-specific aspects of both tasks (“logic problem solving” and “writing”) are possibly more relevant than their cooperative nature. The necessity of reaching a common frame of reference and the more explicit nature of the constraints in the Camp Puzzle contrast with the implic- it assumption of common representations and viewpoints in the Story Task. For cooperation to function optimally, common representations and the ability to make differences and similarities explicit seem to be essential. The nature of the Story Task (open-ended, implicit constraints) leads to a knowledge-telling approach that does not support constructive attempts to reach a common goal.

COOPERATION WITH THE COMPUTER

Model of Cooperative Problem Solving and Dialogue Processing

The kind of tasks being discussed here, in which information exchange is central, contain a very complex relationship between the problem solving process and the dialogue process. The DSA model of cooperative problem solving and information exchange is based on our analyses of dialogue protocols and on similar approaches in the literature (in particular, Carberry, 1985; Fortescue, 1980; Grosz, 1978; Reichman, 1985). The model contains a number of cognitive information process- ing subsystems and specifies the predicted relations between these subsystems as part of the individual student in his or her interaction with the outside world, including the task partner (see also Barnard & Erkens, 1989).

The DSA model is reflected in the modular architecture of the prototype dia- logue monitor for an “intelligent” cooperative system. In principle, the dialogue monitor is meant to be usable for cooperative problem solving in different domains of declarative knowledge and logic. A first version of the dialogue monitor has been implemented for cooperation with the Camp Puzzle. In the domain of biology, we are working on a cooperation task for the learning of animal taxonomy.

Following a description of the architecture of the dialogue monitor, we will dis- cuss some first results of experimentation with the program (for a full report, see Erkens, 1992).

Architecture of the Dialogue Monitor Program

The Dialogue Monitor computer program contains five different modules, each with a specified function. The architecture of the program is presented in Figure 5,

Tab

le

2. C

orre

latio

ns

of

Com

mun

icat

ive

Act

s an

d S

core

s on

T

wo

Coo

pera

tion

Tas

ks

(N =

12)

Sto

ry

Con

stru

ctio

n T

ask

Cam

p P

uzzl

e LE

NG

TH

A

SY

M

TU

RN

A

l-T

EN

T

INF

OR

M

ELI

CIT

A

ES

PO

N

AR

GU

M

SC

OR

E

FA

ULT

S

LEN

GT

H

57

.20

.07

-.18

-.

28

-.O

O

.26

.32

AS

YM

-3

4 .2

1 -.

31

.14

-.04

.1

8 -.

21

.36

.20

TU

RN

-.

27

.12

.15

-.15

.1

8 :

.30

.04

-.13

.0

3 .2

1 -.

06

.15

AT

TE

NT

-.

12

.25

0 .4

1 .0

9 .4

4 .1

5 .1

3 -.

30

-.42

.o

o IN

FO

RM

-.

51

.04

.26

-.20

.4

8 .0

5 .1

5 -.

16

ELI

CIT

-.

42

.33

-.30

.0

3 .lO

-.

34

-.O

l .4

3 .0

2 -.

17

-.44

.5

5 R

ES

PO

ND

-.

33

.27

-.29

.3

0 -.

33

-.22

.0

8 .2

8 .2

8 -.

63

AR

GU

ME

NT

.5

5 .4

2 -.

30

.09

-.49

-.

12

-.27

.1

8 .5

9 S

CO

RE

-.

12

.36

.09

.05

.lO

.24

-.41

.0

4 -.

04

.33

FA

ULT

S

-.02

-.

36

-.52

.3

9 -.

44

-.20

.ll

.0

3 .4

4 -.

31

.20

.14

__~

__

-___

_

Cooperatiorl in problem solving 119

I Dialogue

Pmcessor

dialogue-acts argmntation .

nlenu-based “natural language” ‘r- interface

_______-__----~---~~--~~ received generated utterances utterances

.

Figure 5. Architecture of the Dialogue Monitor: Model of problem solving and dialogue processing.

and represents our model of problem solving and information exchange for a single student in a cooperative task situation. The program is programmed in PROLOG (SD-prolog, version 1.2) and runs on 80286 or higher IBM/MS-DOS machines.

In this model, the labels task information, solution matrix, and partner represent the external sources of information with which the monitor communicates. A simi- lar model of the cooperating partner can be presupposed. The solution matrix is a common information source for both partners. Task information (i.e., one of the letters) is only accessible for one of the partners.

The interface is the channel through which communication with the outside world, the cooperation partner, takes place. Internally, four components are assumed, which process incoming and outgoing information. The arrows indicate the internal information exchange between the various components. The double arrows from the central focusing processor indicate a control function.

The problem solving processor contains knowledge of problem solving proce- dures - that is, rules indicating which (sub)problems are to be tackled and how. About 40 different logical inference procedures and subprocedures are implement- ed in this module. Two faulty inferences are also implemented, and can be option- ally addressed. All intermediate steps are being saved as frames in a limited work- ing memory with an estimation of the certainty level of the inference itself. These frames can be used later for explanation or question answering. In this module also task strategies (which subproblem next?) and information evaluation (what kind of info is interesting?) are performed.

The. dialogue processor contains knowledge about dialogue acts (i.e., about the forms of the nttera~ces with which info~a&io~ is exchanged and reactions from the partner can be elicited). The dialogue processor can also interpret the commw- nicative function of the dialogue acts coming from the partner. Furthermoree, the dialogue processor contains argumentative and exptanatory scenarios.

The importance of ‘%hecking”’ the ~nfo~at~o~ that is put forward by the partner is reflected in our s~rn~~at~o~ model by a checking procedure which operates on every incoming utterance of the partner. By this checking procedure the plausibility of the information transferred is compared with the own knowledge base, before the information is accepted and further used for inferencing. With the exception of checking questions, this procedure accounts for most of the con~~ations~ accep- tances, repeats, den&, and counters observed in the protocols.

The alter component contains same inference rules from which a picture of the partner’s current cognitive and communicative activities might be derived. For example, a partner’s silence could lead to the conclusion that she or he is busy, looking for info~l~tio~~ in her or his own letter

These three components operate on information stored in frame-based working memories, which contain the currently relevant information about problem solving, dialogue process, and partner.

The central part of this model is contained in the central focusing processor. This component determines the flow of information, both between the various compo- nents as well as from and toward the outside world. The focusing pracessor com- bines the results of the various components. The general task of the central focus- ing pracessor is to interpret and check incoming utterances of the partner and to generate utterances itself in a reacting or initiative way. An i~npor~nt task in this context is dete~i~i~g the focus and topic of the current dialogue context.

For the factual interaction with the system a menu-based “natural language” interface has been constructed (see Miller, 1988; Tennant, Ross, Saenz, Thompson, & Miller, 1983). By means of interconnected menus the student can select different constituents of the utterance he wants to make. Connective, type of sentence, sub- ject, predicate (Le., verb), object, and ill~cution can be picked separately and repeatedly. After each selection the utterance made so far is updated in a grammati- cally correct “natural language” form and shown in a window. In this way the inter- face is very fiexible and easy to use. With the interface a large amount of different sentences can be made (about 3.2 million). In working with the interface, the stu- dent makes, in reality, a proposition in the inte~ally used VOS representation of the program. A separate module translates these propositions as well as the propo- sitions venerated by the monitor in “natural” (Dutch) sentences, The advantages of this kind of interface should be obvious: No ambivalent semantic parsing and no typing skill is required. In Figure 6 a screen display, not translated, of the interface is shown, The sentence “But Piet and Jan sleep next to (each other)“’ is being made.

After a short instruction by the experimenter, the student receives a letter (in an envelope) and is asked to type his first name, The rest of the interaction is with the system itself by means of menus. The program asks which Letter the student received, in order to take the other one. Sub~eque~tly~ use of the ~nte~ace is trained by requiring the subjects to construct six different types of sentences (the instruc- tion module is standard courseware). After this instruction the cooperation session starts. The screen is divided in three parts, from top to bottom: (a) solution matrix,

121 Cooperation in problem solving

SOORT ZIN PERSOON/WOEP 'wz#L/NIET ZEKEBBEIB KLMR

Figure 6. Screen display of menu-based “natural language” interface.

(b) dialogue window, and (c) sentence window of the interface. In the dialogue window the current dialogue scrolls up. The menu-based “natural language” inter- face can be activated at any time by any key. In that case, interaction stops until the student finishes the sentence. Task dialogues are logged by the program.

First Results on Student-System Interaction

The program has been experimentally tested with 40 students of the highest grade of two elementary schools (lo-12 years). The students cooperated with the pro- gram under four different conditions representing optional levels of task content and communicative expertise. On the task content level low expertise was simulat- ed by allowing the system also to make faulty (bi-implicational) inferences. Different levels of communicative expertise concerned the tendency of the monitor to check or accept the information exchanged by the partner. At this moment, the resulting task dialogues and problem solving results are not yet fully analyzed. Still, some preliminary remarks on the results and drawbacks of the program can be made. A full report on program and experiments will be given in Erkens (1992).

First, and most important of all, the program was able to maintain a cooperative dialogue with a student for 45 min (adjusted session period) without too many bugs or inconsistencies.

In Figure 7 a dialogue fragment, literally translated from Dutch, is shown of the interaction of the monitor with a 12-year-old student. Utterances made by the stu- dent are prefixed with alter:. Blank lines in the protocol represent pauses of more than 2 s.

As can be seen in the fragment, the dialogue has a rather “natural” flavour with a lot of implicit proposals, ellipsis (“Yes,” “ Oh”), nonspecific replies (13) plausibili- ty checking (2), and unexpected topic shifts (17, 19). In fact, most dialogues look, at first sight, rather similar to student-student dialogues, which is promising for the ends of our research. Although in this fragment the student takes most of the initia- tive, this is not a characteristic feature of all dialogue protocols (in fact, this student was too quick for the monitor to be able to initiate).

The preliminary analyses resulted in some problems with the student-computer version of the task. One important drawback we experienced with the program was

122 Erkem nrrd Amiriessen

Fraumant of dialogue nrotocol (translated fran dutch)_ alter: 12 year-old llrale student - mnnitor: Dialogue lbnitor

1 alter :

2 monitor:

3 alter :

4 nunitor:

5 alter :

6 nxuiitor:

7 monitor: 8 monitor: 9 monitor:

0 alter :

1 monitor: 2 monitor:

3 alter :

4 monitor:

5 alter :

6 nwnitor:

7 alter :

8 awnitor:

9 alter :

‘0 nanitor:

‘1 monitor: 12 monitor: ‘3 tmnitor: ‘4 monitor: 15 monitor: 16 mnitor:

17 alter :

“Chris does gyrmastics; that is in nly letter.”

“Does Chris do gYllMstics ?”

“Yes. ”

“So Chris does gytmastics.”

“Yes.”

“Oh...”

“We can write that : Chris on “Well, I have written it.” ?y=stics.“V le. filled in the solution-matrix “All right !”

“Let’s search : the partner of Jan.”

“Yes...” “With tian does Jan work together ?” [i.e. cooperate with

“And the partner of Jan is a girl; that is in n(y letter.”

“Oh..., the partner of Jan is a girl.

“Yes: that is correct.”

“Oh...”

“From which city does Jan cuinz frum 7”.

“The city of Jan I don’t know: it is not in my letter.”

“Jan does the same spurts as Jill: I am sure of that.”

“So. Jan does the sarmz sports as Jill.”

“Jan dues the saaie snorts as Jill; you said that.” “Jill does winming; that is in a~v letter.” “Then Jan does swinmirg.” “We can write that: Jan on swiaxning.” “It is written 1” [ie. filled in the solution-amtrix] “What else do you know ?”

“Wait a minute...”

Figure 7. Fragment of a dialogue protocol from a 12-year-old student and the dialogue monitor of a cooperative educational system.

that the student-system dialogue was much slower in time than the verbal speech dialogues between students. In the student-student version the reduced Camp Puzzle was solved in about 25 minutes! In addition, the average score of correctly solved subproblems on the reduced Camp Puzzle (24 subproblems) in a student-student version was 20.5, whereas in the student-system version the aver- age score was only 13.2. The lower score is not caused by an abundance of mis- takes and faulty solutions, but simply by the fact that the students are not able to complete the task in the adjusted time period of 45 min.

Another problem was that the students do not seem to make full use of the inter- face. They tend to stick to the same sort of sentences and do not vary very much, especially in the first half of the session. Probably the time allowed for instruction and exercises with the menu-based interface is too short. Most students seem really at ease with the interface only in the last 10 to 15 min of the session. Only then do they start to experiment with the sentences they construct, use various connectives and illocutions, and seem to be more comfortable with the program as a whole.

Cooperation in problem solving 123

This problem could be solved by a separate instruction session to get familiar with the program and its interface.

There may be another factor involved, which is more fundamental and disturb- ing to our research. Quite a few students seem to be impressed by the program; they seem to comply with the actions of the monitor and take little initiatives them- selves. In short, they do not seem to cooperate with an equal partner, but rather with an authority. Although it is said in the instruction that the program can make mistakes as well, most mistakes made in the low-expertise condition are accepted or believed by the students. On the whole, the number of denials, counters, and arguments by the students is very low.

Another complicating factor may be an artifact of the program itself. The con- straint put on the dialogue for the students is that only one utterance is accepted at a time. Accordingly, the length of each turn for the student is set to one utterance. So, although the student is able to relate an argument to another argument in his next turn, he is not allowed to build an “argumentation” in a sequence of directly connected arguments. We did not observe problems or frustrations of the students concerning this constraint. Still, the constraint causes asymmetry in the dialogues as the monitor bypasses this constraint. The constraint was implemented in the pro- gram out of fear of combinatorial explosion when interpreting multiple utterances at the same time.

As a result, the program and session procedure will have to be updated and improved in several respects. The nature of the task dialogues obtained gives rise to optimism about the possibility to construct “intelligent cooperative systems” in the end.

INTELLIGENT COOPERATIVE SYSTEMS: SOME FINAL REMARKS

In research on intelligent computer assisted learning, the following two main approaches can be distinguished:

1. Tutoring systems. One approach is the development of tutoring systems acting as a teacher who guides the student and controls the learning path of the student. Emphasis lies on the development of domain expert and diagnosis modules, and the effectiveness of such tutoring ultimately depends on instructional and cur- riculum expertise in specific knowledge domains.

2. Open learning environments. The other approach is the development of open learning environments in which the student is able to take over control and determine his/her own learning path. These systems can take the shape of a sim- ulation environment, a laboratory, a microworld, and so forth. The effectiveness of this approach depends on the validity of a specific learning principle, that of (guided) discovery learning. Control over the learning path lies either with the student or with the system. Who is in control is not dependent on the ongoing problem solving process nor on the ongoing interaction process.

Presently, however, a third approach seems to emerge: More and more research aims at a cooperative approach. Examples are mixed initiative systems, systems where the student acts as a teacher to the system, advisory systems, and help sys- tems. We propose to call these systems that are meant to cooperate with the student intelligent cooperative systems (KS; Kanselaar, Andriessen, Barnard, & Erkens,

124 Erkens ord Ardriessen

1990). In this approach neither the student nor the system has complete control of the learning path. They both work together as “intellectual partners” on a learning task (Salomon, 1988).

Which Criteria Should ICS Meet?

We can define a cooperative learning situation as one in which two or more stu- dents work together to fulfil an assigned task within a particular domain of learning in order to achieve a joint product. From this definition the following criteria for ICS can be inferred:

1. Complementary abilities or information. Only when the participants have abili- ties or information that are complementary, cooperation can be fruitful. ICS requires tasks that call for cooperation to reach a successful completion. This could also imply that an ICS does not have complete knowledge about a domain and is not able to solve all the problems encountered.

2. Mixed control. In cooperative learning situations none of the participants is able to determine the process one-sided. The participants are dependent on each other’s cooperation. System and student in an KS should both have the opportu- nity to take control of the exchange and processing of information.

3. Mixed initiative. Both system and student have to be able to take the initiative in interaction. They must be able to take initiative in asking questions, in making remarks, in transferring information, in suggesting solutions, and so forth.

4. Common interest and common goal. In cooperation, system and student must have a common interest in solving the problem at hand. They have to reach common goals and common subgoals that determine the flow of the problem solving process.

Research Implications

In this article, we showed important differences in cooperation in three problem solving situations. These differences concern a complex interaction between task strategies and communication processes. Cooperation requires that the cooperating subjects acquire a common frame of reference in order to be able to negotiate and communicate their individual viewpoints and inferences. A problem for coopera- tion is that the processes of representation formation and communication often take place implicitly. Natural language communication is implicit by nature, viewpoints are not always advanced, task strategies are not always open to discussion, and so forth. While implicitness may be ineffective because it masks differences in knowl- edge, viewpoints, and attitudes, it also results in efficient and nonredundant trans- fer of information. Coordination in information transfer is accomplished by multi- functional dialogue acts (e.g., rereading in the Story Construction Task or informa- tives in the Camp Puzzle). With respect to intelligent cooperative systems, this puts a heavy burden on the interpretative power of the program. Most notably, it should deal with the functions of utterances in the situated context.

REFERENCES

Andricssen, J. E. B. (1991). Mirlirntrl strtrtrgies for colrc~rctrt text production. Utrecht, The Netherlands: ISOR.

Cooperation in problem solving 125

Barnard, Y. F., & Erkens, G. (1989). Simulation and analysis of problem solving and dialogue pro- cessing within cooperative learning. In H. Mandl, E. De Corte, N. Bennett, & H. F. Friedrich (Eds.), Learning and instruction. European research in an international context (pp. 181-196). Oxford, England: Pergamon Press.

Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Erlbaum.

Carberry, M. S. (1985). Pragmatic modelling in irlformation system inteeaces. Unpublished doctoral dissertation, University of Delaware.

Erkens, G. (1992). Modelling and representation of cooperative dialogue systems for Intelligent Educational Systems. Unpublished doctoral dissertation, Utrecht University, The Netherlands.

Erkens, G., Kanselaar, G., & Van der Linden, J. L. (1983). Sociaal gedrag en kognitieffunktioneren [Social behavior and cognitive functioning] (Tech. Rep. SVO-project 0515). Utrecht University, Department of Psychology and Pedagogy, The Netherlands.

Fortescue, M. D. (1980). A discourse production moa’el for “Twenty Questions”. Amsterdam: John Benjamins.

Grice, H. P. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and semantics (pp. 41-58). New York: Academic Press.

Grosz, B. J. (1978). Discourse knowledge. In D. E. Walker (Ed.), Understanding spoken language (pp. 229-344). New York: Elsevier North Holland.

Kanselaar, G., Andriessen, J. E. B., Barnard, Y. F., & Erkens, G. (1990). Some issues on interaction in Intelligent Cooperative Systems. In J. Pieters (Ed.), Intelligent tutorial systems and instruction (pp. 45-65). Enschede, The Netherlands: University of Twente.

Miller, J. R. (1988). The role of human-computer interaction in intelligent tutoring systems. In M. C. Polson & J. J. Richardson (Eds.), Foundations of intelligent tutoring systems (pp. 143-191). Hillsdale, NJ: Erlbaum.

Reichman, R. (1985). Getting computers to talk like ~014 and me. Discourse context, focus und semantics (an ATN model). Cambridge, MA: MIT Press.

Salomon, G. (1988). Technology of the mind: Partner nrld rrortrisher. Paper presented at the Educational Research Conference, Leuven, Belgium.

Searle, J. R. (1969). Speech acts, an essay in the philosophy of Ianguage. London: Cambridge University Press.

Tennant, H. R., Ross, K. M., Saenz, R. M., Thompson, C. W., & Miller, J. R. (1983). Menu-based nat- ural language understanding. In Proceedings of the 21st unm4al meeting of the Association for Computational Linguistics (pp. 151-158). New York: Association of Computing Machinery.

Wenger, E. (1987). Artificial intelligence and tutoring systems: Computational and cognitive approaches to the communication of knowledge. Los Altos, CA: Morgan Kaufmann.

Wickelgren, W. A. (1974). How to solve problems: Elements of a theory of problems and problem solving. San Francisco: Freeman.