The Implications of Cognitive Studies for Teaching Physics Edward F. Redish

Department of Physics, University of Maryland, College Par,k MD 20742-4111

This article first appeared in the American Journal of Physics,62(6), 796-803 (1994) ; ©1994, American Association of Physics Teachers.

I. Introduction

Many of us who have taught introductory physics for many years recall with dismay a number of salient experiences: a reasonably successful student who can produce a graph but can't say what it means; a top student who can solve all the problems but not give an overview or simple derivation; many students of varying abilities who memorize without understanding despite our most carefully crafted and elegant lectures.

As physics teachers who care about physics, we have a tendency to concentrate on the physics content we're teaching. We often are most concerned for those students who are like we were -- that small fraction of our students who find physics interesting and exciting and who will be the next generation of professional physicists. But the changes in our society and in the role of technology for the general public mean that we must change the way we are teaching. It no longer suffices to reproduce ourselves. Society has a great need not only for a few technically trained people, but for a large group of individuals who understand science.

During the past decade, data has built up that demonstrates that as physics teachers we fail to make an impact on the way a majority of our students think about the world.[1,2,3,4,5] We have readjusted our testing so that the students can succeed and we have then either fooled ourselves that we are teaching them successfully or lowered our standards by eliminating understanding from our definition of successful learning. Alan van Heuvelen [6] has remarked that in his study of a typical introductory lecture class, 20% of the students entered the first semester of an introductory calculus-based physics class as Newtonian thinkers. The impact of the course was to increase that number to 25%. If we want to reach a substantial fraction of our students, we must pay much more attention to how students learn and how they respond to our teaching. We must treat the teaching of physics as a scientific problem.

A few physicists have begun to perform detailed experiments to determine what our students are thinking and what works in the teaching of physics. Some of their articles are of tremendous importance and I believe should be read by every physics teacher (see refs. 4-5 and references therein). But even among these few articles, only a small fraction of the authors attempt to place their results in a general theoretical framework -- to give us a way of thinking about and organizing the results.[7] Those of us in physics know well that

advancement in science is a continual dance between the partners of theory and experiment, first one leading, then the other. It is not sufficient to collect data into a "wizard's book" of everything that happens. That's not science. Neither is it science to spout high-blown theories untainted by "reality checks". Science must build a coherent and clear picture of what is happening at the same time as it continually confirms and calibrates that picture against the real world.

The time has come for us to begin the development of a framework for understanding and talking about student learning. Some of the results of the past few decades in cognitive studies8 begin to provide such a framework.Cognitive studies focuses on how people understand and learn. It is still an amorphous field, and it is not yet really a single discipline. It overlaps many areas from anthropology to neurophysiology. It may not yet be "a science" as we in physics use the term, but developments in the past few decades have changed drastically what we know about how the mind works.

The issue of how to teach physics is a difficult one: the attempt of a naive student to build a good understanding of physics involves many intricate processes over a long period of time. These processes tend to be much more complex than those most cognitive scholars have addressed.[9] Nonetheless, some of the basic ideas of cognitive studies appear to be both firmly grounded and useful to the teacher of physics.

In this essay I briefly review some of the lessons I have learned from cognitive studies. This is not a review article, but a narrow selection from a small slice of a large field. For those interested in a more substantial introduction to cognitive studies I recommend Howard Gardner's historical overview [10], the collection of articles assembled by Gentner and Stevens [11], and some of the articles in the reprint volume collected by Collins and Smith.[12] These will give an entry point into the modern cognitive literature. The book by Inhelder and Piaget [13] has lots of discussion of experiments on how adolescents learn physics. Some articles by leading educational specialists also can help link to the existing literature.[14] Just for fun, I have to add Donald Norman's delightful book on how people interact with objects in their world.[15] For an introduction to the physics education research literature, Arnold Arons's book [1] and a few review articles [16] provide an appropriate entry point.

I have grouped what I have learned from the cognitivists into four broad principles with elaborative corollaries. One of the things students of cognitive processes have learned about thinking is that it is fuzzy. The sharp, crisp operations of formal logic or the digital computer are not appropriate models for the way most people think. Therefore, it's not correct to call these principles "theorems" or "laws of cognitive science". Nor is it correct to use them as such. They can't provide us with hard and fast rules for what to do. Using them incautiously without reference to experimental data can lead us to the wrong conclusions. But I have found that they help me to organize my thinking about my students and to refocus my attention. Instead of concentrating only on the organization of the physics content, I now also pay attention to what my students are doing when they interact with a physics course. This is not to suggest that an emphasis on content is somehow unimportant or should be neglected. What we are teaching is important, but it must be viewed in the context of what our students learn.

II. Building Patterns: The Construction Principle

The fundamental change that has led to the breakthroughs in cognitive studies in the past few decades has been a new willingness to model what is happening in the mind in terms of inferred structures. For the first half of this century10, studies of thinking were severely constrained by the "behaviorist" philosophy that one should formulate all one's theories only in terms of direct observables. This is like the S-matrix theory of elementary particles which insisted that we should only formulate our theories in terms of observable particles and their scattering amplitudes. Elementary particle physicists only made their breakthrough when they were willing to formulate their ideas in terms of quarks and gluons -- particles which could only be inferred and not be seen directly. Cognitive scholars started to make real progress when they began to be willing to formulate how people were thinking in terms of mental patterns or models that could not be directly observed or measured.

Principle 1: (Weak form) People tend to form mental patterns.

This is the fundamental hypothesis about how the mind works. On some levels, there is direct observation of this mental processing by patterning. For example, it has been demonstrated in detail that we process visual information on a variety of levels to form patterns beginning with the first layer of nerve cells attached to the retina, and the process continues through many stages deep into the brain.[17] I once attended a physics colloquium given by Jerome Lettvin on the subject of blind spots in the visual field. He passed out the standard blind-spot demonstration pages[18] that let us clearly find the blind spot in our eye. We then moved the end of a pencil into our blind spots and saw the end of the pencil disappear as if bitten off. Yet there was no "blank spot" in the visual field. The brain fills in the background -- here the simple white of a blank page. But it will fill in even a quite complex pattern. If the page had been covered with plaid, my brain would still have filled in my blind spot with the appropriate pattern. But note that the patterning was not sufficiently sophisticated to produce the "right" answer! My automatic filling led to my seeing paper in the blind spot, not the rest of the pencil.

The tendency of the human mind to form patterns is not just limited to the analysis of sensory data. This leads me to state the principle in a stronger (and more relevant) form.

Principle 1:(Strong form) People tend to organize their experiences and observations into patterns or mental models.

I use the term mental model for the collection of mental patterns people build to organize their experiences related to a particular topic. I use the term schema (pl. schemas or schemata) to describe the basic elements of these mental models. I think of a schema as a "chunk" or "object" (in the sense of object-oriented programming). It is a collection of closely linked data together with some processes and rules for use. Be careful of the use of the word "model". It tends to convey something clockwork -- a mechanism that has links

and rules and operates in a well-defined way. These are not the characteristics of many mental models.

The characteristics of mental models and schemas are still vigorously debated.[19] Despite attempts to build a general representational system for mental models, none has yet been widely accepted. However, some results are clear.[20]

Properties of mental models

Mental models have the following properties:

1. They consist of propositions, images, rules of procedure, and statements as to when and how they are to be used.

2. They may contain contradictory elements. 3. They may be incomplete. 4. People may not know how to "run" the procedures present in their mental models. 5. Elements of a mental model don't have firm boundaries. Similar elements may get

confused. 6. Mental models tend to minimize expenditure of mental energy. People will often do

extra physical activities -- sometimes very time-consuming and difficult -- in order to avoid a little bit of serious thinking.

This inferred structuring of mental models is distinctly different from what we usually assume when teaching physics. We usually assume that our students either know something or they don't. The view of mental models we learn from cognitive scholars suggests otherwise. It suggests that students may hold contradictory elements in their minds without being aware that they contradict.

I had an interesting experience that illustrated for me vividly the surprising fact that one's mental model may contain contradictory elements. Ron Thornton visited the University of Maryland a few years ago to give a seminar on his now famous work on using the Sonic Ranger to teach the concept of velocity.[3] The Ranger detects the position of an object using sonar and can display the position or the velocity of the detected object on a computer screen in live time. Thornton set up the Ranger to display velocity and had the computer show a pre-set pattern (a square wave). He then called me up to the front of the room to serve as a guinea pig and try to walk so my velocity matched the pre-set pattern.

I had no hesitation in doing this. I had been teaching physics for nearly twenty years and felt perfectly comfortable with the concept of velocity. I did my first trial without thinking; I walked backward until my velocity reached the height of the pre-set square wave. Then I stopped and my velocity dropped to zero immediately! I asked for another chance, and this time, putting my brain in "velocity mode", I was able to reproduce the curve without difficulty.

What this experience said to me was that, for normal walking, I still maintained a naive (but appropriate!) position-dominated proposition in my mental model of motion. I also had a correct proposition for the concept of velocity, but I had to consciously apply a rule

telling me to use it. I've also had personal experiences illustrating characteristic 6. I once spent an hour searching through my hard drive and piles of floppy disks to find a short paragraph (three sentences!) that I needed again. When I found it, I realized it would have taken me only five minutes to have rewritten it from scratch. A nice example of this characteristic is Donald Norman's study of how people use complex calculators (ref. 19).

An important aspect of Principle 1 is that "people ... organize their experiences into ... mental models" with the emphasis on the fact that people must build their own mental models. This is the cornerstone of the educational philosophy known as constructivism. For this reason, I refer to Principle 1 as "The Construction Principle."

An extreme[21] statement of constructivism is: You can't teach anybody anything. All you can do as a teacher is to make it easier for your students to learn. Of course, facilitation can be critical to the learning process. Constructivism shouldn't be seen as disparaging teaching, but as demanding that we get feedback and evaluations from our students to see what works and what doesn't. It asks us to focus less on what we are teaching, and more on what our students are learning.

Implications

A number of interesting corollaries, elaborations, and implications that are relevant for understanding physics teaching come from Principle 1. The first is the realization that what we want our students to get is not simply "the content" but to build their understanding of that content into an accurate and effective mental model.

Corollary 1.1:The goal of physics teaching is to have students build the proper mental models for doing physics.

This helps us identify the point of teaching physics. I really want my students to do three things:

1. develop the ability to reason qualitatively about physical processes; 2. structure that content into coherent and appropriately organized --and appropriately

accessible -- mental models; 3. learn how to apply that model to "do" physics in an expert and creative way.

These goals suggest that we should broaden our evaluation procedures. We traditionally test only the content and part of the student's skill in doing physics, usually (at least at the introductory level), in pre-set limited contexts.

Some of the characteristics of mental models clarify what is happening when students make mistakes. Often in listening to my students explain what they think I used to become confused and sometimes irritated. How can they say x when they know the contradictory principle y? How come they can't get started on a problem when they certainly know the relevant principle? They just told it to me two minutes ago! How come they brought up that particular principle now? It doesn't apply here! The well-documented characteristics of mental models [22] listed above helps me understand that these sorts of errors are natural

and to be expected.Corollary 1.2: It is not sufficient for students to "know" the relevant correct statements of physics. They also have to be able to gain access to them at the appropriate times; and they have to have methods of cross-checking and evaluating to be certain that the result they have called up is truly relevant.We must also test the underlying mental models that the students are developing. Traditional testing fails to do this, because many schemas can produce the correct solution to a problem. Even if the student goes through the same steps as we do, there's no guarantee that their schema for choosing the steps is the same as ours.[23] I once asked a student (who had done a problem correctly) to explain his solution. He replied: "Well, we've used all of the other formulas at the end of the chapter except this one, and the unknown starts with the same letter as is in that formula, so that must be the one to use."

Part of the way we have fooled ourselves with our current testing is that we are interested "in what students know". If they don't access to the right "information" in an exam, we give them clues and hints in the wording to trigger access. But since an essential component of a mental model are the processes for access to information, we are not testing the complete mental model. The student "has" the information, but it is inert and cannot be used or recalled except in very narrow almost pre-programmed situations.

To find out what our students really know we have to give them the opportunity to explain what they are thinking in words. We must also only give exam credit for reasoning, and not give partial credit when a student tries to hit the target with a blast of shotgun pellets and accidentally has a correct and relevant equation among a mass of irrelevancies. To know whether our students access the information in appropriate circumstances we have to give them more realistic problems -- ones that relate directly to their real world experience and do not provide too many "physics clues" that specify an access path for them.

My next corollary relates to the problem that students often seem to listen to us but not to hear what we think we are trying to say.

Corollary 1.3: The student is not a tabula rasa (blank slate). Each one comes to us having had experiences with the physical world and having organized these experiences into mental models.

As physics teachers, we must realize that students come to us with naive mental models for how the physical world works. These are often referred to in the literature on physics education as preconceptions (or misconceptions).[24] Even experienced teachers can be surprised by this. A few years ago, after reading the ground-breaking articles of Halloun and Hestenes [2] in this journal on students' preconceptions in mechanics, I excitedly related a brief description of the results to one of my colleagues -- someone whom I know to be a concerned teacher and a superb lecturer. He was skeptical at the idea that students had trouble learning Newton's first law because they had pre-existing mental models that were friction dominated. He insisted: "Just don't tell them about friction. They won't know about it if you don't tell them." This natural expectation of this experienced teacher is now strongly contradicted by an impressive body of data.

The presence of "false" preconceptions really isn't so surprising if we think about our students' previous experience. Why should we be surprised that students think that any moving object will eventually come to a stop? In their direct personal experience that is always the case. It's even the case in the demonstrations we show in class to demonstrate the opposite! When I slide a dry-ice levitated puck across the lecture table, I catch it and stop it at the end of the table. If I didn't, it would slide off the table, bounce, roll a short distance, and stop. Every student knows that. Yet I ask them to focus on a small piece of the demonstration -- the stretch of about four or five seconds when the puck is sliding along the table smoothly -- and extend that observation in their minds to infinity. The student and the teacher are focusing on different aspects of the same physical phenomena.[25]

Many teachers show surprise in response to the excellent educational physics studies that have graced the pages of this journal demonstrating that students regularly generalize their naive schemas incorrectly. Why should it be surprising that students think cutting off part of a lens will result in their seeing only part of an image?[26] Try it with a magnifying glass! (Yes, I know that's not a real image.) Where do students get the idea that electricity is something that flows out of the wall and is used up in the object?[27] Why don't they think circuits? Although we don't always think about it, most of our students have had extensive experience with electricity by the time they arrive in our classes. When I said the current had to come in one wire and go out the other, one of my students complained: "If all the electricity goes back into the wall, what are we paying for?"

Corollary 1.4: Mental models must be built. People learn better by doing than by watching something being done.

This is sometimes expressed in the phrase: active learning works better than passive learning.[28] In most cases, this means that reading textbooks and listening to lectures is a poor way of learning.[29] This shouldn't be taken as universally true! As physics teachers, most of us have had the experience of having a few "good" students in our lectures -- students for whom listening to a lecture is an active process -- a dialog between themselves and the teacher. Indeed, many of us have been that good student and remember lectures (at least some of them) as significant parts of our learning experience.[30] A similar statement can be made about texts. I remember with pleasure working through texts and lecture notes, reorganizing the material, filling in steps, and posing questions for myself to answer. Yet few of my students seem to know how to do this or even to know that this is what I expect them to do. This leads us to think about a fifth corollary.

Corollary 1.5: Many of our students do not have appropriate mental models for what it means to learn physics.

This is a "meta" issue. People build mental models not only for content, but also for how to learn and what actions are appropriate under what circumstances. Most of our students don't know what you and I mean by "doing" science or what we expect them to do. Unfortunately, the most common mental model for learning science in my classes seems to be:

• Write down every equation or law the teacher puts on the board that is also in the book. Memorize these, together with the list of formulas at the end of each chapter.

• Do enough homework and end-of-the-chapter problems to recognize which formula is to be applied to which problem.

• Pass the exam by selecting the correct formulas for the problems on the exam. • Erase all information from your brain after the exam to make room for the next set

of material.

I used to be flabbergasted to discover that when I stopped a lecture and said: "OK, here's a really important general principle for how to do things. It's not in the book but it comes from my years of experience as a physicist," my students would not write it down or consider it important, even if I wrote it on the board! (Well, after all, it wasn't going to be on the exam.)

I call the bulleted list above "the dead leaves model". It's as if physics were a collection of equations on fallen leaves. One might hold s= ½ gt**2, another F = ma, and a third F = -kx. These are each considered as of equivalent weight, importance, and structure. The only thing one needs to do when solving a problem is to flip through one's collection of leaves until one finds the appropriate equation. I would much prefer to have my students see physics as a living tree!

I like the term mental ecology to describe the mental model that tells students what mental model to apply in what set of circumstances. It is a more important goal to reshape our students' mental ecologies so that they expand their idea of learning to make it more constructive, take it out of the classroom into their everyday lives, and understand what science is and how to apply it, than it is to teach them to parrot back equations or solutions to turn-the-crank problems. One final observation on the first principle is the following:

Constructing our own lectures and teaching materials can prove very useful in producing learning -- in the teacher!

Haven't we all remarked: I only really understood E&M (or classical mechanics, or thermodynamics, or whatever) when I finally taught it? This is really dangerous! For those of us who love learning, the experience of lecturing and teaching is such a powerful learning experience that we don't want to give it up, even when it proves less effective for our students than other methods.

III. Building on a Mental Model: The Assimilation Principle

The second and third principles have to do with the dynamics of modifying and extending one's mental models.

Principle 2: It is reasonably easy to learn something that matches or extends an existing mental model.

This principle states that mental models are not only the way that we organize our interactions with the world, but they also control how we incorporate new information and experiences.31 (The question of how they are first established in young children is interesting -- and controversial -- but it doesn't really concern us here.) I use the term "assimilate" to emphasize adding something smoothly to an existing set.32I pose three restatements and elaborations of this principle as corollaries to show what it means for teaching.

Corollary 2.1: It's hard to learn something we don't almost already know.

All students have things they know (some of which may be wrong!), things they are a bit familiar with, and things they have no knowledge about at all. In the last area my daughter would say they're "clueless".

I like to look at this as an archery target. What they know is the bull's-eye -- a compact black area; what they know a little about is a gray area surrounding the black; and the clueless region is a white "rest of the world". To teach them something, I do best to hit in the gray. A class full of students is a challenge because all of their gray areas are not the same. I want to hit as many of the grays as possible with each paint-tipped shaft of information to turn gray areas black. In communication studies, an important implication of this corollary is called the "given-new principle".33 It states that new information should always be presented in a context that is familiar to the reader and that the context should be established first. The analogous statement is very important in physics teaching, especially at the introductory level. As physicists with years of training and experience we have a great deal of "context" that our students don't possess. Often we are as fish in water, unaware of this context and that it is missing in our students.

There are a number of specifics that we can cite that are given-new problems. We often use terms that students are not familiar with -- or use in a different sense than we do. As a part of their study in the way speakers of English build their meaning of the term "force", Lakoff and Johnson [34] classified the characteristics of common metaphors using the term. Among their list of 11 characteristics, 8 involved the will or intent of an individual! But most of us are so familiar with the technical meaning of "force" that we surprised to learn that a significant fraction of our introductory students do not believe that a table exerts a force on a book it is supporting.[2] Why doesn't the book fall through? The table is just "in the way".

The problem caused by the interpretation of common speech words for technical ones is not a simple one. I know that the terms "heat", and "temperature" are not really distinguished in common speech and are used interchangeably for the technical terms "temperature" (average energy per degree of freedom), "internal energy", and "heat" (flow of internal energy from one object to another). In one class, I stated this problem up front and warned my students that I would use the terms technically in the lecture. Part way through I stopped, realizing that I had used the word "heat" twice in a sentence -- once in the technical sense, once in the common speech sense.[35] It's like using the same symbol to stand for two different meanings in a single equation. You can occasionally get away with it,[36] but it isn't really a good idea!Putting new material in context is only part of the story.

Our students also have to see the new material as having a plausible structure in terms of structures they are familiar with. We can state this as another useful corollary.

Corollary 2.2: Much of our learning is done by analogy.

This strongly counters the image of the student as a tabula rasa. This and the previous corollary make what students know at each stage critical for what we can teach them. Students always construct their knowledge, but what they construct depends on how what we give them interacts with what they already have.

One implication of these results is that we should focus on building structures that are useful for our students' future learning. I state this as a third corollary.

Corollary 2.3: "Touchstone" problems and examples are very important.

By a touchstone problem, I mean one that the student will come back to over and over again in later training. Touchstone problems become the analogs on which they will build the more sophisticated elements of their mental models.

It becomes extremely important for students to develop a collection of a few critical things that they really understand well.[37] These become the "queen bees" for new swarms of understanding to be built around. I believe this is why some problems have immense persistence in the community. Inclined plane problems really aren't very interesting, yet the occasional suggestions that they be done away with are always resisted vigorously. I think the resisters are expressing the (usually unarticulated) feeling that these are the critical touchstone problems for building students' understanding of vector analysis in the plane. Corollary 2.3 is one reason why we spend so much time studying the mass on a spring. It's not really of much interest in itself, but it serves as a touchstone problem for all kinds of harmonic oscillation from electrical circuits up to quantum field theory.

Looking at a curriculum from the point of view of the mental models we want students to develop, their pre-existing mental models, and touchstone problems can help us analyze what is critical in the curriculum, which proposed modifications could be severely detrimental, and which might be of great benefit.

Combining this with the discussion of access and linking above leads us to focus on the presence of a framework or structure within a course. It suggests that building a course around a linked series of touchstone problems could be of considerable assistance in helping students understand the importance and relevance of each element. Such a structure is sometimes referred to as a story line.

IV. Changing an Existing Mental Model: The Accommodation Principle

Unfortunately, if students are not blank slates, sometimes what is written is -- if not wrong -- inappropriate for future learning in physics. Then it can seem as if we have run into a brick wall. This brings us to the next principle. I call this the "accommodation principle" to

emphasize that changes have to be made in an existing structure. (Again, the term goes back to Piaget.)

Principle 3: It is very difficult to change an established mental model substantially.

Traditionally, we've relied on an oversimplified view of Principle 1, the patterning principle, to say: "Just let students do enough problems and they'll get the idea eventually." Unfortunately, the principle doesn't apply in this form if they already have a mental model about the subject. It has been demonstrated over and over again that simply telling somebody something doesn't easily change their deep ideas. Rather, what happens is that a poorly linked element is added with a rule for using it only in physics problems or for tests in one particular class. This and the fact that a mental model can contain contradictory elements is the reason why "giving more problems" can be ineffective. Once students learn how to do problems of a particular type, many will learn nothing more from doing more of them: new problems are done automatically without thinking. This also means that testing by varying homework problems slightly may be inadequate to probe the student's mental models of physics. More challenging tests involving a variety of modalities (problem solving, writing, interpreting, organizing) are required.

A few years ago I learned a lovely anecdote illustrating the barriers one encounters in trying to change a well-established mental model. A college physics teacher asked a class of beginning students whether heavy objects fall faster than light ones or whether they fall at the same rate. One student waved her hand saying "I know, I know". When called on to explain she said: "Heavy objects fall faster than light ones. We know this because Galileo dropped a penny and a feather from the top of the leaning tower of Pisa and the penny hit first." This is a touchstone example for me. It shows clearly that the student had been told -- and had listened to -- both the Galileo and the Newton stories. But she had transformed them both to agree with her existing mental model.[38]

Principle 3 can cause problems, both in getting students to change their mental models, and in getting ourselves to change the preconceptions we have about how students think! Fortunately, "difficult" does not mean "impossible". We have mechanisms that permit us to measure our mental models against the world and change them when we are wrong.[39] It appears as if the mechanism critically involves prediction and observation. The prediction must be made by the individual and the observation must be a clear and compelling contradiction to the existing mental model. A simple contradiction isn't sufficient. Posner et al.40 suggest that changing an existing mental model requires that the change have the following characteristics (which I state as a corollary).

Corollary 3.1: In order to change an existing mental model the proposed replacement must have the following characteristics:

• It must be understandable. • It must be plausible. • There must be a strong conflict with predictions based on the existing model. • The new model must be seen as useful.

The clearer the prediction and the stronger the conflict, the better the effect. A nice example of how this process works concerns physics teachers rather than their students. It explains why the response to the Halloun-Hestenes test has been so great. Many teachers of introductory physics have a mental model that says the teacher is successful in teaching the concepts if the students can average 75% on a traditional exam. These teachers look at the H-H test and predict: "My students will be able to do very well on those problems. They're easy." Then their predictions fail miserably. On some critical questions, students average 20% or less and the conflict with the teacher's existing mental model is very strong. Many of the teachers who have gone through this process appear to be converted to a new way of looking at their students and what they know.[41]

Attempts are being made to combine the assimilation and the accommodation principles to yield new, more effective methods of teaching. John Clement[42] has proposed finding a series of bridges or interpolating steps that would help a student transform his or her mental model to match the accepted scientific one.

V. The Individuality Principle

One might be tempted to say: Fine. Let's figure out what the students know and provide them with a learning environment -- lectures, demonstrations, labs, and problems -- that takes them from where they are to where we want them to be. Since we all know that a few students get there from here using our current procedures, how come it doesn't work for all of them? We do in fact now know that the right environment can produce substantially better physics learning in most of the students taking introductory university physics.[43] But my final principle is a word of warning that suggests we should not be looking for a "magic bullet".

Principle 4: Since each individual constructs his or her own mental ecology, different students have different mental models for physical phenomena and different mental models for learning.

I like to call this the individuality or "line width" principle. This reminds us that many variables in human behavior have a large natural line width. The large standard deviation obtained in many educational experiments is not experimental error, it is part of the measured result! As scientists, we should be accustomed to such data. We just aren't used to its being so broad and having so many variables. An "average" approach will miss everyone because no student is average in all ways.[44]

Implications

One implication of this is that different students can have different reasons for giving the same answer. If we formulate our questions too narrowly we may misinterpret the feedback we are getting. This observation has influenced the style of educational physics research in a way that at first seems strange to a physical scientist.

When we try to take our "first look" at what students are doing, it is very important to consider them one at a time and to interview them in depth, giving them substantial opportunity for "thinking aloud" and not giving them any guidance at all.

This approach is characteristic of many important educational studies. Instead of hoping to "average out" the variation by doing large statistical experiments, one focuses on it and tries to learn the range of possible approaches that students are taking. Of course, at a later stage, one wants to be able to interrogate large numbers of students in order to obtain the frequency with which various modes of thinking are occurring in the population at large. But many valuable studies in educational physics (and in cognitive studies in general) are done with a sample that seems very small to a physicist.

An excellent example is the work of McDermott and Goldberg (ref. 25). They start with extensive interviews of a fairly small number of students, then develop short answer exams based on those observations that can be applied to large groups, and finally test that those exams are giving the same results as the interviews. The various Hestenes tests were developed in a similar fashion.[45] In addition to the fact that students have different experiences and have drawn different conclusions from them, their methods of approach may differ significantly. I state this as a corollary.

Corollary 4.1: People have different styles of learning.

There is by now a vast literature on how people approach learning differently. Many variables have been identified on which distributions have been measured. These include authoritarian/independent, abstract/concrete, and algebraic/geometric to name a few.[46] The first means that some students want to be told, others to figure things out for themselves. The second means that some students like to work from the general to the specific, some the other way round. The third means that some students prefer to manipulate algebraic expressions while others prefer to see pictures. Many of us who have introduced the computer in physics teaching have noted that some students want to be guided step by step, others try everything. These are only a few of the variables.

Once we begin to observe and use these differences in our students, we have to be exceedingly careful about how we use them. A preference does not mean a total lack of capability. Students who prefer examples with concrete numbers to abstract mathematical expressions may be responding to a lack of familiarity with algebra rather than a lack of innate ability. Many of our students preferences come from years of being rewarded for some activities (such as being good memorizers) and chastised for others (such as asking questions the teacher couldn't answer). Expanding our students' horizons and teaching them how to think sometimes requires us to overcome years of negative training and what they themselves have come to believe are their own preferences and limitations!

An interesting observation that has been made by a number of observers,[47] is that physics as a discipline requires learners to employ a variety of modalities (methods of understanding) and to translate from one to the other -- words, tables of numbers, graphs, equations, diagrams, maps. Physics requires the ability to use algebra and geometry and to go from the specific to the general and back. This makes learning physics particularly

difficult for many students. One of our goals should be to have our students understand this, be able to identify their own strengths and weaknesses, and while building on the former, strengthen the latter.

An important implication is the following:

Corollary 4.2: There is no unique answer to the question: What is the best way to teach a particular subject?

Different students will respond positively to different approaches. If we want to adopt the view that we want to teach all our students (or at least as many as possible), then we must use a mix of approaches and be prepared that some of them will not work for some students. An important set of studies that are just beginning to be done ask the question: What is the distribution function of learning characteristics that our students have in particular classes?

Another implication that is very difficult to keep in mind is:

Corollary 4.3: Our own personal experiences may be a very poor guide for telling us what to do for our students.

Physics teachers are an atypical group. We selected ourselves at an early stage in our careers because we liked physics for one reason or another. This already selects a fairly small subclass of learning styles from the overall panoply of possibilities. We are then trained for approximately a dozen years before we start teaching our own classes. This training stretches us even further from the style of approach of the "typical" student. Is it any wonder why we don't understand most of our beginning students and they don't understand us? I will never forget one day a few years ago when a student in my algebra-based introductory physics class came in to ask about some motion problems. I said: "All right, let's get down to absolute basics. Let's draw a graph." The student's face fell, and I realized suddenly that a graph was not going to help him at all. I also realized that it was going to be hard for me to think without a graph and to understand what was going through the student's mind. I never minded doing without derivatives -- motion after all is the study of experimental calculus and you have to explain the concept (maybe without using the word) even in a non-calculus based class; but I have found it difficult to empathize with students who come to physics and can't read a graph or reason proportionately.[5] It takes a special effort for me to figure out the right approach.

This is very natural given the earlier principles. Our own personal mental models for how to learn come from our own experiences. However, to reach more of our students than the ones who resemble ourselves, we will have to do our best to get beyond this. It makes the following principle essential.

Corollary 4.4: The information about the state of our students knowledge is contained within them. If we want to know what they know, we not only have to ask them, we have to listen to them!

VI. Conclusion

The typical university course is a complex structure. It involves physics content, a teacher, perhaps graders or teaching assistants, a classroom, a laboratory, and, for each class, a particular set of students. Above all, it involves expectations and contexts for both the teacher and the students. If we are to make serious progress in reaching a larger fraction of our students, we will have to shift our emphasis from the physics content we enjoy and love so well to the students themselves and their learning. We must ask not only what do we want them to learn, but what do they know when they come in and how do they interact with and respond to the learning environment and content we provide.

The principles we are learning from cognitive studies can provide a framework for how we think about the complex issues of teaching and learning. The four principles that I have presented can help us begin to construct such a framework.

Acknowledgments

This article arises from the valuable discussions I have had with many friends and colleagues, especially Arnold Arons, Seth Chaiklin, Dewey Dykstra, John Guthrie, David Hestenes, Priscilla Laws, Lillian McDermott, Ed Taylor, Ron Thornton, Jack Wilson, and Dean Zollman. I would like to thank John Clement, Priscilla Laws, and Len Jossem for commenting on the manuscript. The suggestions of two very conscientious AJP reviewers were helpful in guiding me towards revisions that improved the paper significantly. Finally, I would like to thank my wife, Dr. Janice C. Redish, for numerous discussions on the subject of cognitive studies. Her expertise has played a major role in the development of my views on these issues. I would also like to thank her for her detailed editorial comments on the presentation. She is not to be held responsible for my persistent grammatical idiosyncrasies.

Endnotes

[1] Arnold Arons, A Guide to Introductory Physics Teaching (Wiley, 1990).

[2] David E. Trowbridge and Lillian C. McDermott, "Investigation of student understanding of the concept of velocity in one dimension", Am. J. Phys., 48 (1980) 1020-1028; "Investigation of student understanding of the concept of acceleration in one dimension", Am. J. Phys. 49 (1981) 242-253.

[3] A. Halloun and D. Hestenes, "The initial knowledge state of college physics students", Am. J. Phys., 53, 1043-1055 (1985); "Modeling instruction in mechanics", Am. J. Phys. 55, 455-462 (1987).

[4] R. K. Thornton and D. R. Sokoloff, "Learning motion concepts using real-time microcomputer- based laboratory tools", Am. J. Phys. 58 (1990) 858-867.

[5] Lillian C. McDermott, "Millikan Lecture 1990: What we teach and what is learned -- Closing the gap", Am. J. Phys. 59, 301-315 (1991); "Guest Comment: How we teach and how students learn-- A mismatch?" Am. J. Phys. 61, 295-298 (1993); and refs. therein.

[6] Alan van Heuvelen, Bulletin of the American Physical Society, 36, 1359 (April, 1991).

[7] One that has been very useful for me is the article by David Hestenes, "Toward a modeling theory of physics instruction", Am. J. Phys. 55, 440-454 (1987).

[8] I use the term "cognitive studies" as opposed to the term "cognitive science", which is more prevalent in the literature, advisedly. The field is extremely broad and cuts across many disciplines. Very little in cognitive science satisfies the scientific criteria we are used to in physics of being precisely stated and well-tested experimentally, as well as useful. So as not to raise the expectations of my scientist readers as to the nature of the information précised here, I use the weaker term (and use it as a singular noun like "physics").

[9] There are a few notable exceptions. Piaget has made extensive studies of how children build their concepts of the physical world, and a number of trained physicists such as Fred Reif, John Clement, and Jose Mestre, among others, could be counted as cognitive scholars.

[10] Howard Gardner, The Mind's New Science: A History of the Cognitive Revolution (Basic Books, 1987).

[11] Derdre Gen�tner and Albert L. Stevens, Eds. Mental Models (Lawrence Erlbaum Associates, 1983).

[12] Allan Collins and Edward E. Smith, Readings in Cognitive Science (Morgan Kauffmann, 1988).

[13] Bärbel Inhelder and Jean Piaget, The Growth of Logical Thinking from Childhood to Adolescence (Basic Books, 1958).

[14] L. B. Resnick, "Mathematics and science learning: A new conception", Science 220, 447-478 (1983); S. Carey, "Cognitive science and science education", Am. Psych. 41, 1123-1130 (1986); R, Driver, "Students' conceptions of the learning of science", Int. J. of Sci. Ed. 11, 481-490 (1989).

[15] Donald Norman, The Design of Everyday Things (Basic Books, 1990). (This book was originally published under the title: The Psychology of Everyday Things.)

[16] Lillian McDermott, "Research on conceptual understanding in mechanics", Phys. Today 37(7), 24-32 (1984); Jose Mestre and Jerrold Touger, "Cognitive research -- what's in it for physics teachers?", The Physics Teacher 27, 447-456 (September, 1989); Robert Fuller, "Solving physics problems -- how do we do it?", Physics Today 35:9, 43-47 (1982) .

[17] David H. Hubel, "The Visual Cortex of the Brain", Scientific American 225:11, 54 (1963); "The Brain", Scientific American 241:3, 44(1979).

[18] The page is blank with two� marks: a black dot and a cross about six inches apart. One closes one eye and focuses on the black dot with the other. The page is oriented so the cross is off to one side. By moving one's head back and forth, one finds a distance at which the cross appears to vanish.

[19] One must take particular care with these terms in the cognitive literature as they are used by different authors in different senses.

[20] This list of properties is based in part on Donald Norman, "Some observations on mental models", in Gentner and Stevens, op. cit., pp. 7-14 and refs. therein.

[21] Actually, a very wide variety of educators and cognitive sp�ecialists consider themselves constructivists. The statement here is not really extreme in the current spectrum of views. Radical constructivists reject the idea that knowledge has any base outside of construction in the human mind, a view which leads to results that could be classified as solipsism.

[22] Andy diSessa has documented and attempted to classify some of the levels of principles that people create and their modes of access to them. Although his papers are rather technical, he uses many examples from physics and includes references to the large literature on the subject. See for example, "Phenomenology and the evolution of intuition", in Genter and Stevens (Eds.), op. cit., 15-33; or "Knowledge in Pieces", Cognition and Instruction 10, issues 2 and 3.

[23] The difficulty is that the mapping from underlying schema to problem solving steps is not one- to-one. As nice example of this is given in a recent article in this journal: J. Bowden, et al., "Displacement, velocity, and frames of reference: phenomenographic studies of students' understanding and some implications for teaching and assessment", Am. J. of Phys. 60, 262-269 (1992) .

[24] I prefer the term preconceptions or naive schemas, with "naive" used in the non-pejorative sense of "untutored". The typical student's naive schema about motion and mechanics, for example, is not wrong. It correctly correlates their observations at low velocities in friction- dominated systems. It just does not provide them with the power and range of the schema we are trying to substitute for it. See refs. 1-5 and the references referred to therein for details and documentation.

[25] This argument is made in a slightly different context in T. S. Kuhn, The Structure of Scientific Revolutions, 2nd ed. (Chicago University Press, 1970).

[26] F. C. Goldberg and L. C. McDermott, "An investigation of student understanding of the real image formed by a converging lens or concave mirror", Am. J. Phys. 55, 108-119

(1987); "An investigation of student understanding of the images formed by plane mirrors", The Physics Teacher 34, 472-480 (1986).

[27] R. Cohen, B. Eylon, and U. Ganiel, "Potential difference and current in simple electric circuits: A study of students' concepts", Am. J. Phys. 51, 407-412(1983).

[28] We have to b�e careful what we mean by "active learning". This shouldn't necessarily be taken to mean that "solving more problems is the answer". See the discussion after principle 3.

[29] J. S. Brown, A. Collins, P. Duguid, "Situated Cognition and the Culture of Lea�rning", Educational Research, 32-42 (Jan-Feb, 1989), and refs. therein.

[30] In many research groups, a seminar more resembles a panel discussion than a lecture. These are very active learning experiences, both for the speaker and the listener.

[31] J. D. Bransford and M. K. Johnson, "Contextutal prerequisites for understanding: Some investigations of comprehension and recall", J. of Verbal Learning and Verbal Behavior 11, 717-726 (1972); "Considerations of some problems of comprehension", in W. Chase (Ed.), Visual Information Processing (Academic Press, NY, 1973).

[32] The term "assimilation" and the matching term "accommodation" used in the next section were introduced by J. Piaget.

[33] H. Clark and S. Haviland, "Comprehension and the given-new contract", in Discourse Production and Comprehension, R. Freedle, ed. (Lawrence Erlbaum, 1975).

[34] G. Lakoff and M. Johnson, Metaphors We Live By (U. of Chicago Press, Chicago, 1980), p. 70.

[35] "If there is no heat flow permitted to the object we can still heat it up by doing work on it."

[36] The energy levels of hydrogen in a magnetic field could be written with an m in the denominator for the electron mass and one in the numerator for the z-component of the angular momentum. Most physicists can correctly interpret this abomination without difficulty.

[37] In addition to giving them centers on which to build future learning, kn�owing a few things well gives the student a model of what it means to understand something in physics. This valuable point that has been frequently stressed by Arnold Arons. It is an essential element in the mental ecology of a scientist.

[38] Of course the students' mental model in this case is in fact correct. Lighter objects do fall more slowly than heavy ones if they fall in air, and few of us have much direct experience with objects falling in a vaccuum. But that observation does not yield a useful

idealization. The observation that objects of very different mass fall in very nearly the same way does.

[39] Unfortunately, these mechanisms appear to atrophy if unused.

[40] G. J. Posner, K. A. Strike, P. W. Hewson, and W. A. Gertzog, "Accommodation of a scientific conception: toward a theory of conceptual change", Science Education 66:2, 211-227 (1982).

[41] A nice description o�f this process is given in Eric Mazur's brief but insightful article on teaching, "Qualitative vs. quantitative thinking: Are we teaching the right thing?", Optics & Phonics News 38 (February, 1992).

[42] John Clement, "Overcoming Students' Misconception�s in Physics: The Role of Anchoring Intuitions and Analogical Validity", Proc. of Second Int. Seminar: Misconceptions and Educ. Strategies in Sci. and Math. III, J. Novak (ed.) (Cornell U., Ithica, NY, 1987); "Not all preconceptions are misconceptions: Finding 'anchoring conceptions for grounding instruction on students' intuitions", Int. J. Sci. Ed. 11 (special issue), 554-565 (1989).

[43] Priscilla Laws, "Calculus-based physics without lectures", Phys. Today 44:12, 24- 31(December, 1991); Alan Van Heuvelen, "Overview, Case Study Physics", Am. J. Phys. 59, 898-907(1991); Richard Hake, "Socratic Pedagogy in the Introductory Physics Laboratory",The Physics Teacher, 33 (1992); L. C. McDermott and P. S. Shaffer, "Research as a guide for curriculum development: An example from introductory electricity. Part I: Investigation of student understanding", Am. J. Phys. 60, 994-1003 (1992); erratum, ibid. 61, 81 (1993); P. Shaffer, and L. C. McDermott, "Research as a guide for curriculum development: An example from introductory electricity. Part II: Design of an instructional strategy", Am. J. Phys. 60, 1003-1013 (1992).

[44] This is analogous to the story of the three statisticians who went hunting deer with bows and arrows. They cam across a large stag and the first statistician shot -- and missed to the left. The second statistician shot and missed to the right. The third statistician jumped up and down shouting "We got him!"

[45] Halloun and Hestenes, Op. Cit.; David Hestenes, M. Wells, and G. Swackhammer, "Force Concept Inventory", The Physics Teacher, 30:3, 141-158(1992); D. Hestenes and M. Wells, "A Mechanics Baseline Test", The Physics Teacher, 30:3, 159-166(1992).

[46] David A. Kolb, Experiential learning: experience as a source of learning and development (Prentice Hall, 1984); Noel Entwistle, Styles of integrated learning and teaching: an integrated outline of educational psychology for students, teachers, and lecturers (Wiley, 1981); Howard Gardner, Frames of Mind (Basic Books, 1983).

[47] Priscilla Laws, David Hestenes, private communication.

©1994, American Association of Physics Teachers.

Maintained by University of Maryland PERG Comments and questions may be directed to E. F. Redish

Last modified 5 February 2002

The Development of Open Models for Teaching Physics to Schools in Dispersed Locations in Russia and Canada

Aleksandr N. Sandalov Natalia A.Sukhareva

Maurice Barry Terry Piper Ken Stevens

Prepared for the EDEN - Lomonosov Conference Moscow State University

Moscow, Russia 3-5 October, 1999

Abstract

Russia and Canada are the two largest countries in the world in terms of geography and in each society, new information and communication technologies are being used to enhance the education of students in dispersed locations. This paper outlines two models for teaching physics from universities to students located in communities that are remote from major centres of population. The first model is being developed in Eastern Canada in which physics students are educated within a digital intranet, the sites of which are all located in rural communities in the province of Newfoundland and Labrador. The second model is Russian in which open distributed learning has been developed between Moscow State University and schools in both Russia and countries of the former Soviet Union. The paper concludes with an examination of the two models and a look at their future development in each country.

Authors:

Aleksandr N. Sandalov is an Associate Professor and Director of the Information Systems and Technology Centre of the Department of Physics, Moscow State University, Russia Email: sandalov@phys.msu.su

Natalia A.Sukhareva is an Associate Professor and Vice-Dean of the Department of Physics at Moscow State University, Russia Email: suhareva@phys.msu.su

Maurice Barry is a TeleTeacher of Physics in the TeleLearning and Rural Education Centre, Faculty of Education, Memorial University of Newfoundland, Canada Email: mbarry@stemnet.nf.ca

Terry Piper is Dean of the Faculty of Education at Memorial University of Newfoundland (until October 1999), then Dean of Education at York University, Toronto, Canada Email: tpiper@mun.ca

Ken Stevens is Professor of Education and Chair of the TeleLearning and Rural Education Centre, Faculty of Education, Memorial University of Newfoundland, Canada Email: stevensk@mun.ca

The Development of Open Models for Teaching Physics to Schools in Dispersed Locations in Russia and Canada

Open classes for students in dispersed locations have become a feature of education in several parts of the developed world this decade. In Iceland (Stefansdottir, 1993), New Zealand (Stevens, 1995a, 1995b) and Finland (Tella, 1995) open classes provide access to an increasing range of learning opportunities for students who live in small and often remote communities. In all parts of the developed world in which information and communication technologies are being applied to the development of new, electronic educational structures, there is a search for appropriate ways of organizing teaching and learning (Goodyear,1999; Kynaslahti and Wager, 1999).

Russia and Canada are the largest and second-largest countries in the world respectively. Both Canada (Industry Canada, 1995;1997) and Russia increasingly depend on advancing, applying and integrating science and technology for the development of their knowledge-based economies and both societies contain significant regions that are remote from major centres of population. Russian and Canadian students who live beyond major urban areas increasingly depend on the application of information and communication technologies (ICT) for the delivery of educational opportunities.

Physics is a key domain of science, engineering and technology. Without access to instruction in Physics, students face reduced access to jobs in many scientific and technological fields. In the past it has been particularly difficult to attract, retain or even justify the appointment of specialist teachers of science in schools in rural communities in which student numbers have been small and access to scientific equipment has been difficult. Accordingly, many rural students have not, in the past, had equal access to scientific and technological occupations unless they completed their education in schools in larger centres.

Advances in ICT and in the electronic networking of schools have improved access to science in remote parts of Russia and Canada. This paper outlines the development of open models for teaching and learning physics in both countries, based on the integration of new information and communication technologies to pedagogy.

One: The Canadian Model - The Teaching of Advanced Placement Physics

In both the United States and Canada Advanced Placement courses are a well-established part of the senior High School curriculum for students who are able to undertake first-year University study. Depending on the level of pass obtained in the final examination, students can, through undertaking AP courses, gain credit towards their degrees before enrolling at a University.

In the Canadian province of Newfoundland and Labrador most schools (66%) are classified as rural. In district No. 8, the Vista School District, in which most schools are in small, rural communities, an Intranet was developed to electronically link eight sites. An integral part of the Vista Digital Intranet was the Centre for TeleLearning and Rural Education, located in the Faculty of Education of Memorial University of Newfoundland in the provincial capital, St Johns. The appointment of a teleteacher in the Centre for TeleLearning and Rural Education to teach Advanced Placement (AP) Physics students in three sites (schools) in the Vista Intranet provided both technological and pedagogical challenges.

The development of an AP Physics course for delivery through the Internet was undertaken over a sixteen-week period during the spring and summer of 1998. The teacher of Physics was assisted by a recent graduate in this discipline who had advanced computing skills in web page design, Java and HTML. Most of the AP Physics course development was through interaction between the lead teacher and the recent graduate, with reference, as appropriate, to professors in Science Education and Physics. This model facilitated interaction between the school system, graduate students and the Faculties of Education and Science of Memorial University of Newfoundland.

The introduction of Internet-delivered AP Physics in the Vista School District Digital Intranet represented two dimensions of change: (a) curriculum and technology and (b) the re-organization of classes within an Intranet.

(a) Curriculum and Technology

Several issues had to be considered in the development of the AP course:

1. What hardware and software resources would be required to deliver instruction of an AP course to high school students in rural schools?

Minimum specifications were adopted for computer hardware and network connectivity. All schools involved in the project had satellite dishes installed. Software had to be identified and evaluated. Software was required for both the development of the resources, and the delivery of instruction.

Software for development of the AP resources:

Front Page 98 was selected as the software package. Additional software was used for development of images, animated gifs, etc. These included Snagit32, Gif Construction Set, Real Video, and similar packages.

Software for student management and delivery of instruction:

Many software packages were evaluated and finallyWebCT was selected. This package enabled the instructor to track student progress, it contained online testing and evaluation, private email, a calendar feature, public bulletin board for use by both instructor and

student, a link to lessons, chat rooms for communication between teacher and student, and more.

For real time instruction Meeting Point and Microsoft NetMeeting were selected. This combination of software enabled a teacher to present real-time interactive instruction to multiple sites.

2. Students require training in the use of the hardware and software.

An orientation session was provided to students in June of the year prior to implementation. Visits by instructors to students were carried out where required. Students has to learn how to communicate with each other and with their instructor using the Internet.

(b) The Re-organization of Classes In an Intranet

1. Problems associated with the delivery of the lab component for the course.

Problems of supervision and safety have had to be considered. This remains an issue that has not been finally resolved. The use of online video files and CD-ROMs are being explored.

2. The nature of the high school students taking the AP course.

The question how students would work in a completely independent learning environment has been prominent in the minds of researchers from the outset of each AP course. There is no teacher present on site and students are not normally used to being alone and largely unsupervised.

3. Synchronous and Asynchronous Teaching and Learning- Establishing a common schedule (time-table) in all schools within the district.

It was recognized early that a common schedule must be adopted throughout the district to allow students to be able to interact with their instructors. The initial plan was to allow for 5 online sessions and 5 offline sessions. This schedule was not followed in all schools. Online sessions are scheduled in the morning when network traffic is at its lowest point. Off line sessions are scheduled in the afternoon. Some schools chose not to be coordinated with the AP schedule for various valid reasons. As a result the instructor might be teaching another subject when the AP student would be in class. Students in different schools therefore have differing access to their AP Instructors. It is anticipated that in future there will be both asynchronous as well as synchronous teaching and learning within the Intranet, but at the present time it is not possible to provide other than synchronous access.

4. The problem of student evaluation.

The delivery of evaluation instruments is still being considered, but the researchers' preference is for electronic delivery. Both traditional paper/pencil and online testing are being evaluated. Student preference at present is for paper/pencil type tests.

Two: The Russian Model

In the Russian education system OD courses are academic resources that are open to public discussion. On the one hand, academic processes may appear to be homogenous, with high standards of content. But, on the other hand, the principally asynchronous regime of educational communications in Russia and NIS make this type of delivery very vulnerable to subjective factors. It is possible to provide support for all "standard procedure" learning processes today through networked education. Networked education includes lectures with audio- video conference support, student classes with interactive tests, laboratory classes with on- line experimental facilities and electronic libraries with context-search systems and advisory board network services.

There is an emerging view that it is possible to mirror elementary and higher education through networked teaching and learning, based, in large part, on changing concepts of "information support." Information support includes not simply shaping the databases of academic courses and multimedia network developments, but through interactive on-line discussions and conferences with global research experimental facilities. Networked teaching and learning provides new and enhanced dimensions of student-teacher interaction and provides a basis for the development of virtual classes.

The changing basis of student - teacher interaction that is encouraged by the creation of new information support systems provides new ways of organizing learning:

1. From personal computers in student's homes; 2. From training auditoriums; 3. From teachers at schools and teachers in universities.

New developments in teacher-student interaction have to take into consideration changes in the direction of global telecommunications systems.

1. The increasing a number of users with mainly low velocities of access (modem connection from 9.600 before 56.400 kb/s);

2. Developing high speed performance links, uniting academic server clusters in the global academic information structure;

The relative simplicity of standard Web applications encourages the conversion of text and multimedia files by students, providing them with enormous amounts of information from a "swollen" network database. This situation is providing difficulties for students who are increasingly required to distinguish between valuable and less-valuable information. A danger is emerging in this situation of a "cold-information cultural layer" as part of educational networking. The problem is that there is, for some students, too much "overflow" or too high a "noise level" in the amount of information that they can access and process. Educators using networked education systems have to keep in mind that it is

important to pursue defined learning goals. Students can be overwhelmed by access to unlimited information.

The unity of the research and education activities inherent in university education has to be reflected in the development of information processing materials and electronic education books and other multimedia developments. It is possible to determine five directions for information support for students as they engage in new, networked educational processes:

1. The formation of information as a "mirror" of the auditorium work of the teacher; 2. The development of accompanying information materials to support the teacher's

work; 3. The creation of on-line education "training machines" (local or global experimental

facilities); 4. The development of On-Line and Out-Line assistance systems and knowledge

testing; 5. The development of unified databases of education materials.

After three years testing information presentation, the following were selected as standards by the Moscow State University network:

1. RealMedia - for direct access presentation to lectures in various disciplines, 2. RealMedia and MPEG3 - for audio accompanying system for foreign language

teaching (English, German, French for Russian students and Russian for foreign students),

3. MPEG1 - for video accompaniment of different education courses, 4. JAVA - terminals for work with remote-access experimental systems, 5. PERL - management for interactive testing and assistance, 6. POSTGRESQL - Database as an integrated base of education developments at

Moscow State University and other academic institutions in Russia.

These selections were based on the criteria of compatibility, speed and information security with different types of hardware and software in various regions of Russia with Moscow State University. Important considerations in the selection of formats for hypermedia resources include multi-platform compatibility, free access to program resources and the possibility of the installation of full server support on small PCs with standard configurations.

The Future Development of Each Model

The future development of the open models for teaching Physics in Russia and Canada will depend on developments in ICT and the extent to which these can be integrated with appropriate pedagogy. To continue to be effective, open models for teaching Physics to dispersed locations will have to be constantly evaluated.

(a) The Canadian Model

The Vista School District Digital Intranet is a prototype for the other ten school districts in the province and for the development of courses in other areas of the curriculum. There are a number of problems to be resolved during the first year and the prototype is being monitored regularly.

The move to an open environment in the teaching and learning of Physics is based upon a transition from an analog to a digital environment. Newfoundland and Labrador has a high rate of use of satellite dishes per capita and there are many schools in this province with LANs. As a province Newfoundland and Labrador provides excellent opportunities for the development of these technologies.

There are a number of teaching and learning issues that have been noted by observers of the Vista Digital Intranet:

Teaching Issues:

1. Teachers' work in Internet-based courses is open to the scrutiny of other members of the profession on the Internet. There is less professional privacy for teachers working in this open, inter-institutional model than they are used to in traditional classrooms.

2. Teachers of Internet-based AP courses are likely to gain increased prominence in their profession in Newfoundland and Labrador, assuming the role of expert practitioner. Will this lead to the emergence of a new category of "lead teacher?"

3. It has been observed that telelearning is the future of distance education (Collis, 1996). Future teacher education programs are likely to have to accommodate open, electronic teaching and learning.

Learning Issues:

1. AP Physics students in the Vista Digital Intranet faced problems adjusting to the difficulty of the subject matter and to a new mode of delivery. The implications of this experience for the development of other, less academically-advanced courses for average students, have to be considered.

2. To what extent can the Internet be used in the teaching of Physics in conventional (face to face) classes?

(b) The Russian Model:

The creation of the Open Distributed Learning (ODL) system for the Universities and Schools of Russia has resulted in four developments:

1. The development an Inter-institutional network of communications to link all education services;

2. The convergence of educational material selection and presentation procedures; 3. The establishment of standards of network presentation using "multi-platform

software applications";

4. The development of standards for hardware and software support in the networking of universities and other academic institutions.

The priority task in the development of educational networking in Russia has been the provision of information support within the educational process. To achieve this aim it has been necessary to find technological solutions by networking compatible hardware and software. The ultimate aim of these developments is to provide access to the intellectual resources of universities and other academic institutions for more people throughout the country through the development of a "network teaching auditorium".

Conclusion

The development of open models for teaching physics to schools in dispersed locations in Russia and Canada is based on the application of contemporary developments in information and communication technologies. In both countries educators are attempting to bring developments in the application of information and communication technologies together through the creation of educational networks. In developing these new electronic educational structures, both Russian and Canadian educators have had to consider appropriate ways of organizing teaching and learning, acknowledging both the possibilities and the limitations of the new technologies with which they are currently working.

Both Russian and Canadian educators have found that they have to be constantly aware of the issue of the compatibility of hardware and software between sites. The effectiveness of the teaching of physics to dispersed sites within networks is dependent on the extent to which hardware and software can be shared between teachers and students.

This paper describes two types of inter-networking for education purposes that have been developed during past three years in Canada and Russia. The specific parameters of each model are influenced by the geography of each country. More than 9 hours time-zone differences in Russia together with unpredictable channel capacity make asynchronous information transfer very important. Canada's location and higher performance networks makes synchronous, real-time networking a high priority.

For both models there is a common zone in a developing - knowledge base and in the creation of global experimental facilities. For both models of ODL teacher education is evolving. The next steps for integrating technology and pedagogy include:

1. Developing the architecture of an open knowledge base in physics; 2. Developing integrated Web-interfaces for knowledge resources; 3. International teacher education to maximize these developments.

Information and communication technologies are, above all, collaborative. The development of new ways of teaching and learning physics in each country has been facilitated by the convergence of information and communication technologies with growing awareness of adjustments that are needed in pedagogy to accommodate technological changes. As Russian and Canadian scientists and educators collaborate to

improve the organization of open models for teaching to dispersed sites, it becomes increasingly possible that for teachers and students, national boundaries will disappear.

References

Collis B (1996) TeleLearning in a Digital World - The Future of Distance Learning, London and Boston, Thompson Computer Press

Goodyear, P (1999) Pedagogical Frameworks and Action Research in Open and Distance Learning,European Journal of Open and Distance Learning, June, pp: 1-12 http://kurs.nks.no/eurodl/eurodlen/index.html

Industry Canada (1995) The Challenge of the Information Highway, Ottawa, Information Highway Advisory Council

Industry Canada (1997) Preparing Canada for a Digital World, Ottawa, Information Highway Advisory Council

Kynaslahti, H and Wager P (1999) Changing Roles of the Teacher in Inter-Institutional Networks of Schools, European Journal of Open and Distance Learning, August, pp: 1-8, http://kurs.nks.no/eurodl/eurodlen/index.html

Stefansdottir, L. (1993). The Icelandic Educational Network - Ismennt, In: Davies, G. and Samways, B. (eds) Teleteaching - Proceedings of the IFIP TC3 Third Teleteaching Conference, Amsterdam, Elsevier Science Publishers, pp: 829-835

Stevens, KJ (1995a) Geographic Isolation and Technological Change: A New Vision of Teaching and Learning in Rural Schools in New Zealand, The Journal of Distance Learning Vol 1, No1

Stevens KJ (1995b) The Technological Challenge to the Notion of Rurality in New Zealand Education - Repositioning the Small School, In: Ian Livingstone (ed) New Zealand Annual Review of Education No 5, pp: 93-102

Tella S (1995) Virtual School in a Networking Learning Environment, Helsinki, University of Helsinki, Department of Teacher Education

THE IMPACT OF BELIEFS ON THE TEACHING OF MATHEMATICS

Paul Ernest

Official reports such as NCTM (1980) Agenda for Action, and the Cockcroft Report (1982) recommend the adoption of a problem solving approach to the teaching of mathematics. Such reforms depend to a large extent on institutional reform: changes in the overall mathematics curriculum. They depend even more essentially on individual teachers changing their approaches to the teaching of mathematics. However the required changes are unlike those of a skilled machine operative, who can be trained to upgrade to a more advanced lathe, for example. A shift to a problem solving approach to teaching requires deeper changes. It depends fundamentally on the teacher's system of beliefs, and in particular, on the teacher's conception of the nature of mathematics and mental models of teaching and learning mathematics. Teaching reforms cannot take place unless teachers' deeply held beliefs about mathematics and its teaching and learning change. Furthermore, these changes in beliefs are associated with increased reflection and autonomy on the part of the mathematics teacher. Thus the practice of teaching mathematics depends on a number of key elements, most notably:

• the teacher's mental contents or schemas, particularly the system of beliefs concerning mathematics and its teaching and learning;

• the social context of the teaching situation, particularly the constraints and opportunities it provides; and

• the teacher's level of thought processes and reflection.

These factors are therefore those which determine the autonomy of the mathematics teacher, and hence also the outcome of teaching innovations - like problem solving - which depend on teacher autonomy for their successful implementation.

The mathematics teacher's mental contents or schemas includes knowledge of mathematics, beliefs concerning mathematics and its teaching and learning, and other factors. Knowledge is important, but it alone is not enough to account for the differences between mathematics teachers. Two teachers can have similar knowledge, but while one teaches mathematics with a problem solving orientation, the other has a more didactic approach. For this reason, the emphasis below is placed on beliefs. The key belief components of the mathematics teacher are the teacher's:

• view or conception of the nature of mathematics, • model or view of the of the nature of mathematics teaching • model or view of the process of learning mathematics,

The teacher's conception of the nature of mathematics, is his or her belief system concerning the nature of mathematics as a whole. Such views form the basis of the philosophy of mathematics, although some teacher's views may not have been elaborated into fully articulated philosophies. Teachers' conceptions of the nature of mathematics by no means have to be consciously held views; rather they may be implicitly held

philosophies. The importance for teaching of such views of subject matter has been noted both across a range of subjects, and for mathematics in particular (Thom, 1973). Three philosophies are distinguished here because of their observed occurrence in the teaching of mathematics (Thompson 1984), as well as in the philosophy of mathematics and science.

First of all, there is the instrumentalist view that mathematics is an accumulation of facts, rules and skills to be used in the pursuance of some external end. Thus mathematics is a set of unrelated but utilitarian rules and facts. Secondly, there is the Platonist view of mathematics as a static but unified body of certain knowledge. Mathematics is discovered, not created. Thirdly, there is the problem solving view of mathematics as a dynamic, continually expanding field of human creation and invention, a cultural product. Mathematics is a process of enquiry and coming to know, not a finished product, for its results remain open to revision.

These three philosophies of mathematics, as psychological systems of belief, can be conjectured to form a hierarchy. Instrumentalism is at the lowest level, involving knowledge of mathematical facts, rules and methods as separate entities. At the next level is the Platonist view of mathematics, involving a global understanding of mathematics as a consistent, connected and objective structure. At the highest level, the problem solving view sees mathematics as a dynamically organised structure located in a social and cultural context.

The model of teaching mathematics is the teacher's conception of the type and range of teaching roles, actions and classroom activities associated with the teaching of mathematics. Many contributing constructs can be specified including unique versus multiple approaches to tasks, and individual versus cooperative teaching approaches. Three different models which can be specified through the teacher's role and intended outcome of instruction are:

TEACHER'S ROLE INTENDED OUTCOME 1. Instructor: Skills mastery with correct performance 2. Explainer: Conceptual understanding with unified knowledge3. Facilitator: Confident problem posing and solving

The use of curricular materials in mathematics is also of central, importance in a model of teaching. Three patterns of use are:

1. The strict following of a text or scheme; 2. Modification of the textbook approach, enriched with additional problems and

activities; 3. Teacher or school construction of the mathematics curriculum.

Closely associated with the above is the teacher's mental model of the learning of mathematics. This consists of the teacher's view of the process of learning mathematics, what behaviours and mental activities are involved on the part of the learner, and what

constitute appropriate and prototypical learning activities. Two of the key constructs for these models are: learning as active construction, as opposed to the passive reception of knowledge; the development of autonomy and child interests in mathematics, versus a view of the learner as submissive and compliant. Using these key constructs the following simplified models can be sketched, based on the child's:

1. compliant behaviour and mastery of skills model, 2. reception of knowledge model, 3. active construction of understanding model, 4. exploration and autonomous pursuit of own interests model.

RELATIONSHIPS BETWEEN BELIEFS, AND THEIR IMPACT ON PRACTICE

The relationships between teachers' views of the nature of mathematics and their models of its teaching and learning are illustrated in the following diagram. It shows how teachers' views

FIG. 1 Relationships between beliefs, and their impact on practice

of the nature of mathematics provide a basis for the teachers' mental models of the teaching and learning of mathematics, as indicated by the downward arrows. Thus, for example, the instrumental view of mathematics is likely to be associated with the instructor model of teaching, and with the strict following of a text or scheme. It is also likely to be associated

with the child's compliant behaviour and mastery of skills model of learning. Similar links can be made between other views and models, for example:

• Mathematics as a Platonist unified body of knowledge - the teacher as explainer - learning as the reception of knowledge;

• Mathematics as problem solving - the teacher as facilitator - learning as the active construction of understanding, possibly even as autonomous problem posing and solving.

These examples show the links between the teacher's mental models, represented by horizontal arrows in the diagram.

The teacher's mental or espoused models of teaching and learning mathematics, subject to the constraints and contingencies of the school context, are transformed into classroom practices. These are the enacted (as opposed to espoused) model of teaching mathematics, the use of mathematics texts or materials, and the enacted (as opposed to espoused) model of learning mathematics. The espoused-enacted distinction is necessary, because case-studies have shown that there can be a great disparity between a teacher's espoused and enacted models of teaching and learning mathematics (for example Cooney, 1985). Two key causes for the mismatch between beliefs and practices are as follows.

First of all, there is the powerful influence of the social context. This results from the expectations of others including students, parents, peers (fellow teachers) and superiors. It also results from the institutionalised curriculum: the adopted text or curricular scheme, the system of assessment, and the overall national system of schooling. These sources lead the teacher to internalise a powerful set of constraints affecting the enactment of the models of teaching and learning mathematics. The socialisation effect of the context is so powerful that despite having differing beliefs about mathematics and its teaching, teachers in the same school are often observed to adopt similar classroom practices.

Secondly, there is the teacher's level of consciousness of his or her own beliefs, and the extent to which the teacher reflects on his or her practice of teaching mathematics. Some of the key elements in the teacher's thinking - and its relationship to practice - are the following.

• Awareness of having adopted specific views and assumptions as to the nature of mathematics and its teaching and learning.

• The ability to justify these views and assumptions. • Awareness of the existence of viable alternatives. • Context-sensitivity in choosing and implementing situationally appropriate teaching

and learning strategies in accordance with his or her own views and models. • Reflexivity: being concerned to reconcile and integrate classroom practices with

beliefs; and to reconcile conflicting beliefs themselves.

These elements of teacher's thinking are likely to be associated with some of the beliefs outlined above, at least in part. Thus, for example, the adoption of the role of facilitator in a problem-solving classroom requires reflection on the roles of the teacher and learner, on the

context suitability of the model, and probably also on the match between beliefs and practices. The instrumental view and the associated models of teaching and learning, on the other hand, requires little self consciousness and reflection, or awareness of the existence of viable alternatives.

I have argued that mathematics teachers' beliefs have a powerful impact on the practice of teaching. During their transformation into practice, two factors affect these beliefs: the constraints and opportunities of the social context of teaching, and the level of the teacher's thought. Higher level thought enables the teacher to reflect on the gap between beliefs and practice, and to narrow it. The autonomy of the mathematics teacher depends on all three factors: beliefs, social context, and level of thought. For beliefs can determine, for example, whether a mathematics text is used uncritically or not, one of the key indicators of autonomy. The social context clearly constrains the teacher's freedom of choice and action, restricting the ambit of the teacher's autonomy. Higher level thought, such as self-evaluation with regard to putting beliefs into practice, is a key element of autonomy in teaching. Only by considering all three factors can we begin to do justice to the complex notion of the autonomous mathematics teacher.

REFERENCES

Cockcroft, W. H. (1982) Mathematics Counts, HMSO, London. Cooney, T. J. (1985) Journal for Research in Mathematics Education, 16(5), 324-336. NCTM (1980) Agenda for Action, NCTM, Reston, Virginia. Thom, R. (1973) in Howson, A. G. (ed.) Developments in Mathematical Education, Cambridge, 194-209. Thompson, A. G. (1984) Educational Studies in Mathematics, 15, 105-127.

This paper was presented as ‘The Impact of Beliefs on the Teaching of Mathematics’ at 6th International Congress of Mathematical Education, Budapest, August 1988. Printed as ‘The Impact of Beliefs on the Teaching of Mathematics’, in P. Ernest, Ed. Mathematics Teaching: The State of the Art, London, Falmer Press, 1989: 249-254. Printed slightly abridged as ‘The impact of beliefs on teaching’, in C. Keitel with P. Damerow, A. Bishop, and P. Gerdes, Eds. Mathematics Education and Society, Paris, UNESCO, 1989: 99-101. Reprinted as ‘The Impact of Beliefs on the Teaching of Mathematics’, in Bloomfield, A. and Harries, T. Eds (1994) Teaching and Learning Mathematics, Derby: Association of Teachers of Mathematics.

COGNITIVE TEACHING MODELS

Brent G. Wilson University of Colorado at Denver

Peggy Cole Arapahoe Community College

To appear in the HANDBOOK OF RESEARCH IN INSTRUCTIONAL TECHNOLOGY

David H. Jonassen, editor New York: Scholastic Press

1996

Educational psychology and instructional design (ID) have had a long and fruitful relationship (Dick, 1987; Merrill, Kowallis, & Wilson, 1981). Educational psychologists like Gagné and Glaser have always shown an interest in issues of design (Gagné, 1968; Glaser, 1976); indeed, they helped establish instructional design as a field of study (Gagné, 1987; Lumsdaine & Glaser, 1960). In recent years, a growing number of cognitive psychologists have shown a renewed interest in design issues, and have tested out their ideas by developing prototype teaching models. These teaching models differ from most educational innovations in that they are well-grounded in cognitive learning theory. Examples include John Anderson's intelligent tutors (Anderson, 1987) and Brown and Palincsar's reciprocal teaching method for teaching reading (for a research review, see Rosenshine & Meister, 1994). Wilson and Cole (1991) reviewed a number of these prototype teaching models, and related them to current ID theory. This chapter continues that agenda by reviewing a number of additional teaching models, and drawing implications for the design of instruction.

Specifically, the purposes of the chapter are to:

1. Argue that the development and validation of teaching models is a legitimate research method, and has been an important vehicle for advancing knowledge in learning and instruction.

2. Show how the development of cognitive teaching models compares to the development of traditional ID theory.

3. Review a number of cognitive teaching models, and discuss a few in detail.

4. Look for insights from these cognitive teaching models that relate to instructional design.

5. Identify issues for future research.

INSTRUCTIONAL PSYCHOLOGY AND DESIGN: AN HISTORICAL OVERVIEW

To provide a context for interpreting the chapter, consider the historical overview provided in Table 1.

Time period 1960-75 1976-1988 1989-present

Dominant paradigm

Behavioral psychology

Information processing psychology

Knowledge construction/

Social mediation

Status of ID

ID emerging

ID engaged in theory/model development

ID engaged in redefinition

Status of instructional psychology

Behaviorist

Moves toward cognitive mainstream

Follows mainstream towards constructivism

Relationship between ID and instructional psychology

ID and instructional psychology closely aligned

ID and instructional psychology diverge

ID and instructional psychology engaged in more dialogue

Table 1. The changing relationship between instructional psychology and ID from the 1960s to the present.

The field of instructional design developed in the 1960s and early 1970s at a time when behaviorism still dominated mainstream psychology. ID shared those behaviorist roots and at the time was closer to mainstream psychology in the U. S. ID theorists such as Gagné, Briggs, Merrill, and Scandura all were educational psychologists. With the cognitive revolution of the 1970s, instructional psychology differentiated itself from ID and drifted more to the cognitive mainstream, leaving ID relatively isolated with concerns of design. In a review of instructional psychology in 1981, Lauren Resnick (who only a few years earlier had developed Gagné-style learning hierarchies) observed:

An interesting thing has happened to instructional psychology. It has become part of the mainstream of research on human cognition, learning, and development. For about 20 years

the number of psychologists devoting attention to instructionally relevant questions has been gradually increasing. In the past 5 years this increase has accelerated so that it is now difficult to draw a clear line between instructional psychology and the main body of basic research on complex cognitive processes. Instructional psychology is no longer basic psychology applied to education. It is fundamental research on the processes of instruction and learning. (Resnick, 1981, p. 660)

In her review, Resnick acknowledged that mainstream instructional psychologists had focused on issues of performance modeling and cognitive task analysis, neglecting the challenge of devising effective instructional strategies, models, and interventions. Even so, she did not look to the ID community to fill the need because "Instructional design theory..., which is directly concerned with prescribing interventions, has developed without much reference to cognitive psychology" (Resnick, 1981, p. 693). Hence, she excluded ID theory entirely from her review. Of course, the ID community was active during this time, with even some attempt at integrating cognitive psychology into their methods (e.g., Low, 1981; Merrill, Wilson, & Kelety, 1981; Merrill, Kowallis, & Wilson, 1981)--Resnick and other mainstream psychologists just weren't reading them! This polarization between ID and psychology continued through the 1980s. In spite of efforts to move ID into the cognitive mainstream, psychologists and designers continued to move in different circles and speak somewhat different languages. Psychologists viewed designers with suspicion because of the eclectic and ad hoc nature of the ID theory base and because of the field's concern for stimulus design over cognitive processes. Likewise, the ID literature often ignored developments in cognitive theory, resulting in theory that was generally divorced from state-of-the-art learning theory (e.g., Reigeluth, 1983, 1987).

Only recently, with the vigorous dialogue on constructivism and situated learning, have psychologists and designers resumed a substantive conversation (Duffy & Jonassen, 1992; Educational Technology, April 1993 special issue on situated learning; Wilson, in press). Psychologists such as Bransford, Perkins, Scardamalia, and Lesgold, who have taken on the challenge of design, have run up against many of the same problems addressed by traditional ID theories. At the same time, the perspectives of psychologists have stimulated reflection and renewal within the ID community. The net result of this interplay is a renewed recognition of the importance of design, as well as an array of new designs that take into account new technologies and theories of learning.

THE ROLE OF TEACHING MODELS IN RESEARCH

Like other scientists, instructional psychologists develop theories and models describing the world, then use accepted methods of inquiry to test and revise those theories. Examples of appropriate research methods include controlled experiments in laboratory settings as well as ethnographic and qualitative studies in field settings. Another legitimate method of testing out concepts and strategies is to develop a prototype teaching model and assess its overall effectiveness in different settings. A teaching model incorporates a complex array of learning/instructional factors into a single working system. For example, John Anderson tested out his ideas of procedural learning by developing intelligent tutoring systems in LISP programming, geometry, and algebra (Anderson, 1987; Lewis, Milson, & Anderson,

1988); Ann Brown and her colleagues (Brown, Campione, & Day, 1981; Brown & Palincsar, 1989) developed reciprocal teaching as a means of testing their work on metacognition and reading.

The development and tryout of practical teaching models would not normally come to mind as a method of "research," yet surely such design and implementation efforts yield important new knowledge about the viability of cognitive theories and models. Perhaps such practical projects could be termed "inquiry" even if they do not fit the traditional connotation of "research." When researchers become interested in the problem of how people learn complex subject matters in realistic learning settings, practical tryout of programs and methods fills a role that no amount of theorizing or isolated-factor research can provide.

Teaching models can derive from direct empirical observation. Collins and Stevens (1982, 1983) closely observed teachers who used a Socratic dialogue approach and, based on the observed patterns, developed an instructional framework for inquiry teaching. Duffy (in press) developed a hypermedia tool to help preservice teachers to develop instructional strategies. The tool displays real-life segments of master teachers' lessons, then offers critiques from a number of perspectives, including those of the observed teachers. It also provides an electronic notepad for each preservice teacher to reflect on strategies used in the teaching episodes. While one could argue that Duffy's work is merely a neutral tool for displaying observed teaching performances, the tool embodies an underlying teaching model that is heavily grounded in actual teaching performances. Such "bottom-up" approaches can complement the heavy influence of "top-down" learning theory as a basis for the design of teaching models.

In summary, the development of teaching models constitutes a unique combination of theory construction and empirical testing. Theoretical abstractions must be carried to a new level of specificity as they become instantiated into an effective teaching program. At the same time, promising theory must be tested against the demands of real-world settings. Thus the development and testing of teaching models helps triangulate findings from more traditional research methods and assures a relevance to the practice of teaching.

In the review of models below, we have purposefully selected a range of models to illustrate the diversity found in the instructional psychology literature. We conclude with a discussion of goals and methods for instruction aimed at bringing some order to the diversity.

IMPROVING TRADITIONAL INSTRUCTION: COGNITIVE LOAD THEORY

For a number of years, John Sweller, an Australian psychologist from the University of New South Wales, has examined instructional implications of a model of memory called "cognitive load theory." Cognitive load theory is based on a straightforward reading of information-processing concepts of memory, schema development, and automaticity of procedural knowledge:

--Human working memory is limited--we can only keep in mind a few things at a time. This poses a fundamental constraint on human performance and learning capacity.

--Two mechanisms to circumvent the limits or working memory are:

--Schema acquisition, which allows us to chunk information into meaningful units, and

--Automation of procedural knowledge.

The first mechanism deals primarily with processing and understanding information; the second deals with the acquisition of skills. Each mechanism helps us overcome the limits of working memory by drawing on our long-term memories, which are very detailed and powerful.

Sweller's model of instructional design is based upon these concepts:

1. Our limited working memories make it difficult to assimilate multiple elements of information simultaneously.

2. When multiple information elements interact, they must be presented simultaneously. This imposes a heavy cognitive load upon the learner of the information and threatens successful learning.

3. High levels of element "interactivity" and their resulting cognitive load can be inherent in the content--e.g., learning language grammar inherently involves more element interactivity than simple vocabulary learning. However, weak methods of presentation and instruction may result in unnecessarily high overhead. An example would be to present a student a figure whose understanding requires repeated consultation of the text. The extra work required in decoding and translating the figure competes with the content for precious working-memory resources as the learner attempts to comprehend the material.

Cognitive load theory leads to some specific predictions for student learning:

--Simple content--i.e., content with relatively few intrinsic interactive elements--is not threatened by weak instructional methods. Learners are generally able to fit the demands of content and instruction within their working memories in such cases.

--Content containing high levels of interactivity among its elements cannot be learned effectively through weak instructional methods--that is, methods that require extra processing by learners. The demands of content and/or the method exceed the limits of the learner's working memory and learning does not occur.

Sweller's cognitive load theory has led to a number of instructional prescriptions, including:

--Carefully analyze the attention demands of instruction. Sweller's method defines "elements" and then counts the number of elements in instructional messages. Processing troubles arise when the learner must attend to too many different elements at the same time.

--Use single, coherent representations. These should allow the learner to focus attention rather than split attention between two places, e.g., between a diagram and the text or even between a diagram with labels not located close to their referents (Chandler & Sweller, 1991; see discussion in Sweller & Chandler, 1994, pp. 192-193).

--Eliminate redundancy. Redundant information between text and diagram have been shown to decrease learning. (See Saunders & Solman, 1984; Reder & Anderson, 1982; Lesh, Landau, & Hamilton, 1983; and Schooler & Engstler-Schooler, 1990 for research on redundant information on other tasks.)

--Provide for systematic problem-space exploration instead of conventional repeated practice (Pierce, Duncan, Gholson, Ray, & Kamhi, 1993).

--In multimedia instruction, present animation and audio narration (and/or text descriptions) simultaneously rather than sequentially (Mayer & Anderson, 1991, 1992; Mayer & Sims, 1994).

--Provide worked examples as alternatives to conventional problem-based instruction (Paas & Van Merriënboer, 1994; Carroll, 1994; in the area of analogical reasoning tasks, see Robins & Mayer, 1993; and Pierce et al., 1993).

In the section below, we present an overview of research on worked examples to illustrate the implications of cognitive load theory for instruction. For more discussion of the other instructional strategies briefly listed above, refer to Sweller (1989) and Sweller and Chandler (1994).

WORKED EXAMPLES

Conventional models of instruction in many domains involve the presentation of a principle, concept, or rule, followed by extensive practice on problems applying the rule. This approach at first glance seems like commonsense--providing ample skills practice is "learning by doing." However, cognitive load theory suggests that such instructional approaches may actually be hurting learners' understanding of the subject matter.

Sweller and Cooper (1985) examined the cognitive-load effects of methods for teaching algebra to high-school students. They hypothesized that when learners confront a conventional end-of-chapter practice exercise, they devote too much attention to the problem goal and to relatively weak search strategies such as means-end analysis. Students already know how to use general search strategies to solve problems; what they lack is the specific understanding of how cases relate to the general rule.

Sweller and Cooper hypothesized that learners might benefit from studying worked examples until they have "mastered" them, rather than working on conventional practice problems as soon as they have "obtained a basic familiarity with new material" (p. 87). The authors developed an alternative teaching model that emphasized the study of worked examples. After learners acquire a basic understanding of the algebraic principle, they study a series of examples; then the teacher answers any questions the learners have. When the learners indicate they understand the problems, they are required to explain the goal of each sample problem and to identify the mathematical operation used in each step of the problem. The teacher provides assistance to any learners who have difficulty with the questions. Then the learners complete similar problems, repeating them until they are solved with no errors; if too much time elapses, the teacher provides the answer.

Sweller and Cooper found that in the worked-examples model, acquisition of knowledge was significantly less time-consuming than in the conventional practice-based model. Furthermore, learners required significantly less time to solve similar problems (i.e., problems identical in structure) and made significantly fewer errors than did their counterparts. There were no significant group differences in solving novel problems. Thus learning was more efficient with no discerned loss in effectiveness. The authors concluded that "the use of worked examples may redirect attention away from the problem goal and toward problem-state configurations and their associated moves" (p. 86).

Sweller (1989) summarizes his position toward problem solving and learning by arguing that:

a. both schema acquisition and rule automation are the building blocks of skilled problem-solving performance...;

b. paradoxically, a heavy emphasis on conveying problem solving is not the best way to acquire schemas or facilitate rule automation because the means-end strategy commonly used focuses attention inappropriately and imposes a heavy cognitive load;

c. alternatives to conventional problem solving such as...worked examples must be carefully analyzed and, if necessary, modified to ensure that they, too, do not inappropriately direct attention and impose a heavy cognitive load; and

d. for the same reasons as for Point C, the format of instructional materials should be organized to minimize the need for students to attend to and mentally integrate disparate sources of information. (Sweller, 1989, p. 465, reformatted)

Sweller's critics might claim that students under the worked-example treatment were indeed actively engaging in problem-solving and practice activities, but that the nature of the practice shifted from traditional word problems to the study of worked examples. Instead of engaging in a multi-task activity (e.g., translating the word problem into one or more formulas, and performing calculations), the task narrowed to articulating the goal of the worked example and the appropriate mathematical operation. Sweller would likely agree with the critic. The point of the research is to suggest that not all "problem-solving" activities are equally effective. Some problem-solving activities actually leave learners at a

loss, forcing them to resort to "weak" problem-solving methods--which they already know--rather than "strong" or domain-specific methods--which they are trying to learn. Bereiter and Scardamalia (1992) discuss this issue:

In novel situations, where no strong methods have been devised, weak methods are all anyone has. We use them all the time, whenever we are stumped. But just because everyone uses them, could hardly survive without doing so, and therefore practices them extensively, there is reason to question the value of teaching them. Teaching problem-solving skills may be an illusion, like teaching babies to talk. (Bereiter and Scardamalia, 1992, p. 528)

If our goal is to teach students certain well-defined domains such as algebra or physics, then giving them problems requiring extensive use of "weak" methods may be counterproductive and may even interfere with learning the domain.

Worked examples and self-explanations. One limitation of the Sweller and Cooper study was that only indirect inferences could be made concerning learners' cognitive processes. Chi and her colleagues (e.g., Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Chi, de Leeuw, Chiu, and LaVancher, 1991; Chi & VanLehn, 1991) addressed this issue in the area of college-level physics. They analyzed the think-aloud protocols of good and poor problem solvers to identify the cognitive processes learners used in studying worked examples in physics. Learners in the study did not differ significantly in their prior knowledge of physics; instead, good and poor problem solvers were identified by their performance on objective tests.

Chi's worked examples differed from those used by Sweller and Cooper (1985) in significant ways. Sweller and Cooper presented worked-example sheets which were not part of a text and which included no verbal commentary. By contrast, the physics examples were part of the text and included step-by-step verbal commentary, although the learners had to infer the "why's and wherefore's" of each step.

Self-explanations were one kind of student response to the worked examples. "A self-explanation is a comment about an example statement that contains domain-relevant information over and above what was stated in the example line itself" (p. 69). Chi et al. (1989) found that good problem solvers generated more self-explanations than poor problem solvers and that poor problem solvers used the examples rotely as prompts for solving subsequent problems. Chi and VanLehn (1991) conjectured that "the act of self-explaining may make the tacit knowledge…more explicit and available for use" (p. 101). They identified two general sources for self-explanations: "deduction from knowledge acquired earlier while reading the text part of the chapter,… [and] generalization and extension of the example statements" (p. 69).

In an intervention study, Chi et al. (1991) found that high-ability and average-ability students benefited equally from being prompted to generate self-explanations. This finding counters other research on strategy training, which has found that such training generally benefits low-ability students while it doesn't benefit and may even interfere with the performance of high-ability students (e.g., Brown & Campione, 1981). These discrepant findings might be partially explained by the fact that the earlier studies tended to teach

skills rather than strategies (see Duffy & Roehler [1989] for a discussion of the confusion about skills and strategies).

Summary. Cognitive load theory bears a strong resemblance to traditional instructional-design theories (Reigeluth, 1983, 1987). The prescriptions for instruction require a careful task analysis that especially considers the memory load implications of different content combinations and instructional methods. The emphasis on well-defined content, worked examples, and careful doses of presented information is reminiscent of Merrill's (1983; Merrill & Tennyson, 1977) Rule-Example-Practice prescriptions for teaching concepts and procedures. The emphasis on careful control over presentation and pacing, and the strongly positive gains attributable to managing cognitive load, serve as prudent reminders of the importance of task and memory variables.

CONTEXTUALIZING INSTRUCTION: COGNITIVE APPRENTICESHIPS

Collins, Brown, and colleagues (e.g., Collins, Brown, & Newman, 1989; Collins, 1991) developed an instructional model derived from the metaphor of the apprentice working under the master craftsperson in traditional societies, and from the way people seem to learn in everyday informal environments (Lave, 1988). The cognitive apprenticeship model rests on a somewhat romantic conception of the "ideal" apprenticeship as a method of becoming a master in a complex domain (Brown, Collins, & Duguid, 1989). In contrast to the classroom context, which tends to remove knowledge from its sphere of use, Collins and Brown recommend establishing settings where worthwhile problems can be worked with and solved. The need for a problem-solving orientation to education is apparent from the difficulty schools are having in achieving substantial learning outcomes (Resnick, 1989).

Emulating the best features of apprenticeships is needed because, as Gott (1988a) noted, lengthy periods of apprenticeship are becoming a rarity in industrial and military settings. She termed this phenomenon the "lost apprenticeship." She noted the effects of the increased complexity and automation of production systems. First, the need is growing for high levels of expertise in supervising and using automated work systems; correspondingly, the need for entry levels of expertise is declining. Workers on the job are more and more expected to be flexible problem solvers; human intervention is often most needed at points of breakdown or malfunction. At these points, the expert is called in. Experts, however narrow the domain, do more than apply canned job aids or troubleshooting algorithms; rather, they draw on the considerable knowledge they have internalized and use it to flexibly solve problems in real time (Gott, 1988b).

Gott's second observation relates to training opportunities. Now, at a time when more problem-solving expertise is needed due to the complexity of systems, fewer on-the-job training opportunities exist for entry-level workers. There is often little or no chance for beginning workers to acclimatize themselves to the job, and workers very quickly are expected to perform like seasoned professionals. True apprenticeship experiences are becoming relatively rare. Gott calls this dilemma--more complex job requirements with less

time on the job to learn--the "lost" apprenticeship and argues for the critical need for cognitive apprenticeships and simulation-type training to help workers develop greater problem-solving expertise.

FEATURES OF COGNITIVE APPRENTICESHIPS

The Collins-Brown model of cognitive apprenticeship incorporates the following instructional strategies or components.

1. Content: Teach tacit, heuristic knowledge as well as textbook knowledge. Collins et al. (1989) refer to four kinds of knowledge:

--Domain knowledge is the conceptual, factual, and procedural knowledge typically found in textbooks and other instructional materials. This knowledge is important, but often is insufficient to enable students to approach and solve problems independently.

--Heuristic strategies are "tricks of the trade" or "rules of thumb" that help people narrow solution paths while solving a problem. Experts usually pick up heuristic knowledge indirectly through repeated problem-solving practice; slower learners usually fail to acquire this subtle knowledge and never develop competence. There is evidence to believe, however, that at least some heuristic knowledge can be made explicit and represented in a teachable form (Chi, Glaser, & Farr, 1988).

--Control strategies are required for students to monitor and regulate their problem-solving activity. Control strategies have monitoring, diagnostic, and remedial components; this kind of knowledge is often termed metacognition (Flavell, 1979).

--Learning strategies are strategies for learning; they may be domain, heuristic, or control strategies. Inquiry teaching to some extent directly models expert learning strategies (Collins & Stevens, 1983).

2. Situated learning: Teach knowledge and skills in contexts that reflect the way the knowledge will be useful in real life. Brown, Collins, and Duguid (1989) argue for placing all instruction within "authentic" contexts that mirror real-life problem-solving situations. Collins (1991) is less forceful, moving away from real-life requirements and toward problem-solving situations: For teaching math skills, situated learning could encompass settings "ranging from running a bank or shopping in a grocery store to inventing new theorems or finding new proofs. That is, situated learning can incorporate situations from everyday life to the most theoretical endeavors" (Collins, 1991, p. 122).

Collins cites several benefits for placing instruction within problem-solving contexts:

--Learners learn to apply their knowledge under appropriate conditions.

--Problem-solving situations foster invention and creativity.

--Learners come to see the implications of new knowledge. A common problem inherent in classroom learning is the question of relevance: "How does this relate to my life and goals?" When knowledge is acquired in the context of solving a meaningful problem, the question of relevance is at least partly answered.

--Knowledge is stored in ways that make it accessible when solving problems. People tend to retrieve knowledge more easily when they return to the setting of its acquisition. Knowledge learned while solving problems gets encoded in a way that can be accessed again in similar problem-solving situations.

3. Modeling and explaining: Show how a process unfolds and tell reasons why it happens that way. Collins (1991) cites two kinds of modeling: modeling of processes observed in the world and modeling of expert performance, including covert cognitive processes. Computers can be used to aid in the modeling of these processes. Collins stresses the importance of integrating both the demonstration and the explanation during instruction. Learners need access to explanations as they observe details of the modeled performance. Computers are particularly good at modeling covert processes that otherwise would be difficult to observe. Collins suggests that truly modeling competent performance, including the false starts, dead ends, and backup strategies, can help learners more quickly adopt the tacit forms of knowledge alluded to above in the section on content. Teachers in this way are seen as "intelligent novices" (Bransford et al., 1988). By seeing both process modeling and accompanying explanations, students can develop "conditionalized" knowledge, that is, knowledge about when and where knowledge should be used to solve a variety of problems.

4. Coaching: Observe students as they try to complete tasks and provide hints and helps when needed. Intelligent tutoring systems sometimes embody sophisticated coaching systems that model the learner's progress and provide hints and support as practice activities increase in difficulty. The same principles of coaching can be implemented in a variety of settings. Bransford and Vye (1989) identify several characteristics of effective coaches:

--Coaches need to monitor learners' performance to prevent their getting too far off base, but leaving enough room to allow for a real sense of exploration and problem solving.

--Coaches help learners reflect on their performance and compare it to others'.

--Coaches use problem-solving exercises to assess learners' knowledge states. Misconceptions and buggy strategies can be identified in the context of solving problems.

--Coaches use problem-solving exercises to create the "teachable moment."

5. Articulation: Have students think about their actions and give reasons for their decisions and strategies, thus making their tacit knowledge more explicit. Think-aloud protocols are one example of articulation. Collins (1991) cites the benefits of added insight and the ability to compare knowledge across contexts. If learners' tacit knowledge is brought to light, that knowledge can be recruited to solve other problems.

6. Reflection: Have students look back over their efforts to complete a task and analyze their own performance. Reflection is like articulation, except it is pointed backwards to past tasks. Analyzing past performance efforts can also influence strategic goal-setting and intentional learning (Bereiter & Scardamalia, 1989). Collins and Brown (1988) suggest four kinds or levels of reflection:

--Imitation occurs when a batting coach demonstrates a proper swing, contrasting it with your swing;

--Replay occurs when the coach videotapes your swing and plays it back, critiquing and comparing it to the swing of an expert;

--Abstracted replay might occur by tracing an expert's movements of key body parts such as elbows, wrists, hips, and knees, and comparing those movement to your movements;

--Spatial reification would take the tracings of body parts and plot them moving through space.

The latter forms of reflection seem to rely on technologies--video or computer-- for feasible implementation.

7. Exploration: Encourage students to try out different strategies and hypotheses and observe their effects. Collins (1991) claims that through exploration, students learn how to set achievable goals and to manage the pursuit of those goals. They learn to set and try out hypotheses, and to seek knowledge independently. Real-world exploration is always an attractive option; however, constraints of cost, time, and safety sometimes prohibit instruction in realistic settings. Simulations are one way to allow exploration; hypermedia structures also allow exploration of information.

8. Sequence: Present instruction in an ordering from simple to complex, with increasing diversity, and global before local skills.

--Increasing complexity. Collins et al. (1989) point to two methods for helping learners deal with increasing complexity. First, instruction should take steps to control the complexity of assigned tasks. They cite Lave's study of tailoring apprenticeships: apprentices first learn to sew drawers, which have straight lines, few pieces of material, and no special features like zippers or pockets. They progress to more complex garments over a period of time. The second method for controlling complexity is through scaffolding. Here the cases or content remains complex, but the instructor provides the needed scaffolding for initial performances and gradually fades that support.

--Increasing diversity refers to the variety in examples and practice contexts.

--Global before local skills refers to helping learners acquire a mental model of the problem space at very early stages of learning. Even though learners are not engaged in full problem solving, through modeling and helping on parts of the task (scaffolding), they can

understand the goals of the activity and the way various strategies relate to the problem's solution. Once they have a clear "conceptual map" of the activity, they can proceed to developing specific skills.

The three teaching models presented below illustrate various features of the cognitive apprenticeship model. The first two are computer-based environments: Sherlock and goal-based scenarios. The third model is the problem-based learning environment developed by medical educators at the University of Illinois. All three models build instruction around problems or cases that are faithful to real-life situations, within which learners learn the details of a subject matter.

SHERLOCK

What happens when you combine extensive cognitive-task analysis of a well-defined, technical domain with a situated-learning philosophy and a teaching model based on intelligent tutoring systems (ITS)? Sherlock is an example of such a teaching tool. Sherlock is a computer-coached practice environment developed by Alan Lesgold and colleagues (e.g., Lajoie & Lesgold, 1992; Lesgold, et al., 1988; Lesgold, Lajoie, Bunzo, & Eggan, 1992) to develop the troubleshooting skills of Air Force electronics technicians. Sherlock was specifically designed to teach the most difficult parts of the troubleshooter's job. Learners are presented a number of troubleshooting problems requiring two kinds of activities:

--The student solves the problem, requesting advice from the intelligent tutor/coach as necessary.

--The student reviews a record of his/her problem-solving activity, receiving constructive critique from the coach. (Gott, Lesgold, & Kane, in press)

Sherlock serves not only as an instructional environment and an assessment device, but also as a laboratory for instructional research (Lajoie & Lesgold, 1992). Assessment is interwoven with instruction so that coaching is highly individualized. This is achieved using expert systems technology to create two types of student modeling: a competency model which is updated throughout the program, and a performance model. Together, they provide the basis for diagnosing learner problems and selecting appropriate instructional mediation relating to goals, operators, methods and strategies.

A central feature of Sherlock is its intelligent hyperdisplay:

When Sherlock constructs a schematic diagram to help illustrate the advice it is providing, that diagram is organized to show expert understanding about the system with which the trainee is working....What is displayed is approximately what a trainee would want to know at that time, but every display component is "hot" and can be used as a portal to more detail or explanation. (Gott et al., in press)

Sherlock's diagrams are dynamic; that is, they are assembled at any point in the program to be sensitive to immediate conditions. The diagrams are adjusted so that conceptually central components are afforded the most space; diagram boxes and circuit paths are color-coded to reflect the learner's prior knowledge about them.

Gott, Hall, Pokorny, Dibble, & Glaser (1992) studied Sherlock learning environments to find out how students made flexible use of their knowledge in novel situations:

Time and again we observed [successful] learners access their existing mental models of equipment structure...and their schema of the troubleshooting task.... They then used these models as flexible blueprints to guide their performance as they crafted solutions to new problems. Their prior models became interpretive structures, and when these models were inadequate, better learners flexibly used them as the basis for transposed and elaborated structures that could accommodate the novel situations. They were ready and willing to construct new knowledge that was grounded in their existing representational and functional competence. (Gott et al., in press)

The current incarnation of the program, Sherlock 2, includes a number of refinements aimed at facilitating students' development of device models and transfer of knowledge. Consistent with the cognitive apprenticeship model, students have a variety of supports and reflective tools available to them:

To complement coached learning by doing, we have developed a collection of tools for post-performance reflection. One provides an intelligent replay of the trainee's actions. A trainee can "walk through" the actions he just performed while solving the problem. In addition, he can access information about what can in principle be known about the system given the actions replayed so far.... Also, he can ask what an expert might have done in place of any of his actions, get a critique of his action, and have his action evaluated by the system....Further, there is an option for side-by-side listing of an expert solution and the trainee's most recent effort. (Gott et al., in press)

One key to Sherlock's success is the extensive and sophisticated cognitive task analysis which provided "critical information necessary for developing the appropriate student models of proficiency" (Lajoie & Lesgold, 1992, p. 381). The program addresses eight dimensions of proficiency proposed by Glaser, Lesgold, and Lajoie (1987) and based on research on expert-novice differences (Lajoie & Lesgold, 1992):

1. knowledge organization & structures,

2. depth of underlying principles,

3. quality of mental models,

4. efficiency of procedures,

5. automaticity to reduce attentional demands,

6. procedural knowledge,

7. procedures for theory change, and

8. metacognitive skills.

According to Gott et al. (in press), the Sherlock team's approach to task analysis is similar but distinguishable from traditional instructional-design approaches. "What is different is that the structure of learning tasks is more authentic, rooted in the needs of practice (or simulated practice) rather than being derived directly from task analysis structure...."

Research has found that learners who used Sherlock improved dramatically in their troubleshooting skills, during training as well as on a posttest (Lesgold et al., 1988; Nichols, Pokorny, Jones, Gott, & Alley, in press). Sherlock 2 yielded effect sizes on posttest measures ranging from .87 to 1.27 (Gott et al., in press).

GOAL-BASED SCENARIOS

Roger Schank and colleagues at the Institute for the Learning Sciences have developed an architecture for the design of learn-by-doing courses and simulations (Schank, Fano, Bell, & Jona, 1993/1994). Goal-based scenarios constitute an alternative to traditional intelligent tutoring systems (ITS) that combine elements of simulation, case-based reasoning, and traditional ITS modeling techniques. Riesbeck (in press) describes the concept of a goal-based scenario :

In a [goal-based scenario], a student is given a role to play, e.g., owner of a trucking company or chief scientist at a nuclear research installation, and interesting problems to solve or goals to achieve. The role and problems should be of real interest to the student, e.g., feeding the world, getting rich, or flying a rocket to the moon, not artificial word problems....

The student engages in a simulation in order to solve the defined problem or achieve the goal. Typically the student interacts with simulated agents and objects within a simulated environment. A goal-based scenario differs, however, from traditional simulations in a number of respects, for example:

When the student gets stuck or in trouble, a tutor, in video form, appears to offer advice, tell stories, and so on. The stories come from a multimedia archive of texts and video interviews of experts in that domain, telling personal experiences similar to the student's simulated situation. These stories are also organized for browsing in a structure we call ASK networks. (Ferguson et al., 1991, cited in Riesbeck, in press)

ASK networks are systems for indexing and archiving stories in a way that makes them useful within the goal-based scenario. Stories are brought into the simulation as the need arises, with the indexing sensitive to the learner's progress and other local conditions.

Three goal-based scenarios that have been developed are briefly reported below.

Broadcast News. High school students collaborate to produce their own simulated TV news broadcast. "The student first sees a brief introduction informing him or her that he or she will be working on a newscast for a particular day in history...[say] May 21, 1991. The student is then given a rough draft of a news story that requires revisions so it can go on the air" (Schank et al., 1993/94, p. 309).

Students often lack the historical or political knowledge to understand the draft script, so they consult tools and resources within the program that provide a context for understanding the script.

The student then needs to revise the script and prepare it for broadcast. Rather than personally re-writing the draft, the student submits specifications for revision back to the writers. The program's experts can provide support and advice through this process. As in real life, two experts' feedback, in fact, may conflict with each other, forcing the student to decide how to interpret suggestions and incorporate them into the revised script.

After the student gives final approval to a story, he or she can then choose to play the role of anchorperson for the newscast as well. The program then acts as a teleprompter and editing booth. The student reads the story as the text rolls by on the screen. A video camera controlled by the computer records the student as he or she plays the role of anchor; the computer also supplies the video accompanying the story. A complete videotape of the student's newscast is ready as soon as the newscast ends. ( Schank et al., 1993/94, pp. 309-310)

The student can watch the tape and compare it with a professional network newscast covering the same event. This comparison fosters reflection and discussion about the process and decisions made.

Sickle Cell Counselor. This is an interactive hypermedia exhibit designed for the Museum of Science and Industry in Chicago (Bell, Bareiss, & Beckwith, 1993/94; Schank et al., 1994, pp. 310-311). The user assumes a role of genetic counselor advising couples of the genetic risks of their upcoming marriage. The student is able to run simulated lab tests, interact via interactive video with the couple, collect data from the couple, and offer advice. Research indicated that museum visitors spent considerably more time with the exhibit than for other exhibits and learned something about genetics (based on both self-report and performance on pre- and posttests).

YELLO. This is a program designed to teach telephone operators how to sell Yellow Pages advertising (Kass, Burke, Blevis, & Williamson, 1993/94). The program follows a framework designed for the teaching of complex social skills. Of particular interest is the interjection of stories into the practice section. The program tracks student performance and retrieves a "story" that matches the student's performance profile, based on a sophisticated indexing scheme. The story--a real-life recounting by an experienced practitioner--is then provided to the student to strengthen motivation and make the task meaningful, as well as to correct the performance.

Schank et al. (1993/1994) outline the principal components of goal-based scenarios, along with criteria for good design. The following four components form the basis of a goal-based scenario.

Mission. The mission is the overall goal of the goal-based scenario. A scenario's mission may relate to process skills or outcome achievement skills. Process skills (e.g., running a trucking company, flying a plane, being a bank teller, etc.) lend themselves to role-play scenarios where the learner assumes the role of a character and learns the knowledge and skills related to that role. Outcome achievement skills (e.g., troubleshooting an engine or building a bridge) lend themselves to a scenario focusing on a specific task or achieving a particular result. By accomplishing the task, the student learns relevant skills along the way.

Mission focus. Schank et al. (1993/94) refer to the mission focus as "the underlying organization" of the activities engaged in by students (p. 327). They identified four mission foci:

--Explanation. Students are asked to account for phenomena, predict outcomes, diagnose systems, etc. Sickle Cell Counselor has an explanation mission focus.

--Control. Students are asked to run an organization or maintain and regulate a functioning system. Examples of a control focus would be managing a software project or running a nuclear power plant (Schank et al., 1993/94, p. 331).

--Discovery. Students enter a microworld and explore the features available. They may be asked to infer the microwold's governing principles or participate in activities available. YELLO offers an example of this type of mission focus.

--Design. Students create or compose some product or create the design specifications for some artifact. An example would be Broadcast News.

Cover story. The cover story provides the specifics of the student's role and the surrounding context. Schank et al. (1993/94) suggest designing a cover story around something the student might like to do (e.g., be president of the United States or fly an airplane) or something the student would have some strong feeling for (e.g., investigate the Chernobyl accident site, help a person threatening suicide). The details of the story are worked out in the design of the cover story, including the "setup" (explanations to students about why the scenario is important, specification of tools available in solving problems, etc.) and the "scenes" (the specific physical settings encountered in the story).

Scenario operations. Specification of scenario operations is the final stage of a goal-based scenario design. Scenario operations are the discrete, specific responses required of students engaged in the program. Examples might include "adjusting a parameter with a dial, issuing a directive in a social simulation, answering a question, using a tool to shape part of an artifact, searching for a piece of information, and deciding between two alternatives" (Schank et al. (1993/94, p. 336).

In many ways, Schank's work continues the tradition of developing instructional simulations. Simulations have been popular forms of instruction since the advent of computer-based instruction in the 1960s, including models of role-playing, control of dynamic systems, and task performance. Schank's contribution has been in the development of a sound theoretical model based on cognitive memory research, and in the creation of a design laboratory that follows a well-defined development model in creating working products. The costs of developing full-blown goal-based scenarios undoubtedly remain high, but they signal important progress in the design of instructional systems.

PROBLEM-BASED LEARNING

As noted above, intelligent tutoring systems rely on extensive and sophisticated cognitive task analysis to develop expert and student models; moreover, they are usually designed for individual learners. Like Sherlock, they tend to address well-structured problems in well-structured domains and build on a broad base of content knowledge. Problem-based learning (PBL) fills a different need, addressing ill-structured problems and/or ill-structured domains. Koschmann, Myers, Feltovich, and Barrows (1994) distinguished between an ill-structured domain, "in which no single concept, or even a small number of conceptual elements, is sufficient for capturing the workings of a typical instance of knowledge application" (Koschmann et al., 1994, p. 231), and an ill-structured problem. According to Korschman et al. (1994, summarizing Barrows & Feltovich, 1987), ill-structured problems have the following characteristics:

--[D]efining the problem requires more information than is initially available;

--the nature of the problem unfolds over time;

--there is no single, right way to get that information;

--as new information is obtained, the problem changes;

--decisions must be made in the absence of definitive knowledge; and

--there may never be certainty about having made the right decision. (p. 231, punctuation altered and formatting added)

Problem-based learning integrates the learning of content and skills, utilizes a collaborative environment, and emphasizes "learning to learn" (as opposed to case-based "learn by doing") by placing most of the responsibility for learning on the learner rather than providing a sophisticated pre-designed instructional system.

The PBL model has been implemented in several areas of higher education, including medicine, business, education, architecture, law, engineering, and social work, as well as in high school (Savery & Duffy, in press). However, the best known applications of PBL are in medical schools, where it was developed in the 1950's, and whose graduates face particularly ill-structured problems in an ill-structured and ever-expanding domain that

requires life-long learning skills. (See, e.g., Williams [1992], Savery & Duffy [in press], and Koschmann et al. [1994] for critical overviews of the use of PBL in medical schools.) More than a hundred medical programs include a PBL option (Duffy, 1994). PBL combines the teacher-directed case method which is used extensively in law and business schools with the discovery-learning philosophy of Jerome Bruner (Lipkin, 1989; Schmidt, 1989).

During the first two years of medical school, students in a PBL curriculum work in small self-directed groups to learn both content (basic science knowledge) and skills (examining and diagnosing patients, and metacognitive skills such as self-monitoring, reflections, and resource allocation). In lieu of traditional lectures and laboratory exercises, the problem-based curriculum presents a series of authentic patient problems; groups of five to seven students work intensely for about a week on each problem, diagnosing and learning to understand its causes. Authenticity is critical in motivating students and in avoiding the "construction of fictional problems...[with] symptoms that cannot coexist" (Williams, 1992, p. 404).

The facts of the problem are presented, just as they were initially to a doctor, as an incomplete set of symptoms that must be evaluated and explained. For practical reasons, the presentation is usually simulated on paper or by an actor trained as a patient. The facts of the complete case are contained in a problem-based learning module (Barrows, 1985; Distlehorst & Barrows, 1982), which "is designed to allow for free inquiry, providing responses for any question, examination, or laboratory test an examiner might request for the actual patient" (Koschmann et al., 1994, p. 241), without cueing any factors that are critical to the case (Savery & Duffy, in press).

A tutor facilitates students' negotiation of five recursive stages of the problem-based methodology (Koschmann et al., 1994):

1. Problem formulation. Students isolate important facts from their rich context, identify the problem, and generate hypotheses.

2. Self-directed learning. Group members identify and address information needed to evaluate hypotheses. This list of needed information sets the learning agenda. For example, they might research basic biological mechanisms that might underlie a patient's problems, question or "examine" the patient, review results of tests they "order," and consult with the medical faculty.

3. Problem re-examination. Group members bring to bear their findings from their self-directed learning activities--adding, deleting or revising hypotheses as warranted.

4. Abstraction. This is an articulation process (cf. Collins, Brown, & Newman, 1989) during which members compare and contrast cases, forming cognitive connections to increase the utility of the knowledge gained in specific contexts.

5. Reflection. At this point the group debriefs the experience and identifies areas for improvement in their learning processes.

The role of the tutor, according to Barrows (1992), is critical to "the success of any educational method aimed at 1) developing students' thinking or reasoning skills (problem solving, metacognition, critical thinking) as they learn, and 2) helping them to become independent, self-directed learners..." (p. 12). The tutor must not provide mini-lectures or suggest solutions but help each member of the group internalize effective metacognitive strategies by monitoring, modeling, coaching, and fading. This role includes

not only moving students through the various stages…but also monitoring group process and the participation of individuals within it, guiding the development of the clinical reasoning process by strategically questioning the rationale underlying the inquiry strategy of the group or individuals, externalizing self-questioning and self-reflection by directing appropriate questions to individuals or the group as a whole and evaluating each student's development. (Koschmann et al., 1994, p. 243)

The tutor externalizes higher-order thinking that students are expected to internalize, by asking students to justify not only their inferences (e.g., "How do you know that's true?") but also any question they ask the patient (Savery & Duffy, in press; Williams, 1992). Thus the students learn to identify and challenge superficial thinking and vague notions. In PBL, the tutor carefully monitors each student's development and forces him/her to remain an active learner.

Medical students appear to be very positive about the approach, particularly in their first year. However, many issues are still to be addressed. Because so much of the work is group-oriented, PBL cannot guarantee that the student will do his/her own thinking. As Koschmann et al. noted:

Teaching methods that depend on group interaction often experience what is termed the polling problem; the opinions of individuals vary as a function of the order in which their views are gathered. Contributions of less dominant members may be suppressed or contaminated by the more dominant members; convictions of any single individual in the group may be inappropriately influenced by other members; individuals can find means to hide or ride on the coattails of other group members. The polling problem, therefore, can result in the suppression of ideas, reducing the multiplicity of viewpoints expressed. (p. 243)

Compared to the computer-based programs reviewed above, PBL is relatively ill-defined; that is, students' specific interactions cannot be prespecified. Because of this, good design and successful implementation become intertwined--Design must happen constantly throughout the course of students' activities. The instructor must remain vigilant to ensure that less dominant members still have a voice within the group. Reluctant learners need to be monitored and encouraged to participate. More than some forms of controlled instruction, PBL depends on high-quality implementation by skilled instructors and participants.

PBL is also time-consuming. Problem-based activities can become tedious and boring, especially for students who have already internalized the clinical reasoning process. These complaints lead one to ask: Is it essential that students discover or seek out primary sources

in the library for every case? Or might learning be just as effective but more efficient if hypermedia programs consolidated answers to relevant questions to reduce the temporal and cognitive loads, at least for some of the cases? A careful analysis to identify the specific activities most valuable in generating new knowledge seems in order (cf. Collins' epistemic games, described below).

Indeed, faculty have begun to modify PBL programs. For example, faculty at the University of New Mexico School of Medicine designed focused cases "to solve these problems by encouraging learning on a single topic rather than leading to the generation of multiple, diverse hypotheses" (Williams, 1992, pp. 403-404). Koschmann et al. (1994) are developing computer-assisted programs to augment PBL, several of which directly or indirectly reduce cognitive load. These programs will facilitate

--students' expression of their honest viewpoints,

--"a retrievable record of the group's deliberations for previously studied cases" (p. 247),

--the tutor's monitoring of the development of each person's progress,

--the collection of authentic cases,

--the selection of cases appropriate to the needs of the students,

--students' communicating outside group meetings,

--students' access to learning resources,

--students' access to information in their notes.

Koschmann et al. (1994) are designing programs that utilize groupware (Stefik & Brown, 1989), hypertext/hypermedia (Conklin, 1987), database technologies, and electronic mail.

TOOLS FOR KNOWLEDGE BUILDING COMMUNITIES

In an article on the design of collaborative learning environments, Pea (1994) describes three metaphors of communication:

1. Communication as transmission of information. This is the dominant idea that communication conveys a message over time and distance from one person to another.

2. Communication as ritual. This refers to the participation and fellowship involved in the sharing of certain forms of expression. Participation in the performing arts such as dance, theater, and music, either as a performer or as an audience member, involves ritualistic aspects of communication. The content of the message is often less important than the

medium and style of expression, informing the audience and strengthening the common bond of group membership. Ritual communication emphasizes the sharing and communal functions of communications, allowing groups to maintain a sense of identity and coherence.

3. Communication as transformation. Both the "sender" and "receiver" of information are transformed as they share a goal of learning and knowledge generation. Participants in transformative communication open themselves up and expect change to occur as part of the process. Communication thus serves as a stimulus to inquiry, observation, and reflection. Transformative communication combines aspects of knowledge sharing and group collaboration, with an emphasis on new experience and learning (Pea, 1994, pp. 287-288)

Transformative styles of communication are characteristic of learning communities, whether in schools, classrooms, workgroups, or families. Pea (1994, p. 289) notes that a number of researchers are presently moving from a cognitive-science base toward a social-cognition framework in their attempt to understand the symbols and discourses of learning communities. Cognitive apprenticeships are an example, as are Brown's (1994) communities of learning and Scardamalia & Bereiter's (1994) knowledge-building communities. The common notion is that groups of people share a goal of building meaningful knowledge representations through activities, projects, and discussion.

Transformative communication seems not to be emphasized in Sherlock or Schank's case-based scenarios. The goal is not mutual change between communicating parties, but more computer-directed change in the student. Similarly, the PBL model expects the students to be transformed, playing roles of both senders and receivers, but the tutor is expected to remain basically detached, monitoring, coaching, and externalizing higher-order thinking. A transformative or learning-community view would suggest that the instructor is a part of the learning community, and should be an active, learning participant in the community.

The models described below are designed as tools to support knowledge-building communities.

EPISTEMIC GAMES

What do learning communities do? Collins and colleagues (Collins & Ferguson, 1993; Morrison & Collins, in press) would respond that learning communities generate new knowledge by participating in certain defined cultural patterns or forms. The products of this work are called epistemic forms. "Completed" forms contain new knowledge and adhere to defined structures accepted by the community. Working together to generate these forms is called participating in epistemic games. The game is the set of rules or conventions that can be followed in generating a given epistemic form.

Collins and Ferguson (1993) suggest three important types of epistemic games, along with several sub-categories shown in Table 2:

1. Structural analysis games. What are the components or elements of a system?

2. Functional analysis games. How are the elements in a system related to each other?

3. Process analysis games. How does the system behave?

Each of these general game types is found in every subject matter. Additional knowledge-building games and activities are found in Collins and Ferguson (1993) and in Jonassen, Bessner, and Yacci (1993). Domain-specific games take on very specific forms, for example, designing a research study or developing a timeline for a project. As games become more domain-specific, they typically become more valuable to participants of that work area.

--------------------------------------------------

--------------------------------------------------

CATALOGUE OF GAMES

STRUCTURAL ANALYSIS GAMES

List. Make a list of answers to a specific question.

Spatial decomposition. Break an entity down into non-overlapping parts and specify topographical relations between them.

Temporal decomposition. Make a list of sequential stages of a process.

Compare & contrast. Compare salient features of entities.

Cost-benefit. Identify the pros and cons of choices.

Primitive elements. Characterize a set of phenomena by their makeup or component elements.

Cross products. Compare listed items across a set of dimensions or attributes.

Axiom systems. Diagram the relationships between a set of formulae and their rules of inference.

FUNCTIONAL ANALYSIS GAMES

Critical-event. Identify causes leading to an event, or the consequences derived from an event.

Cause & effect. Use critical-event analysis, distinguishing between causes and preconditions. Each effect of a cause can become a cause of a new effect.

Problem-centered. Break an event stream into problems and actions taken to solve them. The side effects of the solution may cause new problems.

AND/OR graphs. Create a causal chain diagram showing the logical AND and OR relationships between links in the chain.

Form-and-function. Distinguish between an objects structure and its purpose.

PROCESS ANALYSIS GAMES

Systems-dynamics. Model a system showing how the contributory variables increase and decrease, and how they affect the system via feedback.

Aggregate behavior. Model a system showing how the interactive events between the components affect the behavior of the system.

Constraints. Model a system by creating a set of equations which describe system behavior.

Situation-action. Model a situation by a set of rules to apply in various cases. The situation can change because the world changes, or an agent takes action.

Trend/cyclical. Model the relationships between variables by showing how each changes over a period of time. Variable behavior can be linear, exponential, cyclical or growth.

-------------------------------------------------- Table 2. An outline of several types of epistemic games (from Collins & Ferguson, 1993).

Morrison and Collins (in press) argue the following:

1. Our culture supports numerous ways of constructing knowledge--some domain-specific, and some more general.

2. These different ways of constructing knowledge, which we call epistemic games, are culturally patterned.

3. Different contexts (communities of practice) support different ways of knowing, and therefore different kinds of epistemic games. People are more or less fluent epistemically, depending largely on their contextual experiences, i.e., the sorts of subcultures and communities of practice in which they have participated.

4. An important goal of school is to help people become epistemically fluent, i.e., be able to use and recognize a relatively large number of epistemic games.

5. A key question to ask about particular environments is whether they tend to foster (or inhibit) epistemic fluency.... (Morrison & Collins, in press)

The epistemic-game framework can serve as a language for describing learning activities within constructivist learning environments (Grabinger, this volume; Wilson, in press).

According to Collins and Ferguson (1993, pp. 27-28), the playing of epistemic games exhibits the following characteristics:

1. There are constraints to playing. In playing a list game, for example, the items listed should be similar (that is, on the same scale or level) and yet distinct from one another. The lists should be comprehensive in their coverage (that is, leaving nothing important out), yet brief and succinct. These constraints can serve as the rules or criteria we use to judge the quality or appropriateness of a new list.

2. There are entry conditions that define when and where game-playing is appropriate and worthwhile. The list game, for example, becomes appropriate in response to a question such as "What is involved in X" or "What is the nature of X?" where X can be decomposed or analyzed in simple fashion.

3. Allowable moves are the actions appropriate during the course of the game. List moves include adding a new item, consolidating two items, and rejecting or removing an item from consideration.

4. Players occasionally may transfer from one game to another. For example, a list game may shift to a hierarchy game when the structure of list elements begins to assume a form containing sub-categories.

5. Game playing results in the generation of a defined epistemic form, e.g., lists, hierarchies, processes, etc.

The Collins framework of epistemic games and forms provides a structure and language to articulate what learning communities do when they work together to generate new knowledge. Such a framework can become useful to understanding classroom and workgroup processes, but it also can serve a prescriptive or heuristic role for teachers and designers. Many teachers complain that they want to teach critical thinking, but have failed to find a suitable set of strategies.

Epistemic games can be useful to teachers in either of two ways:

1. Using the framework as a diagnostic or interpretive device. Existing learning activities can be interpreted from an epistemic-game perspective, providing valuable insights into processes and interactions.

2. Targeting game-playing as a learning objective. Our students (Trigg & Sherry, 1995) are presently developing learning materials aimed at teachers, encouraging them to engage students directly in epistemic games.

While empirical research in this area is only in beginning stages, epistemic game-playing seems a promising way to think about knowledge-generating activities. It provides a needed link between cultural forms and cognitive-epistemic points of view. Research must address many questions, such as the extent to which the games encourage knowledge generation rather than rote learning, and the types and amount of scaffolding that are desirable in various learning situations. One could imagine students mindlessly developing a list or guessing at causes where no new knowledge was generated. Rules need to be developed for playing games in a way that is conducive to knowledge generation.

TABLETOP

Tabletop (Hancock & Kaput, 1990a; Hancock & Kaput, 1990 b; Hancock, Kaput, & Goldsmith, 1992; Kaput & Hancock, 1991) is a computer-based tool that allows users to manipulate numerical data sets. By combining features of what Perkins (1992a) calls symbol pads and construction kits, Tabletop provides a "general purpose environment for building, exploring, and analyzing databases" (Hancock et al., 1992, p. 340).

The program allows the user to construct a conventional row-and-column database and then to manipulate the data by imposing constraints on the data with animated icons. Double-clicking on an icon displays the complete record for that icon. Summary computations can be represented in a variety of formats, including scatter plots, histograms, cross tabulations, Venn diagrams, and other graphs.

Tabletop is a product of two major design goals: intelligibility and informativeness. Hancock et al. (1992) compare the role of the individual icons, which allow the learner to identify with them physically, to the role of the Turtle in Papert's Logo programming language. They theorize that the icons provide a "pivotal representation in which kinesthetic/individual understanding can…be enlisted as a foundation for developing visual/aggregate-understanding" (p. 346).

Intelligibility and accessibility are also supported by other aspects of the program. For example, the user constructs and can modify graphs through a series of reversible steps and always has the full database in view. Thus, the user can observe the effect each new constraint has on each member of the database (a feature which also contributes to the program's informativeness).

The intelligibility and informativeness of the program support the learner in negotiating meaning in a real-world iterative process of construction, question-asking, and interpretation. Hancock et al. (1992) describe a case in which a student used Tabletop to graph a hypothesis about data which had not yet been entered in the database.

Tabletop was initially piloted on students aged 8 to 15, and with students aged 11 to 18. The pilot studies provided insight into the kinds of questions, problems and thinking processes that students engage in during all phases of data modeling and confirmed Hancock et al.'s (1992) belief that data creation and data analysis are inextricably intertwined. The description of the thinking students engaged in clearly reveals that in the

"data definition phase" the students were drawing on the "raw data" of their individual experiences.

Tabletop was clinically tested on an eighth-grade class and a combined fifth- and sixth-grade class in six units during one school year. Field observation clearly demonstrated that Tabletop can help students develop their understanding of many kinds of graphs. However, students were less successful in:

a. using that graph to characterize group trends;

b. constructing the graph in order to generate, confirm, or disconfirm a hypothesis;

c. connecting the graph with the data structures necessary to produce it; and

d. embedding the graph in the context of a purposeful, convergent project. (pp. 361-62)

Tabletop is not designed to be a self-contained program for developing skills and concepts in data modeling. Indeed, the developers envision it as a tool in a collaborative learning environment, with students helping each other and receiving appropriate scaffolding and coaching from the teacher, as in a cognitive apprenticeship model. In the pilot and clinical tests of the program, students sometimes were unable to perceive--even with some coaching and scaffolding--that they could not create particular graphs because they had not coded the data in a relevant way. Because of time constraints, the teacher sometimes scaffolded learning by adding a relevant data field "between sessions" (Hancock, Kaput, & Goldsmith, 1992, p. 350). Although the researchers imply that it would have been preferable for the students to discover the solution for themselves, research must still address the issue of whether, when, and how much scaffolding of this sort is beneficial.

As noted above, while Tabletop was generally effective in helping students understand a variety of graphs, it was less effective in helping students use the graphs to support general conclusions. Like Duffy and Roehler (1989), Hancock, Kaput, and Goldsmith (1992) found that learning and incorporating new strategies into one's repertoire requires much time. They concluded that one year was insufficient time for students to develop "authentic, well-reasoned data modeling activities" (p. 353). Even after a year, students' projects lacked coherence and purpose, beginning "without clear questions, and end[ing] without clear answers" (p. 358).

Two lessons can be learned from Tabletop. First, it is an excellent example of a tool that allows ideas and content elements to be manipulated, tested, explored, and reflected upon. Students working with Tabletop have a qualitatively different experience than they would completing exercises at the back of the chapter. Second, students using these kinds of tools need well designed supports, meaningful goals and projects, and attentive teachers to realize the tool's potential. Even when conditions are favorable and care is given to design and support, students cannot be expected to reach higher levels of schema acquisition and problem-solving skill simply by having experience with the tool. Learning environments that allow projects, data manipulation, and exploration require continuing attention to

design in order for students to achieve learning gains (see Jonassen, 1995 for a discussion of other tools useful to learning communities).

COMPUTER-SUPPORTED INTENTIONAL LEARNING ENVIRONMENTS (CSILE)

Scardamalia, Bereiter, McLean, Swallow, and Woodruff (1989) observe:

There has been a history of attempts in computer-assisted instruction to give students more autonomy or more control over the course of instruction. Usually these attempts presupposed a well developed repertoire of learning strategies, skills, and goals, without providing means to foster them. (p. 51)

Scardamalia and Bereiter envision a computer-based learning environment wherein students can learn and exercise these metacognitive skills, giving the name computer-supported intentional learning environments to "environments that foster rather than presuppose the ability of students to exert intentional control over their own learning..." (Scardamalia et al., 1989, p. 52). In a series of studies, Scardamalia & Bereiter (1992) found that children were capable of generating impressive higher-order questions about a new subject, based upon their interest and background knowledge. These questions could then be used to guide students' research and exploration of the topic. Intentional learning environments are designed to support the high-level, knowledge-generating activity resulting from this question-asking process.

The authors have developed a model computer program referred to as "CSILE" (for computer-supported intentional learning environment), the acronym denoting the specific program developed in the laboratory. While CSILE exhibits a number of design features, for space reasons we focus on the program's foundation and philosophy. In an early report, Scardamalia et al. (1989) suggested eleven principles that should guide the design of intentional learning environments:

1. Make knowledge-construction activities overt.

2. Maintain attention to cognitive goals.

3. Treat knowledge deficits in a positive way.

4. Provide process-relevant feedback.

5. Encourage learning strategies other than rehearsal.

6. Encourage multiple passes through information.

7. Support varied ways for students to organize their knowledge.

8. Encourage maximum use and examination of existing knowledge.

9. Provide opportunities for reflection and individual learning styles.

10. Facilitate transfer of knowledge across contexts.

11. Give students more responsibility for contributing to each other's learning.

While many of these principles also are similar to traditional instructional-design prescriptions (e.g., Reigeluth, 1983), there is additional emphasis on knowledge construction consistent with the cognitive apprenticeship model and other constructivist learning theories.

More recently, Scardamalia and Bereiter describe three ideas that are fundamental to intentional learning environments:

1. Intentional learning. Ng and Bereiter (1991) observed students learning computer programming and found three kinds of goals:

--performance goals, i.e., task completion goals;

--instructional goals, i.e., the goals articulated by the instructor and the learning materials;

--learning goals, i.e., the specific goals for learning brought to the situation by the learner. Learning goals usually overlap but do not equate to performance or instructional goals.

Intentional learning depends upon students having learning goals and finding successful avenues to learn based upon those goals.

2. The process of expertise. Scardamalia and Bereiter argue for a view of expertise in process terms rather than strictly as a performance capability. As people gain experience in a domain, simple tasks become routinized, freeing up mental resources for other tasks. If those newly available resources are reinvested back into learning more about the domain, then more and more difficult problems can be mastered. This process of expertise is equally characteristic of serious students and seasoned experts "working at the edges of their competence" (Scardamalia & Bereiter, 1994, p. 266). To become experts, students must demonstrate a disposition and commitment to engage in systematic intentional learning, in addition to having the brute cognitive capacity to learn.

3. Restructuring schools as knowledge-building communities. Scardamalia and Bereiter (1994) contrast what we call static from dynamic learning communities. Members of static communities must adapt to the environment, but once adapted, "one becomes an old timer, comfortably integrated into a relatively stable system of routines..." (pp. 266-267). Traditional schools and even child-centered, individualized instruction are often static in this sense. In contrast, a dynamic learning community requires constant readaptation to other community members. Sports and businesses are examples. "[T]he accomplishments of participants keep raising the standard that the others strive for" (p. 267). In the sciences, for example, the collective knowledge base is continually changing. The challenge in these

environments is to continue growing, adapting, and contributing along with the rest of the community. Dynamic knowledge-building communities engage in the kind of transformative communication suggested by Pea (1994; see discussion above). In large part, the goal of the CSILE research agenda is to find ways to help classrooms and schools become dynamic knowledge-building communities in this respect.

Central to intentional learning environments is the cultivation of a collective knowledge base, explicitly represented in CSILE as a computer database. This knowledge base allows the creation and storage of numerous forms of representation--text, graphics, video, audio--as well as the linking of items together via a hypermedia structure. The knowledge base is built up over a period of time in response to students' questions and their subsequent investigations and reports.

Another key feature of CSILE is the "publication" process, similar to the review process of academic journals:

Students produce notes of various kinds and frequently revise them. When they think they have a note that makes a solid contribution to the knowledge base in some area, they can mark it as a candidate for publication. They then must complete a form that indicates, among other things, what they believe is the distinctive contribution of their note. After a review process (typically by other students with final clearance by the teacher), the note becomes identified as published. It appears in a different font, and users searching the database may, if they wish, restrict their search to published notes on the topic they designate. (Scardamalia & Bereiter, 1994, p. 279)

Thus CSILE emulates in many respects the activities of scholarly knowledge-building communities. Attempts at applying a CSILE-like model to higher education classrooms are reported in Grabinger (this volume).

Like problem-based learning, the CSILE model provides a concrete framework for designers and teachers seeking to break out of traditional conventions and incorporate constructivist principles of instructional design. Two remaining issues for consideration are:

--Matching intentional learning activities to pre-specified curriculum objectives. Every learning environment exists within a larger system of curriculum expectations and learning needs. A student may want to study X while the teacher thinks that Y would be a better choice. Meanwhile, the school district has a policy insisting on Z as the proper content. Negotiating between student-generated study questions and the surrounding system is an important consideration.

--Maintaining motivation. As in every learning environment, designers of intentional environments must develop methods for encouraging thoughtful collaboration while avoiding the damaging effects of competition. Cultivating a cooperative, open spirit among participants requires attention to group dynamics and the chemistry between individuals and within working groups. Maintaining motivation could be a challenge within an environment of widely diverging competencies and expectations.

CONCLUSION

In our previous review of cognitive teaching models (Wilson & Cole, 1991), we were surprised by the diversity of approach and method. In this review, key differences again should be acknowledged between the various models. Cognitive load theory adopts a no-nonsense approach to the efficient and effective teaching of defined content. The packaged computer-based learning environments (Sherlock and the case-based scenarios) are highly effective, controlled environments that filter out much of the world's complexity and provide learners with an authentic-enough environment conducive to learning. The problem-based learning model is simpler in design yet more ambitious in the sense that it departs more radically from established instructional methods. Finally, the tools and models related to learning communities become almost anti-models in the sense that so much is left to the participants, both instructors and students.

In view of the substantial differences between models, generalizing across them is a difficult task. Several key points of reflection, however, are offered below.

Design and implementation are inseparable. There is no doubt that development and tryout of coherent models can yield important outcome information. Knowing, for example, that reciprocal teaching produces, on average, effect sizes of .32 on standardized tests and .88 on locally developed measures (Rosenshine & Meister, 1994), conveys a sense of confidence and reliability for users of the method. At the same time, some of the most valuable lessons learned may come from the real-world experience gained in setting up and administering a program. The implementation can be just as important as the theory-guided design.

An example may be taken from the 1970s research in computer-assisted instruction. One program, the TICCIT project, was shown by an NSF evaluation to achieve its objectives more successfully than traditional classroom instruction (Merrill, Schneider, & Fletcher, 1979). The program failed, however, in getting students to stay with the program; the dropout rate was unacceptably high when compared to traditional classrooms. In this case, the actual development and tryout of working models produced knowledge that would not have been anticipated ahead of time.

Another example is Clancey's Guidon-Manage research in intelligent tutoring systems or ITS (1993). In a remarkable example of self-reflection, Clancey concludes: "After more than a decade, I felt that I could no longer continue saying that I was developing instructional programs for medicine because not a single program I worked on was in routine use..." (p. 7). What did Clancey learn from his research? Apart from his contribution to intelligent tutoring technologies, he learned that research in a laboratory differs from research in the field. "[R]esearchers must participate in the community they wish to influence... (p. 9, italics retained). "As ITS matures, some members of our research community must necessarily broaden their goals from developing representational tools to changing practice--changing how people interact and changing their lives... (p. 9, italics retained). Clancey then reflects on how he might approach the Guidon-Manage research differently today:

--participating with users in multidisciplinary design teams versus viewing teachers and students as my subjects,

--adopting a global view of the context...instead of delivering a program in a...box,

--being committed to provide cost-effective solutions for real problems versus imposing my research agenda on another community,

--facilitating conversations between people versus only automating human roles,...

--relating...ITS computer systems to...everyday practice...versus viewing models...as constituting the essence of expert knowledge that is to be transferred to a student, and

--viewing the group as a psychological unit versus modeling only individual behavior. (Clancey, 1993, p. 17)

Although the specific research agenda is different, the lessons Clancey learned apply very well to cognitive teaching models. Developers of teaching models need to stay close to the context of use and include implementation within their domain of interest.

Like the TICCIT and Guidon projects, outcomes of research are sometimes negative, as happened in the failure of young students to learn abstract concepts via TableTop, or the tendency for some experienced medical students to become bored with PBL activities. We believe that these negative findings can become extremely useful as formative evaluation data, feeding back into future implementations of the model.

Norman (1993) speaks of the power of representation as a stimulus to scientific progress. Repeatedly in the history of science, revolutionary strides are made when a new technology is developed that allows a repicturing of problems in a domain. By analogy, a similar kind of progress is made possible by research on teaching models. Through the careful articulation and construction of actual working methods, new perspectives are made possible. An actual product, once created, may be examined from a variety of angles and for a variety of purposes, many perhaps unintended by the creator. Determining exactly "what is learned" from research of this type may be difficult to articulate yet remain extremely valuable to the scientific and practitioner communities.

Choosing a model is closely tied to the curriculum question. Perkins (1992b) warns of a common fallacy implied by the statement: "What we need is a new and better method. If only we had improved ways of inculcating knowledge or inducing youngsters to learn, we would attain the precise...outcomes we cherish" (p. 44). Instead, Perkins believes that "given reasonably sound methods, the most powerful choice we can make concerns not method but curriculum--not how we teach but what we choose to try to teach" (p. 44). This comment suggests that a fundamental step in instructional design involves the serious consideration of learning goals. A variety of constituencies should be included in this process, including sponsors and members of the learning community itself. Once consensus

is reached about the kind of learning being sought, certain teaching models become unfeasible while others become more attractive.

A basic lesson learned from observing schools is that two teachers may be covering the same ostensive curriculum while what really is taught differs radically between them. And what any two students learn in the same teacher's class may differ just as radically. At its base, the constructivist movement in education involves curriculum reform, a rethinking of what it means to know something. A constructivist curriculum is reflected in many of the models reviewed in this chapter. Thus, if a commitment is made toward rethinking curriculum to expand the roles of knowledge construction and learning communities, then a corresponding commitment needs to be made in rethinking learning activities. Deciding upon a teaching model is not a value-neutral activity. Recognizing this puts the selection of a teaching model squarely into the political realm of policymaking. New issues become important, such as access, equity, representation, voice, and achieving consensus amid diverse perspectives.

As Reigeluth (1983) acknowledges, curriculum and instruction cannot be completely separated. There is a tendency among many institutions to give lip service to higher-order outcomes while maintaining teaching methods that specifically suppress such outcomes. Medical schools that teach students to simply memorize and take tests are an example. Another example is a military school whose mission statement prizes "creativity" in students, yet whose teaching methods and authoritarian culture strictly reinforce conformity and transmission of content.

Deciding upon a teaching model and making decisions within that framework is a highly situated activity. The success of a given implementation will depend more on the local variables than on the general variables contained in the various models described above. Put another way, "the devil is in the details." There is a way to succeed and a way to fail using a whole host of teaching models. All the models reviewed above can succeed if properly implemented. Teachers and students must see the sense of what they are doing, come to believe in the efficacy of the program, and work hard to ensure that the right outcomes are achieved.

This situational perspective conflicts with traditional views. Thinking of instructional design as a technology would lead us to think that a situation gets analyzed, which leads to a technical fix to be implemented, which leads either to a measured solution to the problem or a revision in the fix for the next cycle of intervention. A situated view of instructional design would lead to a different process:

1. A learning community examines and negotiates its own values, desired outcomes, and acceptable conventions and practices.

2. The learning community plans for and engages in knowledge-generating activities within the established framework of goals, conventions, and practices.

3. Members of the learning community, including both teachers and students, observe and monitor learning and make needed adjustments to support each other in their learning activities.

4. Participants occasionally re-examine negotiated learning goals and activities for the purpose of improving learning and maintaining a vital community of motivated learners. This may lead to new goals and methods and cultural changes at all levels, from cosmetic to foundational.

This situated, community-oriented view of instruction takes a more holistic view to the design of instruction. The community is opportunistic in addressing "design" issues at any stage of planning and implementation. Community members, including students, have a voice in determining what happens to them in instruction. In return, they must show the needed commitment and disposition to behave responsibly and in support of learning.

If community members have participated in the establishment of a program, they are more likely to believe in it. If they believe in the program, the chances of success increase dramatically. As Perkins (1992b) suggests, even very imperfect instructional methods can work if the commitment is made to work together and ask the right questions in designing curriculum.

Each teaching model is a particular blend of costs and outcomes. Some kind of costs-benefits analysis--implicit or explicit--happens in designing educational programs. Surveying our teaching models reveals that some have demonstrably high development costs. Sherlock has taken a decade of patient research to achieve its present form. Once developed, however, the prototype model may be replicated at a reasonable cost. Other models, such as problem-based learning, may pose heavy demands in terms of time in the curriculum. Instructional designers (or learning-community members) must then face the questions of how and whether to implement such resource-demanding teaching methods into an existing system and curriculum.

Every decision to adopt one teaching model over another involves such weighing of pros and cons. However, while costs may be objectively measured and estimated, learning benefits are notoriously difficult to reduce down to a number. This inequity of measurability results in a common bias: The cost differences become exaggerated while the potential benefits, because they are harder to measure, tend to be undervalued or ignored. Comparison of alternative teaching models must give full consideration to qualitative differences in learning outcomes in addition to the more visible cost differences in time and money.

Some ideas may be borrowed and inexpensively incorporated into related products or programs. For example, if an instructor becomes excited by Schank's case-based scenarios, she may choose to incorporate case histories and classroom simulations into her teaching. While the resulting lessons may bear only a passing resemblance to the computer-based scenarios, they are heavily influenced by Schank's principles of case-based, interactive instruction. Many of the principles discussed above, including those of cognitive

apprenticeships and intentional learning communities, can be efficiently adapted into instruction in a number of ways, depending on local circumstances and resources.

Instruction should support learners as they become efficient in procedural performance and deliberate in their self-reflection and understanding. Virtually all of the teaching models under review--Sweller's research notwithstanding--emphasize the grounding of instruction in complex problems, cases, or performance opportunities. Yet organizing instruction around problems and cases should not mask the importance of perception, reflection, and metacognitive activity. Indeed, these two aspects of human performance (problem solving and perception) can be seen as inherently complementary and equally necessary. Contrary to the suggestion of Dreyfus and Dreyfus (1986), experts are more than mere automatic problem-solvers. Rather, experts become experts through a progressive series of encounters with the domain, each involving an element of routine performance and a corresponding element of reflection and deliberation. This is the process of expertise spoken of by Scardamalia and Bereiter (1994) and discussed above.

Prawat (1993) makes this point well. While there is a tendency among cognitive psychologists to make problem solving central to all cognition, Prawat reminds us that schemas, ideas, and perceptual processes hold an equally important place. Learning how to see --from a variety of points of view--is as important as solving a problem once we do see. Principles of perception, whether from ecological psychology (Allen and Ott, this volume), connectionism, or aesthetics, need to have a place within successful teaching models. This includes teaching students how to represent problems and situations, but also how to appreciate and respond to the aesthetic side of the subject, how to reflect upon one's actions, and how to "raise one's consciousness" and recognize recurring themes and patterns in behavior and interactions.

Successful programs must seek to make complex performance do-able while avoiding the pitfalls of simplistic proceduralization. The art of "scaffolding" complex performance is a key problem area that surprisingly is still not well understood. How does a coach entice a young gymnast to perform just beyond her capacities, overcoming the fear and uncertainty that normally accompany new performances? How does the coach know just when and where to step in, preserving the integrity of the task (and the learning) while not letting the athlete fall on her head? These are questions of appropriate scaffolding or support for learning. Once a teacher begins believing the constructivist agenda and the importance of authentic, meaningful tasks, then the challenge of supporting novice performance within a complex environment becomes a central concern. As Sweller's research makes clear, poorly supported problem-solving activities force learners to rely on weak methods that they already know. Appropriate and wise scaffolding makes problem-solving activities more efficient because learners stay focused within the critical "development" zone between previously mastered knowledge and skills beyond their reach (Vygotsky, 1978). Developing a technology for optimizing this kind of support is an area in need of further research and development.

This same concept of scaffolding can be directed to the implementation of the teaching model itself. Instructional designers and teachers need proper supports and aids in designing according to a particular model or tradition. At the same time, they should be

cautioned against simplistically "applying" a model in a proceduralized or objectivist fashion. Postmodernists would say that in such cases, the model "does violence" to the situation. The complexities of a situation should not be reduced down to the simple maxims of a teaching model. Any model that is forced upon a situation and made to fit, will lead inevitably to unintended negative consequences. The negative fallout will happen at those points of disjuncture or lack of fit between model and situation. As we have stressed, the details of the situation need to be respected and taken into account when adapting a model to a situation.

This, perhaps, is a more appropriate way of thinking about implementation: Rather than applying a particular teaching model, a teacher necessarily adapts that model to present circumstances. Learning how to adapt abstractions to concrete realities is a worthy task for both students and teachers, and indeed, may lie at the heart of some forms of expertise.

Each of these points is worthy of continued research. As psychologists continue to develop models for teaching that embody their best thinking and theories, the field of instructional design will have opportunities for reflection and growth as they re-examine their own models and methods. Our hope is that the dialogue may continue to flourish and expand, resulting in a "transformation" of both communities.

REFERENCES

Anderson, J. R. (1987). Methodologies for studying human knowledge. Behavioral and Brain Science, 10, 467-477.

Barrows, H. S. (1985). How to design a problem-based curriculum for the preclinical years. New York: Springer.

Barrows, H. S. (1992). The tutorial process. Springfield, IL: Southern Illinois University School of Medicine.

Barrows, H. S., & Feltovich, P. J. (1987). The clinical reasoning process. Journal of Medical Education, 21, 86-91.

Bell, B., Bareiss, R., & Beckwtih, R. (1993/94). Sickle Cell Counselor: A prototype goal-based scenario for instruction in a museum environment. The Journal of the Learning Sciences, 3 (4), 347-386.

Bereiter, C., & Scardamalia, M. (1992). Cognition and curriculum. In P. Jackson (Ed.), Handbook of Research on Curriculum (pp. 517-542). New York: MacMillan.

Brown, A. L. (1994). The advancement of learning. Educational Researcher, 23 (3), 4-12.

Brown, A. L., & Campione, J. C. (1981). Inducing flexible thinking: A problem of access. In M. Friedman, J. P. Das, & N. O'Connor (Eds.), Intelligence and learning (pp. 515-529). New York: Plenum.

Brown, A. L., Campione, J. C., & Day, J. D. (1981). Learning to learn: On training students to learn from texts. Educational Researcher, 10 (2), 14-21.

Brown, A. L., & Palincsar, A. S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Erlbaum.

Carroll, W. M. (1994). Using worked examples as an instructional support in the algebra classroom. Journal of Educational Psychology, 86 (3), 360-367.

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293-332.

Chi, M. T. H., de Leeuw, N., Chiu, M-H, & LaVancher, C. (1991). The use of self-explanations as a learning tool. Pittsburgh, PA: The Learning Research and Development Center, University of Pittsburgh.

Chi, M., & VanLehn, K. A. (1991). The content of physics self-explanations. The Journal of the Learning Sciences, 1(1), 69-105.

Chi, M. T. H., & Bassok, M. (1989). Learning from examples via self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 251-282). Hillsdale NJ: Erlbaum.

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Chi, M. T. H., de Leeuw, N., Chiu, M-H, & LaVancher, C. (1991). The use of self-explanations as a learning tool. Pittsburgh PA: The Learning Research and Development Center, University of Pittsburgh.

Chi, M., & VanLehn, K. A. (1991). The content of physics self-explanations. The Journal of the Learning Sciences, 1(1), 69-105.

Clancey, W. J. (1993). Guidon-Manage revisited: A socio-technical systems approach. Journal of Artificial Intelligence in Education, 4 (1), 5-34.

Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Erlbaum.

Collins, A., & Ferguson, W. (1993). Epistemic forms and epistemic games: Structures and strategies to guide inquiry. Educational Psychologist, 28 (1), 25-42.

Collins, A., & Stevens, A. L. (1983). A cognitive theory of inquiry teaching. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 247-278). Hillsdale NJ: Erlbaum.

Conklin, J. (1987). Hypertext: A survey and introduction. IEEE Computer, 20(9), 17-41.

Dick, W. (1987). A history of instructional design and its impact on educational psychology. In J. A. Glover, R. R. Ronning (Eds.), Historical foundations of educational psychology (pp. 183-200). New York: Plenum Press.

Distlehorst, L. H., & Barrows, H. S. (1982). A new tool for problem-based self-directed learning, Journal of Medical Education, 57, 466-488.

Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York: The Free Press.

Duffy, T. M. (in press). Strategic teaching framework: An instructional model for learning complex, interactive skills. In C. Dills & A. Romiszowski (Eds.), Perspectives on instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Duffy, T. M. (1994, October 22). Workshop on Problem-based Learning. Denver CO: University of Colorado at Denver.

Duffy, T. M., & Jonassen, D. H. (Eds.). (1993).Constructivism and the technology of instruction: A conversation . Hillsdale NJ: Erlbaum.

Duffy, G. G., & Roehler, L. R. (1989). Why strategy instruction is so difficult and what we need to do about it. In C. McCormick, G. Miller, & M. Pressley (Eds.), Cognitive strategy research: From basic research to educational applications (pp. 133-154). New York: Springer-Verlag.

Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34, 906-911.

Gagné, R. M. (1968). Learning hierarchies. Presidential Address of APA Division 15. Educational Psychologist, 6 (1), 1-9.

Gagné, R. M. (Ed.). (1987). Instructional technology: Foundations. Hillsdale NJ: Erlbaum.

Glaser, R., Lesgold, A., & Lajoie, S. P. (1987). Toward a cognitive theory for the measurement of achievement. In R. Ronning, J. Glover, J. C. Conoley, & J. C. Witt (Eds.), The influence of cognitive psychology on testing, Buros/Nebrasks Symposium on Measurement: Vol. 3 (pp. 41-85). Hillsdale NJ: Erlbaum.

Glaser, R. (1990). The reemergence of learning theory within instructional research. American Psychologist, 45 (1), 29-39.

Glaser, R. (1976). Components of a psychology of instruction: Toward a science of design. Review of Educational Research, 46, 1-24.

Gott, S. P. (1988a, April). The lost apprenticeship: A challenge for cognitive science. Paper presented at the meeting of the American Educational Research Association, New Orleans.

Gott, S. P. (1988b). Apprenticeship instruction for real-world tasks: The coordination of procedures, mental models, and strategies. In E. Z. Rothkopf (Ed.), Review of Research in Education, 15, 97-169.

Gott, S. P., Hall, E. P., Pokorny, R. A., Dibble, E., & Glaser, R. (1992). A naturalistic study of transfer: Adaptive expertise in technical domains. In D. K. Detterman & R. J. Sternberg (Eds.). Transfer on trial: Intelligence, cognition, and instruction. (pp. 258-288). Norwood NJ: Ablex Publishing Corp.

Gott, S. P., Lesgold, A., & Kane, R. S. (in press). Tutoring for transfer of technical competence. In B. G. Wilson (Ed.), Constructivist learning environments: Cases studies in instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Hancock, C. M., & Kaput, J. (1990a). Annual report of hands on data (NSF MDR-#8855617). Cambridge, MA: TERC.

Hancock, C. M., & Kaput, J. (1990b). Computerized tools and the process of data modeling. In G. Booker, P. Cobb, & T. N. deMendicuti (Eds.), Proceedings of the 14th International Conference on the Psychology of Mathematics Education (Vol., 3, pp. 65-172). Mexico: The Program Committee of the 14th PME Conference.

Hancock, J. J., Kaput, L. T., & Goldsmith, L. T. (1992). Authentic inquiry with data: critical barriers to classroom implementation. Educational Psychologist, 27(3), 337-364.

Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking. Columbus OH: Charles Merrill.

Jonassen, D. H., Bessner, K., & Yacci, M. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge. Hillsdale NJ: Erlbaum.

Kaput, J. J., & Hancock, C. M. (1991). Cognitive issues in translating between semantic structure and formal record structure. In F. Furinghetti (Ed.), Proceedings of the 15th International Conference on the Psychology of Mathematics Education (Vol. 2, pp. 237-244). Genova, Italy: Dipartimento di Matematica dell' Università de Genova.

Kass, A., Burke, R., Blevis, E., & Williamson, M. (1993/94). Constructing learning environments for complex social skills. The Journal of the Learning Sciences, 3 (4), 387-427.

Koschmann, T. D. (1994). Toward a theory of computer support for collaborative learning. The Journal of the Learning Sciences, 3 (3), 219-225.

Koschmann, T. D., Myers, A. C., Feltovich, P. J., & Barrows. (1994). Using technology to assist in realizing effective learning and instruction: A principled approach to the use of computers in collaborative learning. The Journal of the Learning Sciences, 3 (3), 227-264.

Lajoie, S. P., Lesgold, A. M. (1992). Dynamic assessment of proficiency for solving procedural knowledge tasks. Educational Psychologist, 27(3), 365-384.

Lajoie, S. P., & Lesgold, A. (1989). Apprenticeship training in the workplace: Computer coached practice environment as a new form of apprenticeship. Machine-Mediate Learning, 3, 7-28.

Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday life. Cambridge UK: Cambridge University Press.

Lesgold, A., Lajoie, S., Bunzo, M., & Eggan, G. (1992). A coached practice environment for an electronics troubleshooting job. In J. Larkin & R. Chabay (Eds.), Computer assisted instruction and intelligent tutoring systems: Establishing communications and collaboration (pp. 201-238). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Lesh, R., Landau, M., & Hamilton, E. (1983). Conceptual models and applied mathematical problem solving research. In R. Lesh & M. Landau (Eds.), Acquisition of mathematical concepts and processes (pp. 263-343). New York: Academic Press.

Lewis, M. W., Milson, R., & Anderson, J. R. (1988). Designing an intelligent authoring system for high school mathematics ICAI: The teacher apprentice project. In G. P. Kearsley (Ed.), Artificial intelligence and instruction: Applications and methods (pp. 269-301). New York: Addison-Wesley.

Lipkin, M. (1989). Education of doctors who care. In H. G. Schmidt, M. Lipkin, Jr., M. W. de Vries, & J. M. Greep (Eds.), New directions for medical education (pp. 3-16). New York: Springer-Verlag.

Low, W. C. (1981). Changes in instructional development: The aftermath of an information processing takeover in psychology. Journal of Instructional Development, 4(2), 10-18.

Lumsdaine, A. A., & Glaser, R. (1960). Teaching machines and programmed learning. Washington D. C.: National Educational Association.

Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83 (4), 484-490.

Mayer, R. E., & Anderson, R. B. (1992). The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84 (4), 444-452.

Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86 (3), 389-401.

Merrill, M. D. (1983). Component display theory. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 282-333). Hillsdale, NJ: Lawrence Erlbaum Associates.

Merrill, M. D., Kowallis, T., & Wilson, B. G. (1981). Instructional design in transition. In F. Farley & N. Gordon (Eds.), Psychology and education: The state of the union. Chicago: McCutcheon.

Merrill, M. D., Schneider, E., & Fletcher, K. (1979). TICCIT. Englewood Cliffs NJ: Educational Technology Publications.

Merrill, M. D., Wilson, B. G., & Kelety, J. C. (1981). Elaboration theory and cognitive psychology. Instructional Science, 10, 217-235.

Merrill, M. D., & Tennyson, R. (1977). Teaching concepts: An instructional design guide (1st ed.). Englewood Cliffs NJ: Educational Technology Publications.

Morrison, D., & Collins, A. (1995). Epistemic fluency and constructivist learning environments. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Ng, E., & Bereiter, C. (1991). Three levels of goal orientation in learning. The Journal of the Learning Sciences, 1 (3), 243-271.

Nichols, P., Poknorny, R., Jones, G., Gott, S. P., & Alley, W. E. (in press). Evaluation of an avionics troubleshooting tutoring system. AL/HR TR-94-XX. Brooks AFB TX.

Paas, F. G. W. C., & Van Merriënboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86 (1), 122-133.

Pea, R. D. (1994). Seeing what we build together: Distributed multimedia learning environments for transformative communications. The Journal of the Learning Sciences, 3 (3), 285-299.

Perkins, D. N. (1992a). Technology meets constructivism: Do they make a marriage? In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction (pp. 45-55). Hillsdale, NJ.

Perkins, D. (1992b). Smart schools: From training memories to educating minds. New York: The Free Press (MacMillan).

Pierce, K. A., Duncan, M. K., Gholson, B., Ray, G. E., & Kamhi, A. G. (1993). Cognitive load, schema acquisition, and procedural adaptation in nonisomorphic analogical transfer. Journal of Educational Psychology, 85 (1), 66-74.

Prawat, R. S. (1993). The value of ideas: Problems versus possibilities in learning. Educational Researcher, 22 (6), 5-16.

Reder, L., & Anderson, J. (1982). Effects of spacing and embellishment on memory for main points of a text. Memory and Cognition, 10, 97-102.

Reigeluth, C. M. (Ed.). (1983). Instructional-design theories and models: An overview of their current status. Hillsdale, NJ: Erlbaum.

Reigeluth, C. M. (Ed.) (1987). Instructional theories in action: Lessons illustrating selected theories and models . Hillsdale, NJ: Erlbaum.

Riesbeck, C. K. (in press). Case-based teaching and constructivism: Carpenters and tools. In B. G. Wilson (Ed.), Constructivist learning environments: Cases studies in instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Resnick, L. B. (1981). Instructional psychology. Annual Review of Psychology, 32, 659-704.

Robin, S., & Mayer, R. E. (1993). Schema training in analogical reasoning. Journal of Educational Psychology, 85 (3), 529-538.

Rosenshine, B., & Meister, C. (1994). Reciprocal teaching: A review of the research. Review of Educational Research, 64 (4), 479-530.

Saunders, R., & Solman, R. (1984). The effect of pictures on the acquisition of a small vocabulary of similar sight-words. British Journal of Educational Psychology, 54, 265-275.

Savery, J. R., & Duffy, T. M. (in press). Problem based learning: An instructional model and its constructivist framework. In B. G. Wilson (Ed.), Constructivist learning environments: Cases studies in instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Scardamalia, M., Bereiter, C., McLean, R. S., Swallow, J., & Woodruff, E. (1989). Computer-supported intentional learning environments. Journal of Educational Computing Research, 5 (1), 51-68.

Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media. The Journal of the Learning Sciences, 1 (1), 37-68.

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3 (3), 265-283.

Schank, R. C., Fano, A., Bell, B., & Jona, M. (1993/1994). The design of goal-based scenarios. The Journal of the Learning Sciences, 3 (4), 305-345.

Schmidt, H. G. (1989). The rationale behind problem-based learning. In H. G. Schmidt, M. Lipkin, Jr., M. W. de Vries, & J. M. Greep (Eds.), New directions for medical education (pp. 105-111). New York: Springer-Verlag.

Schooler, J., & Engstler-Schooler, T. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36-71.

Stefik, M., & Brown, J. S. (1989). Toward portable ideas. In M Olson (Ed.), Technological support for work group collaboration (pp. 147-166). Hillsdale NJ: Lawrence Erlbaum Associates, Inc.

Sweller, J. (1989). Cognitive technology: Some procedures for facilitating learning and problem solving in mathematics and science. Journal of Educational Psychology, 81 (4), 457-466.

Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59-89.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12 (3), 185-233.

Trigg, M., & Sherry, L. (1995). Creating and playing epistemic games: A teacher's guide to knowledge generating activities. Manuscript in preparation.

Vygotsky, L. (l978). Mind and society. Cambridge, MA: Harvard University.

Williams, S. M. (1992). Putting case-based instruction into context: Examples from legal and medical education. The Journal of the Learning Sciences, 2(4), 367-427.

Wilson, B. G. (Ed.). (in press). Constructivist learning environments: Case studies in instructional design. Englewood Cliffs NJ: Educational Technology Publications.

Wilson, B. G., & Cole, P. (1991). A review of cognitive teaching models. Educational Technology Research & Development Journal.

AUTHOR NOTES

Brent Wilson (bwilson@castle.cudenver.edu) is associate professor of instructional technology at the University of Colorado at Denver. His interests include cognition and instruction and the design of constructivist learning environments. Peggy Cole (bwilson@castle.cudenver.edu) is instructor of English at Arapahoe Community College in Littleton Colorado. Her professional interests include constructivist models of instructional design and the use of analogies and journals in instruction. We wish to thank Dr. David Jonassen, a former mentor and colleague, for his continued support and encouragement in the process of completing the chapter.

COGNITIVE TEACHING MODELS Brent Wilson Peggy Cole

UNIVERSITY OF COLORADO AT DENVER

Wilson, B., & Cole, P. (1991). A review of cognitive teaching models. Educational Technology Research and Development, 39 (4), 47-64. Also available at:

http://www.cudenver.edu/~bwilson

Abstract

The purpose of this paper is to review from an instructional-design (ID) point of view nine teaching programs developed by cognitive psychologists over the last ten years. Among these models, Collins' cognitive apprenticeship model has the most explicit prescriptions for instructional design. The paper analyzes the cognitive apprenticeship model, then uses components of the model as an organizing framework for understanding the remaining models. Differences in approach are noted between traditional ID prescriptions and the cognitive teaching models. Surprisingly, we were unable to identify common design strategies common to all of the model programs. Key differences among programs included: (1) problem solving versus skill orientation, (2) detailed versus broad cognitive task analysis, (3) learner versus system control, and (4) error-restricted versus error-driven instruction. The paper concludes by arguing for the utility of continuing dialogue between cognitive psychologists and instructional designers.

The field of instructional design (ID) emerged more than 30 years ago as psychologists and educators searched for effective means of planning and producing instructional systems (Merrill, Kowallis, & Wilson, 1981; Reiser, 1987). Since that time, instructional designers have became more clearly differentiated from instructional psychologists working within a cognitivist tradition (Glaser, 1982; Glaser & Bassok, 1989; Resnick, 1981). ID theorists tend to place priority on developing explicit prescriptions and models for designing instruction while instructional psychologists focus on understanding the learning processes in instructional settings.

Of course, the distinction between designers and psychologists is never clear-cut. Over the years, many ID theorists have explored learning processes, just as many psychologists have put considerable energy into the design and implementation of experimental instructional programs. However, because the two fields support different literature and theory bases, communication is often lacking. Cognitive psychologists have tended to overlook contributions from the ID literature, and likewise much design work of psychologists has gone unnoticed by ID theorists.

(A happy exception to this trend is the pointed dialogue on constructivism in the May and September issues of Educational Technology magazine.)

Collins, Brown, and colleagues (e.g., Collins, 1991; Collins, Brown, & Newman, 1989) have developed an instructional model derived from the metaphor of the apprentice working under the master craftsperson in traditional societies, and from the way people seem to learn in everyday informal environments (Rogoff & Lave, 1984). They have called their model cognitive apprenticeships, and have identified a list of features found in "ideal" learning environments. Instructional strategies, according to the Collins-Brown model, would include modeling, coaching, scaffolding and fading, reflection, and exploration. Additional strategies are offered for representing content, for sequencing, and for maximizing benefits from social interaction.

Of course, many of Collins' recommended strategies resemble strategies found in the ID literature (e.g., Reigeluth, 1983a). Clearly both fields could benefit from improved communication concerning research findings and lessons learned from practical tryout. With that goal in mind, the purpose of this paper is to do an ID review of programs and strategies developed by instructional psychologists, using the cognitive apprenticeship model as a conceptual framework. Several teaching systems employing cognitive apprenticeship ideals are described. The resultant review should prove valuable in two ways: cognitive psychologists should be able to make a better correspondence between their models and current ID theory, hopefully seeing areas needing improvement, and ID theorists also should be able to see correspondences and differences, which may lead to revision or expansion of our current models.

The Need for Cognitive Apprenticeships

The cognitive apprenticeship model rests on a somewhat romantic conception of the "ideal" apprenticeship as a method of becoming a master in a complex domain (Brown, Collins, & Duguid, 1989). In contrast to the classroom context, which tends to remove knowledge from its sphere of use, Collins and colleagues recommend establishing settings where worthwhile problems can be worked with and solved. The need for a problem-solving orientation to education is apparent from the difficulty schools are having in achieving substantial learning outcomes (Resnick, 1989).

Another way to think about the concept of apprenticeship is Gott's (1988a) notion of the "lost apprenticeship," a growing problem in industrial and military settings. She noted the effects of the increased complexity and automation of production systems. First, the need is growing for high levels of expertise in supervising and using automated work systems; correspondingly, the need for entry levels of

expertise is declining. Workers on the job are more and more expected to be flexible problem solvers; human intervention is often most needed at points of breakdown or malfunction. At these points, the expert is called in. Experts, however narrow the domain, do more than apply canned job aids or troubleshooting algorithms; rather, they have internalized considerable knowledge which they can use to flexibly solve problems in real time (Gott, 1988b).

Gott's second observation relates to training opportunities. Now, at a time when more problem-solving expertise is needed due to the complexity of systems, fewer on-the-job training opportunities exist for entry-level workers. There is often little or no chance for beginning workers to acclimatize themselves to the job, and workers very quickly are expected to perform like seasoned professionals. True apprenticeship experiences are becoming relatively rare. Gott calls this dilemma-more complex job requirements with less time on the job to learn-the "lost" apprenticeship, and argues for the critical need for cognitive apprenticeships and simulation-type training to help workers develop greater problem-solving expertise.

A Brief Review of ID Models

It is assumed that readers will have some prior knowledge of ID models and theories; however, we offer a short overview to allow a clear contrast with certain cognitive approaches. ID models come in two generic varieties: procedural models for systems design (e.g., Andrews and Goodson, 1980) and conceptual models that incorporate specific instructional strategies for teaching defined content (Reigeluth, 1983a, 1987). The procedural models often are represented as flowcharts reflecting a series of project phases, progressing from needs and problems analyses to product implementation and maintenance. Procedural ID models depend less on learning theory and more on systems theory and project management methodologies (Branson & Grow, 1987). Of greater interest for our purposes are the instructional-strategy models. All such models are based on Robert Gagné's conditions-of-learning paradigm (Gagné, 1966), which in its time was a significant departure from the Skinnerian operant conditioning paradigm dominant among American psychologists. The conditions-of-learning paradigm posits that a graded hierarchy of learning outcomes exists, and for each desired outcome, a set of conditions exists that leads to learning. Instructional design is a matter of clarifying intended learning outcomes, then matching up appropriate instructional strategies. The designer writes behaviorally specific learning objectives, classifies those objectives according to a taxonomy of learning types, then arranges the instructional conditions to fit the current instructional prescriptions. In this way, designers can design instruction to successfully teach a rule, a psychomotor skill, an attitude, or piece of verbal information.

A related idea within the conditions-of-learning paradigm claims that sequencing of instruction should be based on a hierarchical progression from simple to complex learning outcomes. Gagné developed a technique of constructing learning hierarchies for analyzing skills: A skill is rationally decomposed into parts and sub-parts; then instruction is ordered from simple subskills to the complete skill. Elaboration theory uses content structure (concept, procedure, or principle) as the basis for organizing and sequencing instruction (Reigeluth, Merrill, Wilson, & Spiller, 1980). Both methods depend on task analysis to break down the goals of instruction, then on a method of sequencing proceeding from simple to gradually more complex and complete tasks.

Some of the teaching models being offered by cognitive researchers bear strong resemblance to traditional ID models. Larkin and Chabay (1989), for example, offer design guidelines for the teaching of science in the schools (pp. 160-163):

1. Develop a detailed description of the processes the learner needs to acquire.

2. Systematically address all knowledge included in the description of process.

3. Let most instruction occur through active work on tasks.

4. Give feedback on specific tasks as soon as possible after an error is made.

5. Once is not enough. Let students encounter each knowledge unit several times.

6. Limit demands on students' attention.

By any standard, these design guidelines are very close to the prescriptions found in component display theory, elaboration theory, and Gagné's instructional-design theory. The strong correspondence can be seen as good news for ID theories: Many current cognitive researchers seem to agree on some fundamentals of design that also form the backbone of ID models.

On the other hand, other cognitive teaching models emphasize design elements that traditional ID models historically have under-emphasized, such as learner-initiated inquiry and exploration, social "scaffolding," cooperative learning methods, and empathic feedback (Lepper & Chabay, 1988). Some cognitive teaching models explicitly differentiate themselves from traditional instructional development. Constructivist theorists, for example, have offered alternatives to the conditions-of-learning paradigm. Based on a neo-Piagetian framework, Case (Case, 1978; Case & Bereiter, 1985) has advocated an approach that focuses more explicitly on individual learners' misconceptions and knowledge structures. Bereiter (1991) develops a connectionist argument that suggests that human performance cannot be explained by explicit rules, but rather by a pattern

matching process. Hence, the "family of instructional theories in which rules, definitions, logical operations, explicit procedures, and the like are treated as central (Reigeluth, 1983)...has produced an abundance of technology on an illusory psychological foundation" (p. 15).

Other theorists, influenced by Vygotsky's concept of a zone of proximal development, think of tasks differently from traditional ID. Newman, Griffin, and Cole (1989) characterize traditional design methods:

First, the tasks are ordered from simple or easy to complex or difficult. Second, early tasks make use of skills that are components of later tasks. Third, the learner typically masters each task before moving onto the next. This conception has little to say about teacher-child interaction since its premise is that tasks can be sufficiently broken down into component parts that any single step in the sequence can be achieved with a minimum of instruction. Teacherless computerized classrooms running "skill and drill" programs are coherent with this conception of change. (p. 153)

The authors contrast this task-analysis approach with a more teacher-based approach where simplicity is achieved by the social negotiation between teacher and learner:

The teacher and child start out doing the task together. At first, the teacher is doing most of the task and the child is playing some minor role. Gradually, the child is able to do more and more until finally he can do the task on his own. The teacher's actions in these supportive interactions have often been called "scaffolding,"... suggesting a temporary support that is removed when no longer necessary....There is a sequence involved ...but it is a sequence of different divisions of labor. The task-in the sense of the whole task as negotiated between the teacher and child-remains the same. (p. 153)

Several of the cognitive teaching models reviewed below embody similar constructivist concepts. It is not yet clear, however, precisely how such concepts would translate themselves into guidelines for designing teaching materials. The cognitive apprenticeship model discussed at length below includes some well-worn design elements-such as modeling and fading-and others that are relatively neglected by ID models-such as situated learning, exploration, and the role of tacit knowledge.

FEATURES OF COGNITIVE APPRENTICESHIPS

In this section, the design elements of the cognitive apprenticeship model (based primarily on Collins, 1991) are reviewed and related to traditional ID concepts.

1. Content: Teach tacit, heuristic knowledge as well as textbook knowledge Collins et al. (1989) refer to four kinds of knowledge that differ somewhat from ID taxonomies:

Domain knowledge is the conceptual, factual, and procedural knowledge typically found in textbooks and other instructional materials. This knowledge is important, but often is insufficient to enable students to approach and solve problems independently.

Heuristic strategies are "tricks of the trade" or "rules of thumb" that often help narrow solution paths. Experts usually pick up heuristic knowledge indirectly through repeated problem-solving practice; slower learners usually fail to acquire this subtle knowledge and never develop competence. There is evidence to believe, however, that at least some heuristic knowledge can be made explicit and represented in a teachable form (Chi, Glaser, & Farr, 1988).

Control strategies are required for students to monitor and regulate their problem-solving activity. Control strategies have monitoring, diagnostic, and remedial components; this kind of knowledge is often termed metacognition (Paris & Winograd, 1990).

Learning strategies are strategies for learning; they may be domain, heuristic, or control strategies, aimed at learning. Inquiry teaching to some extent directly models expert learning strategies (Collins & Stevens, 1983).

ID taxonomies (Gagné, 1985; Merrill, 1983) pertain primarily to domain or textbook knowledge, although the distinction is not explicit. Gagné's "cognitive strategies" fits the last three strategies listed by Collins et al. Merrill's "find a principle" or "find a procedure" seems closest to those three strategies. Both Gagné and Merrill are least specific about instruction of this type of learning, although they both acknowledge its importance. They recently have developed a framework for integrating learning goals into more holistic structures (Gagné and Merrill, 1990).

2. Situated learning: Teach knowledge and skills in contexts that reflect the way the knowledge will be useful in real life. Brown, Collins, and Duguid (1989) argue for placing all instruction within "authentic" contexts that mirror real-life problem-solving situations. Collins (1991) is less forceful, moving away from real-life requirements and toward problem-solving situations: For teaching math skills, situated learning could encompass settings "ranging from running a bank or shopping in a grocery store to inventing new theorems or finding new proofs. That is, situated learning can incorporate situations from everyday life to the most theoretical endeavors" (Collins, 1991, p. 122).

Collins differentiates a situated-learning approach from common school approaches:

We teach knowledge in an abstract way in schools now, which leads to strategies, such as depending on the fact that everything in a particular chapter uses a single method, or storing information just long enough to retrieve it for a test. Instead of trying to teach abstract knowledge and how to apply it in contexts (as word problems do), we advocate teaching in multiple contexts and then trying to generate across those contexts. In this way, knowledge becomes both specific and general. (Collins, 1991, p. 123)

Collins cites several benefits for placing instruction within problem-solving contexts:

Learners learn to apply their knowledge under appropriate conditions.

Problem-solving situations foster invention and creativity (see also Perkins, 1990).

Learners come to see the implications of new knowledge. A most common problem inherent in classroom learning is the question of relevance: "How does this relate to my life and goals?" When knowledge is acquired in the context of solving a meaningful problem, the question of relevance is at least partly answered.

Knowledge is stored in ways that make it accessible when solving problems. People tend to retrieve knowledge more easily when they return to the setting of its acquisition. Knowledge learned while solving problems gets encoded in a way that can be accessed again in problem-solving situations.

Several of the models reviewed below, including some not cited by Collins et al. (1989), place instruction within a problem-solving context and at least some simulation of a real-life setting.

Taken in its extreme form that would require "authentic," real-life contexts for all learning, the notion of situated learning is somewhat vague and unrealistic. Instruction always involves the dual goals of generalization and differentiation (Gagné & Driscoll, 1989). In its more modest form, however, the idea of context-based learning has considerable appeal. Gagné and Merrill (1990) have pointed to the need for better integration of learning goals through problem-solving "transactions." The notion of situated learning, however it is viewed, challenges the conditions-of-learning paradigm that prescribes the breaking down of tasks to be taught out of context.

3. Modeling and explaining: Show how a process unfolds and tell reasons why it happens that way. Collins (1991) cites two kinds of modeling: modeling of processes observed in the world and modeling of expert performance, including covert cognitive processes. Computers can be used to aid in the modeling of these processes. Collins stresses the importance of integrating both the demonstration and the explanation during instruction. Learners need access to explanations as they observe details of the modeled performance. Computers are particularly good at modeling covert processes that otherwise would be difficult to observe. Collins suggests that truly modeling competent performance, including the false starts, dead ends, and backup strategies, can help learners more quickly adopt the tacit forms of knowledge alluded to above in the section on content. Teachers in this way are seen as "intelligent novices" (Bransford et al., 1988). By seeing both process modeling and accompanying explanations, students can develop "conditionalized" knowledge, that is, knowledge about when and where knowledge should be used to solve a variety of problems.

ID models presently incorporate modeling and demonstration techniques. Tying explanations to modeled performances is a useful idea, similar to Chi and Bassok's (1989) studies of worked-out examples. Again, the emphasis is on making tacit strategies more explicit by directly modeling mental heuristics.

4. Coaching: Observe students as they try to complete tasks and provide hints and helps when needed. Intelligent tutoring systems sometimes embody sophisticated coaching systems that model the learner's progress and provide hints and support as practice activities increase in difficulty. Burton and Brown (1979) developed a coach to help learners with "How the West Was Won," a game originally implemented on the PLATO system. Anderson, Boyle, and Reiser (1985) developed coaches for geometry and LISP programming. Clancey (1986) and White and Frederiksen (1986) include coaches in their respective programs on medical diagnosis and electric circuits, both discussed below.

Coaching strategies can be implemented in a variety of settings. Bransford and Vye (1989) identify several characteristics of effective coaches:

Coaches need to monitor learners' performance to prevent their getting too far off base, but leaving enough room to allow for a real sense of exploration and problem solving.

Coaches help learners reflect on their performance and compare it to others'.

Coaches use problem-solving exercises to assess learners' knowledge states. Misconceptions and buggy strategies can be identified in the context of solving problems; this is particularly true of computer-based learning environments (Larkin & Chabay, 1989).

Coaches use problem-solving exercises to create the "teachable moment." Posner, Strike, Hewson, and Gertzog (1982) present a 4-stage model for conceptual change: (1) students become dissatisfied with their misconceptions; (2) they come to a basic understanding of an alternative view; (3) the alternative view must appear plausible; and (4) they see the new view's value in a variety of new situations (see also Strike & Posner, in press).

Coaching probably involves the most "instructional work" (cf. Bunderson & Inouye, 1987) of any of the cognitive apprenticeship methods. Short of one-on-one tutoring, coaching is likely to be partial and incomplete. Cooperative learning and small-group learning methods can provide some coaching support for individual performance. And computers can help tremendously in monitoring learner performance and providing real-time helps; yet presently coaching is only fully implemented in resource-intensive intelligent tutoring systems. Much work is being done to model the essentials of coaching functions on computer systems; we continue to need resource-efficient methods for achieving the coaching function. Still at issue is when and how coaching should be incorporated into instruction, particularly as it relates to learner errors and misconceptions. For example, should learners be permitted to fail, perhaps even required to learn from failure, as in Clancey's medical tutor?

5. Articulation: Have students think about their actions and give reasons for their decisions and strategies, thus making their tacit knowledge more explicit. Think-aloud protocols are one example of articulation (Hayes & Flower, 1980; Smith & Wedman, 1988). Collins (1991) cites the benefits of added insight and the ability to compare knowledge across contexts. As learners' tacit knowledge is brought to light, that knowledge can be recruited to solve other problems.

John Anderson (1990) has suggested that procedural knowledge-the kind that people can gain automaticity in-is initially encoded as declarative or conceptual knowledge, which later fades as the skill becomes proceduralized. Methods for articulating tacit knowledge help to restore a conscious awareness of those lost strategies, enabling more flexible performance. Traditional ID models suggest practicing problem solving to learn problem solving, but are surprisingly lacking in specific methods to teach learners to think consciously about covert strategies.

6. Reflection: Have students look back over their efforts to complete a task and analyze their own performance. Reflection is like articulation, except it is pointed backwards to past tasks. Analyzing past performance efforts can also involve elements of strategic goal-setting and intentional learning (Bereiter & Scardamalia, 1989). Collins and Brown (1988) suggest four kinds or levels of reflection:

Imitation occurs when a batting coach demonstrates a proper swing, contrasting it with your swing;

Replay occurs when the coach videotapes your swing and plays it back, critiquing and comparing it to the swing of an expert;

Abstracted replay might occur by tracing an expert's movements of key body parts such as elbows, wrists, hips, and knees, and comparing those movement to your movements;

Spatial reification would take the tracings of body parts and plot them moving through space.

The latter forms of reflection seem to rely on technologies-video or computer- for convenient instantiation. Collins (1991) uses Anderson et al.'s Geometry Tutor as an example of reflective instruction; Clancey's (1986) GUIDON medical tutor requires reflective thinking because so much metacognitive responsibility is given to learners. Both are computer-based programs.

Articulation and reflection are both strategies to help bring meaning to activities that might otherwise be more "rote" and procedural. Reigeluth's (1983b) concern with meaningful learning is indicative of the need; however, much of traditional ID practice tends to undervalue the reflective aspects of performance in favor of getting the procedure down right. The risk of ignoring reflection is that learners may not learn to discriminate in applying procedures; they may fail to recognize conditions where using their knowledge would be appropriate, and may fail to transfer knowledge to different tasks.

7. Exploration: Encourage students to try out different strategies and hypotheses and observe their effects. Collins (1991) claims that through exploration, students learn how to set achievable goals and to manage the pursuit of those goals. They learn to set and try out hypotheses, and to seek knowledge independently. Real-world exploration is always an attractive option; however, constraints of cost, time, and safety sometimes prohibit instruction in realistic settings. Simulations are one way to allow exploration; hypertext structures are another.

As Reigeluth (1983a) notes, discovery learning techniques are less efficient than direct instruction techniques for simple content acquisition. The choice must depend, however, on the goals of instruction: For near transfer tasks (cf. Salomon & Perkins, 1988), direct instruction may occasionally be warranted; for far transfer tasks, learners must learn not only the content but also how to solve unforeseen problems using the content; in such cases, instructional strategies allowing exploration and strategic behavior become essential McDaniel & Schlager, 1990).

Having thus represented a traditional ID mode of thinking about exploration, we are still left unsatisfied. Following a constructivist way of thinking, there is something intrinsically valuable about situated problem-solving activity that makes learning more effective than straight procedural practice. This is an area that needs better articulation, including a rationale for appropriate goals for exploratory and inquiry-based instruction, and effective strategies for reaching those goals.

8. Sequence: Present instruction in an ordering from simple to complex, with increasing diversity, and global before local skills.

Increasing complexity. Collins et al. (1989) point to two methods for helping learners deal with increasing complexity. First, instruction should take steps to control the complexity of assigned tasks. They cite Lave's study of tailoring apprenticeships: apprentices first learn to sew drawers, which have straight lines, few pieces of material, and no special features like zippers or pockets. They progress to more complex garments over a period of time. The second method for controlling complexity is through scaffolding; for example, group or teacher support for individual problem solving.

Increasing diversity refers to the variety in examples and practice contexts.

Global before local skills refers to helping learners acquire a mental model of the problem space at very early stages of learning. Even though learners are not engaged in full problem solving, through modeling and helping on parts of the task (scaffolding), they can understand the goals of the activity and the way various strategies relate to the problem's solution. Once they have a clear "conceptual map" of the activity, they can proceed to developing specific skills.

The sequencing suggestions above bear a strong resemblance to those of elaboration theory (Reigeluth & Stein, 1983) and component display theory (Merrill, 1983). The notion teaching of global before local skills is implicit in elaboration theory; simple-to-complex sequencing is the foundation of elaboration theory. The notion of increasing diversity is the near-equivalent to the prescription to use "varied-example" and practice activities in concept or rule learning. The cognitive apprenticeship extends these notions beyond rule learning to problem-solving contexts. At the same time, a number of research findings call into question the simple-to-complex formula. For a more complete discussion of sequencing issues, see Wilson & Cole (1992).

A SURVEY OF COGNITIVE TEACHING MODELS

In the section below, we briefly review several instructional systems developed by cognitive psychologists. Some of these systems were cited by Collins and

colleagues as exemplifying cognitive apprenticeship features. Others were cited by Glaser and Bassok (1989) as incorporating the best new knowledge coming out of cognitive psychology. Still others were selected because they embody noteworthy design concepts. After each program is described individually, a comparison is made between the models in an effort to identify common and not so common themes for discussion.

Anderson's Intelligent Tutors

John Anderson accounts for cognitive performance through the ACT* model of information processing (see, e.g., Anderson, 1990). According to this model, declarative knowledge is compiled into procedural skill through repeated practice performing a task. Whereas a novice must keep a procedure's steps in mind while performing, an expert turns several steps into an integrated routine, similar to the way subroutines work together in computer programming. Once knowledge is compiled from declarative to procedural knowledge, it may be performed with a minimum of allocated conscious attention.

Instruction based on the ACT* model would therefore emphasize guiding the learner through repeated practice opportunities to proceduralize the skills of the curriculum. Anderson and colleagues at Carnegie Mellon University have tested out this instructional approach through a series of intelligent tutoring systems aimed at teaching well-defined procedural skills, including programming in LISP (Anderson, Farrell, & Sauers, 1984), generating proofs for geometric theorems (Anderson, Boyle, & Yost, 1985), and solving algebraic equations (Lewis, Milson, & Anderson, 1988). Based on a cognitive analysis of the task, a model of the ideal student performance is developed for varying stages of competence, including assorted "buggy" procedures. As instruction proceeds, the system fits the learner's response pattern to its performance model, selects problems to minimize errors and optimize learning, and provides feedback and remediation accordingly. The technique of comparing learners' performance with a preexisting performance model is termed model tracing. The program does not make available the model's production rules to students directly; rather, the production rules trigger various instructional events, most notably intervention and feedback following incorrect performance.

Because new knowledge is best learned in the context of solving relevant problems, instruction is centered around problem-solving practice. Glaser and Bassok (1989) summarize the intelligent tutoring approach:

The CMU group advocates shortening preliminary instructional texts, and restraining from elaborated explanations. Texts should focus on procedural information, and students should begin actual problem solving as soon as possible.

Although textual instructions should be carefully crafted to maximize correct encoding, the inevitable misunderstandings should be corrected [immediately] during problem solving. ( p. 638)

Clancey's Intelligent Tutoring Environments

Clancey's (1986) program GUIDON and its descendents use heuristic classification methods as the basis for an intelligent tutoring environment for medical diagnosis. The program differs from ACT* model prescriptions in several ways. First, it assumes that the learner has a basic understanding of terms, concepts and disease processes. Second, it assumes that learning is more efficient if the student determines what he/she needs to know next without being explicitly controlled by the system. Third, and most significantly, it is failure driven; that is, primary instruction occurs in the form of feedback to student errors.

GUIDON requires the student to make a diagnosis, then to justify it with reasons. When the student's diagnosis "fails," he/she must take steps to correct the reasoning that led to it. Thus while the student develops expertise in medical diagnosis in a realistic problem-solving context, he/she also learns to detect and correct buggy procedures and misconceptions. The program is nearly completely learner-controlled; at any point, the student can choose to browse through the expert taxonomies and tables, examine the expert's reasoning during problem solving, ask questions, or request explanations. But ultimately the student must generate the appropriate links in a solution graph for each case.

Qualitative Mental Models

White and Frederiksen's (1986) program to teach troubleshooting in electrical circuits emphasizes the relationship between qualitative models and causal explanations. White and Frederiksen believe that mastery of qualitative reasoning should precede quantitative reasoning. Their program builds on students' intuitive understandings of the domain, carefully sequencing "real-world" problems that require the student to construct increasingly complex qualitative models of the domain. Although the program encourages students to engage in diverse learning strategies (exploring, requesting explanations, viewing tutorial demonstrations or problem solving), it tries to minimize errors. Thus, like the ACT* model, it does not directly address buggy algorithms and misconceptions.

Reciprocal Teaching

Brown and Palincsar (1989; Palincsar & Brown, 1984) have developed a cooperative learning system for the teaching of reading, termed reciprocal teaching. The teacher and learners assemble in groups of 2 to 7 and read a paragraph together silently. A person assumes the "teacher" role and formulates a question on

the paragraph. This question is addressed by the group, whose members are playing roles of producer and critic simultaneously. The "teacher" advances a summary, and makes a prediction or clarification, if any is needed. The role of teacher then rotates, and the group proceeds to the next paragraph in the text. Brown and colleagues have also developed a method of assessment, called dynamic assessment, based on successively increasing prompts on a realistic reading task. The reciprocal teaching method uses a combination of modeling, coaching, scaffolding, and fading to achieve impressive results, with learners showing dramatic gains in comprehension, retention, and far transfer over sustained periods.

Procedural Facilitations for Writing

Novices typically employ a knowledge-telling strategy when they write: They think about their topic, then write their thought down; think again, then write the next thought down, and so on until they have exhausted their thoughts about the topic. This strategy, of course, is in conflict with a more constructive, planning approach in which writing pieces are composed in a more coherent, intentional way. To encourage students to adopt more sophisticated writing strategies, Scardamalia and Bereiter (1985) have developed a set of writing prompts called procedural facilitations, that are designed to reduce working-memory demands and provide a structure for completing writing plans and revisions. Their system includes a set of cue cards for different purposes of writing, structured under five headings: new idea (e.g. "An even better idea is..."; "An important point I haven't considered yet is..."), improve ("I could make my point clearer..."), elaborate ("An example of this..."; "A good point on the other side of the argument is..."), goals ("My purpose..."), and putting it together ("I can tie this together by..."). Each prompt is written on a notecard and drawn by learners working in small groups. The teacher makes use of two techniques, soloing and co-investigation. Soloing gives learners the opportunity to try out new procedures by themselves, then return to the group for critique and suggestions. Co-investigation is a process of using think-aloud protocols that allow learner and teacher to work together on writing activities. This allows for more direct modeling and immediate direction. Bereiter and Scardamalia (1987) have found up to tenfold gains in learning indicators with nearly every learner improving his/her writing through the intervention.

Schoenfeld's Math Teaching

Schoenfeld (1985) studied methods for teaching math to college students. He developed a set of heuristics that were helpful in solving math problems. His method introduces those heuristics, as well as a set of control strategies and a productive personal belief system about math, to students. Like the writing and reading systems, Schoenfeld's system includes explicit modeling of problem-solving strategies, and a series of structured exercises affording learner practice in

large and small groups, as well as individually. He employs a tactic he calls "postmortem analysis," retracing the solution of recent problems, abstracting out the generalizable strategies and components. Unlike the writing and reading systems, Schoenfeld carefully selects and sequences practice cases to move learners into higher levels of skill. Another interesting technique is the equivalent to "stump the teacher," with time at the beginning of each class period devoted to learner-generated problems that the teacher is challenged to solve. Learners witnessing occasional false starts and dead ends of the teacher's solution can acquire a more appropriate belief structure about the nature of expert math problem solving. Schoenfeld's positive research findings support a growing body of math research suggesting the importance of acquiring a conceptual or schema-based representation of math problem solving.

Anchored Instruction

John Bransford and colleagues at Vanderbilt University have developed several instructional products using video settings. Young Sherlock Holmes or Indiana Jones may provide macrocontexts within which problems of various kinds may be addressed. For example, when Indiana Jones quickly replaces a bag of sand in place of the gold idol, the booby trap is tricked into thinking the idol is still there. This scene opens up questions of mass and density: If the idol were solid gold, how big must a sand bag be to weigh the same, and could Indy have escaped as he did carrying a solid-gold idol of that size? Based on a single macrocontext, learners may approach a variety of problems that draw on science, math, language, and history. Bransford and colleagues have applied the name anchored instruction to this approach of grounding instruction in information-rich situations, and have reported favorable findings in field-based and laboratory studies (The Cognition and Technology Group at Vanderbilt, 1990). Recently, several videodisc-based macrocontexts called the Jasper Series have been developed as a basis for research and classroom instruction (The Cognition and Technology Group at Vanderbilt, in press; Sherwood and The Cognition and Technology Group at Vanderbilt, 1991; Van Haneghan et al., in press).

Cognitive Flexibility Hypertexts

Spiro and colleagues (Spiro & Jehng, 1990) have developed hypertext programs to address problems typically associated with acquiring knowledge in complex, ill-structured domains. The programs utilize videodiscs that construct multiple "texts" (audio/video mini-cases, about 90 seconds each) of a domain. The multiple representations in KANE (Exploring Thematic Structure in Citizen Kane) induce students to restructure their thinking as they grapple with the thematic issues which are presented through the mini-cases. The program codes various themes and automatically generates appropriately juxtaposed sequences when the learner selects a theme to explore. It relies on the learner to generate the relationships

which are embodied in a set of mini-cases. Spiro's programs are best thought of as "rich" environments that allow sophisticated learners to pursue their learning goals in a flexible way. They do not typically include skill practice in a traditional sense, but instead rely on learner purposes and externally imposed assignments to give meaning to student browsing. There is an authoring shell, however, built into the system for teachers or students to use. Students, for example, may construct a series of mini-cases into a "visual essay" illustrating a theme not present on the Theme menu.

Like Clancey's intelligent tutoring environments, KANE assumes that students have basic knowledge of the content (in this instance, they must have seen the full movie in its natural sequence at least once).

The idea of mini-cases is central to Spiro's approach. Spiro and Jehng (1990) emphasize the differences between actual experiences and mini-cases:

a. the cases are immediately juxtaposed (hours and days do not pass between nonroutine cases);

b. the cases are thematically related (whereas there is no guarantee of instructive relatedness across naturally occurring adjacent cases);

c. the cases are stripped down to structurally significantly features, making it easier to extract dimensions of structural relatedness;

d. the cases are accompanied by expert commentary and guidance; and, finally,

e. because the cases are short, they are each easier to remember and more of them can be presented in a short amount of time, facilitating the recognition of relationships across cases. (p. 202; reformatted)

Spiro and Jehng emphasize that this instructional approach is difficult-it places great metacognitive demands on learners-but it addresses goals which are often overlooked in instruction precisely because they are difficult.

Instruction designed around mini-cases is similar to Schank's use of stories in instruction (Schank & Jona, 1991; Riesbeck & Schank, 1989). Schank's approach includes videotaping and assembling a database of short "stories" or cases from experts, then accessing and sequencing those stories within an instructional framework according to expressed feelings, perceptions, and needs of the learner. Case-based instruction is a highly appealing design concept, and we look forward to reports of working programs from Schank and colleagues.

Learning Through Design Activities: Computer Tools

Harel and Papert (1990) and Lippert (1988, 1990) have examined the effect of learning a content domain while simultaneously learning other domains, particularly software design.

Instructional Software Design Project.

Harel and Papert (1990) created a "total learning environment" in which fourth-graders simultaneously learned fractions and Logo programming; the students' goal was to design and develop a Logo program to teach something about fractions. "The 'experimental treatment' integrated the experimental children's learning of fractions and Logo with the designing and programming of instructional software" (Harel & Papert, p. 7), four hours a week during a 15-week semester. The students in the first control group spent the same amount of time learning Logo, but with a different purpose. The second control group programmed only once a week. Each group completed a two-month unit on fractions as part of the regular math curriculum.

The experimental treatment purposely cultivated a sense of meaningful activity, collegiality, and metacognitive awareness. The students had primary control over the task of designing and developing individual programs to "'explain something about fractions' to some intended audience" (Harel & Papert, p. 3). Initially Harel guided the students and teacher in a discussion of instructional design and educational software; she also provided models of designs, flowcharts and screens from her previous projects and shared notes and excerpts from programs developed by "real" designers and engineers. As needed, the teacher and Harel led discussions on issues related to understanding and teaching fractions, as well as on specific Logo programming skills.

On each day that the students worked on the project, they initially spent 5-7 minutes writing designs and plans in special notebooks and 45-55 minutes working on their computers, and then wrote about the problems they had encountered and changes they had made; occasionally they also included designs for the next session's activity. "The only two requirements were: (1) that they write in their Designer Notebooks before and after each working session; and (2) that they spend a specific amount of time at the computer each day" (Harel & Papert, p. 4) in order to fit the project into the schedule.

In many respects the students, teacher and researcher were like a community of scholars jointly in pursuit of knowledge, or a community of craft apprentices. At all times students had access to "experts" and to continual evaluation; they could compare their work with each others' work and to the work of experts (all representing various stages of expertise).

By the end of the study, not only did the experimental group significantly exceed both control groups in mastery of fractions, but on some items the students scored twice as high as sixth- to eighth-graders. The students were also more persistent than students in both control groups in trying to solve various Logo programming problems, and developed more metacognitive sophistication. They could find problems, develop plans, discard or revise inefficient plans, and control distractions and anxiety.

Harel and Papert hypothesize that no single factor alone contributed to the students' achievements. Instead they conclude that several factors interrelated in a holistic approach which may include but is not limited to:

the affective side of cognition and learning; the children's process of personal appropriation of knowledge; the children's use of LogoWriter; the children's constructivist involvement with the deep structure of fractions knowledge; the integrated-learning principle; the learning-by-teaching principle; and the power of design as a learning activity. (Harel & Papert, p. 30)

Expert Systems.

Lippert (1988, 1990) describes the way in which an expert system shell can be used as both an instructional and a learning strategy to "facilitate the acquisition of procedural knowledge and problem-solving skills in difficult topics" (1988, p. 22). Again, students learn by designing-developing an expert system individually or in groups, on their own or under the guidance of a teacher. According to Lippert, the strategy can be used with students as young as grade 6 and in any domain whose knowledge base can be expressed in productions.

Like the Harel and Papert approach, developing an expert system forces students to construct a meaningful representation of the domain. Most expert systems are systems which reduce a content domain to a set of IF-THEN rules. According to Lippert's scheme, the knowledge base is the key component and includes four parts: decisions which define the domain; questions which extract information (answers) from the user; rules that relate the answers to the decisions; and explanations (of questions or rules), which require the developer to understand the relationships among the various elements of the domain-the learner must understand "why" and "when," not merely "what." The developer constructs the knowledge base which the system then evaluates; if the system finds inconsistencies or redundancies, the developer must revise the knowledge base. In doing so, the learner must be reflective and articulate his or her implicit knowledge. Developing such a system helps students confront their misconceptions of the content.

In designing the system, students can experiment, trying out ideas and revising them as necessary until they are satisfied with their representation of the domain. They can initially restrict their system to one component of a domain and expand it to accommodate their expanding knowledge base. As in Harel and Papert's Logo environment, when students work in groups, they are exposed to and stimulated by different representations of the domain. And like the students in the Logo environment, students who design an expert system learn much incidental information and often become enthusiastic about learning the content.

Several issues remain unresolved with respect to learning through design activities. Like many prototype programs, factors contributing to the success of the programs are largely embedded within the expertise of the researchers and teachers. The critical factors in the instructional design need to be further explicated; this will allow the approach to be more easily identified and exported to different settings. A key issue is the question of efficiency: Can learning through design make more efficient use of time and resources, or does a design-oriented approach require extra time and resources? Theoretically, learning through design represents a fundamental alternative to traditional instructional designs, and deserves, therefore, continued attention among ID theorists.

IMPLICATIONS FOR INSTRUCTIONAL DESIGN

In reviewing the set of teaching models described above, one is struck by the diversity of approaches and features, and particularly by the key features not common to all the model programs.

Problem-solving versus Skill Orientation

Overwhelmingly, the models approached instruction from a problem-solving orientation. The main exception is Anderson's LISP Tutor, which essentially teaches procedural skill, with little need for metacognitive reflection. Near transfer is the goal of Anderson's tutoring systems. "Problem solving" in this environment is something like working out problems at the end of a textbook chapter. We do not consider this kind of activity genuine problem solving in the full sense of the word. It is worthy of note that the LISP Tutor is the only program that does not include explicit guidance to encourage metacognitive thinking.

Spiro's KANE hypertext is another unusual problem-solving environment. Specific problems are not posed within the program; learners do not "practice" solving interpretation problems per se. Instead, learners are provided a rich environment to explore and study. Problem solving in this setting depends heavily on the predisposition and prior knowledge of the learner. Thus the explicit instructional guidance for problem solving is relatively weak, which would likely result in failure for the learner lacking in metacognitive and preexisting content knowledge.

The rest of the programs include higher levels of guidance for problem solving and practice on procedural skills.

A problem-solving orientation toward instruction is not implied by the conditions-of-learning paradigm; in fact, the reverse seems to be true. In the words of an anonymous reviewer of a previous draft of this article, "this is a content issue, not a method issue." Traditional instructional designers "would say that the needs analysis should determine whether or not you have a problem-solving orientation or a skill orientation (or both, or neither-perhaps an understanding orientation)." Hence, from an ID point of view, problem-solving instruction is only one of several kinds, each meeting different needs. A more constructivist point of view, however, would suggest that virtually all instruction somehow should be embedded within a problem-solving context (e.g., Bransford & Vye, 1989; Wilson, 1989). The heavy bias toward problem solving in the models reviewed seems to be evidence of the viability of such an emphasis.

Detailed versus Broad Cognitive Task Analysis

Most of the models base instruction on a careful and detailed cognitive task analysis. These analyses go beyond behavioral observation; subtle and covert mental heuristics are sought and identified. Again, however, there is considerable variation among the teaching models. Intelligent tutors demand an exhaustive representation of the content; by contrast, the skills taught by group methods such as reciprocal teaching or the fractions software design project are not so well defined in advance.

There is no pretense among any of the projects' designers that the goals of instruction are completely captured by analysis. Rather, the instructional materials and interactions provide a setting wherein students can develop the subtle and often unidentified knowledge needed to succeed in the given task. The distinction between the identified learning goals and the knowledge actually acquired through learning activities is an important one that is often blurred in traditional ID theories. Given the right design, incomplete learning specifications can nonetheless lead toward complete learning outcomes. This attitude toward instructional outcomes helps deflect the criticism that ID is reductionistic because it pretends to capture and transmit expertise outside of the expert (cf. Dreyfus & Dreyfus, 1986; Bereiter, 1991).

Cognitive science has opened up an array of tools for performing task/content analyses, including think-aloud protocols, computer modeling, networks, maps, production systems, and cognitive process analysis (Gardner, 1985; Nelson, 1989; Smith & Wedman, 1988). Instructional designers can represent content with greater precision and accuracy than before. Yet as important as these analytic tools are, of

greater importance to ID are theories of knowledge and learning that are presently being developed within cognitivist and philosophical traditions (e.g., Bereiter, 1991; Lakoff, 1987; Paul, 1990). Faulconer and Williams (1990), for example, outline a Heideggerian perspective that helps to resolve the dualistic bias inherent in the current "objectivism" versus "constructivism" debate (Duffy & Jonassen, in press). Instructional designers can do more to be sensitive to the variety of conceptions of knowledge and learning, and to develop teaching models in line with those varying conceptions.

Authentic versus Academic Contexts

Brown, Collins, and Duguid (1989) define authentic activity as "the ordinary practices of the culture" (p. 34). We take this to mean that they have some bearing on real-life, everyday activities. There is a problem, however, in determining what counts as authentic activity. Apparently, the theorizing and inquiry activities of a nuclear physicist or historian qualify as authentic. For the term to have any precise referent, there needs to be a boundary drawn between authentic and inauthentic activity, and we are not sure where that boundary lies. Examining the set of teaching models, however, there surely seems to be some sort of authenticity continuum. Some programs relate more closely to real-life situations (e.g., the Jasper series), but most are more typically academic work (e.g., Anderson's LISP tutor). Some programs seem to fall in the middle (Spiro's KANE or Harel and Papert's fractions project). Clearly, the resemblance between the instructional setting and real-life settings is not a constant among the models.

ID theories have tended toward academic contexts. For example, the Reigeluth volumes (1983a, 1987) contain surprisingly little discussion of simulation as an instructional strategy. The irony is that large numbers of ID practitioners are heavily involved in developing "authentic" training either in job sites or through simulations. Clearly more attention needs to be given to prescriptions for designing simulations that go beyond present models for direct instruction.

Learner versus System Control

There is also considerable variability on the learner control dimension. Some programs are tightly controlled (e.g., LISP Tutor) while several others are very flexible (e.g., GUIDON, KANE, fractions project). There seems to be a correspondence between the degree of learner control and the metacognitive goals of the instruction: Where metacognitive development is a goal, there is by necessity some flexibility allowing learners to assume responsibility over the "zone of proximal development" (Scardamalia & Bereiter, 1991). Where metacognitive skill is not a goal of instruction, learner control is less critical.

Yet even this generalization is not complete. Schoenfeld's math program is very much oriented toward metacognitive development, yet problem cases are carefully planned and sequenced in advance by the instructor. Within the framework of the cases, however, learners are able to negotiate control with the instructor. The planned sequencing of cases may be accounted for by the well-definedness of the content, which allows for a systematic progression from simple to complex cases. The same can be said for White and Frederiksen's progression of mental models. Learners may experiment and control actions within levels, but movement from level to level is planned in advance according to content parameters. With less defined content (e.g., reciprocal teaching), sequence of cases is, at least on the surface, much less important. Another view of reciprocal teaching, however, is that the text being read is not the content; rather, the content lies in the skills being modeled and practiced, and the sequencing, under the informal control of the teacher, does proceed from simple to complex.

Error-restricted versus Error-driven Instruction

GUIDON is the prime example of a program whose instructional presentations are triggered by the learner's errors. Presentations and explanations thus take the form of feedback to learner actions. In contrast, several programs seek to control errors through careful presentations and sequencing. Errors within the LISP Tutor or White and Frederiksen's troubleshooting tutor would be due to mismatches between the learner and the instructional plan; the learner can then be corrected and brought back on track. Most of the programs seem to fall in the middle on this feature; for example, errors in Schoenfeld's math program are expected, even modeled by the instructor, but the instruction is not completely keyed around diagnosis and response to student errors.

Feedback is an old topic of research, but we know surprisingly little about it. It has long been recognized that the best way to ensure learners' understanding is to have them respond actively. At the same time, old maxims such as "provide immediate knowledge of results" are being challenged (Salmoni, Schmidt, & Walter, 1984; Corbett & Anderson, 1991; Lepper & Chabay, 1988). There is some evidence that continuous detailed feedback sometimes may actually serve to supplant the self-reflective processes that a learner may otherwise apply to a problem. Error-driven (i.e., feedback-heavy) instruction should be designed so as not to interfere with such learning processes.

CONCLUSION

This paper has been concerned with the relationship between two related disciplines: cognitive psychology and instructional design. ID, the more applied discipline, faces the challenges of constantly re-inventing itself as new research in basic psychology sheds light on learning processes. It is no cause for concern that

ID models need continual self-review; indeed, there would be cause for concern if the situation were otherwise. To provide some sense of context, however, we would like to call attention to some basic principles of cognitive apprenticeship articulated by the philosopher Erasmus more than 500 years ago. In a letter to a student friend, Erasmus offers some advice:

Your first endeavor should be to choose the most learned teacher you can find, for it is impossible that one who is himself no scholar should make a scholar of anyone else. As soon as you find him, make every effort to see that he acquires the feelings of a father towards you, and you in turn those of a son towards him . . . . Secondly, you should give him attention and be regular in your work for him, for the talents of students are sometimes ruined by violent effort, whereas regularity in work has lasting effect just because of its temperance and produces by daily practice a greater result than you would suppose . . . . A constant element of enjoyment must be mingled with our studies so that we think of learning as a game rather than a form of drudgery, for no activity can be continued for long if it does not to some extent afford pleasure to the participant.

Listen to your teacher's explanations not only attentively but eagerly. Do not be satisfied simply to follow his discourse with an alert mind; try now and then to anticipate the direction of his thought . . . Write down his most important utterances, for writing is the most faithful custodian of words. On the other hand, avoid trusting it too much . . . . The contests of minds in what we may call their wrestling ring are especially effective for exhibiting, stimulating, and enlarging the sinews of the human understanding. Do not be ashamed to ask questions if you are in doubt; or to be put right whenever you are wrong. (Erasmus, 1974, pp. 114-115)

At a time when terms like metacognition, scaffolding , and cognitive apprenticeship are being invented to describe the learning process, it is humbling to be reminded of the insights of former paradigms. ID theorists may chafe at the continuing need to revise their theories in light of advances in psychological theory, but it is good for both fields for the dialogue to continue. Indeed, the interaction between basic psychology and applied instructional design can be expected to continue for some time to come.

AUTHOR NOTES

We would like to thank Jim Cole for calling our attention to the works of Erasmus. We also thank suggestions made by anonymous reviewers. Please send requests for reprints to Brent Wilson, University of Colorado at Denver/Campus Box 106/P. O. Box 173364/Denver CO 80217-3364.

REFERENCES

Anderson, J. R. (1990). Cognitive psychology and its implications (3rd ed.). New York: Freeman.

Anderson, J. R., Boyle, C. F., & Yost, G. (1985). The Geometry Tutor. In Proceedings of the international joint conference on artificial intelligence, pp. 1-7. Los Altos CA: International Joint Conference on Artificial Intelligence.

Anderson, J. R., Farrell, R., & Sauers, R. (1984). Learning to program in LISP. Cognitive Science, 8, 87-129.

Andrews, D. H., & Goodson, L. A. (1980). A comparative analysis of models of instructional design. Journal of Instructional Development, 3(4), 2-16.

Bereiter, C. (1991, April). Implications of connectionism for thinking about rules. Educational Researcher, 10-16.

Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Erlbaum.

Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser, (pp. 361-392). Hillsdale NJ: Erlbaum.

Bransford, J. D., Goin, L. I., Hasselbring, T. S., Kinzer, C. K., Sherwood, R. D., & Williams, S. (1988). Learning with technology: Theoretical and empirical perspectives. Peabody Journal of Education, 64 (1), 5-26.

Bransford, J. D., Sherwood, R. D., Hasselbring, T. S., Kinzer, C.K., & Williams, S. M. (1990). Anchored instruction: Why we need it and how technology can help. In D. Nix & R. Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology (pp. 115-141). Hillsdale NJ: Erlbaum.

Bransford, J. D., & Vye, N. J. (1989). A perspective on cognitive research and its implications for instruction. In L. B. Resnick & L. E. Klopfer (Eds.), Toward the thinking curriculum: Current cognitive research. Alexandria VA. 1989 ASCD Yearbook.

Branson, R. K., & Grow, G. (1987). Instructional systems development. In R. M. Gagné (Ed.), Instructional technology: Foundations. Hillsdale NJ: Erlbaum.

Brown, J. S., Collins, A., & Duguid, P. (1989, January-February). Situated cognition and the culture of learning. Educational Researcher, 32-42.

Brown, A., & Palincsar, A. S. (1989). Guided cooperative learning and individual knowledge acquisition. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in Honor of Robert Glaser (pp. 393-451). Hillsdale NJ: Erlbaum.

Bunderson, C. V., & Inouye, D. K. (1987). The evaluation of computer-aided educational delivery systems. In R. M. Gagné (Ed.), Instructional technology: Foundations. Hillsdale NJ: Erlbaum.

Burton, R. R., & Brown, J. S. (1979). An investigation of computer coaching for informal learning activities. International Journal of Man-Machine Studies, 11, 5-24.

Case, R. (1978). A developmentally based theory and technology of instruction. Review of Educational Research, 48, 439-463.

Case, R., & Bereiter, C. (1984). From behaviourism to cognitive behaviourism to cognitive development: Steps in the evolution of instructional design. Instructional Science, 13, 141-158.

Chi, M. T. H., & Bassok, M. (1989). Learning from examples via self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in Honor of Robert Glaser (pp. 251-282). Hillsdale NJ: Erlbaum.

Chi, M. T. H., Glaser, R., & Farr, M. J. (1988). The nature of expertise. Hillsdale NJ: Erlbaum.

Clancey, W. J. (1986). From GUIDON to NEOMYCIN and HERACLES in twenty short lessons: ONR final report 1979-1985. AI Magazine, 7(3), 40-60.

Cognition and technology group at Vanderbilt. (1990, August-September). Anchored instruction and its relation to situated cognition. Educational Researcher, 2-10.

Cognition and technology group at Vanderbilt. (in press). Anchored instruction approach to cognitive skills acquisition and intelligent tutoring. In J. W. Regian and V. J. Shute (Eds.), Cognitive approaches to automated instruction. Hillsdale NJ: Erlbaum.

Collins, A. (1991). Cognitive apprenticeship and instructional technology. In L. Idol & B. F. Jones (Eds.), Educational values and cognitive instruction: Implications for reform. Hillsdale NJ: Erlbaum.

Collins, A., & Brown, J. S. (1988). The computer as a tool for learning through reflection. In H. Mandl & A. Lesgold (Eds.), Learning issues for intelligent tutoring systems. New York: Springer.

Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and arithmetic. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale NJ: Erlbaum.

Collins, A., & Stevens, G. (1983). Inquiry teaching. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 247-278). Hillsdale NJ: Erlbaum.

Corbett, A. T., & Anderson, J. R. (1991, April). Feedback control and learning to program with the CMU Lisp Tutor. Paper presented at the meeting of the American Educational Research Association, Chicago.

Dreyfus, H., & Dreyfus, S. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. New York: The Free Press.

Duffy, T., & Jonassen, D. H. (in press). Instructional principles for constructivist learning environments. Hillsdale NJ: Erlbaum.

Eckert, P. (1990). Adolescent social categories-Information and science learning. In J. Gardner, J. G. Greeno, F. Reif, A. H. Schoenfeld, & A. A. diSessa, & E. Stage (Eds.), Toward a scientific practice of science education. Hillsdale NJ: Erlbaum.

Erasmus, D. (1974). The correspondence of Erasmus. Volume 1: Letters 1 to 141, 1484 to 1500. Translated by R. A. B. Mynors and D. F. S. Thomson. Toronto: University of Toronto Press.

Faulconer, J. E, & Williams, R. N. (1990). Reconsidering psychology. In J. E. Faulconer & R. N. Williams (Eds.), Reconsidering psychology: Perspectives from continental philosophy. Pittsburgh: Duquesne University Press.

Gagné, R. M. (1966). The conditions of learning (1st ed.). New York: Holt, Rinehart, & Winston.

Gagné, R. M., & Driscoll, M. P. (1989). Essentials of learning for instruction (2nd ed.). Englewood Cliffs NJ: Prentice-Hall.

Gagné, R. M., & Merrill, M. D. (1990). Integrative goals for instructional design. Educational Technology Research and Development, 38 (1), 23-30.

Gardner, M. K. (1985). Cognitive psychological approaches to instructional task analysis. In E. W. Gordon (Ed.), Review of Research in Education, 12, 157-195.

Glaser, R. (1982). Instructional psychology: Past, present, and future. American Psychologist, 37 (2), 292-305.

Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction. Annual Review of Psychology, 40, 631-666.

Gott, S. P. (1988a, April). The lost apprenticeship: A challenge for cognitive science. Paper presented at the meeting of the American Educational Research Association, New Orleans.

Gott, S. P. (1988b). Apprenticeship instruction for real-world tasks: The coordination of procedures, mental models, and strategies. In E. Z. Rothkopf (Ed.), Review of Research in Education, 15, 97-169.

Harel, I., & Papert, S. (1990). Software design as a learning environment. Interactive Learning Environments, 1, 1-32.

Hayes, J. R., & Flower, L. S. (1980). Identifying the organization of writing processes. In L. W. Gregg & E. R. Sternberg (Eds.), Cognitive processes in writing. Hillsdale NJ: Erlbaum.

Lakoff, G. (1987). Women, fire, and dangerous things: What categories reveal about the mind. Chicago: University of Chicago Press.

Larkin, J. H.& Chabay, R. W. (1989). Research on teaching scientific thinking: Implications for computer-based instruction. In L. B. Resnick & L. E. Klopfer (Eds.), Toward the thinking curriculum: Current cognitive research. Alexandria VA. 1989 Yearbook of the Association for Supervision and Curriculum Development.

Lepper, M. R., & Chabay, R. W. (1988). Socializing intelligent tutors: Bringing empathy to the computer tutor. In H. Mandl & A. Lesgold (Eds.), Learning issues for intelligent tutoring systems (pp. 242-257). New York: Springer-Verlag.

Lippert, R. C. (1988, March), An expert system shell to teach problem solving. TechTrends, 22-26.

Lippert, R. (1990). Teaching problem solving in mathematics and science with expert systems. Journal of Artificial Intelligence in Education, 1(3), 27-40.

McDaniel, M. A., & Schlager, M. S. (1990). Discovery learning and transfer of problem-solving skills. Cognition and Instruction, 7(2), 129-159.

Merrill, M.D. (1983). Component display theory. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status. (pp. 279-333). Hillsdale NJ: Erlbaum.

Merrill, M. D., Kowallis, T., & Wilson, B. G. (1981). Instructional design in transition. In F. Farley & N. Gordon (Eds.), Psychology and education: The state of the union. Chicago: McCutcheon.

Nelson, W. A. (1989). Artificial intelligence knowledge acquisition techniques for instructional development. Educational Technology Research and Development, 37(3), 81-94.

Newman, D., Griffin, P., & Cole, M. (1989). The construction zone: Working for cognitive change in school. Cambridge: Cambridge University Press.

Palincsar, A. M., & Brown. A. L. (1984). Reciprocal teaching of comprehension- fostering and monitoring activities. Cognition and Instruction, 1 (2), 117-175.

Paris, S. G., & Winograd, P. (1990). How metacognition can promote academic learning and instruction. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction. Hillsdale NJ: Erlbaum.

Paul, R. W. (1990). Critical and reflective thinking: A philosophical perspective. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction (pp. 445-494). Hillsdale NJ: Erlbaum.

Perkins, D. N. (1990). The nature and nurture of creativity. In In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction. Hillsdale NJ: Erlbaum.

Posner, G. K., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982) Accommodation of scientific conception: Toward a theory of conceptual change. Science Education, 66, 211-227.

Reigeluth, C. M. (Ed.). (1983a). Instructional-design theories and models: An overview of their current status. Hillsdale NJ: Erlbaum.

Reigeluth, C. M. (1983b). Meaningfulness and instruction: Relating what is being learned to what a student knows. Instructional Science, 12(3) 197-218.

Reigeluth, C. M. (Ed.). (1987). Instructional theories in action: Lessons illustrating selected theories and models. Hillsdale NJ: Erlbaum.

Reigeluth, C. M., Merrill, M. D., Wilson, B. G., & Spiller, R. T. (1980). The elaboration theory of instruction: A model for structuring instruction. Instructional Science, 9, 195-219.

Reigeluth, C. M., & Stein, R. (1983). Elaboration theory. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status (pp. 335-381). Hillsdale NJ: Erlbaum.

Reiser, R. A. (1987). Instructional technology: A history. In R. M. Gagné (Ed.), Instructional technology: Foundations. Hillsdale NJ: Erlbaum.

Resnick, L. B. (1981). Instructional Psychology. Annual Review of Psychology, 40, 631-666.

Resnick, L. B. (1989, February). Learning in school and out. Educational Researcher, 13-20.

Riesbeck, C., & Schank, R. (1989). Inside case-based reasoning. Hillsdale NJ: Erlbaum.

Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New York: Oxford University Press.

Rogoff, B., & Lave, J. (Eds.). (1984). Everyday cognition: Its development in social context (pp. 95-116). Cambridge MA: Harvard University Press.

Salmoni, A. W., Schmidt, R. A., & Walter, C. B. (1984). Knowledge of results and motor learing: A review and critical reappraisal. Psychological Bulletin, 95(3), 355-386.

Salomon, G., Globerson, T., & Guterman, E. (1989). The computer as a zone of proximal development: Internalizing reading-related metacognitions from a reading partner. Journal of Educational Psychology, 81(4), 620-627.

Salomon, G., & Perkins, D. N. (1988). Rocky roads to transfer: Rethinking mechanisms of a neglected phenomenon. Educational Psychologist, 24(2), 113-142.

Scardamalia, M., & Bereiter, C. (1985). Fostering the development of self-regulation in children's knowledge processing. In S. F. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills: Research and open questions (pp. 563-577). Hillsdale NJ: Erlbaum.

Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media. The Journal of the Learning Sciences, 1(1), 37-68.

Schank, R. C., & Jona, M. Y. (1991). Empowering the student: New perspectives on the design of teaching systems. The Journal of the Learning Sciences, 1(1), 7-35.

Schoenfeld (1985). Mathematical problem solving. New York: Academic Press.

Sherwood, R. D., & The Cognition and Technology Group at Vanderbilt. (1991, April). The development and preliminary evaluation of anchored instruction environments for developing mathematical and scientific thinking. Paper presented at the meeting of the National Association for Research in Science Teaching, Lake Geneva WI.

Smith, P. L., & Wedman, J. F. (1988). Read-think-aloud protocols: A new data-source for formative evaluation. Performance Improvement Quarterly, 1(2), 13-22.

Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1988, October). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains (Technical Report No. 441). Champaign IL: University of Illinois at Urbana-Champaign, Center for the Study of Reading.

Spiro, R. J., & Jehng, J-C. (1990). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D. Nix & R. J. Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology (pp. 163-205). Hillsdale NJ: Erlbaum.

Strike, K. A., & Posner, G. J. (in press). A revisionist theory of conceptual change. To appear in R. Duschl & R. Hamilton (Eds.), Philosophy of science, cognitive science, and educational theory and practice. Albany NY: SUNY Press.

Van Haneghan, J. P., Barron, L., Young, M. F., Williams, S. M., Vye, N. J., & Bransford, J. D. (in press). The Jasper Series: An experiment with new ways to enhance mathematical thinking. In D. Halpern (Ed.), The development of thinking skills in the sciences and mathematics. Hillsdale NJ: Erlbaum.

Wertsch, J. V. (1989). Vygotsky and the social formation of mind. Cambridge MA: Harvard University Press.

White, B. Y., & Frederiksen, J. R. (1986). Progressions of quantitative models as a foundation for intelligent learning environments. Technical Report # 6277, Bolt, Beranek, & Newman.

Wilson, B. G. (1987). Computers and instructional design: Component display theory in transition. In M. R. Simonson & S. Zvacek (Eds.), Proceedings of selected research presentations (pp. 767 - 782). Washington, D. C.: Association for Educational Communications and Technology, Research and Theory Division.

Wilson, B. G. (1989). Cognitive psychology and instructional design. In S. Thiagarajan (Ed.), Proceedings of the National Society for Performance and Instruction, 317-333.

Wilson, B. G., & Jonassen, D. H. (1989). Hypertext and instructional design: Some preliminary guidelines. Performance Improvement Quarterly, 2(3), 34-49.

Wilson, B., & Cole, P. (1992). Returning the 'theory' to Elaboration Theory: Strategies for organizing instruction based on cognitive conceptions of learning. Paper presented at the meeting of the American Educational Research Association, San Francisco, manuscript in preparation.

Teaching Molecular Biology on the Web Martinez J. Hewlett

Associate Professor

Department of Molecular and Cellular Biology

University of Arizona, Tucson, Arizona, USA

Abstract

A large, upper division course in molecular biology was taught online for the first time in the spring semester of 2000. About 150 students enrolled in this course, which included Web interactive modules, online chat rooms, Listserv communication and other features common to many distance learning protocols. The student performance in the course was subjectively judged to be improved over the traditional lecture approach. In addition, student attitude toward the course, measured objectively, became more positive from the beginning to the end of the semester. Instructor pedagogical goals were easier to achieve, although it appears subjectively to be more challenging to teach in the absence of live student/faculty interaction.

Introduction and Background

Molecular Biology (MCB 411) is a 3 unit, upper division course required of all majors in the programs Molecular and Cellular Biology and Biochemistry at the University of Arizona. The course is taught in both the fall and spring semesters and enrolls an average of 140 students each time. The course has been taught for many years by a number of faculty in both of the programs. During the last seven years the author has been one of these faculty.

For most of this time the course has been a didactic, lecture course presented in a large auditorium style lecture hall. Presentation of material originally employed the standard visual aids (overheads, slides, etc.). Some years ago the author moved to the use of the Web for all of his courses, establishing home pages for course resources such as syllabi, lecture outlines, exam archives and exam results and grades.

During the spring semester of 2000 the course was converted to an online format. There were two reasons for this change:

• to prepare for a time when the course would be offered to students at any of the three state universities and

• to increase the student engagement with the course material.

Course Logistics

The online course consists of the following elements:

1. The course home page (http://www.blc.arizona.edu/marty/411) with links to the syllabus and general course information.

2. A Course Access link for registered students, using a name and PIN, so that course activity can be tracked. Guest access is available using the following information --Last Name: guest, PIN: 9999.

3. A textbook (not a hypertext). 4. A set of modules covering the material of the course (in lieu of lectures). 5. Interactive questions, both graded and non-graded, imbedded within the

modules. 6. Online home works assignments. 7. A course Listserv to which all students are subscribed. 8. Exam archives from previous semesters. 9. Access to current exam results and grades. 10. Online sessions (one hour each week, 20-25 students per session) in the Old

Pueblo MOO, a chat environment maintained for teaching by the University of Arizona computer center.

11. Transcripts of all MOO sessions.

Because the majority of the students in the course were on campus or located within the Tucson area, certain "live" features were included:

• exam review in preparation for semester and final examinations.

• live examinations. It was not possible to secure examinations in a Web format and the size of the class precluded the use of a computer laboratory to administer the tests. For those students who were taking the course from remote sites and could not be in Tucson for the examination, arrangements were made for proctoring.

During the initial semester a project was undertaken by Ms. Kim Bissell and Dr. Lee Sechrest of the Psychology Department (Evaluation Group for Data Analysis), University of Arizona. The objective was to measure student attitudes toward a distance learning course of this type and to correlate these findings with a number of student demographic

and performance indicators. The data were obtained from available student records and from online surveys performed according to the following schedule:

Survey Number Survey Dates Course Time

Frame

1 1/13/00 Course orientation (first meeting).

2 2/3/00 - 2/14/00 Immediately prior to the first exam.

3 3/7/00 - 3/23/00 Midpoint of the course.

4 4/25/00 - 5/1/00 Immediately after last semester exam.

5 5/2/00 - 5/14/00 During final exam period.

At each time the students were asked to respond to two questions to determine the attitude variable:

• "At this time how do you feel about being in this web-based course?" Choices ranged from "Would much prefer to be in a traditional course" to "Do not have any preference" to "Am very glad to be in this web-based course."

• "Please rate how you currently feel about taking this web-based course." Choices range from "Very negative" to "Neither particularly positive nor negative" to "Very positive."

Their responses could range from 1 to 7 in both cases.

Analysis

Student Demographics

During spring 2000, just over 150 students registered for this course. Of these 147 students completed the semester and were assigned grades in the course. The age distribution of these students is shown in Figure 1 and the number of science courses that these students have taken is shown in Figure 2. Figure 3 shows the the way in which the students rated other university courses in general or science courses in particular. These data were obtained from questions asked in the survey at the beginning of the course.

Figure 1 Figure 2

Figure 3

Student Performance

The academic data for these students includes their grade point average (Figure 4) as well as the grade earned in this course (Figure 5).

Figure 4 Figure 5

While there was no direct comparison of performance in a traditional course versus the online instruction in this course, there was subjective evidence that the students performed at a higher level. In past versions of the course, when examinations deviated from the questions that were available as practice examples in the archive, the average on the examination would be lower than usual. In this case, about 40% of the examination material was new and could not be found in the archival examples. Instead of the expected drop in examination average, there was a consistent rise over the usual score.

Part of the reason for this may be found in the extent to which students engaged the material. Use of the Web site and attendance in the online chat sessions was "compelled" by the availability of a certain number of points toward the final grade. As a result, the student "time on task" was increased. It may be concluded that this increased the students’ preparation for examination on the material of the course.

Student Attitude

Figure 6 shows the mean responses to the question that determines an attitude variable. The increase from a mean response of 3.3 in the initial survey to a mean response of 4.6 in the final survey was significant (t=5.29, p<0.01).

Figure 6

A meta-analytic analysis of the growth curve was performed to look for correlates between student attitudinal change and student demographic or performance indicators. In this technique an initial starting point (intercept) and change throughout the course (slope) are estimated for each individual. The mean estimated intercept was 3.18 and the mean slope over the semester was 0.32. The growth curve analysis also attempts to predict the intercept and slope in the attitudinal variable over time.

Five predictor variables (age, number of science courses, GPA, rating of courses and grade) were included in models attempting to predict the intercept and slope. The growth curve analysis of student preference revealed that age, number of science courses, course rating or GPA did not predict the beginning attitude (intercept) of the students nor the rate of change (slope) throughout the course. However, the final grade achieved in the course was a significant predictor of the rate of change (slope) in student attitude throughout the course.

When correlations between individual time points in the survey were analyzed it suggested that student preference is not set at the beginning of the course. Examining the intercorrelation matrix for the various time points revealed low correlations between earlier time points and higher correlations between later time points. This suggests that preferences tended to congeal as the course progressed.

Pedagogy

Questions with respect to the delivery of course material and instructor experience are, of course, subjective and anecdotal. Nonetheless it may be worth noting some observations.

The use of the Web for the presentation of course materials normally delivered in a traditional lecture format allows a much greater breadth of coverage. It is possible, for instance, to link students to a wider range of experiences, including data analysis, enrichment sources and advanced information. In a lecture class either time constraints or the need to address the student population in the room may prevent this. On the Web, the instructor is effectively ‘speaking" to each student as an individual.

In the traditional large lecture class (>100 students) it is difficult to know more than a handful of students. Usually this is limited to those students who speak up in class or who choose to come to faculty office hours. Using the online chat sessions in the MOO ironically allows for a much richer student/faculty interaction. Students were met in small groups. In addition, the instructor can be online as the students enter the virtual space. This allows time for one-on-one discussion. By the end of the semester the experience is such that the instructor has a conversational relationship with more than half the class. When this is coupled with the "live" review sessions, a real learning community seems to develop.

One negative side of the Web course is the lack of human contact that is experienced by both student and faculty. Electronic contact is obviously necessary when students are engaging the material from remote sites. However, when students are on the same campus as the instructor this situation can become frustrating. As a result, the current version of the course includes more available "live" interactions, such as an optional discussion in a classroom. This has become an outlet for those students for whom the Web is too impersonal. In addition it gives the faculty member some live time with at least a few students.

A final possible negative is the investment of faculty time for a course of this magnitude. The lecture version of the course required 3 in-class hours per week. The Web version required a minimum of 8 online hours per week. This kind of time commitment can discourage faculty from taking on this type of course. This may be overcome by a team approach, where the online discussion sections are managed by more than one instructor.

Conclusions

The experience of one semester is hardly sufficient to draw firm conclusions about the value of distance learning for teaching upper division science. However, some preliminary statements can be made.

The student demographic predictor variables coupled with the growth curve analysis of student attitudes indicates that age is not a predictor of initial attitude (intercept) or rate of change of attitude during the course (slope). While there were only a few students who were older than average (Figure 1), these results do suggest that there was not an effect of younger versus older generations and their experience with technology.

The fact that student attitudes about the course tend to be flexible at the outset and to solidify during the semester suggests that faculty engagement and performance at the outset of an online course is critical to its perceived success.

Finally, convincing other faculty in a department to take on this kind of teaching requires a dedicated support system to reduce the startup effort and may involve a team approach to keep the online hours with reason.

Improving teaching in Molecular Biology

What's New? | Welcome | Our Schedule | Teaching Manual | Contact your AI | Talk To Us |

What's New? Welcome to the home page for the Molecular Biology Teaching Pilot Program. This page will contain resources and information related to our projects and events, which are designed to help improve the quality of teaching in Princeton's Mol. Bio. department.

Watch this space for updates, news, and more information!

Welcome! The Teaching Pilot Program is a joint effort between the department of Molecular Biology and the Graduate School to enhance discipline-specific training of graduate students. The program was initiated to enhance the quality of instruction that undergraduates receive by promoting a positive and educational teaching experience for the graduate student instuctors. Therefore the TPP will focus on the following goals:

Preparing AIs for their teaching experience: Being an AI can be both challenging and rewarding. To help graduate students face this challenge, we have designed and implemented a training program for incoming AIs. This program will be held biannually, in January and September, with the AIs for the coming semester. The program will try to focus on laboratory training in the winter session and precept courses in the fall in order to complement the courses offered by the department.

Fostering faculty/AI interactions: Graduate AIs are an essential part of the teaching in the department of Molecular Biology. However, AIs are most successful when the faculty and AIs are aware of each other's expectations and goals. To increase communication between AIs and faculty, we have asked professors each semester to meet with the AIs to discuss course goals and expectations, responsibilities, scheduling, and grading. In addition, we are encouraging faculty to actively mentor their AIs in teaching skills.

Creating an infrastructure of teaching with the department of Molecular Biology:We hope to create continuity through the exchange of personal experiences and information. A successful part of this program includes a lunch at the start of each semester for previous and current AIs of the courses being taught. These lunches have, for example, prepared AIs for their first weeks in a course, given them ideas for how to lead precepts and to efficiently execute a lab assignment.

Creating a program of assessment and improvement of graduate teaching: It is only through experience and evaluation that AIs can improve their teaching skills. We believe it is important to be evaulated during the teaching process and have implemented both a mid-term evaluation for each course and also a system for web-based feedback. In addition, videotaping services can be provided.

Making available materials relevant to teaching courses within the department: We have written an AI Training Manual for Molecular Biology. We are in the process of assembling materials relevant to teaching specific courses. Plus, we have identified materials from other universities that can be accessed through the world wide web.

Developing teaching opportunities for graduate students interested in teaching: We hope to encourage the department to develop more teaching opportunities on our campus, other campuses, and in the community. We currently organize programs and discussions to help students interested in teaching careers to develop their own pedagogical skills and ideas.

Our Schedule Our schedule: We're working on a cooperative learning experience with a Princeton High School biology course. Look for more information later.

Talk To Us We'd love to hear your suggestions and comments about how we might improve teaching within the department.

Contact Toh Hean Ch'ng or Jill Penn , student co-chairs of the Pilot Program committee or any of the other student members:

• Nicole Clouse • Cemile "Blue" Guldal • Nate Hafer • Brenden Rickards • Megan Sullivan • Michi Taga

with your thoughts on teaching, this web page, or other related areas.

Useful Science Education Links

www.princeton.edu/~aiteachs

science-education.nih.gov/homepage.nsf

science.coe.uwf.edu/narst.html

www-ctl.stanford.edu/ta.html

vector.cshl.org

lyco.lycoming.edu/~newman/models.html

Women in Cell Biology: Teaching Science in High School from the ASCB Newsletter, Vol. 23 No. 8, August 2000.

www.accessexcellence.com/

http://www.wisc.edu/cels/

What's New? | Welcome | Our Schedule | Teaching Manual | Contact your AI | Talk To Us |

Last modified 9/7/01

Teaching models prescribe tested steps and procedures to effectively generate desired outcomes. In general, models can be classified along a continuum from instructor-directed, to student-instructor negotiated, to student-directed (see Figure below).

To help you choose a model, you should know that instructor-directed models are the norm for college campuses, and are criticized for not actively involving students and for often failing to challenge students to think at high levels (e.g., analysis, synthesis, evaluation). Student-directed models require considerably more interest and effort from the students and are criticized for consuming a lot of class time to cover smaller amounts of information at greater depths. A greater chance exists for higher-level thinking via student-directed models, however. Social models provide a common ground between the two extremes. The models listed below are classified by this three-tiered taxonomy.

Top-Down, Teacher Delivered, "Direct" Instruction

• The Audio Tutorial Approach is a mastery learning model relevant for those working with large lecture classes or wishing to develop independent study materials. Relevant for most fields, including linguistics, languages, and performance arts; originally designed for sciences.

• The Personalized System of Instruction is a mastery learning model, emphasizing independent study and proctored-testing. Relevant for those instructors wishing to develop self-paced teaching resources (e.g., web modules).

• In Goal-Based Scenarios, instructors develop a motivational goal for learners that requires the application of certain skills and competencies. Relevant for most fields.

Blend, from "Direct" to "Social"

• Case-Based Teaching can be used to engage students in critical thinking and decision making about realistic problems in a discipline. Relevant for most fields; used extensively in law, business, medicine,education, architecture, and engineering.

• In Guided Design models, students work independently on mastery learning materials (typically outside of class), then apply this knowledge to authentic problems within class. Relevant for instructors wishing to spend less class time on basic course facts and concepts, opting instead for discussion and application to more elaborate problems.

"Social" Models, Student-Teacher Negotiated

• Anchored Instruction prescribes learning through realistic problems and allows students to experience the same professional dilemmas faced by experts in a given field. Relevant for most fields; originally designed for mathematics.

• Cognitive Apprenticeship involves instructors in outlining, then modeling expert-like processes for students. Students apply the processes on their own, utilizing teacher-developed scaffolds. Students discuss their reasoning processes with other learners. Relevant for some fields (e.g., writing, mathematics).

• Cooperative Learning involves students in collaborative or team-based tasks (e.g., group problem solving, paper writing, projects). Students are responsible for their own and for their teammates' understanding. Relevant for any discipline; valuable for promoting reasoning and critical thinking.

Blend of "Social" and "Radical"

• Constructionist; Project-Based Models suggest students learn about topics by developing materials or completing a design task (e.g., Web pages, videos, models). This "learning by creating" approach is applicable to engineering, architecture, history, social sciences, and other fields where students interpret and re-present information in varied forms (i.e., two or three interpretations of the same event, two or three solutions or designs for the same problem).

Bottom-Up Models, Student-Centered, "Radical"

• Problem-Based Learning engages student teams in advanced problem solving. Teams are responsible for problem analysis, research, and solutions. Instructors coach teams as needed. Relevant for most fields; used extensively in medicine and business.

• Learning Environments support students on authentic problem-solving tasks through extensive resources, tools, and scaffolds. Relevant for any field with complex problems or any instructor wishing to engage learners in critical thinking.

Education Panels

Innovative Learning Environments and Teaching Paradigms

Learning on the WWW: A Case Study Daniel Perron

Using the WWW for a Team-Based Engineering Design Class Jack Hong, George Toye, and Larry Lefier

MendelWeb: An Electronic Science/Math/History Resource for the WWW Roger B. Blumberg

Integrating Simulations and W3 Courseware Calum Smeaton and Alan Slater

Building a Research-based Interactive Discovery Gallery for Education: The NCSA Bridge Project

R. M. Panoff, M. South, B. Davenport Bell Atlantic - West Virginia WORLD SCHOOL

Kathryn H. Hilts

K-12 Education

Designing a Server with a K - 8 School in Mind Bonnie Thurber and Bob Davis

Organizing Information in Mosaic: A Classroom Experiment Robert E. Wray, III, Ronald Chong, Joseph Phillips, Seth Rogers, William Walsh, and John Laird

The Earth System Science Community Curriculum Testbed Michael J. Keeler

Uniting International Students with WWW and NCSA Mosaic Pat Ridge

A Web of Students, Teachers, Projects and Resources: Current K-12 Use in the United States

A Web of Students, Teachers, Projects, and Resources: Current K-12 Use in the United States

John Clement and Janice Abrahams A Sampling of Collaborative K-12 Projects on the Web

Louis Gomez Useful Information Resources for Education

Jane Smith

Information Organization and Presentation for an Improved Learning Environment

A Web of Resources for Introductory Computer Science Samuel A. Rebelsky

Improving Education through a CS Digital Library with Three Types of WWW Servers E. Fox and N. Dwight Barnette

Courseware for Parallel Computing using Mosaic and the World Wide Web S. Hurley, A. D. Marshall, S. N. McIntosh-Smith and N. M. Stephens

World Explorer System John Schmitz

Distance Education

The Virtual Community: Distance Education and Community Master Planning Josef Stagg

Combining WWW/Mosaic with Realtime Multimedia Conferencing in Distance Education Per Einar Dybvik and Hakon W. Lie

Uses of Mosaic in a University Setting Judy Rice, Rosina Bignall, Dalinda Bond, and Phillip J. Windley

Posters

An Individualised Course for the C Programming Language Robert J. Kummerfeld and J. Kay

Innovative Courseware Development with Mosaic and WWW Rajiv Tewari

Empowering Students in the Information Age Pat Ward and Kristopher Davis

Discovery System John Schmitz, Marsha Woodbury, Mary Connors, Aaron Buckley, and Chip Aubry

The School Development Resource System (SDRS) Critical Issues Server James M. Nauta

Spinning a Useful Weblet Al Globus and Chris Beaumont

Mosaic and Tele-Amphi, Hypertext Hypermedia Services with Interactive Work Tools for Interactive Multimedia Distance Teaching

Nathalie Begoc, Bernard Bouchare, Alain Guittot, Yannick Lecomte, Jean-Francois Leprovost and Jean Seguin

Using WWW as an Information System to Reduce the Average Period of Study by Better Information Providing and to Relieve Administration

Bernd Mueller WILT: A WWW based Interactive Language Teaching Tool

Jonathan A. Lieberman and Nadine O'Connor DiVito Hypermedia for HyperKids

Judy Cossel Rice Communicating Information about Virus Structure and Biology Via the World Wide Web

Stephan M. Spencer, Max L. Nibert, and Jean-Yves Sgro

The Use of Performance Models in Higher Education: A Comparative International Review

Janet Atkinson-Grosjean University of British Columbia

Garnet Grosjean University of British Columbia

Abstract Higher education (HE) administrators worldwide are responding to performance-based state agendas for public institutions. Largely ideologically-driven, this international fixation on performance is also advanced by the operation of isomorphic forces within HE's institutional field. Despite broad agreements on the validity of performance goals, there is no "one best" model or predictable set of consequences. Context matters. Responses are conditioned by each nation's historical and cultural institutional legacy. To derive a generalized set of consequences, issues, and impacts, we used a comparative international format to examine the way performance models are applied in the United States, England, Australia, New Zealand, Sweden, and the Netherlands. Our theoretical framework draws on understandings of performance measures as normalizing instruments of governmentality in the "evaluative state," supplemented by field theory of organizations. Our conclusion supports Gerard Delanty's contention, that universities need to redefine accountability in a way that repositions them at the heart of their social and civic communities.

I. Introduction

In recent years, the imposition of performance models on institutions of higher education has become a widespread practice. National systems are in place in France, Britain, the Netherlands, Scandinavia, Australia, and New Zealand. In federations like Germany, the US, and Canada, individual Länder, states, and provinces have taken the initiative (Brennan, 1999; Woodhouse, 1996). Performance models include, but are not limited to, social technologies like performance indicators. They are situated within broader, ideological mechanisms variously characterized as public sector reform, new public management (NPM), or what Neave, in the context of higher education (HE), calls "the evaluative state" (Neave, 1998; 1988). These mechanisms attempt to impose accountability on public sector institutions and improve service provision, by measuring performance against managerial, corporate, and market criteria. Accountability and service improvement are common goals of all HE performance models. But different national systems adopt different combinations of supplementary goals. These include stimulating internal and external institutional competition; verifying the quality of new institutions; assigning institutional status; justifying transfers of state authority to institutions; and facilitating international comparisons (Brennan, 1999:223). The particular combination of goals depends on specific national contexts, and the balance within them of accountability, markets, and trust (Brennan, 1999; Trow, 1998). But the foundations of these structural changes extend beyond ideological reform of public-sector institutions. They are rooted, as well, in the post-war transition from élite to mass systems of higher education (Scott, P. 1995). Arguably, the momentum of massification alone would have enforced restructuring of the HE system in most jurisdictions (Neave, 1998; Dill, 1998). The combination of HE expansion and the emergence of the evaluative state produces international convergence around the implementation of performance models. Furthermore, convergence proceeds at a far-from-uniform rate. It is modulated by path-dependent national institutions and entrenched cultural traditions, and the divergent starting points of each national system. Broadly speaking, public universities in the Anglo-Saxon countries are moving from a position of strong autonomy to one of subordination to centralized, state control. For continental Europe and Scandinavia, where strong state control was the norm, more control of higher education is being ceded to the institutions. These apparently contradictory trajectories converge at the level of institutional performance and accountability (Henkel and Little, 1999) where, as Newson (1998:113) has pointed out, "criteria such as 'efficiency,' 'productivity,' and 'accountability' are becoming embedded in the routine day-to-day decision-making that takes place in 'local' units throughout the university." At this level, the proliferation of a few dominant models can be explained, in part, by the operation of isomorphic forces within institutional fields, whereby "lead" organizations set the pace for "followers" (Powell & DiMaggio, 1983.) Performance models have now been in place long enough for studies of consequences to be undertaken (Neave, 1998; Dill 1998). For example, a recent 15-country OECD study, under the direction of John Brennan and Tarla Shah of Britain's Open University, considers the impact of performance models in 40 participating

institutions. On the basis of early analyses, Brennan (1999) reports that while impacts are conditioned by the nature of the individual institution and the distribution of authority in the HE system, performance mechanisms appear to have raised the profile of teaching and learning in HE institutions. He finds that overall impact is increased when the mechanisms gain legitimacy at the faculty and department level, and that increased centralization and managerialism is characteristic at the level of the institution. In some countries, Brennan suggests, evaluation and assessment mechanisms tilt the distribution of power away from faculty and towards senior managers and administrators. But in other countries, where the management layer is traditionally weak, the impacts of external evaluations are more important. A potential weakness of this otherwise exhaustive study is its reliance on institutional self-reports. By surveying a wide range of methodologically diverse studies from different national contexts, we hope to distill a robust set of findings. We first construct the theoretical framework of the "evaluative state," through which to view the policy and administrative implications of performance models. We then consider the theoretical importance of accounting tools in performance measurement, before defining the terms and trends in performance-based HE management. Next, utilizing a comparative international format, we summarize the impact of HE performance models in the United States, England, Australia, New Zealand, Sweden, and the Netherlands. Where appropriate, we add the results of cross-national studies. Finally, we attempt to synthesize our findings into a generalized set of consequences, identifying system-level effects, technical performance issues, institutional effects and management issues, impacts on teaching and research, and on faculty and academic departments.

II. The Evaluative State

Fundamental changes in the policies and practices of most OECD countries have followed a cultural shift in the public management paradigm over the last two decades. Public sector reforms induced fundamental changes, not only in policies and practices, but also in the culture underlying the public administration of nation-states (Strange, 1996; Aucoin, 1995; Charih and Daniels, 1997; OECD, 1995; Keating 1998). This new culture took as axiomatic market-like principles of cost-recovery, competitiveness, and entrepreneurship in the provision of public services (Power 1996; Charih and Rouillard, 1997). Criteria of economy and efficiency were supported by “broad accusations of waste, inefficiency, excessive staffing, unreasonable compensations, freeloading, and so forth” (Harris 1998:137). "Rational" corporate management techniques were installed incorporating accounting, auditing, accountability, and performance criteria. The intent was not only to make public institutions less costly and more effective, but also to normalize and entrench private sector principles (Hood, 1991, 1995; Savoie, 1995; Harris, 1998). The application of these criteria to HE produced elaborate exercises in "visioning," "re-engineering," and "quality assurance," structured on the basis of transparent and auditable accountability for performance (Power, 1996). International convergence around these ideals renders the putative retreat of the state somewhat illusory (Dominelli and Hoogvelt, 1996; Strange, 1996; Dale, 1997).

Rather than regulating directly, however, the state now regulates from a distance, assuring accountability through refined forms of "remote control" or steering (Burchell et al., 1991; Barry et al., 1996; Power, 1995). Neave neatly points to the paradox: “what some regard as a lighter form of surveillance…goes hand in hand with a veritable orgy of procedures, audits, [and] instruments of administrative intelligence which, in their scope and number…make those which upheld the state-control model appear rustic” (1998:266). By using these mechanisms to steer from a distance, the state ensures its performance agenda is internalized by the institution. Thus regulation becomes self-regulation, and state control becomes self-control—a type of self-disciplining Foucault (1978) called "governmentality." In his study of Continental European HE systems, Maassen (1997) empirically identified this move. In the countries Maassen studied, detailed regulation of the inputs and processes of HE is no longer practiced. Instead, institutions themselves create the conditions for achieving the outcomes required by the state, thereby demonstrating the effects of “remote steering” (Maassen 1997:125). To induce self-regulation and self-surveillance in institutions, Maassen found that European governments are also abandoning existing rigid legal frameworks—a move Neave (1998) calls "dejuridification"—in favour of "framework laws." Maassen suggests that European HE is undergoing the most far-reaching transition since that from élite to mass systems. What we are seeing, he speculates, might be “only the beginning of a long-term trend that will change HE far more fundamentally than we can imagine” (1997:125). According to Neave, the beginning of this long-term trend was the emergence of the evaluative state “from two very different discourses, the one European and political, the other mainly American and economic” (1998:278). In the first discourse, control of universities mirrored broader democratic issues, while the second was a direct bid to substitute market control for state control. The former tended to predominate in France, Sweden, Belgium, and Spain, according to Neave, while the latter dominated in the UK and the Netherlands and rooted itself earlier. Both discourses converged, Neave says, around three major displacements in HE. One displacement is increasing concentration on strategic planning and systems development. Another marks the emergence of powerful, intermediary "buffer bodies" to serve as the state's agents in evaluation and surveillance. The third is the proliferation of increasingly demanding performance models, including quality assessment and assurance; continuous improvement; performance-based funding, budgeting, and management; strategic planning and budgeting; and total quality management. In one way or another, all these models rely on measurements or "indicators" of performance.

III. Issues in Measuring Performance

Paradoxically, the evaluative state's self-regulating "governmentality" requires fidelity devices to measure and induce compliance. Largely, these calculative practices (Miller, 1994) or rituals of verification (Power, 1995) employ accounting tools, such as budgets, cost/benefit analyses, cost-centre comparisons, financial audits, and an increasing array of performance and compliance audits (Power, 1995; Porter, 1995;

Harris, 1998). Accounting tools enable “actions on the actions of others…to remedy deficits of rationality and responsibility” (Miller: 1994:29). They are characterized by their surveillance and control capacities, i.e. ability to determine norms, then discipline performance against them (Hoskin and Macve, 1993). Despite appearances, accounting techniques and numbers are not neutral reflections of "reality." Rather, they selectively construct reality from complex webs of social and economic negotiations. An accounting "fact" is actually a contingent and partial accomplishment. Yet contingency and partiality disappear in inscription. Tabulated, calculated, and double-underlined, accounting "facts" appear incontrovertible—the very essence of stability, objectivity, and impartiality. In a university setting, the apparent objectivity of such "facts" can undermine autonomy, “open[ing] up the routine evaluation of academic activities to other than academic considerations, and…mak[ing] it possible to replace substantive judgements with formulaic and algorithmic representation” (Polster and Newson 1998:175). A financial calculus thus underpins the discourse of performance in HE, and constitutes its instrumental logic. The instrumentalities include performance indicators, quality indices, and benchmarking standards. In a detailed study of institutions in three commonwealth countries, Miller (1995:1) found that these market-based, managerial instrumentalities “have modified or come to dominate the governance and culture of universities in Australia, the United Kingdom, and Canada”. Commenting on the lack of faculty resistance, Miller argues that as academics become constrained, monitored, and documented by performance criteria, they come to collude in the construction of their own fate (cf Harley and Lowe 1999). Performance indicators (PIs) are the key instrumentality. Watts (1992) studied the major OECD countries, looking at accountability and performance measures. Of the eight commonalities he found, PIs were by far the most significant. PIs replace traditional input measures, like the number of students enrolled, with goal- or result- oriented estimates of outcomes or value-added, such as the quality and employability of graduates. Identifying one of their most contentious aspects, Watts (1992:87) comments that “many of these efforts have found...real problems in trying to measure quantitatively the unmeasurable.” Harris (1998:136) reminds us that despite their objectified and factual appearance, much of the accounting and other data used to construct PIs derives from the subjective exercise of judgement. Similar judgements are also exercised on the indicators themselves, which are interpreted to infer "facts" that then “create the domain of the factual” (Harris, 1998: 136). Because PIs focus on readily quantifiable inputs and outputs, they tend to neglect the more complex social variables that resist measurement (Newson, 1992; Harris, 1998). And, because of the difficulty of linking measurable outputs to inputs and processes, there is a danger is that “targeted goals, as reflected in indicators, often become ends rather than means” (Harris, 1998:136). El-Khawas and colleagues note that “academics have resisted the move towards performance indicators, arguing that [they] are reductionist, offer inaccurate comparisons, and are unduly burdensome” (1998:9). As a result, she notes, some governments are introducing PIs incrementally, requiring universities to generate an increasing amount of quantitative data for intermediary bodies. Others have embedded PIs in institutional contracts or other forms of conditional funding. While debate

continues on their appropriate use, she says, in most countries public officials advocate the development of a few relevant performance indicators, together with comparisons among institutions and over time. She differentiates England, which “took a further step by linking the amount of research funding to performance scores of academic departments” (El-Khawas et al., 1998:9). In the studies cited later, we will find more variation than El-Khawas suggests in the numbers and types of indicators tracked. We will also see that the pattern of linking funding to performance extends beyond research to HE budgets more generally. And we will find performance-linked funding in, for example, the United States, Australia, and New Zealand as well as in England. While there is no single, agreed-upon definition of PIs, the one developed by Cave, Hanney, and Kogan (1991:24) is still applicable:

a performance indicator is an authoritative measure—usually in quantitative form—of an attribute of the activity of a higher education institution. The measure may be ordinal or cardinal, absolute or comparative. It thus includes both the mechanical applications of formulae (where the latter are imbued with value or interpretative judgements) and such informal and subjective procedures as peer evaluation or reputational rankings.

One of the principal causes of controversy surrounding the use of PIs is their link to performance-related funding and budgeting. It is important to differentiate between these terms. According to Burke and Serban (1998:2), “the advantages and disadvantages of each are the reverse of the other. In performance funding, the tie between results and resources is clear but inflexible. In performance budgeting, the link is flexible but unclear.” Performance funding ties separate and usually small allocations of funding directly to institutional performance against a normally limited number of indicators. In performance budgeting, a longer list of indicators provides an overall picture of institutional performance; this then supplies the context in which a decision on the institution's total budget allocation is made. The former enhances the incentive to improve performance, but punishes circumstances beyond institutional control. Further, the small sums allocated are disproportionate to the effort required to generate the data. The flexibility of the latter allows for extenuating circumstances, but diminishes specific incentives to improve (Burke and Serban, 1998.) Johnstone (1998) confirms these differences, and notes that both are rooted in conceptions of administrators as "rational actors" who will maximize whatever is rewarded. According to Johnstone, conventional budget drivers—particularly full-time equivalent enrollments—induce institutions to "over-enroll" at the cost of quality and can lead to a concentration on popular programs that can be taught cheaply (1998:16). In contrast, performance-based budgets use criteria such as degrees awarded, time to completion, graduates' external performance, faculty success in attracting competitive research grants, and faculty reputations with peers. However, says Johnstone, proponents of performance criteria are beginning to realize that there is a need to balance “multiple, difficult-to-measure, and not always compatible goals” (Johnstone, 1998:16). For example, to maximize student accessibility, institutions are encouraged to accept promising but less-qualified students. This goal is incompatible with maximizing

completion rates or postgraduate examination performance. The offsetting advantages and disadvantages of performance funding and performance budgeting helps to explain why increasing numbers of states in the U.S.A. are adopting both systems (Burke and Serban, 1998). While examples of performance models could be found in some states (e.g. Tennessee) as early as the 1970s, by 1998 they were utilized in half the states in the U.S.A. Reported intentions predict that 70% of states will have adopted performance funding or budgeting models by 2002 (Burke and Serban, 1998). There is more than rational judgement at work here; a "bandwagon" is rolling. Organizational theory assists our understanding of this phenomenon. Powell and Dimaggio (1983), for example, have pointed to the role of isomorphic forces in stabilizing institutional and organizational fields around a dominant model. The forces at work may be regulative, normative, cognitive, or any combination thereof, depending on the nature of the field (Scott, R. 1995). Thus the particular combinations of state policy, programs, and funding (regulative); academic values and norms of accountability (normative); and the way the social purpose of HE is framed (cognitive) might be expected to produce fairly similar institutional responses to performance criteria that may, nevertheless, differ in important respects in different national and sub-national contexts. Further, formal organizations like universities and colleges tend to adopt prevailing "rituals of rationality" to increase their legitimacy and chances for survival (Meyer and Rowan, 1977; Kaghan, 1998). These rituals of rationality increasingly include principles of profitability and "good management" derived from the private sector. Public universities and colleges, therefore, can be situated in a larger institutional framework where the system of organizations is isomorphically aligned around ideological commitments to private sector principles of rationality. But as Kaghan (1998:172) points out, institutional theories tend to focus at the macrostructural level and pay little attention the "microdynamics" of specific practices. To attend to this level of detail, we now consider the way performance models are enacted in different national contexts. A comprehensive examination of US and UK experiences is followed by less detailed analyses of Australia, New Zealand, Sweden and the Netherlands.

IV. Performance Models in Context

1.State Models in the United States Policy-makers in the U.S.A. were among the first to experiment with monitoring the performance of publicly funded institutions of higher education. In the 1960s and 1970s, state officials began examining possibilities of allocating resources to institutions according to how well they achieved state objectives and outcomes (Layzell, 1998). Tennessee was the first state to implement performance funding in higher education. Well regarded in the US, the program is considered a success. The Tennessee State Higher Education Board initiated a pilot program in 1975. By 1979, state officials, working with advisory groups, had developed a set of ten performance criteria. These,

and the associated measurement and reporting procedures, were applied to all public universities and colleges (El-Khawas, 1998). During 1980-81, public institutions were able to earn up to 2 percent above formula allocations, based on performance against these criteria (Albright, 1997). The plan has been reviewed and updated at five-year intervals since then. Today, the amount of discretionary funding available to reward good performance stands at 5.5 percent of an institution's overall budget. Explicit goals are targeted over an extended period of time, allowing institutional behaviour to be shaped towards desired ends. Because of isomorphic forces, the success of the Tennessee program led to the development of similar programs in Arkansas, Missouri and Ohio (El-Khawas, 1998). But conformity is far from total. Texas is among several states that have studied, proposed, and rejected performance funding—largely because of a lack of support from state legislators, combined with cumbersome reporting requirements, and reduced institutional autonomy (Albright 1997). On the other hand, the State of South Carolina has adopted measures that ties allocation of the state's entire budget for public higher education to institutional performance against 37 specific indicators (Burke and Serban, 1998) . One notable characteristic of Tennessee-style performance funding is that it is non-competitive. All institutions can access these supplemental "bonus" funds. If one fails to obtain its share of the supplementary funds, the others do not benefit. Generally, however, policy-makers today are less favourably inclined to voluntary institutional improvement; systems of mandated public accountability are becoming the norm. As with the introduction of the Tennessee model, we see a tendency to copy other states' systems, in an attempt to develop a common core of indicators to address common problems. A study by the National Association of State Budget Offices (NASBO, 1996) reviews measures adopted by 38 states in addressing calls for HE improvement and accountability. These include budget reforms, restructuring of governance, performance-based funding, and privatization of teaching hospitals. We cannot report on this study in detail, or present the responses of all the participating states. However, certain states can be considered "indicators" of the changes induced by performance models in all states. Arizona's Budget Reform Act of 1993 resulted in the development of a master list of state government programs in 1995, complete with mission statements of institutions, functional program descriptions, goals, performance measures, funding and staff information. This was the first opportunity for state analysts to determine budgets and funding sources for higher education. Subsequently, in an attempt to increase graduation rates without increasing the budget, a "short" Bachelor's Degree program (three-years) was implemented at Northern Arizona University. As well, certain programs implemented a twelve-month academic year. Faculty could elect to take their break in either fall or spring instead of summer. To ensure a steady supply of enrollees, the Arizona Legislature introduced a bill to provide HE scholarships to students who graduated high school in three consecutive academic years and retained a GPA of at least 3.0 (out of 4.0). State funding would be shifted from the K-12 system to the HE system to fund the new measures. In 1995, Arkansas moved from an enrollments-based funding policy to one focused on productivity outcomes. The Institutional Productivity Committee and the State Board

of Education developed sixteen performance measures. Amendments to the Revenue Stabilization Law resulted in the creation of a Higher Education Institutions Productivity Fund, authorized to provide an additional $5 million and $10 million in fiscal years 1996 and 1997 respectively, on the basis of institutional performance on these measures. Also in 1995, the Governor of California agreed to provide lump-sum funding to the University of California, and California State University for a period of three years for general support, capital outlays, and to service debt requirements. In exchange, the universities were required to increase enrollments and the portability of courses between institutions; implement new productivity and efficiency increases each year; improve student graduation times; and restore faculty salaries to competitive levels. Meanwhile, in the Kansas fiscal 1997 budget, and the Kentucky 1994-1996 Appropriations Bill, appropriation increases to higher education were based on performance funding concepts and principles. On July 1, 1995, Minnesota merged three of the state's public, post-secondary systems under a single governance structure. For 1995 and 1996, a portion of state appropriations to the University of Minnesota and the state's colleges and universities was made contingent upon achievement of performance goals. For example, for the University of Minnesota, $5 million of the 1996 appropriation was placed in a performance incentive account, to be released in $1 million increments for achieving each of five performance measures. The measures related to: a) recruitment and retention of freshman students with high academic averages in 1995; b) increase in the intake of minority students in 1996; c) increase in the number of women and minority faculty hired in 1995-96; d) increase in graduation rates between 1994 and 1996; and e) increase in the number of credits offered through telecommunications between 1995 and 1996. Missouri adopted policies that ensure the recognition of institutional performance through appropriate incentive funding. In fiscal years 1995 and 1996, funding was appropriated to reward institutions based on their attainment of certain goals: a) assessment of graduates; b) graduation of minority students; c) number of students pursuing graduate education; d) teacher-education graduates scoring in the upper half of national exams; and e) job placement rates in major field. In fiscal year 1996, more that $7 million of the ongoing untargeted funding for four- year institutions was distributed according to these performance goals. While other states, including New Mexico, New York, North Carolina, North Dakota, Oklahoma, South Carolina, Utah, Washington, and Wyoming have all undergone budget reform, restructuring, and the implementation of performance measures, none has gone to the extreme of South Carolina. In 1996, at the urging of a group of prominent business leaders, the State Commission for Higher Education implemented the most significant performance-based funding program to date. The program was phased-in. By the 2000 fiscal year, as stated earlier, 100% of state HE funding will be allocated on the basis of institutional performance on 37 specific indicators. This high number of indicators, as well as the total linking of funding to performance, runs counter to conventional wisdom on performance models.

Agendas beyond Performance The above review of performance models makes evident the extent to which they

can be used to advance state agendas other than those strictly concerned with accountability and performance. In the case of Minnesota and Missouri, for example, performance models are used to address state requirements for equity and equality in public institutions. Thus the state can use these models to force HE institutions to advance compliance with long-range state objectives. If the institutions successfully comply, they are rewarded. Otherwise, there is an implicit threat that the state will step in and take control of budgets and governance structures. But state policy is subject to change with each election. In between, there may be insufficient time for political objectives to be fully integrated into an institution's governance and funding structure. A recent study by the State Higher Education Executive Officers (SHEEO, 1997), provides a snapshot of the experience of 48 states in implementing performance measures. The study indicates that:

• thirty-seven states used performance measures in some way • this is more than double the number three years previously • twenty-six states plan to expand or refine current efforts • most states adopt performance measures for accountability purposes • twenty-three states use performance measures to inform consumers about higher

education • twenty-three states use performance measures to distribute state funds to higher

education institutions (Network News, 1998:1-2).

Most of the performance models referred to in this study fail to differentiate between longer-term state interests and short-term public demands. As well, in the twenty-three states where performance measures supply information to consumers of HE, the information reported is deemed more useful to policy-makers, than for assisting individual consumers to make informed educational choices.

Responses to US Performance Models The SHEEO and the NASBO studies cited above seem to indicate a shared understanding between state officers and HE institutions about the importance of performance models. This may not be the case. In a survey of higher education policy issues (Ruppert, 1998), a total of 1008 respondents, consisting of political leaders (n=519) and higher education leaders (n=489), from 12 Midwestern states were asked to identify the most critical issues facing post-secondary education in the approach to the 21st century. Keeping higher education affordable was a major concern for both groups, but political leaders ranked it as their first priority, while higher education leaders ranked it second. Overall, how to pay for higher education (funding policies) was considered the Midwest's second highest priority. For higher education leaders this was the number one priority, while political leaders ranked it sixth out of nine issues. Capacity for change was the third priority for higher education leaders while political leaders ranked this item fifth. Not surprisingly, political leaders ranked ensuring accountability second, and productivity and cost efficiency third priority, while higher education leaders ranked these sixth and eighth respectively. With such disparities on the relative priorities of key issues, will the two groups support one another? Or is the

stage set for increased tensions, in the form of either active or passive resistance to state mandated measures? In analyzing responses to the SHEEO survey, Albright (1998) reports that in states implementing performance-based funding, HE institutions accrue certain advantages. They benefit from increased communication with, and support from political leaders; the funding provides an alternative to enrollment-based subsidies, and acts as an incentive to improve performance. By aligning planning goals with budgets, institutions can respond to calls for accountability and reinforce confidence in higher education. However, the design and implementation of a performance model is not accomplished without difficulty. Ways must also be found to balance decreasing institutional autonomy and increasing state review and control. Qualitative methods must be used to supplement quantitative measures when studying institutional processes. There is a need to overcome the complexities of measuring "quality," particularly as it pertains to student learning, and to find measures that adequately reflect differences in institutional missions. While some states have been more successful than others in introducing performance measures, it is still too early to attempt to identify a single "best" US model. In terms of future prospects, a survey of state finance officers reports data on legislative action plans for 1999 (McKeown-Moak, 1999). From the perspective of state officials, the financial outlook for US higher education is better now than in years. State appropriations reached the highest level ever in FY99, increasing four times faster than the Consumer Price Index. HE's share of state general funds increased for the first time in over a decade. Average tuition fees are rising steeply. State officials proclaim that such positive economic conditions for higher education have not existed in the last two decades. At the same time, administrators in HE institutions prepare for reduced appropriations and increases in the use of performance models. Student debt loads continue to rise at an alarming rate, and institutions that originally welcomed new federal tax credits now face the added costs of compliance and record keeping. Added to this, are increased competition for state resources; demands for up-to-date curricula that keep pace with the economic and market change; approaching reelection campaigns for state legislators; tensions with faculty and staff about internal restructuring to accommodate performance criteria; and threats to restructure HE governance. Taken together, these factors indicate the prospect of continuing struggle for US higher education leaders. A final note: Congress enacted changes to the Higher Education Act in October 1998. Beginning in the 2001 academic year, colleges and universities must submit comprehensive reports on attendance costs for students, to the National Committee on the Cost of Higher Education. NCHE will then publish trend information on tuition fees and financial aid by institution, and compare this information with the Consumer Price Index. Failure to comply will net the recalcitrant institution a fine of $25,000. Compared to the burgeoning costs of reporting, some might consider the fine the more fiscally prudent option for financially- starved institutions.

2. England In England, performance models were first introduced in the early 1980s as an

ideological initiative of the Thatcher government. Continuing under Thatcher's successor, John Major, they then, as in other countries, transcended the partisan divide into Tony Blair's New Labour administration. A number of intermediary agencies are responsible for administering the performance agenda. These include the Higher Education Funding Councils of England (HEFCE), Wales (HEFCW), and Scotland (SHEFC), which administer the Research Assessment Exercise (RAE), and the Higher Education Quality Council (HEQC). Under the recommendations of the Dearing Report, the latter was succeeded by the Quality Assurance Agency for Higher Education (QAAHE) in 1997. QAAHE administers quality audits and the Teaching Quality Assessment (TQA). The Research Assessment Exercises and Teaching Quality Assessments represent longstanding programs of performance assessment. Both are controversial, for various reasons. The purpose of the former is the highly selective distribution of funding in support of high-quality research. It evaluates on the basis of perceived national and international standards. The latter justifies public support on the basis of quality and quality improvement, and rewards "excellence" in these areas. TQA evaluations are mission-dependent. They inform rather than determine funding, and are less oriented to quantitative data than the RAE, although both programs use performance indicators. TQA indicators include student entry profiles; expenditures per student; progression and completion rates; qualifications obtained; and subsequent destinations. Institutions are assessed on six core aspects rated on a four-point scale. The RAE looks for indicators relating to research publications; research grant income; numbers of assistants and students employed; and the research environment. It rates seven categories and relies on the subjective judgements of peer panels concerning the national and international standing of the research departments assessed (Stanley and Patrick, 1998). In contrast to this "arm's length" determination, TQAs involve site visits by external assessors and encourage critical self-assessment of weaknesses as well as strengths. Much of the criticism focused on the RAE stems from the statistical ranking of institutional performance and the publication of those rankings in the media, with subsequent reputational and funding effects. Criticism is also leveled at the underlying methodology, the emphasis on outputs and the reliance on statistical data rather than qualitative assessments, as well as the additional workload institutions face in complying with performance models. The 1997 National Committee of Enquiry into Higher Education (Dearing, 1997) made performance requirements even more explicit. Dearing recommended the development of performance indicators and benchmarks for "families" of institutions with similar characteristics, on the principle that the interpretation of performance should take account of sector context and diversity. In response, the Higher Education Funding Council (HEFCE) set up a Performance Indicators Study Group (PISG) to develop indicators and benchmarks of performance, rather than descriptive statistics. The latter, while they are “helpful in the management of institutions, can only be judged in the light of the missions of institutions and do not purport to measure performance” (PISG, 1999:8). In this regard, the group comments disparagingly on the publication of "misleading and inaccurate" league tables. In the first stage of its study, the group focused on producing indicators for the government and funding councils that would also inform institutional management and

governance. Its immediate priority was the publication of institutional-level, output-based indicators for research and teaching. Process indicators, such as the results of TQAs, were rejected. By the time of its first report (PISG 1999), the group had prepared proposals for indicators relating to: participation of under-represented groups; student progression; learning outcomes and non-completion; efficiency of learning and teaching; student employment; research output; and HE links with industry. All except the latter related to both institutional and sector-levels. Responding to Dearing's concerns about interpretive contexts, the group developed a set of "context statistics" for each indicator to take account, for example, of an institution's student intake, its particular subject mix, and the educational backgrounds of students. These will allow “the results for any institution to be compared not with all institutions in the sector, but with the average for similar institutions” (PISG, 1999:6). The next stage of the study will look at the information needs of other stakeholders, particularly students and their advisers. The third stage will respond to a call from the Chancellor of the Exchequer to improve the indicators on student employment outcomes. The PISG acknowledges that PIs in HE are “complicated and often controversial” and that “the interpretation of indicators is generally at least as difficult as their construction” (1999:12). They note that PIs require agreement about the values (inputs) that make up the ratio, reliable data collection, and a consensus that a higher ratio is "better" or "worse" than a lower ratio. The literature supports that none of these is easily negotiable nor guaranteed in advance.

Faculty Responses to Performance Models in the UK Among faculty and at the institutional level, responses to performance mechanisms tend to follow a "strategy of accommodation" that focuses on technical rather than normative aspects, and involves participation in the development of measures to make them "more meaningful or less harmful" (Polster and Newson, 1998). Consequences of this strategy in the UK include: the imposition of performance accounting systems for rating faculty productivity; favouring of research that attracts funding; a competitive transfer market in the CVs of "high performing" researchers; heavier and lighter teaching loads for "less productive" and "more productive" researchers respectively; an associated deterioration in teaching conditions; and a reordered system of state-appointed buffer bodies to allocate funding on the basis of externally determined criteria (Polster and Newson, 1998: 177). These elements recur in the following detailed discussion of the findings of two UK studies. Each examines the implications of performance models for faculty in English universities Henkel (Henkel, 1997) studied seven disciplines across six different types of universities, interviewing 105 adminstrators and academics at various levels in the hierarchy. The study sought the implications of three performance policies: the research assessment exercise (RAE); the Higher Education Quality Council's (HEQC) academic audits for quality assurance; and the Higher Education Funding Council for England's (HEFCE) teaching quality assessments (TQA). In five of the universities studied, Henkel found a significant trend to "centralized decentralization"—strong central management coupled with maximum devolution of responsibility. This involved the creation of well-defined new roles at the centre, and the proliferation of non-academic support units. In part, these were to mediate the state's performance expectations and

policies, now interpreted as corporate standards. Budgets were being devolved, usually to the department level, and the iteration between the centre and departments was deemed increasingly important. The new challenges were creating adaptation and status problems for administrators in some universities. But in others, administrative roles were expanding to meet the requirements of the new state policies. One administrator referred to his new authority to “open the black box of academic decision making” (Henkel, 1997:140). While those at the centre spoke of iteration, individual faculty and the basic units were more aware of centralized authority. Many academics expressed "bitter resentment" about the inordinate administrative requirements necessary to comply with performance models, and strongly objected to the amount of time taken away from academic work (141). Many expressed nostalgia for the élite system, and saw the new models as attempting to compensate for the consequences of that system's disappearance. Thus, performance models were viewed as connected with “an undervaluing of individualization, excellence, and risk, espousing instead a "predictable mediocrity"” (ibid). Some also saw the new models as facilitating instrumentalism and "satisficing" behaviour on the part of students, as well linking with market values of consumerism and customer-led education. At issue as well was the emergence of differentiated contracts “based on competitiveness, insecurity, the casualization of academic employment, and…the attenuation of institutional loyalty” (142). Henkel's findings are affirmed in a study of what Dominelli and Hoogvelt (1996) describe as the "Taylorization" of academic labour. Taylorization is achieved through the fragmentation, sequencing, and commodification of faculty work “into component parts or activities, each part being translated or "operationalized" into empirically identifiable and quantifiable indicators or measures” (79) These discrete "technical competencies" may then be “subject to cost-efficiency scrutiny and put up for tender” (79). The elimination of professional autonomy is another key aspect. Functional analysis defines "competences," which are then further defined by performance criteria—the assessable outcomes. What are the consequences of "Taylorization" and performance models for academics? Dominelli & Hoogvelt describe increased workloads; shrinking resources; dramatic declines in social status; and truncation of functions. They cite the following statistics:

• between 1987 and 1993, student numbers in HE increased by 50% while academic staff numbers increased by only 10%, and total spending per student fell by 50%. (p.82 and fns 35 and 36)

• in the same period, core staff increased by 1.2% while staff employed on temporary and short-term contracts increased 23% (p83)

• in the OECD, between 1980 and 1990, the UK was the only country with real negative growth in pay (-3.8%) for academic teachers (p83)

Echoing Henkel's findings, these writers suggest that the English performance model is built on the following characteristics: (1) decentralized budget management; (2) peer pressure and peer scrutiny of "performance"; and (3) flexible production techniques.

The UK's Research Assessment Exercise (RAE) The RAE is a major and recurring evaluation of research performance. For a comprehensive Foucauldian analysis of the RAE as a routine operation of surveillance and assessment dependent on coercion and consent, see Broadhead and Howard (1998). The last RAE was 1996; the next will be in 2001. The RAE directly affects the allocation of funds from the higher education funding councils. Council research budgets have not increased for some years so, for institutions, competition for research funds is a zero-sum game with winners and losers. And, since the binary system of universities and polytechnics was unified in 1992, this "flat" amount of funding now has to be allocated to more than 40 institutions—twice the original number (McNay, 1999). Reporting on the consequences of the 1992 and 1996 RAEs, McNay found that “money was a great driver in participating in the RAE and the money that flows from it was the main means by which it exercised influence for behaviour change” (1999:192). Institutional submissions to the RAE describe research performance and plans for each academic area, and list by area all "research-active" staff, together with details of their research output—publications, discoveries, patents, and so on. A series of panels then judge performance—by a variety of different and not necessarily compatible means—against approximately 70 criteria. The scale runs from 1 (research of little consequence) through 5 (research of international renown), to 5* (outstanding) (Williams, 1998). Funds to support research in a particular institution are subsequently calculated from an aggregation of these determinations. Units that do well have funding for the next five years, while poorly rated units try to limit the damage resulting from lost income (ibid.). To discover the impacts of the RAE, McNay conducted 30 institutional case studies; surveyed administrative and academic staff in 15 institutions; and interviewed external stakeholders in the funding councils, industry, learned societies, and professional bodies. Overall, he finds that the RAE's impacts extend beyond funding, to affect “institutional strategies, priorities, and use of general resouces, not just those flowing from RAE (1999:199). He reports the following institutional-level impacts (1999:195-6). First, he found more refinement of research policy and strategy, with research now focused in a smaller number of priority areas. Next, the research function is better managed and more efficient but administrative requirements have increased, with an increase in centralized research management and the number of committees. Third, these changes are primarily expressed through strategic policies and practices relating to research staffing. For example, some universities adopted more exclusionary recruitment criteria favouring "proven" researchers, and used the same exclusionary criteria to designate some existing research staff "non-active." Contradicting other studies, McNay finds “some spending on attracting "stars" [the CV transfer market] but this was marginal” (1999:196). Next, participation in the RAE caused an organizational restructuring that gradually but effectively separated research from teaching. Research centres freed staff from teaching responsibilities and graduate schools focused on research, leaving undergraduate teaching responsibilities to the departments. Overall, 71% of unit heads reported the RAE's positive impact on research, while 62% report its negative impact on teaching. These results are hardly surprising since, as McNay states, “the Dearing

enquiry takes the breach [between teaching and research] as a fait accompli” (1999:198). Finally, and paradoxically, the RAE generated a virement (reallocation) of funds from higher-graded to lower-graded departments. This reallocation was policy in several of the institutions studied. Largely, the virement is a strategic response to an anomaly in the RAE framework. RAE funding flows from "improvement." Top-rated departments have no room for improvement on the RAE scale so receive no increase in funding. But lower-rated areas can improve their performance and increase their funding. Therefore “financially, improvers were better than star performers at the funding ceiling” (McNay, 1999:196). McNay also found internal reallocations of teaching funds to support research activities. At the unit level, heads of research units were generally positive about the impact of the RAE on productivity but expressed concerns about the related increase in stress. Other concerns included: inhibition of new research areas and interdisciplinary research; increasingly conservative approaches to research; and the aforementioned rupture between teaching and research. Two other issues were important at the unit level. First, concern was expressed at the rewarding of publication rather than dissemination. It was felt that the RAE focused too exclusively on prestige journals “mainly read by other academics, including panel members making RAE judgements”, whereas dissemination could often be more effectively achieved through professional and popular journals read by end-users (1999:198). McNay points out that there is a risk of “the academic world…talking only to itself and so sterilising its work” (201). Second, staff management was a major issue for unit heads—both the determination of researcher status (active or inactive), and the reorganization of individual researchers into teams. At the individual researchers' level, only 34% in McNay's study believed the RAE had improved the quality of their research. Most said the exercise had had little or no impact on them, apart from the stress and time-loss associated with the administration of performance exercises. Nevertheless, half now worked more in teams and about a third reported some constraint on choice of research topics. About 58% believed that the research agenda and priorities were defined by people other than researchers, “despite the peer-review process of RAE and the prominence of academics in committees of the research councils and other funding bodies” (199). Williams (1998:1079), a medical researcher involved in leading the RAE exercise for his research group, takes a more combative stance. He believes the RAE uses “restrictive, flawed, and unscientific criteria” and produces “a distorted picture of research activity that can threaten the survival of active and productive research units”. He says the exercise is “unaccountable, time-consuming, and expensive” and should be made more objective. Williams identifies a number of major flaws in the RAE: restrictive survey criteria; dubious performance indicators; loopholes and abuses; inefficiencies and unnecessary expense; subjective unaccountable panel reviews; bias towards established groups; and damage to other aspects of scholarship like teaching. McNay finally considers a number of system level impacts of the RAE. Through what Williams (1998:1079) calls “the double blessing of money and prestige”, and the RAE's competitive nature, the state seems to have succeeded in increasing research achievements in exchange for little if any growth in the overall research budget. However, the costs are no less real.

McNay believes the research/teaching split was at least anticipated and probably intended. Each was funded and assessed separately and held separately accountable. Staff could be designated "teaching only" as well as "research only. And, increasingly, research and teaching were organized in different forms. McNay notes that in the 1996 RAE, the education panel was the only one that would accept teaching material as evidence of research output, and that “the teaching curriculum is being affected as senior staff in universities withdraw support from [departments] with low RAE grades, so that taught courses close” (200). Increasingly, staff rewards are research driven and some teaching funds are being reallocated ("raided") to finance research. Yet, as McNay points out, 80% of HE funding is for teaching. He questions the privileging of the "scholarship of discovery" over the "scholarship of transmission." Another empirically based study investigated the RAE's impact on academic work in two social science and two business disciplines (Harley and Lowe 1999). In the study, some 80% of respondents identified changes and recruitment patterns in their discipline generally. Of these, three- quarters attributed the changes directly to the RAE and a further 18% held the RAE partly responsible. A quarter of the sample characterized the changes in terms of less emphasis on teaching skills; just under two-thirds in terms of greater emphasis on research; and just over two-thirds in terms of greater emphasis on publication. More than three-quarters of the sample cited changes in recruitment and selection policies in their own departments as a result of the RAE. Asked about the changes taking place in their disciplines, 52% characterized them as "bad," 18% as "good and bad," and 23% as "good." In terms of impacts on their own work, 53% said the RAE had influenced it and only 10% indicated no influence whatsoever. 3. Australia In Australia, the country's 40 public research universities and two private institutions are subject to a common framework of funding and regulation, that provides some 60% of their total funding and subjects them to the performance requirements of the Higher Education Funding Act (Marginson, 1998). Reform commenced in 1988, with the abolition of the binary divide between universities and colleges of advanced education, and has continued since that time. Reform included a number of early initiatives: a system of discipline reviews conducted by panels of experts reporting to the minister; the development and testing of a system of performance indicators; allocation of special funds to support performance initiatives; and establishment of a fund to improve teaching (Harman, 1998). There was strong emphasis on managerial modes of operation, adequate levels of accountability, and maximum flexibility in decision-making (Meek and Wood, 1998). Resulting changes have proved so extensive, the process is often referred to as the "Australian Experiment." During 1993-95, a number of innovative performance features were introduced under the rubric of an annual academic audit focused on processes and outcomes (Harman, 1998). Participating universities would conduct a self-evaluation and prepare a detailed portfolio. Peer-review panels would visit and assess the institution's effectiveness in performance outcomes and processes. Universities would be ranked on the basis of effectiveness and outcome excellence and the rankings, together with detailed reports, would be published annually. As in England's RAE, these rankings and their publication were by far the most controversial element of the scheme. Results were widely reported in the media. High-ranked universities found their prestige had

increased, while those who performed poorly experienced reputational damage. Finally, the process would be driven by the incentive of incremental performance funding, allocated according to the rankings, to a maximum of 5% of annual budgets for the top-ranked institutions (Harman, 1998). Institutions have welcomed the additional funding and the program has garnered the support of institutional leadership and others who saw a need for management reforms and a greater client focus. Criticism has been severe however, much focused, as in England, around the contentious ranking system which favours the older, more-established universities; the underlying methodology and the reliance on narrow statistical data; the additional workload; and the negative effects on less-favoured institutions. Some have argued that, especially in teaching and learning, results are temporary. Others share Dill's (1998) opinion, that the cost/benefit ratio of the whole exercise is flawed, especially for the lower-ranked institutions where the consumption of scarce resources on these initiatives has bred staff resentment. Nevertheless, the new government elected in August 1996 committed itself to continuing performance models, albeit with a 5% reduction in operating grants and other funding restraints (Meek and Wood, 1998). The Higher Education Council was made responsible for the government's new program, which includes the integration of various models; institutional reviews of performance improvements every three to four years; and public reporting of performance improvements. As of 1997, universities had been asked to submit a copy of their strategic plan, together with information on the key indicators they used to judge their own performance; current outcomes and intended improvements; and improvements since the last evaluation (Harman 1998:345). A survey by Taylor and colleagues (Taylor et al., 1998) of Australian academics in three universities sought perceptions of the impacts of these and earlier reforms. The survey revealed a high level of concern in many areas and a fairly dismal assessment of future prospects for teaching and research, as well as of the standard of undergraduate students and the extent of academic freedom. The quality of new students, teaching, and research are all identified as in decline, while the undervaluing of teaching in comparison with research persists. Changes in university management to a more corporate style are seen as a threat to academic freedom. More established research universities are concerned that scarce research funds are being stretched too widely. This perception is leading to new divisions in the unified higher education sector. The writers believe that “the tension between staff desire for academic freedom—with its often time-consuming collegial decision-making—and management's need for flexibility is set to continue” (269). Academics' entrenched distrust of administration “will not be ameliorated by the growing managerial desire to conceive of higher education as a corporate service industry”. They conclude that “there is a real danger that management and academic staff will polarize” (ibid.) Another study (Marginson, 1998) coined the term "new university" to capture the institutional impact of the constellation of changes introduced under the reform agenda. This extensive study of 17 universities found: the emergence of a new kind of strategic leader in the presidential office; eclipse of collegial decision-making and emergence of management-controlled, "post-collegial" mechanisms; changes in research management with consequent effects on academic work; commonalities and variations among the "new universities"; and that the changes corresponded with systems of "new public

management." These results are confirmed in the study of governance and management by Meek & Wood (1998). Currie and colleagues (Currie, 1998; Currie and Vidovich 1998) conducted a qualitative study based on interviews of 153 Australian and 100 American academics at six universities: Sydney, Murdoch, and Edith Cowan in Australia; Arizona, Florida State, and Louisville in the US. Additional data were drawn from studies and interviews in Canada and New Zealand. Currie's theoretical framework was constructed around Foucault's concept of governmentality; Lyotard's ideas on performativity; and theories of globalization and pervasive neoliberal market ideals. The focus was managerialism in Australian and US universities. A large majority (+85%) of respondents in the study reported increases in accountability and surveillance over the last five years. There was a sense that performance data were being gathered without any clear perception of how they were to be used. Other perceptions included: declining budgetary control by faculty; predominance of private-sector approaches to management; the sense that universities no longer thought of themselves as primarily educational institutions; and a suspicion that salary and administrative costs for senior and middle management were burgeoning. Divisions between faculty and central administration were reported to be widening, with the academic function becoming subordinated to the administrative function. Full-cost recovery was a major theme (Fisher and Rubenson, 1998), as were efforts to run the university like a business. Those areas closer to the market flourished while the rest had to battle for survival. A majority of faculty (73% in US; 59% in AUS) said decision-making had become “more bureaucratic, top- down, centralized, autocratic, and managerial” (Currie, 1998:26). Of the rest, 19% in the US and 17% in AUS identified democratic decision-making as present at the unit level, while bureaucratic and corporate managerial procedures predominated at the institutional level.

4. New Zealand New Zealand's 32 post-secondary institutions currently enroll some 200,000 students, just over half at the seven national universities. In September 1997, the New Zealand government released a green paper on tertiary (higher) education. The proposals were radical enough to prompt student protests in the streets of Auckland, Christchurch, and Wellington. Some 74 students were arrested attempting to break through a police barricade at the Parliament Buildings in Wellington. A student leader said that the proposals, if enacted, would turn the NZ into the “most right-wing country in the world” in terms of HE funding (Cohen, 1997:A44). An earlier, leaked version of the document used the term "corporatization," and painted a picture of “voucher-bearing students attend[ing] higher education institutions that were more private than public. The institutions would be expected to turn a profit” (ibid.). The language of the official version was more temperate. Its release was followed by a year of extensive consultation and policy development—almost 400 submissions were received—culminating in a November 1998 white paper. In substance, the new policies have been compared to the UK's Dearing Report. Both the UK and NZ documents “suggest a future in which institutions will bear much more responsibility for their own affairs, particularly their financial affairs” (Cohen, 1997:A44). The white paper establishes the ground rules for what the

government calls “a high-performing tertiary sector” (Creech, 1998). The policy direction follows the "evaluative state" model long established in New Zealand. It calls on universities to “lock-in quality” and sets up a number of mechanisms to ensure performance will occur. A new intermediary body—Quality Assurance Authority New Zealand (QAANZ)—will “rigorously test” the teaching and research of every institution in the sector. Funding will depend on performance tests being met. As well, university governance will be reformed. Governing councils will be limited to twelve members, including faculty, outside experts, and students. The government reserves the right to intervene in the affairs of any institution deemed at risk, whether academically or financially, “to protect the taxpayers' investment”. All institutions will have to demonstrate their financial viability before receiving further government funding. The awarding of government funds for research will also be modified, along the lines of Britain's RAE, to introduce competition. Of the $100 million annual research budget, 20% will be set aside initially as a “contestable pool”. To qualify, researchers will need a demonstrated track record in their fields and a "strategic" focus that both benefits the national interest and is cost-effective. In 2001, after a review of the country's research requirements, the plan is to increase the contestable portion of the annual budget to 80%. These recent moves continue the process of cultural change in the New Zealand Higher Education System, that began with the "neoliberal experiment" in 1984. In a program of radical social and economic restructuring, successive governments have reconfigured the country once called "the welfare capital of the world" (Roberts, 1998:3). As in Australia, and Britain under Thatcher and Major, welfare benefits were slashed, user-pay systems were introduced in the public sector, and state assets privatized. The public sphere was transformed by the introduction of quasi-markets (Marginson, 1997). The trend towards devolution with strong state steering is that of the "evalulative state." Bureaucrats now talk the language of "inputs," "outputs," and "throughputs" (Roberts, 1998). Students pay a higher proportion of their educational costs and are designated as "customers." The teacher-student relationship has become contractual rather than pedagogic (Codd, 1997). The emphasis on performance and accountability for results is pervasive. The discourse is of "international competitiveness" and "enterprise culture" (Roberts, 1998:3). Transforming educational institutions into corporate entities “geared toward the ideal of making a profit or at least minimizing losses and efficiencies” has been an important objective (Roberts, 1998:3). Regular performance reviews—based on a variety of performance indicators—are mandated for all levels of the institution, to ensure efficiency objectives are met. The development of a National Qualifications Framework, which breaks down the "educational product" into "unit standards," facilitates the Taylorization (Dominelli and Hoogvelt, 1996) and commodification (Peters and Marshall, 1996) of higher education in New Zealand. 5. Sweden The evaluation movement arrived in Sweden later than elsewhere in Europe, with performance models first appearing on the political agenda towards the end of the 1980s (Nilsson and Naslund, 1997). It is also developing somewhat differently than in other Nordic countries with a clear trend linking program reviews, institutional evaluations,

and national evaluations. Considerable movement can be detected away from the system of highly centralized state control of HE, that saw the country through the expansive period of the 1960s and 1970s. Decentralization was the motif of the 1980s. In 1989, the Minister of Education appointed a national commission to begin investigating the quality of higher education. The Liberal-Conservative government of 1991-94 signalled continuing commitment to deregulation of HE policy, with their 1992 proposition: Universities and Colleges of Higher Education—Freedom for Quality. They disbanded the central HE authority (Universitets-och höhskoleämbetet—UHÄ) and allowed individual insitutions to communicate directly with the Ministry of Education regarding funding. Infused with neoliberal ideology, the new government sought to provide institutions with more autonomy in their dealings with the state. They established a national Secretariat for Evaluation of Universities and Colleges (subsequently to become the Office of the Chancellor) with a mandate to determine “various indicators of quality which can be used as the basis for allocating funds for undergraduate education” (SFS, 1992, cited in Nilsson & Naslund, 1997: 7). When this proved unrealistic at a national level, each institution was given responsibility for establishing a program of quality development. With the institution of the 1994 proposition (Teaching and Research—Quality and Competitiveness) 5% of each institution's resource allocation was based on an evaluation of its quality development program and iimplementation efforts (Nilsson and Naslund, 1997). When the Social Democratic government assumed power in 1994 they did away with this premium, declaring that “quality enhancement is not simply something that is expressed in special programmes but is basically an attitude which must characterize the day-to-day work of each institution (Nilsson and Naslund, 1997:7). The Social Democratic Government also restructured the intermediate authority into separate free-standing units— including the National Agency for Higher Education (Högskoleverket)—to ensure that institutional performance programs were reviewed regularly. Thus, beginning in 1995, efforts to improve the quality of performance, rather than the quality of education, became the focus of assessment. Concurrent with this decision came the announcement that total funding of undergraduate education was being cut by 10%. Bauer and Kogan (1997) argue that while there appears to be a general trend in devolution of authority from the state to institutions, and while the notion of a national system of performance indicators has been abandoned, the State has actually increased its performance requirements. Feedback of results is an important function in the new steering system. Greater autonomy has thus been obtained at the costs of increased demands for accountability, and a more systematic approach to assurance. This is described by Wahlén (1998), as a shift from a system of management by rule, to one of management by goals or results. The system includes the evaluation of individual educational subjects at a National level, the evaluation of education programs for accreditation, and an emphasis on the development of a professional culture in which university staff take responsibility for their work and its results. Recently, as well, a new requirement calls on universities to report student outcomes according to class, ethnicity, and gender. In performance models generally, social engineering ambitions are never far away. Finally, all 36 institutions of higher education in Sweden must undergo a quality

audit to ensure that mechanisms are in place, before the year 2000, for the efficient use of resources. From early indications, university reactions to these moves are mostly positive (Wahlen, 1998:38). In a study of performance systems in the Nordic countries, Smeby & Stensaker (1999) found evidence in all four countries of balance between internal institutional needs and external societal needs. None of the countries link assessment with resource allocation nor are there direct attempts at political steering. Rather, the intent seems to be ameliorative and, as such, may bolster academics' trust in these systems (1999:13). Despite surface similarities, however, differences in design and practice are apparent, reflecting the differing institutional and political endowments of each country. While the authors accept that performance models represent the new "meta-discourse" of HE policy, they suggest that “the processes involved imply, at least in the Nordic countries, very incremental changes to existing structures of power within higher education” (1999:13). In Norway and Finland, for example, these systems are considered "policy experiments." In Denmark, the process is undergoing reassessment at the end of the first round, while in Sweden the history of decentralization and delegation predates the new meta- discourse, extending back to 1977. The authors conclude that “changes to the existing external and internal "power balance" between state and institutions…occur very slowly in all four countries” (ibid.). This study therefore supports a "historical institutionalist" interpretation of path-dependent policy change (Hall, 1997).

6. The Netherlands Together with France and Great Britain, the Netherlands was among the first European countries to institute a formal performance model system in the mid-1980s. The original approach combined self-evaluation with peer review by visiting expert committees. The focus was the program, rather than the institution. The state strongly advocated performance indicators, but these were resisted by universities. The model was refined in the Ministry of Science and Education's 1985 publication Higher Education Autonomy and Quality, which set out a new coordination relationship between the HE sector and the state (Maassen 1998). More autonomy would be granted, but in exchange for cooperation in the development of a comprehensive system designed to regularly assess the performance of university performance. The state would not completely devolve its authority, but would be selective about the arenas of its involvement. As well, the coordination relationship was open to other stakeholders such as employers and local authorities. According to Maassen, the system incorporated a drift towards market-oriented criteria (1998:20). Universities were to develop strategic, performance-based self-knowledge—institutional profiles—and were encouraged to adopt managerial modes of behavior and business principles. Originally, the state intended the Inspectorate of Higher Education to administer the performance model. But through a compromise deal in 1986, the universities and higher professional schools (the Netherlands has a dual system) were able to involve their own representative organizations in the process, and the IHO was bypassed. In practice, two separate systems were developed: one for universities coordinated by the Association of Cooperating Universities in the Netherlands (VSNU); the other for the higher professional sector coordinated by the HBO-Council (Maassen, 1998:21-2). Both emphasized the dual performance goals of quality improvement and accountability. The

VSNU's pilot project began in 1988 and the full system became operational in 1989. While adapted from the North American model, the Dutch system differs because it is collectively owned by the institutions. Largely because of this, over time, the emphasis has shifted from the accountability end of the spectrum towards the improvement end. As well, evaluation results do not feed into the policy or funding process; there are no political consequences. It is felt direct links would lead to strategic behaviour and tend to undermine the improvement process (Maassen, 1998:25). This creates something of a dilemma since real incentives are lacking, yet if incentives were introduced, power games would prevail. According to Maassen, the Ministry's response has been to abstain from short-term interventions, but with the threat of medium- to long-term consequences in the absence of results. Thus the IHO plays a meta-evaluative, monitoring role. So far, the trust invested in institutions appears not to have been misplaced. Faculties and departments seem to take their responsibilities under the system seriously. But, in the absence of incentives, what does "taking responsibilities seriously" mean? Has the low-key approach to performance produced any real change? A study of Dutch higher education by Frederiks & Westerheijden (1994) concluded that the quality of teaching is receiving considerably more attention than before the reforms. Many programs and faculties now have “special committees or specially appointed staff members for the quality management of education” and the topic “has certainly gained an important place on the agenda of [university} decision makers” (1994:200). As well, in contrast to the former singular focus on pedagogy, the input and output characteristics of education—informing potential students, and investigating the labour market prospects for graduates—are now receiving attention. Frederiks & Westerheijden suggest that a "quality culture" is emerging in Dutch higher education. In terms of responses to self-evaluations and the recommendations of visiting peer-review committees, the authors find that while measures are taken to address outstanding issues, the relation between taking measures and observing improvement is obscure. There is no evidence that “the large amount of resources invested leads immediately to an equally large improvement in the quality of education” (ibid.). Nevertheless, the authors find a surprisingly high level of satisfaction with the Dutch performance model. Surprising for two reasons: the traditional reluctance of autonomous organizations to submit to external scrutiny, and the heavy administrative burden involved in constructing an adequate self-evaluation. Despite generally high levels of satisfaction, however, Maassen forecasts change. Specifically, this relates to Holland's role in the EU, and the general harmonization of HE under EU rules. Some type of accreditation approach may well replace the peer review system in the coming decade.

V. Summary and Conclusions

The politics of performance is deeply embedded in the "evaluative state" and the trend to performance measurement is unlikely to be reversed. Indeed, with the normalization of performance expectations and the broadening of knowledge missions

beyond teaching and research, accountability and performance criteria are likely to become ever more complex and embedded. Gibbons predicts “new bench-marking methodologies and the production of a range of bench-marking studies right across the higher education sector” and the use of quality indicators to rank universities “by region, by country and even globally” (1998: 50). With the globalization of performance in prospect, our study shows deep flaws in the conceptualization, measurement criteria, and impacts of these models (see Appendix for more details.) At the technical level, for example, we report lack of clarity in definitions of what constitutes "good performance," and absence of agreement on the adequacy of specific indicators. At the broad system level, we identify increasing differentiation and stratification as universities were defined by their performance rankings as "good," "bad," or "indifferent" performers, and as either "research" or "teaching" institutions. Increasingly, teaching and research are being defined as measurable products rather than processes of learning or enquiry. The proliferation of buffer bodies to mediate compliance with performance models was a feature of all systems studied. In terms of institutional effects, we find a performance- linked focus on missions and visions that promote increased efficiency and calls for more effective, centralized management. Funding is increasingly linked to performance on various measures, variously defined, few of which account for traditional moral or social imperatives. A consistent complaint is the amount of time and expense involved in conforming to proliferating compliance requirements. Individual departments and faculty members report erosion of disciplinary boundaries and decline of collegiality, as well as polarization between departments and the locus of administrative control. Throughout, we find a strong consensus that the costs of compliance with performance regimes far outweigh the benefits. Our review of the experience of different states and institutions raises a number of empirical questions deserving of further study. Is there any evidence that performance-based funding will actually improve institutional performance in the long run? Is the money allocated in these programs a large enough incentive for participation, or is the implied threat of greater state intervention and the loss of autonomy sufficient motivation? Does compliance indicate agreement with the concept and process? Are the ways states deal with non-compliance effective? Do attempts to meet general, institution-level performance measures create goal dissonance and other difficulties at different internal levels? To what extent is the increased demand for detailed reporting an additional burden? Will institutions engage in aggressive competition in attempts to demonstrate compliance? If funding is at stake, is there a possibility that quality of education will be sacrificed in the rush to meet external standards and access additional funds? Only longitudinal empirical research can answer questions like these, and determine whether performance models have enduring value for the conduct of higher education. Further study is clearly needed. Given the evidence to date, there seems to be no "ideal" model or mix. However, if one country stands out, it is the Netherlands. Of those national systems reviewed here, the Dutch seem to have mastered the positive aspects of performance models while avoiding many of the more negative consequences. This is the reason, no doubt, that many countries in Continental Europe

follow a "softer" Dutch-style model, involving qualitative measures and far less prominence for performance indicators than in the UK and US. States, territories, and provinces that have yet to implement these models, might want to consider the contrasting understandings of "performance" in the European and Anglo-Saxon systems, and review relative strengths and weaknesses, before committing resources. In conclusion, few would argue against the ethic of accountability that animates performance models, nor would they disagree that what performance models measure is important. But the "fatal flaw" of performance models is that they reduce performance to what is measurable, when so much of importance is not. Because performance models focus on instrumental and utilitarian concerns, the fear is that the intrinsic value of education may be lost. As it becomes more accountable in a "knowledge society," can the university survive in its traditional form? Survival may depend on a much broader definition of accountability, according to Delanty (1999); one that encompasses public and civic commitment. The best way to guarantee the future of the university, he says, is to reposition it at the heart of the public sphere, “establish[ing] strong links with the public culture, providing the public with enlightenment about the mechanisms of power and seeking alternative forms of social organization.” Further, with university knowledge becoming such a central social, economic and political resource, why be “a tool of the state and market forces”? Why not, instead, become an agent of social and political change? (ibid.). The central task, we would argue, is to embrace a social mission, banish lingering élitism, and advance the democratization of knowledge.

Appendix: Summary of issues and impacts of performance models internationally

In the tables below, we itemize the consequences, impacts, and issues attached to the performance models we reviewed in a set of tables. As this article makes clear, some of these effects are more pronounced in Anglo-Saxon systems, others in European systems. We do not differentiate among the systems nor do we make a determination whether the consequences are good, bad, or indifferent, since these are open to interpretation and will be conditioned by the reader. We have organized the effects into five categories: (i) overall system-level effects; (ii) technical performance issues; (iii) institutional effects and management issues; (iv) impacts on teaching and research; and (v) impacts on faculty and academic departments. Clearly, many of the effects "spill-over" into other categories and may even appear mutually contradictory. It is worth reiterating that, whatever the commonalities, legacies count. Whether cultural, institutional, national, or ideological, the differences between systems are as great as the convergence among them. Finally, the classification scheme is both provisional and heuristic and should not be read otherwise. No attempt is made to rank-order the effects or to exhaustively reproduce every element previously discussed. We try, instead, to convey generalities.

System-level effects

• possible differentiation of universities into research institutions and teaching

institutions • increased stratification, as rankings differentiate "good," "bad," and "indifferent"

performers • more isomorphism as valid differences are erased by conformance to a limited

number of indicators • "newcomers" have to compete with established institutions for limited funds • established institutions have to share "steady state" funding with newcomers • proliferation of external intermediary bodies to administer performance and

quality programs and mandate consequences of noncompliance and "poor performance"

• more "rational" basis for funding decisions therefore better justifications for HE funding

• bilateral systems unified • social engineering ambitions • broad frameworks replace regulation (dejuridifation) • proliferation of stakeholders to be accommodated

Technical performance issues

• lack of agreed-on definitions of what constitutes "good performance" (quality) • lack of agreement concerning the adequacy of specific performance indicators • incompatibilities between performance measures, so that maximizing some

means underperforming on others • inability of quantitative measures to capture contextual and institutional

differences • use of dubious proxies of performance • reduction of complexity • subjective bias in construction and interpretation of measures • appearance of "objective" neutrality • more, and more directly useful data; revelations about previously unknown

aspects of performance • increased ability to "prove" accountability for public funds • susceptibility of measures to changing political agendas

Institutional effects and management issues

• increased efficiency and more effective management • focus on "missions," priorities, and identification of strengths • growth of non-academic management-support functions with the power to

intervene in academic decisions • funding increasingly linked to performance, on various measures, variously

defined • increased competition, both within and between institutions • increased surveillance, both internal and external • centralized, corporate decision making, supported by budgetary and

performance-based criteria

• increased time and costs to administer and conform to proliferating compliance requirements

• possibility that short term gains from compliance will produce "long-term pain" • possibility that the "short term pain" of compliance will produce long term gains • evidence that universities are becoming more market-like; strategic behaviour to

maximize market gains • evidence that universities are abandoning traditional societal and moral

imperatives • better understandings of institutional missions and new, more dynamic

perspectives on the management of institutions • better responsiveness to the needs of public, political, and other stakeholders • limited financial incentives

Impacts on teaching and research

• performance defined as measurable product (publications; external research funding; job-ready graduates) rather than process (learning; inquiry)

• separation of research and teaching • more-rigorous definitions of "active research" • focus on quantity rather than quality of research • focus on quantity rather than quality of publications • devaluation of teaching in some systems, with shift of resources to research • less time for performing teaching and research due to conforming with

compliance procedures • peer-reviewer "burn-out" as more are called on to participate in assessments and

audits • preference for research with measurable outcomes, within a defined time frame,

that carries external funding • shift in pedagogical emphasis as students demand more "relevance" • value-for-money approach: students are no longer learners in pursuit of

understanding, but customers taking delivery of a commodity • impact of cost/benefit and cost-recovery constraints on course diversity • narrow definitions of research performance discourage risk-taking and

innovation

Impacts on faculty and academic departments

• erosion of disciplinary boundaries • decline of collegiality • individual projects discouraged in favour of "team efforts" • polarization between faculties/departments and central administration • detrimental effect of compliance exercises on faculty workloads • decreased faculty time for students and community service • increased stress, anxiety, uncertainty, and resentment • resistance to the measures although this tends to be passive rather than active • "Taylorization" of faculty work means more short-term contracts and less

security • loss of autonomy over individual work • demands for more productivity

Note

The authors wish to acknowledge the financial support of the Humanities and Social Science Federation of Canada, which funded the foundational study for this paper. The authors also thank Professors Kjell Rubenson and Donald Fisher, co-directors of the Policy Centre, for their guidance.

References

Albright, B. N. (1997) Of Carrots, Sticks, and State Budgets, available online at <http://www.nacubo.org/website/accountable/agb.html>. In Accountability and Regulation on Public Higher Education. August 16

Albright, B. N. (1998) Performance-based Funding. Network News, 17(1), February

Aucoin, P. (1995) The New Public Management: Canada in Comparative Perspective. Montreal: The Institute for Research on Public Policy

Baker, J. (1997) Conflicting Conceptions of Quality - Policy Implications for Tertiary Education. Paper presented at. AIC Tertiary Education in New Zealand Conference, 27 & 28 May 1997

Banta, T., Rudoph, L. B., Van Dyke, J. & Fisher, H. S. (1996) Performance Funding Comes of Age in Tennessee. Journal of Higher Education, 67(1)

Barry, A., Osborne, T. & Rose, N., Eds. (1996) Foucault and Political Reason: Liberalism, Neoliberalism, and Rationalities of Government. Chicago: University of Chicago Press

Bauer, M. & Kogan, M. (1997) Evaluation Systems in the UK and Sweden: Successes and Difficulties. European Journal of Education, 32(2)

Bjarnason, S. (1998) "Buffer" Organizations in Higher Education: Illustrative Examples in the Commonwealth, Study supported by the Commonwealth Fund for Technical Cooperation. London: Association of Commonwealth Universities, available online at <http://wwwacuacuk/chems/onlinepublications>

Brennan, J. (1999) Evaluation of Higher Education in Europe. In Changing Relationships Between Higher Education and the State, ed. M. Henkel & B. Little, pp. 219-35. London: Jessica Kingsley Publishers

Broadhead, L. and Howard, S. (1998). "The Art of Punishing": The Research

Assessment Exercise and the Ritualisation of Power in Higher Education. Education Policy Analysis Archives, 6(8). (Entire issue.) Available online at http://epaa.asu.edu/epaa/v6n8.html.

Burchell, G., Gordon, C. & Miller, P., eds. (1991) The Foucault Effect: Studies in Governmentality. Chicago: University of Chicago Press

Burke, J. c. & Serban, A. M. (1998) Funding Public Higher Education for Results: Fad or Trend? Results from the second annual survey, July 1998. State University of New York: The Nelson A Rockefeller Institute of Government

M., Hanney, S. & Kogan, M. (1991) The Use of Performance Indicators in Higher Education: A Critical Analysis of Developing Practice, Second Edition. London: Jessica Kingsley Publishers

Charih, M. & Daniels, A. (1997) Introduction: Canadian Public Administration at the Crossroads. In New Public Management and Public Administration in Canada, ed. M. Charih & A. Daniels, pp. 13-24. Toronto: The Institute of Public Administration of Canada

Charih, M. & Rouillard, L. (1997) The New Public Management. In New Public Management and Public Administration in Canada, ed. M. Charih & A. Daniels, pp. 27-45. Toronto: The Institute of Public Administration of Canada

J. (1997) Knowledge, Qualifications, and Higher Education: A critical view. In Education Policy in New Zealand: The 1990s and Beyond, ed. K. M. Matthews & M. Olssen. Palmerston North: Dunmore Press

Cohen, D. (1997) New Zealand prepares for a major shake- up of its higher education system. Chronicle of Higher Education, December 19, A44, International section

Creech, W. (1998) Students, Quality, and Fairness— Key to Future for Tertiary Education, Minister of Education's Press Statement on the Tertiary Education White Paper, November 18. Wellington: Government of New Zealand

Currie, J. (1998) Globalization Practices and the Professoriate in Anglo-Pacific And North American Universities. Comparative Education Review, 42(1), February, 15-29

Currie, J. & Vidovich, L. (1998) Microeconomic Reform through Managerialism in American and Australian Universities. In Universities and Globalization: Critical Perspectives, ed. J. Currie & J. Newson. Thousand Oaks: Sage

R. (1997) The State and the Governance of Education: An Analysis of the Restructuring of the State-Education Relationship. In Education: Culture, Economy, and Society, ed. A. Halsey, H. Lauder, P. Brown & A. S. Wells. Oxford: Oxford University Press

Dearing, Lord (1997) Report of the National Committee of Enquiry into Higher Education, Hayes, Middlesex: NCIHE

Delanty, G. (1999) The transformation of knowledge and the ethos of the university: Outline of a theory of epistemic shifts. Plenary paper. Re-Organizing Knowledge/Transforming Institutions: The University in the XXI Century. University of Massachusetts-Amherst, September 17-19

D. D. (1998) Evaluating the "Evaluative State": Implications for Research in Higher Education. European Journal of Education, 33(3), September, 361- 78

Dominelli, L. & Hoogvelt, A. (1996) Globalization, Contract Government and the Taylorization of Intellectual Labour in Academia. Studies in Political Economy, 49(Spring), 71-100

El-Khawas, E. (1998) Strong State Action but Limited Results: Perspectives on University Resistance. European Journal of Education, 33(3), September, 317- 31

El-Khawas, E., DePietro-Jurand, R. & Holm-Nielsen, L. (1998) Quality Assurance in Higher Education: Recent Progress; Challenges Ahead. Preparation supported by the World Bank; available online at <www.worldbank.org/html/extdr/educ/postbasc.htm>. UNESCO World Conference on Higher Education. Paris, France, October 5-9, 1998

Fisher, D. & Rubenson, K. (1998) The Changing Political Economy: The Private and Public Lives of Canadian Universities. In Universities and Globalization: Critical Perspectives, ed. J. Currie & J. Newson. Thousand Oaks: Sage

Foucault, M. (1978) Governmentality. In The Foucault Effect: Studies in Governmentality, 1991, ed. G. Burchell, C. Gordon & P. Miller, pp. 87-104. Chicago: University of Chicago Press

Frederiks, M. & Westerheijden, D. (1994) Effects of Quality Assessment in Dutch Higher Education. European Journal of Education, 29(2), 181-200

Gibbons, M. (1998) Higher Education Relevance in the 21st Century. Paper prepared for UNESCO World Conference on Higher Education, Paris, France, October 5-9, 1998. Secretary General, Association of Commonwealth Universities, supported by the World Bank. 70 pages

Hall, P. (1997) The Role of Interests, Institutions, and Ideas in Comparative Political Economy of the Industrialized Nations. In Comparative Politics: Rationality, Culture, and Structure, ed. M. I. Lichbach & A. S. Zuckerman. Cambridge: Cambridge University Press

Harley, S. & Lowe, P. (1999) Academics Divided: The Research Assessment Exercise and the Academic Labour Process. Paper presented at. Critical Perspectives on

Accounting. Baruch, CUNY, April 25-27

Harman, G. (1998) Quality Assurance Mechanisms and their Use as Policy Instruments: Major International Approaches and the Australian Experience since 1993. European Journal of Education, 33(3), September, 331- 49

Harris, J. (1998) Performance Models. Public Productivity & Management Review, 22(2), December, 135-40

Henkel, M. (1997) Academic Values and the University as a Corporate Enterprise. Higher Education Quarterly, 51(2), April, 134-43

Henkel, M. & Little, B. (1999) Introduction. In Changing Relationships Between Higher Education and the State, ed. M. Henkel & B. Little, pp. 9-22. London: Jessica Kingsley Publishers

Hood, C. (1991) A Public Management for All Seasons? Public Administration, 69(Spring), 3-19

Hood, C. (1995) The "New Public Management" in the 1980s: Variations on a Theme. Accounting, Organizations and Society, 20(2/3), 93-109

Hoskin, K. W. & Macve, R. H. (1993) Accounting As Discipline: The Overlooked Supplement. In Knowledges: Historical and Critical Studies in Disciplinarity, ed. E. Messer-Davidov, D. R. Shumway & D. J. Sylvan, pp. 25-53. Charlottesville, VA: University Press of Virginia

Johnstone, D. B. (1998) The Financing and Management of Higher Education: A Status Report on Worldwide Reforms. Preparation supported by the World Bank; available online at <www.worldbank.org/html/extdr/educ/postbasc.htm>. UNESCO World Conference on Higher Education. Paris, France, October 5-9, 1998

Kaghan, W.N. (1998), "Court and Spark: Studies in Professional University Technology Transfer Management," Diss, Seattle, WA, University of Washington.

Keating, M. (1998) Public Management Reform and Economic and Social Development. Paris: OECD

Maassen, P. A. M. (1997) Quality in European Higher Education: Recent Trends and their Historical Roots. European Journal of Education, 32(2), 111- 25

Maassen, P. A. M. (1998) Quality Assurance in the Netherlands. In Quality Assurance in Higher Education: An International Perspective, ed. G. H. Gaither, pp. 19-27. San Francisco: Jossey Bass

Marginson, S. (1997) Markets in Education. Sydney: Allen and Unwin

Marginson, S. (1998) The Best of Times and the Worst of Times; Research Managed as a Performance Economy—the Australian Case. Paper presented at. Association for the Study of Higher Education. Miami, FL, November 4-5

McDaniel, O. C. (1996) The Theoretical and Practical Use of Performance Indicators. Higher Education Management, 8(3), November

McKeown-Moak, M. P. (1999) Financing Higher Education: An Annual Report from the States, online publication at <http://www.sheeo.org/pubs/pubs- finance99.htm>. SHEEO

McNay, I. (1999) The Paradoxes of Research Assessment and Funding. In Changing Relationships Between Higher Education and the State, ed. M. Henkel & B. Little, pp. 191-218. London: Jessica Kingsley Publishers

Meek, V. L. & Wood, F. (1998) Higher Education Governance and Management: Australia. Higher Education Policy, 11(2-3), June, 165-81

Meyer, J.W. and Rowan, B. (1977), "Institutional Organizations: Formal Structure as Myth and Ceremony," in W.W. Powell and P. DiMaggio (Eds.), The New Institutionalism in Organizational Analysis, 1991, Chicago, University of Chicago Press.

Miller, H. D. (1995) The Management of Change in Universities: Universities, State and Econonomy in Australia, Canada, and the United Kingdom. Buckingham: Open University Press and Society for Research into Higher Education

Miller, P. (1994) Accounting as a Social and Institutional Practice: An Introduction. In Accounting as a Social and Institutional Practice, ed. A. G. Hopwood & P. Miller, pp. 1-39. Cambridge: Cambridge University Press

NASBO. (1996) State Innovations in Higher Education Finance and Governance. National Association of State Budget Officers Information Brief, vol. 4, No.1, April. Washington, DC: NASBO

Neave, G. (1988) On the Cultivation of Quality, Efficiency, And Enterprise: An Overview of Recent Trends in Higher Education in Western Europe 1986-1988. European Journal of Education, 23, 7-23

Neave, G. (1992) On Bodies Vile and Bodies Beautiful: The Role of "Buffer" Institutions between Universities and State. Higher Education Policy, 5(3)

Neave, G. (1998) The Evaluative State Reconsidered. European Journal of Education,

33(3), September, 265-85

Network News. (1998) State Survey on Performance Measures: 1996-7. Network News: A Quarterly Bulletin of the SHEEO/NCES Communication Network, Vol.17, No.1 (February)

Newson, J. (1992) The Decline of Faculty Influence: Confronting the Effects of the Corporate Agenda. In Fragile Truths: Twenty-Five Years of Sociology and Anthropology in Canada, ed. W. K. Carroll, L. Christiansen-Ruffman, R. Currie, F & D. Harrison, pp. 227- 46. Ottawa, ON: Carleton University Press

Newson, J. (1998) The Corporate-Linked University: From Social Project to Market Force. Canadian Journal of Communication, 23(1), Winter, 107-24

Nilsson, K. A. & Naslund, H. (1997) Towards a Swedish Evaluation and Quality Assurance System in Higher Education. Http://www.evaluat.lu.se/dokument/evaluation.htm. August 15 1999

OECD (1995) Governance in Transition: Summary, PUMA Committee, 1990-95. Paris: OECD

OECD (1999) IMHE: Institutional Experiences of Quality Assessment in Higher Education. Available online at <http://www.oecd.org//els/edu/imhe/instexp.htm>, for information contact: John Brennan at <J.L.Brennan@open.ac.uk>. August 14 1999

Peters, M. & Marshall, J. (1996) The Politics of Curriculum: Busnocratic Rationality and Enterprise Culture. Delta, 48(1), 33-46

PISG (1999) Performance Indicators in Higher Education, First report, Performance Indicators Steering Group, First report—February. London: Higher Education Funding Council for England

Polster, C. & Newson, J. (1998) Don't Count your Blessings: The Social Accomplishments of Performance Indicators. In Universities and Globalization: Critical Perspectives, ed. J. Currie & J. Newson, pp. 173-91. Thousand Oaks: Sage

Porter, T. M. (1995) Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton: Princeton University Press

Powell, W. W. and DiMaggio, P. (1983) “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields,” in W.W. Powell and P. DiMaggio (eds) The New Institutionalism in Organizational Analysis, 1991, Chicago, University of Chicago Press

Power, M. (1995) The Audit Society: Rituals of Verification. Oxford: Oxford University

Press

Power, M. (1996) Making Things Auditable. Accounting, Organizations and Society, 21(2/3), 289-315

Roberts, P. (1998) Rereading Lyotard: Knowledge, Commodification and Higher Education. Http://www.sociology.org. Electronic Journal of Sociology, 3(3)

Ruppert, Sandra (1998) Survey Results of State Political and Higher Education Leaders. Available online at <http://www1.umn.edu/mhec/summit/issues.htm>

Savoie, D. J. (1995) What is Wrong with the New Public Management? Canadian Public Administration, 38(1), Spring, 112-21

Scott, Peter (1995) The meanings of mass higher education, Buckingham: Open University Press

Scott, W.R. (1995), Institutions and Organizations, Thousand Oaks, Ca, Sage.

SHEEO. (1997) Focus on Performance Measures, available online at <http://www.sheeo.org/network/nn- feb98.htm>

Smeby, J.-C. & Stensaker, B. (1999) National Quality Assessment Systems in the Nordic Countries: Developing a Balance between External and Internal Needs. Higher Education Policy, 12(1), March, 3-14

Stanley, E. C. & Patrick, W. J. (1998) Quality Assurance in American and British Higher Education: A Comparison. In Quality Assurance in Higher Education: An International Perspective, ed. G. H. Gaither. San Francisco: Jossey Bass

Strange, S. (1996) The Retreat of the State: The Diffusion of Power in the World Economy. Cambridge: Cambridge University Press

Taylor, T., Gough, J., Bundrock, V. & Winter, R. (1998) A Bleak Outlook: Academic staff Perceptions of Changes in Core Activities in Australian Higher Education. Studies in Higher Education, Oct98, Vol 23 Issue 3, P255, 14p, 23(3), October, 255-69

Trow, M. (1998) American Perspectives on British Higher Education under Thatcher and Major. Oxford Review of Education, 24(1), March, 111-30

Wahlen, S. (1998) Is there a Scandinavian Model of Higher Education? Higher Education Management, 10(3)

Watts, R. (1992) Universities and Public Policy. In Public Purse, Public Purpose: Autonomy and Accountability in the Groves of Academe, ed. J. Cutt & R. Dobell, pp. 75-91. Ottawa, ON: Institute for Research on Public Policy and Canadian Comprehensive

Auditing Foundation

Williams, G. (1998) Misleading, Unscientific, and Unjust: The United Kingdom's Research Assessment Exercise. BMJ: British Medical Journal, 316(7137), 4 April, 1079-83

Woodhouse, D. (1996) Quality Assurance: International Trends, Preoccupations, and Features. Assessment and Evaluation in Higher Education, 21(4), December, 347-57

About the Authors

Janet Atkinson-Grosjean

Centre for Policy studies in Higher Education & Training 2125 Main Mall University of British Columbia Vancouver, BC, Canada, V6T 1Z4

604-377-8155 Email: janetat@interchange.ubc.ca

Janet Atkinson-Grosjean is conducting interdisciplinary research at the intersection of science and innovation policy, higher education policy, science studies, and institutional and organizational sociology. Her dissertation examines the status of "post-academic" science at the public/private divide, where intellectual property rights move publicly-funded discoveries into the private domain. Her fieldwork concerns Canada's Network of Centres of Excellence program. Other research interests include accountability, performance and governance in higher education, and the sociology of the professions. She holds a doctoral fellowship from the Social Sciences and Humanities Research Council of Canada and has presented her work at national and international venues.

Garnet Grosjean Research Coordinator Centre for Policy Studies in Higher Education and Training

Garnet Grosjean's research draws on the policy and practice of adult and higher education, to study co-operative education as a bridge between the academy and workplace. The central focus is the effect of learning context on university students' conceptions of learning and work. Other research interests include governance and performance of higher education, and the changing vocational role of the university. He is a member of Canada's Western Research Network on Education and Training.

Copyright 2000 by the Education Policy Analysis Archives

The World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-0211. (602-965-9644). The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .

EPAA Editorial Board

Michael W. Apple University of Wisconsin

Greg Camilli Rutgers University

John Covaleskie Northern Michigan University

Alan Davis University of Colorado, Denver

Sherman Dorn University of South Florida

Mark E. Fetler California Commission on Teacher Credentialing

Richard Garlikov hmwkhelp@scott.net

Thomas F. Green Syracuse University

Alison I. Griffith York University

Arlen Gullickson Western Michigan University

Ernest R. House University of Colorado

Aimee Howley Ohio University

Craig B. Howley Appalachia Educational Laboratory

William Hunter University of Calgary

Daniel Kallós Umeå University

Benjamin Levin University of Manitoba

Thomas Mauhs-Pugh Green Mountain College

Dewayne Matthews Western Interstate Commission for Higher Education

William McInerney Purdue University

Mary McKeown-Moak MGT of America (Austin, TX)

Les McLean University of Toronto

Susan Bobbitt Nolen University of Washington

Anne L. Pemberton apembert@pen.k12.va.us

Hugh G. Petrie SUNY Buffalo

Richard C. Richardson New York University

Anthony G. Rud Jr. Purdue University

Dennis Sayers Ann Leavenworth Center for Accelerated Learning

Jay D. Scribner University of Texas at Austin

Michael Scriven scriven@aol.com

Robert E. Stake University of Illinois—UC

Robert Stonehill U.S. Department of Education

David D. Williams Brigham Young University

EPAA Spanish Language Editorial Board

Associate Editor for Spanish Language Roberto Rodríguez Gómez

Universidad Nacional Autónoma de México

roberto@servidor.unam.mx Adrián Acosta (México) Universidad de Guadalajara adrianacosta@compuserve.com

J. Félix Angulo Rasco (Spain) Universidad de Cádiz felix.angulo@uca.es

Teresa Bracho (México) Centro de Investigación y Docencia Económica-CIDE bracho dis1.cide.mx

Alejandro Canales (México) Universidad Nacional Autónoma de México canalesa@servidor.unam.mx

Ursula Casanova (U.S.A.) Arizona State University casanova@asu.edu

José Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es

Erwin Epstein (U.S.A.) Loyola University of Chicago Eepstein@luc.edu

Josué González (U.S.A.) Arizona State University josue@asu.edu

Rollin Kent (México) Departamento de Investigación Educativa-DIE/CINVESTAV rkent@gemtel.com.mx kentr@data.net.mx

María Beatriz Luce (Brazil) Universidad Federal de Rio Grande do Sul-UFRGS lucemb@orion.ufrgs.br

Javier Mendoza Rojas (México) Universidad Nacional Autónoma de Méxicojaviermr@servidor.unam.mx

Marcela Mollis (Argentina) Universidad de Buenos Aires mmollis@filo.uba.ar

Humberto Muñoz García (México) Universidad Nacional Autónoma de Méxicohumberto@servidor.unam.mx

Angel Ignacio Pérez Gómez (Spain) Universidad de Málaga aiperez@uma.es

Daniel Schugurensky (Argentina-Canadá) OISE/UT, Canada dschugurensky@oise.utoronto.ca

Simon Schwartzman (Brazil) Fundação Instituto Brasileiro e Geografia e Estatística simon@openlink.com.br

Jurjo Torres Santomé (Spain) Universidad de A Coruña jurjo@udc.es

Carlos Alberto Torres (U.S.A.) University of California, Los Angeles torres@gseisucla.edu

Return to my Home Page

Vertical Degree Differentiation Within Academic Organizations: The Problematic Introduction of the Three-Level System of Higher Education* in Bulgaria

Paper Presentation

INTRODUCTION:

OVERHEAD 1 (Title):

Initially, I requested this one quarter of an hour with the idea that I would simply introduce my research project. However, although the research continues, I have developed a paper on my findings and in this brief presentation I will attempt to summarize its main points. The paper focuses on the Vertical Degree Differentiation Within Academic Organizations: The Problematic Introduction of the Three-Level System of Higher Education in Bulgaria.**

OVERHEAD 2 (Overview):

In the following I am going to start with a brief introduction of my research project discussing the problems that initiated it as well as its significance. Next I will approach the analysis of the policy on the introduction of the three-level system of education in Bulgarian higher education through descriptions of the general policy environment in Bulgaria, the educational reform movements, and the reasons for the formulation of that specific policy. Then, I will attempt to provide some insight into successful vertically differentiated models of higher education. Next will come an explanation of the special form of descending vertical degree differentiation of higher education in Bulgaria. I will finally focus on the factors I found to be inhibiting the effective implementation of the policy for the introduction of different higher education tiers.

OVERHEAD 3 (The New Degree System):

The 1995 Law on Higher Education introduced the three-tier system of bachelor, master, and doctor into the existing (mono-phased) structure of higher education. It is important to note that the levels of certification were the first step in this conceptualization; the levels, and content, of training that would correspond to them were to follow. As a basic component of university reform in Bulgaria, this vertical

degree differentiation policy was supposed to serve as a mechanism which would facilitate the universities' adaptation to their changing environment and would synchronize Bulgarian higher education with other more open and efficient international educational models. The biggest challenge in the new system was presented in the effective institutionalization of the undergraduate, bachelor's, level. As you can see, this level did not have a corresponding degree level in the period before 1995. In this sense, a special form of a descending vertical degree differentiation was intended through which the traditional mono-phased higher education was to be divided into two sequential tiers. This mono-phased system of higher education in Bulgaria, inherited by the period of socialism, ended with a diploma which provided the necessary document for entering a specific profession. Getting a job was predetermined through centralized governmental planning of the number of graduates to be needed each year. Academia, on its part, rejuvenated itself from within, directing a few selected students beyond the diploma into the doctoral studies, the so-called "aspirantura," which ended with the defense of a dissertation and the "kandidat na naukite" (Candidate of Science) degree. Achieving this degree was usually through combining one's studies with work at the same time (assistant positions teaching practical seminars). This type of education responded well to the needs of the society it was designed for.

Confusion around the main objectives of the policy for the introduction of the three-level system of education has inhibited its implementation within deadline (supposed to happen by the end of 1996 as postulated in the Additional Regulation N12 of the 1995 Law on Higher Education) which was postponed till 1999. The delay was due mostly to the belated preparation of the Unified State Requirements for the bachelor programs by the Ministry of Education. The 1999 Law on the Changes and Amendments of the 1995 Law on Higher Education again addressed the vertical degree differentiation backed by the experience gained during the first years of bachelor degree programs in Bulgaria.

OVERHEAD 4 (My Research Project):

Through the case-study methodology, I analyze the introduction of the three-level system of higher education at large Bulgarian universities. The selected sites are: Sofia University, Plovdiv University, and Veliko Turnovo University. I am interested in the way this higher education policy on the vertical degree differentiation was formulated and implemented in the academic organizations. The ultimate goal of the research is twofold: to provide a critical evaluation of the impact of the implementation of this policy upon the institutions' structure and academic content and to examine the relationship between intended change and educational policy. A number of reasons justify the need for a study of this kind. On the one hand, for forty-five years, the Bulgarian socialist government, through political control and central planning of goals, structure, and content of higher education, imposed principles of university administration and operation that had profound effects not only on the institutional structures, but on the individuals working and studying in them. In this connection, it is widely believed that academic institutions are resistant to change; according to some (Burgen 1996, Popov 1994), stabilizing measures have not been achieved yet

within the Bulgarian higher education system. In this context, it will be important to assess the institutional effectivity in implementing change.

On the other hand, it is indisputable that universities have a critical role to play in the development of the Bulgarian civil society. As history shows, calls for the transformation of education have always been a part of a profound social change. However, institutions of higher education can be instrumental in transforming society only if they reform themselves. The initiated reform is a part of the transformation towards more democratic models of education and it is important to analyze the institutional reform achievements.

Theses:

The introduction of the three-level system of higher education in Bulgaria presents a major conceptual shift away from the existing structure and academic organization of the institutions of higher education. Prescriptive and regulatory normative acts, institutional organization, and legacies from communism have been counter-productive to the effective implementation of the policy of vertical degree differentiation. At the same time, the Bulgarian higher education sector has been committed to integrate the system within the European structures. As a result, a mode of thinking and organization inherited from the past has been framed in a new hierarchically ordered structure that has led to little qualitative change of higher education. Rather than introduce a more holistic, qualitatively different change in the organization and structure of the educational process and setting, the introduction of the policy has presented more or less a reproduction of the old educational process with a new name. More specifically, the way in which bachelor programs were created has not affected the qualitative nature of the process and the contents of study but merely the quantitative aspects of the degree such as the number of years, the course hours, the institutions which could offer the different degrees. In practice, old study plans and programs have been reshuffled and then re-ordered within the new levels. The implications are mostly felt at the bachelor's level where narrow specialization has been preserved thus limiting the student's options already from the start; moreover, there exists a lack of clear qualitative differentiation between the objectives of the training of a specialist (a level which was incorporated into the higher education system with the 1995 Law on Higher Education) and the bachelor degree. As a result, limited is also the institutions' potential for creating mechanisms for the effective adaptation to the changing environment, the fluctuating market, and mass education.

Policy Analysis:

A) The Policy Environment - a Backdrop for the Problems

1. The System of Higher Education Policy in Bulgaria

Political judgment, complemented by recourse to bureaucratic norms, has been the distinct way for characterizing university political formation. This tendency has been quite strong during the communist period but has remained intact during the past 10

years after the 1989 political upheaval. To my mind, this approach has resulted in inconsistency of higher education policy development and governance.

OVERHEAD 5 (The Reform Movements)

2. The Pendulum of the Higher Education Reform Movement

After 1989, higher education reform in the country can be broken down into three different stages: before the Law on Academic Autonomy of 1990, between the 1990 Law and the 1995 Law on Higher Education, and after the 1995 Law and the 1999 Law on the Changes and Amendments to the 1995 Law on Higher Education (the latter will be effective from the 1999-2000 academic year). During each of these stages, different normative documents, prepared and approved by the respective political governments, have directed the reform in diverse ways. At the present moment, the state authorities (external to the system of higher education) play an overriding role in directing university change. Current developments in Bulgarian higher education are characterized by a more invasive step in of the government with regards to curricula, finances, student selection, defining of specialties and programs.

OVERHEAD 6 (Reform Stages and Reasons Behind the Policy)

B) Reasons Behind the Formulation of the Policy for the Vertical Degree Differentiation of Bulgarian Higher Education:

- to synchronize Bulgarian higher education with more open and efficient systems;

- to stimulate in-ward restructuring of the institutions of higher education in order to facilitate their adaptation to the changing social, political, and economic environment;

- to provide means for broadened entry of increased number of students without affecting the whole system of higher education;

- to "mend" some of the effects from the previously active Law on Academic Autonomy (1990-1995) through providing opportunities for the combination of the high number of newly-created narrow specialties at the undergraduate level; it could also provide the basis for future mergers thus reducing the increased number of institutions;

- as a reaction to previous institutional experiments with program cycles and degree differentiation.

C) Main Characteristics of Successful Vertical Differentiation Models (the Anglo-American Examples)

The main components of vertical differentiation have been: general education, research, and practical training. Throughout the years, the more specialized,

professional, graduate degrees of higher education, those of a master and a doctor, gradually developed and were incorporated into the educational systems. At the same time, the lower, bachelor, level became the place where the disciplines of the graduate school could be taught as a background for, or introduction to, further professional studies, or as a means of general education. Some conclusions from these examples can be that effective conceptualization of a bachelor's degree would involve the provisions of some basis of specialized knowledge, transferable skills, and personality development.

The Bulgarian Peculiarities: The Form of Descending Vertical Degree Differentiation in Bulgarian Higher Education

Until very recently, Bulgarian higher education followed the Soviet mono-phase model: students studied for five years usually, progressing in large unchangeable cohorts through fixed curricula; graduates received a professional degree designed to place them into a specific type of job in parallel with the planned economy. The intended further vertical differentiation of the system of higher education in Bulgaria is expressed mainly in the introduction of the undergraduate, the bachelor's, level. In other words, the intended change would efficiently break into two different levels the long higher education period. In this sense, the implementation of the policy presents a form of descending vertical differentiation of education. A second feature of this descending differentiation is the inclusion of the so-called semi-higher (poluvisshe) education (a type of vocational skill), in the new higher education structure through the introduction of an even lower, vocationally oriented degree.

Factors Inhibiting the Effective Implementation of the Policy:

1. The challenges of over-specialization and fixed curricula (emphasizing the lecture method of teaching and relying on memorization and reproduction at exams); almost no interdisciplinary courses have been developed; student choice of course is extremely limited within the prescribed curricula.

2. The challenges of the normative framework: decisive for the vertical differentiation of higher education are the 1995 Law on Higher Education, the Law on Changes and Amendments to the 1995 Law on Higher Education, the Unified State Requirements, and the State Registry of Specialties. The latter two documents are approved by the Council of Ministers (Law on Higher Education 1995, Article 9, point 3, paragraphs 4 and 5). Whereas the major two laws do not have clear objectives to this reform directive, the last two normative acts perpetuate the over-specialization character of higher education.

3. The challenges of university organization expressed in the lack of co-operation amongst different faculties and the narrow specialization of the single departments.

____________________________

* As it stands now, the different degree levels of higher education are four after the normative incorporation of the "specialist" level, previously known as the semi-higher level (polu-visshe) into the system of higher education (Law on the Change and Amendment of the 1995 Law on Higher Education from June 30, 1999).

** The concept of vertical differentiation within institutions of higher education belongs to Burton Clark who distinguished between "the horizontally aligned units as sections and the vertical arrangement as tiers. Among institutions, the lateral separations are called sectors; the vertical, hierarchies." (The Higher Education System 1983, p. 36).

References:

Dobrow-King, M. (1998). "New Structure of Bulgarian Higher Education". WENR, November/December 1998, Volume 11, issue 6, pp. 10-11.

Clark, B. (1983). The Higher Education System. Berkeley: The University of

California Press.

Georgieva, P. (1999). "Development or Accountability: Orientation of the Bulgarian Higher Education Quality Assessment System". In Boyadjieva, P. (editor), University Autonomy and Academic Responsibility. Sofia: Open Society Institute.

Gill, J. and Saunders, L. (eds.) (1992). Developing Effective Policy Analysis in Higher Education, New Directions for Institutional Research, N. 76, Winter 1992. San Francisco: Jossey-Bass Publishers.

Kirp, David L. (1983). "Professionalization as a Policy Choice: British Special Education in Comparative Perspective" in World Politics..

Law for the Change and Amendment of the 1995 Law on Higher Education (June 30, 1999). State Newsletter, N 60.

Law on Academic Autonomy of Higher Education Institutions (January 25, 1990). State Newsletter, N 10.

Law on Higher Education (December 12, 1995). State Newsletter, N 12.

Law on Higher Education - including all amendments (July 22, 1997). State Newsletter, N 8, page 17.

Ministry of Education and Sciences (1992). White Book of Bulgarian Education and Sciences. Sofia: University Press "St. Kl. Ohridski."