Design Expertise Amongst Student

Designers NIGEL CROSS, HENRI CHRISTIAANS and

KEES DORST

Abstract This report of a series of protocol studies of designers at varying levels of

competence examines the pedagogical system of integrating developing skills with increasing project complexity as a definition of design expertise. It comments on

different kinds of problem-solving strategies which are used to ensure the appropriateness of the aims of design education, and assesses the relative

importance that novelty and creativity, uncertainty and information, imagination and constructive thought, and drawing and modelling play in the maturation of

designers. The results show a lack of predictability, but that the abilities needed for design expertise can be articulated and developed as educational programmes.

Introduction Educational programmes in schools of design, whether they are schools of architecture, engin- eering or industrial design, are based on the premise that students gradually develop expert- ise as a designer. As students progress through their school’s programme, not only does their specialised domain knowledge become more extensive, but they are also expected to inte- grate this knowledge with their developing skills of design in projects of increasing com- plexity. The principal pedagogical method for the development of design expertise in students is the setting of gradually more complex pro- jects. Senior students are expected to display more developed expertise than junior students. But what constitutes this expertise?

Relatively few studies have been conducted into the nature of design expertise, and even fewer into the development of that expertise in student designers. One study of students, how- ever, was that by Lawson [l l , who investigated the problem-solving abilities of architectural students in comparison with science students. He concluded that the designers used ‘solution-

focused’ problem-solving strategies, whereas the scientists used ‘problem-focused strategies. That is, the scientists preferred to try analysing the problem before attempting to generate sol- utions, whereas the designers proceeded by generating solutions without attempting fully to ‘understand’ the problem. Lawson also found that these differences in problem-solving behaviour seemed to be a result of the different educational programmes that the students had experienced. Cross [21 has suggested that the designer students’ problem-solving strategy was an appropriate one for the ill-defined nature of design problems. The architectural students’ design education, therefore, did seem to have cultivated appropriate cognitive skills relevant to the development of design expertise.

In recent years there has been renewed inter- est in trying to understand and build models of ‘how designers think’ 131. This interest grows from the recognition that we know very little about the ‘natural intelligence’ of design, but that such knowledge would not only help in the development of ‘artificial intelligence’ in design but also in the development of natural design

0 NSEAD. 1994

40 N. CROSS, H CHRISTIAANS and K. DORST Desian ExDertise Amonqst Student Desiqners

skills and expertise through design education. One of the major research tools used in trying to understand design thinking has been that of protocol studies [41, and there have been several such studies reported in recent design research literature [51[61[71I81.

Our own interest in such research in design thinking stems from methodological and peda- gogical issues of the development of design ability. We would like to be sure that the aims of design methods and design education are to reach high levels of design ability, and that the proposed means of reaching these levels are soundly based. As part of our research pro- gramme in the Faculty of Industrial Design Engineering at the Delft University of Tech- nology, we have therefore conducted a series of protocol studies of designers at varying levels of competence, including studies of junior (2nd year-level) students and senior (4th year-level) students. In this paper we summarise some of the background theory that has influenced our research, and report on the experiments with student designers.

Aspects of design expertise From reviews of research studies in ‘design thinking’ [91[101 it is possible to identify some aspects of the cognitive behaviour of designers that we can say exemplify expertise in design. For example, from interview studies we know that the following aspects can be expected to be demonstrated by experienced, successful designers.

Expert designers produce novel, unexpected solutions It is usually stated that designing is pre-emi- nently focused on generating creative solutions for ill-defined problems [ill. Designers them- selves make comments such as, ‘The solution that I came up with wasn’t a solution to the problem at all! But it worked. . .from a com- pletely different point of view’ 1121. However, neither the concept of creativity nor the acqui- sition of competence in resolving ill-defined problems is clearly understood. In order to observe whether and how student designers

display this aspect of expertise, in our study cre- ativity was defined as product creativity - i.e., the creativity of the product designs produced by the students was assessed. This product cre- ativity assessment was also used as a dependent variable to test other aspects of the student designers’ developing expertise.

Expert designers tolerate uncertainty, working with incomplete information

Because of the ill-definedness of the design problem it is not certain what available infor- mation is relevant to its solution. Furthermore, the designer’s information-processing capability is not so fast and sure that all new information that might be relevant to the problem can be structured, stored and accessed. Designers are therefore used to ‘living with uncertainty’ and they adopt the solution-focused strategy of resolving uncertainty through the early gener- ation of solution ‘conjectures’. In our experi- ments, information on the set design problem was available to the student designers on demand. We could therefore record which information was actually requested and trace the apparent patterns of information pro- cessing.

Expert designers apply imagination and constructive thought

Designers frequently comment on what they see as the crucial role of ‘intuition’ in the resol- ution of design problems. Of course, what expert designers also rely upon is their know- ledge of solution prototypes or ‘protomodels’ which they draw upon for solution conjectures [131. From the discussion above, we can see that the ill-structuredness of the problem and the inherent uncertainty of the strategy for its resol- ution force designers to use their imagination and not to wait until all relevant information has been gathered and ‘processed. They use their imagination to construct a productive relation- ship between the problem as given and emerg- ent models of its solution. This is not so much ‘intuition’ as a necessary rationality for design- ing [141. By recording the ‘thinking aloud’ proto- cols of our student designers we were able to

0 NSEAD. I994

N CROSS, H. CHRISTIAANS and K DORST Design Expertise Amongst Student Designers 41

look for evidence of the development of these appropriate, ‘designerly’ ways of thinking.

Expert designers use drawings and other modelling media as means of problem-solving The principal working method of designers is the sketch drawing, and this use of drawings is always regarded as crucial by expert designers - they ‘think with their pencil’ U51. The early sketches produced by a designer during prob- lem-solving are not meant as communications with anyone else, but only with oneself. The drawings themselves are therefore a kind of ‘thinking aloud’ and are an important element in the analysis of the experiment protocols. They are a reflection of cognitive processing, by which information can be reduced and kept under control.

Experiments In our study of undergraduate student design- ers, we sought to compare ‘novice’ (junior student) and ‘intermediate’ (senior student) levels of design expertise. The subjects were ten students at 2nd year-level and ten students at 4th year-level in the School of Industrial Design Engineering, University of Delft. Half of each group of students was selected from those with relatively low previous grades in design project work, and half with relatively high previous grades.

Each subject worked alone on the same set design project for a period of approximately two-and-a-half hours, in a room where unob- trusive video cameras and microphones recorded their actions and comments. The sub- jects were first given a short training problem, to introduce them to the protocol study exper- imental technique of thinking aloud. An exper- imenter was also in the room, to provide further problem information on request and to stimu- late ‘thinking aloud if this lapsed.

The set design project was to produce one or more concepts for a new ‘litter collection sys- tem’ to be installed in new railway carriages for the Dutch railways. Although framed as a ‘sys- tem’ problem, most subjects concentrated on the principal component, the litter bin. Subjects

were given a short design brief outlining the problem, the various stakeholders involved (e.g. the railway company, manufacturer, pass- engers, cleaners), and their own role as the designer asked to produce initial concepts.

All other information that might be required for the project was available in a set of infor- mation sheets. If a subject wanted to know something, they asked the experimenter, who then handed them the appropriate information sheet. This provided a quick and natural flow of information which also allowed a record to be kept of the type and amount of information asked for. (The information on the sheets was presented as coming from different natural sources, such as trade catalogues, technical sources, research studies, etc.)

When the set of twenty individual protocols was completed, the design concepts generated by the subjects were prepared in a uniform for- mat and submitted to a panel of expert assessors (design tutors) who were asked to grade each concept design for overall quality of the solution and various constituent aspects such as technical feasibility, ergonomic suit- ability and creativity of the concept. Previous work of Christiaans [161 had confirmed the use of such panels of experts as an experimentally- reliable technique of assessment.

Results As suggested in our introduction, we are par- ticularly interested in the pedagogical develop- ment of key aspects of design expertise. Here, we will concentrate on observations from our protocol studies of the development of the aspects of expert designer behaviour as ident- ified by Cross [see 91, i.e. (i) the production of novel, unexpected solution concepts, (ii) the ability to tolerate uncertainty, working with incomplete information, (iii) the use of imagin- ation and constructive thought, and (iv) the use of drawings and other modelling media as means of problem-solving.

(i) Novelty and creativity For this aspect we are able to use measures of the creativity of the solution concepts from the

0 NSEAD, I994

42 N CROSS, H CHRISTIA4NS and K. DORST Design Expertise Amongst Student Designers

TABLE I Average ‘Product Creativity Ratings‘ for groups of subjects by level of expertise and previous design grade

Low design High design Overall Average PCR grades grades averages Novices (junior students) 5.9 4.6 5.3 Intermediates (senior students) 6.3 5.6 5.9 Overall Averages 6.1 5.1 5.6

panel of assessors. ‘Novelty’ of the concept is known to be one dimension of this perceived creativity. A ‘product creativity rating’ (PCR) was computed for each subject’s proposed sol- ution concept from the individual scores of the panel of experts.

Table I shows how the average PCRs for the subjects relate to both the assumed level of design expertise (junior or senior students) and the assumed design ability (previous low or high grades for project work) of the subjects. The results show a small, but not significant, positive effect of expertise in the expected direction - i.e. the seniors scored slightly higher than the juniors on creativity. From these results, the ability to produce novel, creative design solution concepts cannot be said to have developed significantly through the progression of these students from novice to intermediate levels of expertise.

However, the effect of assumed (previous) design ability is greater, but in the opposite direction from that expected - i.e. students of lower ability tended to score higher on creativ- ity. Performance in the formal educational sys- tem by these students therefore does not seem to be a predictor of creativity.

(ii) Uncertainty and information

There is reason to believe that information-ask- ing behaviour is somehow related to feelings of uncertainty. Some subjects seemed to find it difficult to ‘live with uncertainty’, and tried to gather lots of information, perhaps in the hope that solution ideas would somehow be found amongst the data. There was a significant differ- ence between juniors and seniors in the average amounts of time spent on gathering infor- mation. Table I1 shows the averages and stan- dard deviations for the times spent gathering information in the first and second half-hours of the protocols (very little information gathering occurred later than this).

In general, junior students collected and used substantially less information than seniors. This does not necessarily mean that they were better at coping with uncertainty. We believe that it indicates that they were still unused to having to deal with large amounts of data, and indeed were unaware even that certain types of infor- mation were available to them if they asked. Only two juniors, who got rather stuck in the brief, asked for a lot of information.

Amongst senior students, we found that

TABLE I I Time spent in gathering information (minutesj

Junior students Senior students Time period Mean s.d. Mean s.d.

1-30 minutes 10.9 5.4 12.5 3.6 31-60 minutes 5.8 4.8 8.7 6.5

ONSEAD, 1094

N CROSS, H. CHRISTIAANS and K . DORST Design Expertise Amongst Student Designers 43

several of them spent large amounts of time simply gathering information. This did not usu- ally lead to better solution concepts - gathering data was sometimes just a substitute activity for actually doing any design work. The more suc- cessful senior students tended to vary their attention quite rapidly between gathering small amounts of information and reflecting upon it, building a structured representation of require- ments, constraints, etc.

This is illustrated clearly in the activityhime chart of Subject 6 (senior student, solution con- cept judged highly creative) in comparison with Subject 1 (also senior student, solution concept judged low in creativity) shown in Fig 1. The activity pattern of Subject 6 oscillates rapidly between gathering information and reflecting; whereas that of Subject 1 remains fixed on information gathering for rather long periods at a time, with relatively little reflection.

The activity pattern exhibited by Subject 6 is one that we would expect to be typical of inter- mediate and more experienced designers, who have learned to live with the inherent uncer- tainty of design activity, and who are used to quickly gathering and processing information as they develop solution ideas and deeper under- standing of the problem; they involve them- selves in interaction with the problem situation and the available data, rather than simply immersing themselves in unstructured data, as Subject 1 appears to have done. Whilst we were able to observe the expected pattern of behav- iour in some senior students, it was not dis- played by them all, and clearly is not a behav- iour pattern that has been developed consistently by their education.

(iii) Imagination and constructive thought We are able to reflect on the subjects’ thinking strategies by looking in more detail at their pro- tocols. For example, Subject 15 (junior student, solution concept judged highly creative) pro- duced the proposal shown in Fig 2. How did this novel proposal to place the litter bin on the backs of the seats arise?

In the subject’s drawings (Figs 3-5) we can see that he started, like almost everyone, with consideration of the existing litter bin design.

What is significant is that he then proceeded to consideration of the context of the bin, and sketched some interior views of the train car- riage. This led him to consider alternative pos- itions for the bin, such as beside the window rather than below it (the existing position) (Fig 31, and the idea also occurred to him of being able to slide the bin along a rail. This idea was later translated into the proposal of placing the bin on the seat tops and able to slide along (Fig 4). Having had the ‘seat-top’ idea, the protocol transcript shows that the subject then con- sidered other, ‘crazy’ ideas such as placing the bin on the ceiling, but concluded that that would be ‘exaggerated (Figs 3-51.

This seems to be a classic case of applying a ‘rule’ of creative thinking, which is to widen one’s view of the problem and expand the search space of possible solutions, even allowing consideration of ‘crazy’ ideas. Whether or not the subject applied such a rule con- sciously (and his consideration of the ceiling location seems to suggest that it was conscious), this ‘rule’ of creative or imaginative thinking can be taught, and there is therefore reason to sup- pose that this aspect of design ability can indeed be developed through design education.

We would expect to see evidence of similar creative or imaginative thinking increasing from novice through intermediate to expert perform- ance. Novices might not be expected to be con- sciously aware of how to use ‘rules’ of creative thinking, whereas intermediates might be expected to apply such rules rather (self-) con- sciously, and experts to use them with fluency and in a rather unconscious way. In the event, our example, above, is of a novice using the ‘rule’ in a conscious way: we did not observe any similar occurrences amongst the inter- mediate-level subjects.

(iv) Drawing and modelling All subjects made use of drawing, making sketches usually from about halfway through the session. Other modelling facilities were also available, including plastic foam and cardboard. However, apart from two junior students who made foam models, there was very little use made of these facilities.

ONSEAD, 1994

44 N CROSS, H CHRISTIAANS and K. DORST Design Expertise Amongst Student Designers

Reflecting -

Sketching -

Gathering information-

Reflecting

Sketching

Gathering informatior

Reading brief

Time in minutes

Subject 7

Subject 6

5 10 15 20 25 30 35 40 45 50 55 60

t- Reading brief

0 5 10 15 20 25 30 35 40 45 50 55 60

Time in minutes

FIGURE 1 Activity/trme charts of two senior students - one with a high creativity design (SuQect 6) and one with a low-creativity design [SubJect 1 )

0 NSEAD, I994

N. CROSS, H. CHRISTIAANS and K. DORST Desian Expertise Amonast Student Desianers 45

Reflecting

Sketching

Gathering information

Reading brief

Subject 75

Reflecting

Sketching

Gathering information

Reading brief

~ L l ~ ~ L U ~ ~ 0 5 10 15 20 25 30 35 40 45 50 55 60

Time in minutes

Subject 70

~LLLLLlrLlr~LLLlLLLLlrLI~III~l~LlrLlllJllllllLLLLllllLLLlllll~lIILW~ILlJllLLIL1 I l L I I ~ I l I ~ I J

0 5 10 15 20 25 30 35 40 45 50 55 60

Time in minutes

FIGURE 7 ActivityAime charts of two junior students - one with high creativity design (Subject 15) and one with low-creativity design (Subject 1 0)

ONSEAD, 1994

46 N. CROSS, H. CHRISTIAANS and K. DORST Design Expertise Amongst Student Designers

i C a, D 3 u IA

i

0 C 3 3

D a, Y 2 U 0 h

0 NSEAD, I994

N CROSS, H CHRISTIPANS and K DORST Design Expertise Amongst Student Designers 41

h

s 3

3 -4

3

--

3 I' I

h 0

ONSCAD. I994

~ ~~ ~~

48 N CROSS, H CHRISTIAANS and K DORST Desiqn Expertise Amonqst Student Desiqners

a, 2

ONSCAD I991

N CROSS, H CHRISTIAANS and K DORST Design Expertise Amongst Student Designers 49

,

3 - 3 -% 3

\'

\

a

\

0 NSWD. I994

~ ~~~~~~~~

50 N CROSS, H CHRISTIAANS and K DORST Design Expertise Amongst Student Designers

We expected to see the use of sketching start from an early stage of the protocols, particularly amongst the more experienced subjects as they applied a solution-focused strategy to help them develop solution concept ideas and use these ideas in turn to help them identify and structure relevant information. However, this pattern was not evident amongst the senior stu- dents; for example, both Subjects 6 and 1 (senior students) used sketching very little in the first hour of their protocols (see Fig l), with both of them just making a little sketching activity at around twenty-five minutes into the session.

Although most subjects made fairly extensive use of drawing, the type of drawing made was noticed to be influential in terms of the quality (as assessed by creativity) of the proposed SOL ution concept. Reference has been made above to the use of drawings by Subject 15 to help widen his search space by considering the sur- rounding context in which the litter bin is placed. In contrast, Subject 10 (also a junior student, but solution judged low on creativity (see Fig 6) spent a lot of time drawing, from very early in the session. (See the activityhime charts for Subjects 15 and 10, Fig 7). From the actual sketches made by Subject 10 (see Figs % ll), it is clear that he remained narrowly fixed throughout the session on the litter bin itself, and merely developed a variant of the exist- ing bin.

We would expect drawing ability to develop throughout the course of a designer’s edu- cation, and even to continue to develop well into many years of practice. However, we noticed that drawing ability remained rather patchy amongst both junior and senior students, with some of the seniors making it clear that they did not regard themselves as having a well- developed ability in drawing. This did not necessarily mean that they produced worse sol- ution concepts, however. Some senior students produced high-scoring solution concepts even though they were not able to draw well. And as we have seen in our comparison of the draw- ings of Subjects 15 and 10, it is the type of draw- ing that is perhaps most influential on the cre- ative quality of the product; merely producing

lots of sketches is, of course, no guarantee of quality of solution concept.

Discussion and conclusions In this study we have tried to relate the devel- opment of expertise in design students, from ‘novice’ to ‘intermediate’, to certain identified aspects of ability that are exhibited by expert designers. Previous work on problem-solving strategies by Lawson had suggested that the pedagogical methods of design education could be expected to develop increasingly ‘designerly’ cognitive strategies in students. Our more com- prehensive study of a range of abilities suggests a more complex picture. In some cases, our expectations of the pattern of development of expertise in students were confirmed, but in others they were not, and sometimes were con- tradicted.

In terms of our subjects’ abilities t o produce novel, unexpected and creative solution con- cepts, we found that the senior students perfor- med only slightly (and non-significantly) better than the juniors. An unexpected result was that level of design ability, as measured by perform- ance in previous student project work, was negatively correlated with the creativity of the solutions produced in the experiments. Perhaps creativity is not rated highly as a criterion in these students’ normal design project-work assessment procedures.

Junior students gathered less problem infor- mation than the senior students, but this is per- haps more to do with naivety than with the skill of coping with uncertainty. What emerged more clearly, and confirmed our expectations, was that merely gathering data was not a successful design strategy. It seems from the protocols that successful strategies involve an interaction between data gathering and reflection. This ability was not uniformly developed however, with some senior students getting trapped into apparently gathering information for its own sake, rather than beginning to construct ‘conjec- tures’ which might guide their information searching.

We found that imaginative thinking was not widespread amongst the students. From one

0 NSEAD, 1994

N CROSS, H CHRISTIAANS a n d K DORST Design Expertise Amongst Student Designers 5 I

0 NSkAD. I994

52 N CROSS, H CHRISTIAANS and K. DORST Design Expertise Amongst Student Designers

s 1

A-

s,

1

c \ 4 d

a

ONSLAD. 1994

N. CROSS, H. CHRISTIPANS a n d K DORST Desian Expertise Amonast Student Desianers 53

33

--

.‘I f n

‘-5 %

u C

c

0 NSEAII, I994

~

5 4 N CROSS, H. CHRISTIAANS and K. DORST Design Expertise Amongst Student Designers

.- --r 4

R -.

0 NSTAU. 1794

N CROSS, H. CHRISTIAANS and K DORST Design Expertise Amongst Student Designers 5 5

\

ONSEAAD, 1994

~

56 N CROSS, H CHRISTIAANS and K DORST Design Expertise Amongs t S tudent Designers

example only, we suspect that the conscious use of creative thinking techniques can help student designers. The example was an appar- ently conscious attempt to ‘widen the search space’ by considering the broader context of the product being designed.

We had expected that the use of drawing would figure earlier in the protocols than it did. What was clear was that it was the type of draw- ing which was used that was more influential than simply the amount of drawings produced. In concept generation, drawing is an aid to thinking and not a substitute for it. We were surprised that development of drawing skills seemed to be patchy amongst the students, and apparently uncorrelated with design perform- ance.

Whilst we have identified some aspects o f design expertise, and shown that these can be tracked in protocol study experiments, we have not been able to show that these aspects o f design expertise developed in a predictable way throughout these students’ education. The educational programme itself may be faulty, o f course, but students’ individual abilities are also notoriously variable. What we hope to have shown is that abilities relevant to design expert- ise can be articulated and that educationists can begin to develop educational programmes that more consciously and deliberately seek t o develop these abilities in students.

Nigel Cross, Henri Christiaans

and Kees Dorst

References LdWSOn, €3. (1979) Cognitive Strategies in Architectural Design, ETonomics, 22 N o 1, pp. 59-68 Cross, N. (1982) Designerly Ways of Knowing, Design Studies, 3 N o 4 , pp. 221-227 Cross, N., C. Dorst, and N. Roozenburg (ed.) (1992) Research in Design minking, Delft University Press, Delft, The Netherlands Ericsson, K.A. and H.A. Simon (1984) Protocol Analysis: Verbal Reports as Data, MIT Press, Cambridge, Mass., USA Akin, 0. (1986) 7%e Psychology of Design, Pion Ltd, London Stauffer, L.A. and D.G. Ullman (1988) A Comparison of the Results of Empirical Studies into the Mechanical Design Process, Design Sludies, 9 No 2, pp. 107-114 Eckersley, M. (1988) The Form of Design Processes: a protocol analysis study, Design Studies, 9 N o 2, pp. 86-94 Chan, C. (1990) Cognitive Processes in Architectural Design Problem Solving, Design Studies, 11 N o 2, pp. 60-80

-

9

10

11

12

13

14

15

16

Cross, N. (1990) The Nature and Nurture of Design Ability, Design Studies, 1 1 N o 3, pp.

Cross, N. (1992) Research in Design Thinking, in N. Cross et ul. (eds), Research in Desip7 minking, Delft University Press, Delft, The Netherlands, pp. 3-10 Thomas, J.C. and J.M. Carroll (1979) The Psychological Study o f Design, Design Studies, 1 No 1, pp. 5-11 Davies, K. (1985) A Psychological Enquicy into the Origination and Implementation o f Ideas, MSc. mesis, UMIST, University o f hkanchester Schon, D.A. (1988) Designing: rules, types and worlds, Design Studies, 9 N o 3, pp. 181-190 Roozenburg, N. (1993) On the I’attern o f Reasoning in Innovative Design, Dfsign Studies, 14 N o 1, pp. 4-18 Herbert, D.M. (1988) Study Drawings in Architectural Design: their properties as a graphic medium, Journal of Architectuml Education, No 41/2, pp. 2&38 Christiaans, H. (1992) Creutivitv in Deyign, Lemma, Utrecht, The Netherlands

127-140

0 NSEAD. 1994