Development and application of a framework for comparing early design methods for young children

Interacting with Computers 23 (2011) 70–84

Contents lists available at ScienceDirect

Interacting with Computers

journal homepage: www.elsevier .com/ locate/ intcom

Development and application of a framework for comparing early designmethods for young children

R.J.W. Sluis-Thiescheffer a,⇑, M.M. Bekker a, J.H. Eggen a, A.P.O.S. Vermeeren b, H. de Ridder b

a Eindhoven University of Technology, Department of Industrial Design, Den Dolech 2, 5600 MB Eindhoven, The Netherlandsb Delft University of Technology, Faculty Industrial Design Engineering, Landbergstraat 15, 2628 CE, Stevinweg 1, 2628 CN Delft, The Netherlands

a r t i c l e i n f o a b s t r a c t

Article history:Received 18 December 2008Received in revised form 9 October 2010Accepted 15 October 2010

Keywords:DesignChildrenUser centered designDesign space explorationFrameworkDesign skills

0953-5438/$ - see front matter � 2010 Elsevier B.V. Adoi:10.1016/j.intcom.2010.10.002

⇑ Corresponding author. Address: Eindhoven Univement of Industrial Design, Den Dolech 2, 5600 MB Eind+31 641396021.

E-mail addresses: [email protected] (R.J.W. Sluistue.nl (M.M. Bekker), [email protected] (J.H. Egge(A.P.O.S. Vermeeren), [email protected] (H. de Ridder

When designing with young children, designers usually select user centred design methods based on thechildren’s required level of engagement and the inspiration expected to be created according tothe designer. User centred design methods should be selected for their suitability for children and forthe quality of the output of the design method. To understand the suitability of design methods, a frame-work was developed to describe design methods in terms of required design skills as identified by theTheory of Multiple Intelligences. The proposed framework could provide the basis for a tool to comparedesign methods and to generate hypotheses about what design method would work optimally withchildren in a specific school grade. The initial examination of the viability of the framework is a compar-ison of design methods by the number of skills involved; earlier work showed that the involvement ofmore skills (as with, e.g. low-fi prototyping) could result in more options for a design problem thanthe involvement of fewer skills (as with e.g. brainstorming). Options and Criteria were counted to under-stand the quality of the method in terms of the amount of design-information. The results of the currentpaper indicate that 8-to-10-year-old children generate significantly more options in prototyping sessionsthan when they are involved in sessions applying a Nominal Group Technique. The paper indicates that(a) with the framework we can generate hypotheses to compare design methods with children and (b)that the outcome of various design methods, which might lead to very different representations, canbe compared in terms of Options and Criteria. Further usage of the framework is expected to result inempirical support for selecting a design method to be applied with young children.

� 2010 Elsevier B.V. All rights reserved.

1. Introduction

Until recently, the focus in the field of designing with childrenwas mostly on acquiring methodological knowledge, examiningthe question how to design with children, through product evalua-tion or product design (Druin, 1999b; Jensen and Skov, 2005). Nowthat the field has described the role of children in the designprocess, the focus in literature starts to shift from how to apply adesign method, to why to apply a specific design method, to findjustification for the choice for a certain method. Markopoulosand Bekker (2003) propose criteria for comparative assessment ofmethods for children. They suggest to assess methods on threedimensions: (1) the components that constitute the method (e.g.the number of participants, the procedure, data capture, etc.), (2)the measures for assessing a method (e.g. robustness, reliability

ll rights reserved.

rsity of Technology, Depart-hoven, The Netherlands. Tel.:

-Thiescheffer), m.m.bekker@n), [email protected]).

and efficiency) and (3) the special characteristics of children as testparticipants (e.g. verbalization skills, concentration span andgender differences). Their conclusion is that many comparisonsare based on the usage of the first two dimensions, but the charac-teristics of the target user group (i.e. the children) are hardly evertaken into account. This paper proposes a framework forcomparing design methods based on relating characteristics ofchildren with characteristics of design methods. Subsequently,the paper describes a study that examines a hypothesis based onthe framework.

1.1. Children’s characteristics affecting design sessions lack aframework

In the literature on designing with children, evidence is re-ported on how children’s characteristics may affect design ses-sions. For example, gender has an effect on the design methods,as boys have a different behaviour than girls, and also, their behav-iour in one-gender groups differs from that in mixed-gendergroups (Hou et al., 2006; Isomursu and Still, 2004; Stienstra,2003). Furthermore, gender-based behaviours differ per age group.

http://dx.doi.org/10.1016/j.intcom.2010.10.002

mailto:[email protected]






http://dx.doi.org/10.1016/j.intcom.2010.10.002

http://www.sciencedirect.com/science/journal/09535438

http://www.elsevier.com/locate/intcom

R.J.W. Sluis-Thiescheffer et al. / Interacting with Computers 23 (2011) 70–84 71

Greenbaum (1988) suggests that at a young age (under 8 years ofage) boys and girls do not like each other enough to cooperate wellin a design session. However, when they grow older to becometeenagers, they may pay too much attention to each other therebyagain disturbing the design sessions. As far as we are aware, Green-baum’s suggestion has not been investigated in relation to designstudies. Reported observations in a design study with small groupsof young children (two to three) used single-gender groups to be-gin with (e.g. Ruland et al., 2006; Scaife et al., 1997). In design stud-ies with larger groups (e.g. Verhaegh et al., 2006; Bekker et al.,2003) this observation has not been reported as far as we know,neither was gender mentioned as a problem by Druin in ‘‘The de-sign of children’s technology’’ (Druin, 1999a). Group size is anotherpoint of attention. The advantage of designing with one child, orwith a small group is that a sufficient amount of attention can bepaid to each participant and less effort is required to structurethe design session (Heary and Hennessy, 2002). Power structurescould also play a role in design sessions. Power structures (like inan adult–child relationship) have an influence as some childrenare more likely to speak freely in for example friendship groups(Heary and Hennessy, 2002). However, the literature is not conclu-sive as power structures do not seem to have a negative effectwhen talking about technology (Pardo et al., 2005). Speeking freelyand generating ideas in design sessions might be affected by cul-tural differences. For example Moraveji et al. (2007) indicates thatthe effectiveness of brainstorming with young children might besubject to cultural practices in rural China. Also the choice ofinvolving children as co-designers or informants determines howdesign sessions are held. Druin (1999b) proposes to involve chil-dren in different roles, for example as co-designers, giving themresponsibility over a part of the design process, e.g. in conceptdevelopment or requirements gathering. As co-designers the chil-dren are actively involved. Scaife et al. (1997) report that childrencan better be involved as informants than as co-designers. As aninformant the child is more distantly involved, for which differentmethods are more optimal than for a role as a co-designer. Fun andmotivation is investigated in a study on the design method calledKidReporter (Bekker et al., 2003). This study assumes that childrenwho can choose from a set of design activities, will have more funand show more motivation than children who cannot choose a pre-ferred design activity. Looking into the cognitive developmentof children is suggested in studies by for example Antle (2007),Gelderblom (2004) and Wyeth et al. (2003). Especially inGelderblom (2004) and Antle (2007) it is argued that designersneed to be informed about the cognitive development of youngchildren to reach informed design decisions when designing forchildren. However, the developmental issues raised in Antle(2007) and Gelderblom (2004) are not yet further developed intopractical guidelines for how to design with children. This article de-scribes the development of a framework, that matches children’scapabilities to skills required for design activities. Inspired by the-ories from developmental psychology, the framework is a basis forcreating hypotheses about the expected outcome of design meth-ods in relation to the developmental characteristics of children ina specific age group.

1.2. Assessment of early design methods is not explicit enough

The studies on designing with children can also be improved interms of better defined measures for the assessment of the outputof early design methods. Early design methods are idea generationmethods used early in the design process. In the early phases of de-sign, the main objective is typically an exploration of the designspace (Sas and Dix, 2006), a generative phase or information gath-ering phase (Sanders and Stappers, 2008) neatly summarized asthe ‘‘fuzzy front end’’ of a design process (Sanders, 2005).

For adult designers, there are examples of measures for describingoutput of design sessions. For example MacLean et al. (1996)created a method to describe conversations between GUI-designersduring a product development process. Another example is the workof Shah et al. (2003), who explain in much detail measurementsto assess ideation effectiveness. In the field of designing withchildren, we only found examples of measures in studies on usabilityevaluation methods. For example the use of verbal or non-verbalbehaviour in detecting usability problems is studied by Barendregtand Bekker (2005) and Donker and Reitsma (2004). We have notfound such an explicit and replicable assessment of children’soutput for the early stages of design. This is coherent with Jensenand Skov (2005), who found that most research on designing withchildren is done on engineering and evaluation.

Bekker et al. (2003) studied the quality and the characteristicsof the output of design methods, depending on the engagementof the children in KidReporter. Despite the systematic approachof the study, they only assessed the output by the implicit criteriaof only one designer. Kelly et al. (2006) developed Bluebells to opti-mize the relation between designers and children as co-designers.They show that a more iterative involvement of children in shortco-design activities is perceived by designers as more informativeand/or inspirational, than one or two larger design sessions. Never-theless, the criteria on which the designers decided that the activ-ities were more informative or inspirational were not madeexplicit.

The present article applies the work on Questions, Options andCriteria of MacLean et al. (1996) to design conversations with chil-dren. MacLean et al. (1996) found that discussions between design-ers can be analyzed in terms of Options and Criteria. Olson et al.(1992) found that designers explore the design rationale by bring-ing up Options for a design solution and evaluating those Optionsby questioning them and evaluating them with Criteria (furtherreferred to as the QOC-model). In designing with children, weassume the designer (a grown-up) to feed the design conversationwith questions about design ideas. Although the children will inev-itably ask each other questions too, we focus on the answers of thechildren. We explore whether the answers can also be described interms of Options and Criteria and whether design methods have ameasurable effect on the design conversations with children interms of Options and Criteria.

1.3. Development and assessment of a framework for design sessionswith children

Section 1.1 describes several studies that examined parametersaffecting design sessions with children. However, these assess-ments were all based on the success of the processof designing withthe children, rather than the success of the output of designing withchildren. Kelly et al. (2006) stress the importance of a frameworkthat describes the effectiveness of different design methods andthe context in which these methods are useful. Inspired by theorieson developmental psychology, we take a developmental approachto propose a framework that describes design methods in terms ofskills required for design. The framework uses terminology basedon the Theory of Multiple Intelligences by Gardner (1999). Bymatching the skills required to execute a certain design methodwith the children’s skills, we can create hypotheses about the suit-ability of design methods for particular age groups. In the followingsections, the framework and its development process are pre-sented. Furthermore we explain how the framework can be usedto generate hypotheses to study the output of early design meth-ods. Finally, the framework’s use is illustrated with a study thatcompares the outcome of nominal-group-technique (NGT) sessionswith the outcome of prototyping sessions.

72 R.J.W. Sluis-Thiescheffer et al. / Interacting with Computers 23 (2011) 70–84

To observe the differences between prototyping and the Nomi-nal Group Technique, we apply well-defined measures. As ex-plained above, rather than to focus on the design solutions persé, we observe the explanations of the design solutions. In observ-ing the explanations in terms of Options and Criteria (see previousparagraph), the design solutions generated in different media (e.g.design solutions from the Nominal Group Technique and prototyp-ing) become comparable. By comparing the Options and Criteriagenerated through different methods, we can report on the differ-ences in children’s contributions for each design method.

2. Designing with children from a developmental psychologyperspective

To develop a framework for designing with children, we need todescribe characteristics of children and characteristics of designmethods. The common denominator between children and designactivities lies in human behaviour. Therefore we base the frame-work on a psychological theory. To describe the versatile skillsrequired for design, we based our work on the Theory of MultipleIntelligences, developed by Gardner (1983). Gardners theorydescribes human behaviour from nine distinctive perspectives,unlike most psychological theorists, who usually take one or onlya few perspectives (Vasta et al., 1995; Gardner, 1999). First, we willdetermine which of these intelligences are relevant for design,then, we will propose a framework describing design methodsdefined by the intelligences they involve. We will conclude thisSection 2 with a proposal to apply this framework in designingwith children.

2.1. Design requires a versatile approach

In this work, we focus on the early stages of design, in whichdivergence is necessary to explore the design space of a given de-sign problem. Divergence requires an exploration of the designproblem from different angles. In literature on creativity we findthis idea reflected in the work of, for example, De Bono’s ThinkingHats (Bono, 1985). De Bono proposed creative sessions in whichpeople approach a given problem from six predefined perspectives,e.g. an optimistic perspective vs. the perspective of the ‘‘devil’sadvocate’’. Sternberg (2003) explains that to be creative, the skillsavailable to the ‘‘creator’’ should be deployed by different thinkingstyles, e.g. thinking globally vs. locally. In literature, designing isdescribed as different ways to approach a design problem. Forexample, Kelley and Littman (2006) describe 10 personas as arche-types of the members of a design team: the learning, the organiz-ing and the building personas. Their point is that members of adesign team need to differ substantially and have specific qualitiesto come to successful, innovative design. Dorst (2003) explains thatdesigners are strong in working quickly towards a number of dif-ferent solutions to one design problem, hence exploring the designspace from different angles. Similarly Hummels and Frens (2008)explain that designers should be educated to not only use theiranalytic minds in looking for a solution to a design problem, butalso their implicit and intuitive knowledge in the creation of multi-ple artefacts that embody design solutions. Thus in designing withchildren, the children should be facilitated in finding differentways to solve a design problem.

At this point the question arises, what different approaches tosolve a design problem do design methods facilitate? Since weare looking into designing with children, our inspiration sourceswere developmental theories. However, many prominent develop-mental theories focus on describing one or only a few skills. Forexample, Freud described the development of the ego, Piagetdescribed stages in cognitive development and Vygotski focused

on development from a socio-cultural perspective (Vasta et al.,1995; Anderson, 2000). Since we need something more versatile,we looked into the more holistic Theory of Multiple Intelligences,by developmental psychologist Gardner (1999, 1983). In the nextsection we will explain the theory, followed by a discussion onopportunities and limitations for using particularly this theory.

2.2. Different intelligences described by the Theory of MultipleIntelligences

This section introduces the psychological Theory of MultipleIntelligences and describes to what extent this theory is supportedin the field of psychology, in the field of education, and in the fieldof design.

2.2.1. The Theory of Multiple Intelligences explainedThe Theory of Multiple Intelligences (Gardner, 1983, 1999) is

developed on the basis of biological data, of data on neurologicaldeficits, and of developmental theories. Gardner makes two com-plementary claims (Gardner, 1999), (1) the theory ‘‘is an accountof human cognition in its fullness’’ and (2) each human being has‘‘a unique blend of (these) intelligences’’. The theory distinguishesnine different intelligences: (1) linguistic: comprises rhetoricalskills, mnemonic skills (to help remember information), explana-tory skills and meta-linguistic skills (the ability to use languageto reflect on language); (2) logical–mathematical: the ability toapply (formal) rules of logic to quantities and to model complexproblems; (3) musical: the ability to perceive and produce pitch(melody), rhythm, timbre and compositional form; (4) spatial-visual: the capacity to perceive the visual world accurately, toperform transformations and modifications on that perceptionand to re-create (aspects of) a visual experience, even in theabsence of relevant physical stimuli; (5) bodily-kinaesthetic: mas-tering the process of translating intention into bodily action andknowledge of ‘‘what comes next’’ to allow smoothness of perfor-mance; (6) intrapersonal: the sense of self; (7) interpersonal: thesense of others, and since 1999; (8) naturalistic: a talent for recog-nition and classification of the numerous species in the flora andfauna of the environment; and (9) existentialistic intelligence: thecapacity to locate oneself to fundamentally human features like‘‘the significance of life’’, ‘‘the meaning of death’’ or ‘‘ultimate fate’’(Gardner, 1999).

2.2.2. Discussion on why the Theory should be considered for designIn the field of psychology this theory had great impact for two

reasons. First of all, as aforementioned, many psychologists seekto explain human behaviour from one or only a few perspectives.The fundamental holistic approach by Gardner is therefore innova-tive. His theory allowed psychologists to consider children’s abili-ties beyond the scope of established sets of tests on logical,linguistic and spatial-visual tasks (Anderson, 2000). Secondly, psy-chometric scientists relying on constructs like the IntelligenceQuotient (Becker, 2003b) are challenged by this theory. The theorybehind the Multiple Intelligences is considered good scientificwork, but to implement a bodily-kinaesthetic intelligence similarlyto the implementation of, for example, logical thinking in the Stan-ford-Binet test is a challenge that has not yet been overcome. Thedebate of whether or not all the multiple intelligences can beunderstood as intelligences in itself has contributed to the theory’sreputation (D.H. Feldman in Sawyer et al. (2003)).

Both the holistic and the psychometric point cause debate in thepsychological field. Some oppose the holistic view with the ideathat intelligence is something general and cannot take differentforms (Brody, 1992; Carroll, 1993). These psychologists suggestthat the acceptability of the theory is a matter of how intelligenceis defined. They think that Gardner should have called the


intelligences talents (Scarr, 1985), taxonomies (Brody, 1992), abili-ties (Carroll, 1993) or skills (Sawyer et al., 2003). On the other hand,the field of cognitive science assumes that intelligence as a prob-lem solving ability can be embodied in a yet undefined numberof different forms. In that respect Gardners ideas are consideredas an early contribution to cognitive science (Calvo and Gomila,2008).

In this paper the Theory of Multiple Intelligences is a startingpoint for the development of a framework of skills that are re-quired for design. It is possible that in future work this frameworkwill be extended with design skills from other theories to facilitatea better selection of design methods for children. For this paper,Gardners theory provides a defined set of intelligences that donot have an overlap in skills. Therefore the theory provides a suit-able set of skills with which we can start creating and exploringsuch a framework, similar to how the theory is applied ineducation.

In education, teachers recognized the applicability of the Theoryof Multiple Intelligences. The holistic approach allowed them toclassify children in different ‘‘types of smart’’ rather then ‘‘more’’or ‘‘less’’ smart. Furthermore, teachers experience that childrenhave a dominant intelligence in which they are most comfortableto learn and express themselves (Anderson, 2000; Antle, 2007).The Theory of Multiple Intelligences is a tool to approach a childwith media in their dominant intelligences, while providing a men-tal exercise in the other. For example, a child whose dominantintelligence is bodily-kinaesthetic, is likely to become bored byan exercise in logical–mathematical skills on paper. An exercisethat teaches the child logical skills through movements is morelikely to be successful. The strategy is to introduce the logicalskills with the media of the preferred intelligence (like bodily-kinaesthetic in the example), and then make a transition to themore conventional media (like pen and paper in the example).

The Theory of Multiple Intelligences can be useful in the field ofdesign in the same manner: to optimize the choice for a designmethod for the dominant intelligence of children at a specificage. Each child will develop a personal profile in terms of the intel-ligences. However, developmental theories show that childrenhave common transitions in their development, and that certainskills develop in a universal order (like the stages of cognitivedevelopment as described by Piaget (in Vasta et al. (1995)). A de-sign method chosen to suit the common development of skills ismost likely to be successful with children. For example it makessense to execute a prototyping method once the children’s fine mo-toric skills have developed.

Currently, in the field of design, this theory is receiving in-creased interest, often to support design decisions (Cuthbertsonet al., 2007; Iversen et al., 2007; Kang et al., 2007; Raffle et al.,2007) or to design applications for education to teach children intheir preferred learning style, based on their dominant intelligence,e.g. Becker (2003a) and Kelly and Tangney (2006). However, as faras we are aware, nobody has applied this theory to support theselection of an appropriate design method. To choose an appropri-ate design method, the available methods to choose from should bespecified according to the kind of skills required to perform thedesign activity. Then, we can examine at what age children havesufficiently acquired the necessary skills for a particular designmethod. With such an analysis, we can predict which designactivities are optimal for a certain age group.

3. Developing a framework for optimizing design activities withchildren

In this section, we explain how we developed a framework tomatch design methods to the skills of children in various age

groups. Using the Theory of Multiple Intelligences as a startingpoint for the framework, we describe design methods in terms ofrequired skills. The Teele Inventory of Multiple Intelligences (TIMI)can describe children in terms of intelligences related to thoseskills. By matching the skills required for the design methods tothe skills of children in particular age groups, we can then generatehypotheses about the relation between the suitability of designmethods and children of a certain age.

3.1. A framework of design skills to explain three aspects of designing

To develop a framework for characterizing design skills, wematched 28 commonly used design methods (Bekker et al., 2003;Langford and McDonagh, 2003; Muller, 2003; Dindler et al.,2005; Eggen et al., 2003) with the intelligences as design skills.To verify the results of our matching process, we asked nine ex-perts in design to choose which skills they would associate witha particular design method. The design experts, are professionalsin the area of user centred design, and fulfilled the followingcriteria:

� Graduated with a degree in user interface design, of at least auniversity master’s level, for example from– Industrial design, with a portfolio that shows projects which

dealt with the development of user interfaces;– a postmaster course in User System Interaction;– Media Interaction.� Currently working in this area, with active knowledge about

and experience with design methods.

We provided the designers with descriptions of the methodsand the definitions of the intelligences. Furthermore, we askedthem to motivate their choices. The result is shown in Table 1 giv-ing an overview of which method is associated with which intelli-gence. The table’s columns show the intelligences and the rowsrepresent the design methods. A particular intelligence is consid-ered required for a method, if at least five out of nine observersincluded that intelligence. The scores represent the number ofobservers that included that intelligence, the shading is consistentwith the score, the more observers included that intelligence, thedarker the shading. The table further shows the results of the clus-ter analysis reported below and the division in skills we createdbased on the observations of the designers.

To interpret the data, a cluster analysis using a binary measurewith squared Euclidian distance was performed (Caraux andPinloche, 2005; Everitt et al., 2001). If we accept a level ofclustering for which all 28 methods belong to a cluster, the analysisshows four groups of methods. The four clusters of methods interms of the intelligences are:

1. The linguistic methods (nine methods);2. the linguistic/interpersonal methods (eight methods);3. the spatial-visual methods (eight methods); and4. the linguistic/interpersonal/bodily-kinaesthetic methods (three

methods).

Depending on the type of clustering applied, the linguistic/interpersonal methods are clustered with group one (average link-age) or with group four (complete linkage/linkage using the Wardmethod), therefore we consider this group as a fourth distinctgroup. The four groups are represented in Table 1. The clustersshow that distinctive features for design methods are the type ofrequired skill (linguistic vs. spatial-visual) and the number of re-quired skills: one, two and three or more skills. The table alsoshows that the most important skills for designing are linguistic,interpersonal, spatial-visual and bodily-kinaesthetic skills.

Table 1The matrix shows 28 design methods (rows) and skills (columns) required to perform the method as such. The methods are grouped by four clusters, linguistic methods (top: L),linguistic and interpersonal methods (second group: L–I), spatial-visual methods (third group: S-V) and methods involving more than three skills (bottom group: >3). Skills in theMultiple Intelligence Theory that are not associated with a design method (musical, natural and existential) are represented by the ‘‘(. . . )’’ column in the matrix. The second andthird row groups the skills by the type of involvement in designing (1) communication, (2) the act of designing and (3) method specific skills. The numbers represent the numberof designers (out of nine) that associated a skill with a design method, 9/9 (darkest shade of grey) means that all designers associated a skill with a design method; 5/9 (noshading) means that just more than half of the designers associated a skill with a design method.


A benefit from having clustered the methods is that we canchoose one method as a representative of a cluster of methodsfor the comparison studies. This means that instead of conductingcomparisons on a method level we can conduct comparisons on amethod-cluster level, reducing the number of required comparisonstudies.

Our observations, the interpretations of the observing designersand the cluster analysis revealed that a design method requiresskills in three distinguishable aspects:

1. The participant’s skills in communication, i.e.� comprehension of instructions, and� expression of the contribution (e.g. Barendregt and Bekker,

2005; Sternberg, 2003).2. Executing design activities3. Method specific or product domain-related skills

We had not asked the designers to consider the design methodswith these distinctions, but in the motivations for their choiceseven of them spontaneously made that distinction: theyexplained they always included the communications skills, or leftthem out to state only the skills that were required for the designactivity per sé. Using the Theory of Multiple Intelligences as aframework of skills, we can explain and examine design skills inthese three aspects. The communication skills (1) are interpretedas linguistic and interpersonal skills, the skills for design activities(2) are mostly linguistic, interpersonal, spatial-visual and bodily-kinaesthetic, the skills in the area of the product domain (3) de-pend on the (subset of) characteristics of the product domain that

the designer is interested in to explore with the target user group.Since the Theory of Multiple Intelligences is ‘‘an account of humancognition in its fullness’’ we assume that it is capable of describingthe skills required in the product domain.

During the development of the framework designers describeddesign methods based on the Theory of Multiple Intelligences. Thisdescription provides the means to connect the field of psychologyto design. The advantage is that psychological constructs can nowbe applied in the context of design research. This framework is anattempt to explore the possibilities of such a connection in the fieldof design research. Furthermore, the chosen versatile approachincludes many of the parameters that were studied before inisolation. For example gender differences affecting the designsession (see for example Hou et al., 2006; Isomursu and Still,2004; Stienstra, 2003) would be an interpersonal issue, andverbalisation skills (measured in a usability study by forexample Barendregt and Bekker (2005)) would be a linguisticparameter.

To examine and optimize design methods, it is important to dis-tinguish between the aforementioned three types of skills (com-munication skills, executing design activities and productdomain-related skills) for two reasons: (1) to focus on the skillsthat are necessary for the design act as such and (2) to differentiatethe necessary skills for communication and product domain. Forexample, for a musical application, it is required to have musicalskills to come up with creative and relevant ideas. However, musi-cal skills are not required for a design method per sé. Such is alsothe case for the naturalistic and existentialistic skills. For this work,we focus on (2), the children’s execution of the design method. We


propose some directions with which designers can optimize theirchoice to design with children, inspired by the Theory of MultipleIntelligences. Through design studies we want to find empiricalevidence for these directions based on the information that designmethods provide. In such studies, we have to make sure that theskills required for communication and for the product domain re-main comparable.

3.2. Children’s development in terms of skills

To develop a framework that indicates which method wouldwork best at what age, we need to know what the children arecapable of at a certain age. Currently there are two pen and paperinstruments available that we are aware of that can describe chil-dren’s skills in terms of Gardners multiple intelligences; the TIMI(the Teele Inventory of Multiple Intelligences (Teele, 1992, TIMI))and the MIDAS (The Multiple Intelligence Development Assess-ment Scales for Children). The MIDAS seems to score better thanthe TIMI on internal consistency (Shearer, 1997; McMahon et al.,2004), but the TIMI seems to be the most widely used, amongprimary schools in the USA (Teele, 1996), in high schools (Diaz-Lefebvre, 1999) or in different research fields, for example, to findgender differences in dominant intelligences (Loori, 2005).

The TIMI is a forced choice pictorial inventory that contains 56numbered pictures of panda bears representing characteristics ofeach of the seven skills. The TIMI asks children to associate them-selves with the intelligences. Each intelligence is represented ineight pictures. The children perform a paired comparison of eachpossible combination of any two intelligences. They match theirown behaviour with the behaviours depicted in the test. Basedon which picture they feel describes them best, they choose oneof the two pictures in each comparison. An overview of the resultsbased on 6000 answer sheets from schools in the United States areshown in Table 2. The table shows which skills are the most or theleast dominant for a certain school grade. The dominance level isranked by the numbers one to four, where one means most domi-nant (Teele, 1996).

The table shows the main intelligences of children in a particu-lar school grade. An empty cells means that the correspondingintelligence plays a less dominant role in a child’s developmentduring this school period. By addressing children with media (oractivities) that comply with their dominant intelligences it is morelikely that children in that age group are motivated to get involvedin the activity.

The combination of Tables 2 and 1 allows a theoretical predic-tion of whether a design activity will match successfully with chil-dren of a particular school grade. Similar to how the Theory ofMultiple Intelligences is applied in education to select an educa-tional method for children, these tables can be used to select a de-

Table 2The change in dominance levels of the intelligences per school grade according to Teelerepresented by different shading of grey. The darkest shade of grey (1) means most domdevelopment of a child during this school period. The second and third column groups the idesigning and (3) method and/or product specific skills.

sign method for children. The next step to the development of sucha framework is to explore which opportunities and limitationsthis approach holds for design practice. In this paper we want tomake a start with exploring these opportunities and limitations.First we will explain in more detail how such a framework couldwork.

3.3. Generating hypotheses from children’s characteristics and methodcharacteristics

Comparing Table 1 (skills required for design methods) to Table2 (dominance of the intelligences vs. the different school grades)leads to interesting hypotheses about the extent to which a designmethod is appropriate for children at a specific educational stage.Table 2 suggests that until the fifth grade of a primary school(equivalent to around 11 years of age) children should best be in-volved in spatial-visual design activities. At a later stage, the inter-personal intelligence becomes dominant, which suggests that amethod from the clusters which include the interpersonal intelli-gence would be more appropriate.

From this observation, the most interesting hypotheses arethose concerned with the suitability of design activities for partic-ular age groups, based on a match between the skills required for adesign method and the dominant skills of that particular agegroup. However, to validate such hypothesis requires a study inwhich different age groups use different design methods. We firstwant to explore our framework with a simpler approach to findevidence that this framework is fruitful. Therefore we focus on ahypothesis concerned with the number of skills that are used dur-ing a design activity.

Our initial study compares two methods based on the numberof skills that are required. The prerequisites for comparing designmethods are (1) a research methodology that allows designmethods to be comparable, (2) a new instrument for measuringchildren’s design output, and (3) an estimation of whether theskill-based framework proves fruitful in selecting a method fordesigning with children.

4. A study on the number of skills involved in design spaceexploration

The first study assumes that the more skills a design methodinvolves in an early-design session, the better the design space isexplored. In the following sections, we will first explain thishypothesis, then the hypothesis is operationalized as a comparisonbetween a NGT method and a prototyping method. Initial insightsfrom this study, based on a small subset of the data (4 of 13 groupsfrom this study was presented in Sluis-Thiescheffer et al. (2007)).

(1996). The dominance level per grade is ranked by the numbers one to four andinant. An empty cell means that the intelligence plays a less dominant role in the

ntelligences by the type of involvement in designing: (1) communication, (2) the act of


4.1. Hypothesis: The more skills involved in the design method, themore Options and Criteria generated in a design session

In the early stages of the design process, the design space needsthorough exploration to find innovative solutions for the designproblem at hand. In the previous paragraph we explained that inan early design phase, divergence is essential. To reach divergence(in terms of our framework), we hypothesize that the more skillsinvolved in a design method, the more approaches are availablein that method. A method that only involves linguistic skills, leavesfor the development of ideas only a linguistic approach throughe.g. reading, speaking aloud or writing. A method that involves alsospatial-visual skills facilitates a designer also in a spatial-visualway, allowing to approach a design problem with sketching, draw-ing or creating a map (e.g. a cognitive map).

As a result, we expect that the more skills involved in a designmethod, the better the design space is explored. For a design meth-od involving a limited number of skills we expect a more limitedexploration. For a design method involving a larger number ofskills, we expect a more thorough exploration of the design space.

4.2. Comparing the Nominal Group Technique (NGT) and prototyping

This is a first exploration of the framework. Therefore wewanted to select a method that is (1) well established, (2) workswith children and (3) fits our hypothesis of being a method requir-ing a minimal or a maximum set of skills. We initially explored themethod of brainstorming. Brainstorming is one of the most estab-lished and well recognized methods in the early phases of design,and although it requires two skills, two is still a low number, andwe had no reason to believe that it would not work with children.Brainstorming has a long history and is most recognized by thedefinition of Osborn (1942) ‘‘a conference technique by which agroup attempts to find a solution for a specific problem by amass-ing all the ideas spontaneously by its members.’’

However, a pilot study (Sluis-Thiescheffer et al., 2007, reportedin) showed that in a brainstorming session, children copied openlythe ideas of their best friends, or claimed they generated the sameidea as the most dominant child in the group. This effect isdescribed as cognitive tuning in working with adults. Cognitivetuning is the phenomenon that members of a group ‘‘tune into’’each others mindset, limiting the potential diversity in ideas (Fern,2001). To avoid cognitive tuning with a method that resemblesbrainstorming, we exchanged brainstorming for the NominalGroup Technique (NGT). The NGT allows the children to developtheir ideas on their own, before sharing them with a group (Fern,2001) hence avoiding cognitive tuning.

Our considerations for choosing NGT are therefore not onlydriven by the framework. On the upside; from the perspectiveof the framework it is a method with a minimal set of requiredskills, on the downside: it is arguably the most representativeof the linguistic skills. However at this point in the developmentof the framework, (1) the aspect of a minimal number of skills ismore important than the linguistic representativeness of theskills and (2) we think that practical drivers are equally impor-tant; we want to make a point with a method that is recognized,widely used and that avoids a bias in measuring the design skillsof children.

The cluster of methods associated with the largest number ofskills is the cluster with linguistic, interpersonal and bodily-kinaesthetic skills. Within that cluster, prototyping (Muller, 2003)is associated with the largest number of skills. Prototyping is definedby Muller (2003): ‘‘Creating an understanding for an artefact andchanging the artefact to explore understandings of one another’spositions, to question one another’s approaches and to accommo-date heterogeneity of views and interests.’’ For the study we explore

the difference in output between the Nominal Group Technique andprototyping. In the next section, we will explain how the differencein output is measured, despite the differences between the media ofthe generated design solutions.

4.3. Measuring design space exploration in terms of Options andCriteria

In the introduction, we explained that design method assess-ment can be improved by using explicit measures. At the sametime we want to make design output comparable. To comparebrainstorming and prototyping, we want children to discusstheir ideas and support them with arguments, so we can mea-sure information in the design discussion. Design discussionswith adults have been explored by, for example, MacLean et al.(1996) and Olson et al. (1992). They analysed discussionsbetween designers in terms of MacLean’s model of Questions(Q), Options (O) and Criteria (C). By creating an Options andCriteria diagram, designers are better able to formulate Ques-tions (Q) to come to a more explicit design rationale. In ourstudy, we adopted the QOC-model to describe a discussion onearly designs.

The difference with MacLean’s study and our study is thatMacLean studied conversations among designers. In that situa-tion each designer tried to understand the design space. There-fore, all participating designers generated Questions to gainunderstanding from each other, and they generated Optionsand Criteria to explain their understanding to each other. Inour case, the designer, or moderator of the design session, mainlytries to gain understanding from the children, for the children notmuch is at stake to gain a thorough understanding from eachother. Therefore we do not focus on the questions of the children,but rather on the Options and Criteria they generate. Beforeexplaining more about the setting in which we studied theconversations, we will first explain the design problem that weshared with the children.

A telecommunication company provided a design problemthey were working on, two realistic cases to design a device forchildren in a primary school. The goal of the design cases wasto establish live communication between a child at home andthe class in school. The purpose of the live connection was tofacilitate attending class for a child that has almost recoveredfrom an illness. The child is not yet able to come to school, butis mentally fit enough to attend at least some hours of class.We want the children to generate ideas about a device locatedin class and one at home. The design space was focused on find-ing interface solutions in the home situation and at school. Theschool situation was further specified for two common class situ-ations, thus providing the two design cases. In one case the focuswas on a collaborative activity, where the child would work in asmall group of peers (the ‘‘group work’’ case). In the other casethe focus was on a class activity, where the child pays attentionto the teacher, who is actively teaching in front of the classroom(the ‘‘teaching’’ case).

The following example explains how we applied the QOC-model to this design problem. A device that could solve this designproblem is a TV, which would be an option. An ordinary TV wouldnot suffice, as interaction with the class is required. The TV is there-fore only a good solution if it can record who is watching (criterion1) and what the person is doing (criterion 2). The criteria are thesuccess descriptors of the options.

With the QOC-model as a coding scheme for the informationthat the children will generate, we hypothesize that the more skillsinvolved in a particular design method with children, the moreOptions and Criteria the children will bring forward.

Table 3Overview of the combinations of design cases and methods dealt with in a focusgroup session.

Methods ? NGT 1st; Prototyping 1st;Cases ; Prototyping 2nd NGT 2nd

Teaching 1st NGT for teaching; Prototyping for teaching;Group work 2nd Prototyping for group work NGT for group work

Group work 1st; NGT for group work; Prototyping for group work;Teaching 2nd Prototyping for teaching NGT for teaching


4.4. Method

4.4.1. Focus group settingAs explained above, the basis for comparing NGT-output with

prototyping output is the conversations about the generateddesign solutions. For the setup of the study we have to choosebetween group conversations and one-on-one conversations.Hennessy and Heary (2004) report that in a focus group thedynamic interaction of five children will elicit more informationthan a one-on-one interview because there is less emphasis onthe adult–child relationship. Moreover, design activities withuser-participants are often reported to be group-activities(Langford and McDonagh, 2003). For the group size, Hennessyand Heary (2004) report that a focus group session is done bestwith a maximum of five children. A group of five children is largeenough to have at least one talkative child, and yet small enough togive each child sufficient attention.

Because of the structure of a focus group conversation, the anal-ysis can be done on a group level and on a per child basis. On thegroup level, all observed Options and Criteria during the groupconversation are taken into account. However, although thegroup-design has its advantages for the involvement of children,the group work could average out the differences that are the focusof this study. Therefore we will also perform an analysis on thebasis of each child’s explanation.

4.4.2. ParticipantsThe study we present in this paper focuses on children of 8–

12 years of age, in marketing terms also referred to as ‘‘tweeners’’.They are the group who are often targeted for product develop-ment, because they firstly adopt ‘‘leftover’’ technology by their par-ents, they have relatively much freetime, they receive pocketmoney and they are greatly influencing buying decisions (Dillen,2001; Acuff and Reiher, 1997).

Thirteen groups of five 10-year-old children participated in thestudy (a total of 65 children). All groups were of a mixed gendercomposition (at least two of each gender). All children were ingrade five of a Dutch primary school. Three primary schools, onefrom the centre of the Netherlands and two schools from the southof the Netherlands took part in the study.

Recruiting 65 children for this study might seem a large num-ber. However, each group of five children produces essentiallyone data point for our analysis: 13 data points are collected. Usingrepeated measures to analyse the data, usually six or seven datapoints should suffice Vickers (2003). For this study, a within-subject design is used (see below) and when an analysis on poten-tial order and interaction effects is performed, seven data pointseasily break down into a marginally small number of data points;resulting in not enough statistical power to exclude a bias. Further-more, the 13 groups include some over-recruitment to be able tocompensate for possible loss of data. Therefore, the 65 childrenare a reasonable number of participants.

4.4.3. Within-subject designA within-subject design was applied. Each group of five children

participated in both an NGT session and a prototyping session. Atypical session would have the following setup:

1. General introduction2. Introduction of the first case3. Introductory questions to warm up for idea generation4. Individual time to write ideas (NGT)5. Group discussion on the written ideas6. Introduction of the second case7. Introductory questions to warm up for idea generation8. Individual time to create a prototype

9. Group discussion on the prototypes10. Rounding off

In the group discussions, the experimenter would ask a maxi-mum of two questions to keep the discussion on one idea going.This was included to treat talkative and less talkative children sim-ilarly, hence avoiding a bias in feeding the discussions by theexperimenter.

The discussed case is assumed not to have an effect on the num-ber of Options and Criteria, the two cases were included to avoid afeeling of performing a redundant activity among the participants.To avoid an influence of order and possibly of case, we performedthe sessions in four variations of method (NGT or prototyping) andcase (group work setting or teaching setting) illustrated in Table 3:

In total there are four possible variations of Table 3 to controlfor each form of interaction of case, method and order, resultingin a total of 16 different sessions. However, we are only interestedin the difference between NGT and prototyping, therefore thestudy was performed using an incomplete block design, i.e. weonly conducted the 4 of 16 possible variations as displayed in Table3. In the analysis, a possible order effect will be investigated tounderstand whether an incomplete block design was justified.

4.4.4. Session lengthInitially, the plan was to allow the children 15 min to come up

with ideas or a description in each creative sessions. However, apilot study with adults showed that in an NGT session most partic-ipants finished their description after approximately 10 min andbecame rather bored during the remaining 5 min. In the prototyp-ing session the participants were frustrated, because they felt theyhad insufficient time to finish a proper model of their design idea.Therefore we decided to shorten the time for the NGT session with5 min and to lengthen the time for the prototyping sessions with5 min. Our studies with the children showed that the majority ofthe group members finished within these time limits. By allowinga different time window for both methods, we potentially intro-duce a confounding factor of time to generate ideas. However,we still think the sessions are comparable for two reasons. The firstreason is that in both sessions the timeslot is now compatible withthe time the participants need to finish the activity hence avoidingbias by boredom or frustration. The second reason is that in thediscussion following the activity, we allow the same amount oftime and prompts to explain the ideas developed by both methods,hence allowing the same window to express a number of ideas. Inthe pilot study we found that the group of participants used the fulltime window after both activities.

Each session was recorded using video and audio recordingdevices. For the video recording we used a mobile usability labconsisting of two digital cameras, a video mixer, and a digitalencoder (for converting the analog video signal) connected to alaptop via a Universal Serial Bus (USB) connection. Each childwas equipped with a wireless microphone, thus being recordedon an individual sound track, a necessity for the transcriptions.The audio tracks were transcribed using the audio editor Audacity(Audacity, 2006). One of the authors and one research assistant


took care of the transcriptions. The transcriptions contained atimestamp for each utterance, a number indicating the speakerand a code to indicate overlapping speech. Comments of the tran-scriber on an utterance were noted between square brackets; forexample [laugh] for laughter, [unint] for unintelligible or [takespicture] when the moderator took a picture of the prototype.Fig. 1 shows an example of coding the conversation.

5. Analysis

All conversations were transcribed verbatim and two observersscored for Options and Criteria. As an extra check, one transcriptwas observed by a third observer to perform an inter-rater agree-ment analysis. Furthermore we checked whether the groups ofchildren were comparable in terms of their intelligence profilesaccording to the Theory of Multiple Intelligences.

5.1. Any-two agreement of 61% on average in observing Options andCriteria

Several measures are available to determine inter-coder reli-ability, for example, Cohen’s kappa and the any-two-agreementmeasure. Cohen’s kappa (Cohen, 1960) is a commonly used mea-sure that estimates the proportion of agreement between two eval-uators after correcting for the proportion of chance agreement.However, Cohen’s kappa is based on each evaluator classifyingthe same observation points. In the case of free detection of Op-tions and Criteria evaluators may not have observed the samenumber of items to be coded thus resulting in different observationpoints. An any-two-agreement calculation (Hertzum and Jacobsen,2001) is not restricted by a fixed set of observation points andtherefore more suitable to give an estimation of the inter-coderconsistency in our study.

To determine the consistency of the proposed coding schemethree observers independently coded both the prototyping andthe NGT session of one group of five children. In the conversation,we observed Options and Criteria and repetitions of Options andCriteria. Therefore the possible outcomes of the comparison were:

1. Agreement: both observers coded a phrase as an option or as acriterion.

2. Disagreement: only one observer coded a phrase as new, whilethe other did not code it.

3. Disagreement: only one observer coded a phrase as new, whilethe other coded it as a repetition of an aforementioned option.

When describing the design space in Options and Criteria, a rep-etition does not add new information to the design space. Hence ifone observer did not score anything where the other observer

Fig. 1. A piece of transcript of a prototyping session where E stands forexperimenter, and 3 indicates child 3 talking. [On] codes an option, [Cn�n] codes acriterion.

scored a repetition, no information is lost. Table 4 shows that theagreement between the three observers is 61%. We consider thisagreement sufficient to continue to the results.

5.2. Representative profiles of multiple intelligences

The hypothesis on which activities would suit the skills of thechildren best, was based on the profiles as identified by Teele(1996). An extremely deviant group profile of multiple intelli-gences would be a confounding factor to our results. For example,in the case that none of the children would associate themselveswith the linguistic intelligence, this could have a strong influenceon the results.

Since we based our hypothesis on addressing the multiple intel-ligences of children, we asked them to fill in the TIMI individually.Fig. 2 provides an overview of the results. Each column indicatesthe average rating of an Intelligence by the children (64 children),using the TIMI. The 0.5 – confidence intervals shows the spreadingof the ratings. To examine the coherence between the children’sratings, we calculated W, Kendall’s coefficient of concordance(Siegel and Castellan, 1988). All 67 children were asked to com-plete the TIMI, unfortunately three TIMI-forms were returnedincomplete. For the remaining 64 children (k = 64) and sevenintelligences (N = 7), the coefficient of concordance W = 0.440.Using the table of Critical Values for Kendall’s W (Siegel and Castel-lan, 1988), W is significant with a = 0.01, meaning that the ratersare not independent, the children are sufficiently coherent in ratingthemselves. Given Kendall’s W, we accept Fig. 2 as the generaloverview of dominant intelligences of the children in this study.

All groups associated themselves mainly with the interpersonaland the bodily-kinaesthetic intelligence. The profile of these Dutchchildren in grade five is not identical to the profile for grade fivepresented by Teele on a study with American schools (see Table2), but still consistent in terms of having the same set of dominantskills. It is also compliant with the common view in developmentalpsychology indicating that the children’s motorical and socialdevelopment is dominant in this age group (e.g. Breeuwsma,1994).

6. Results

Below we will present the results of the study. First, we deter-mined whether there was a difference in the number of Optionsand Criteria generated by the children during the two design meth-ods. Second, we will explore possible interaction effects and ordereffects.

6.1. Significantly more options after prototyping session, no differencefor the Criteria

The protocols of 13 groups, 11 groups of five children and twogroups of six children, have been coded. During one session, how-ever, one child did not complete the entire session, therefore that

Table 4Number of agreements, number of unique observation points for each evaluator, andnumber of disagreements in coding Options and Criteria in the transcripts of thedesign conversations.

Observerpair

Agree (1)(%)

Unique A (2)(%)

Unique B (2)(%)

Disagree on rep(3) (%)

1 � 2 66.7 7.3 17.3 8.71 � 3 56.0 14.6 20.2 9.32 � 3 60.7 14.6 13.9 11.1Average 61 12 17 10

Fig. 2. Overview of the average intelligence profile of the participating children in the 13 groups. Each bar represents the relative dominance level of intelligences. The errorbars show the 0.05 confidence interval around the mean.


session was left out of the group analysis, leaving 12 groups for theanalysis.

The conversations after prototyping showed a significantly lar-ger number of options (Mprot-options = 37.000) than the conversa-tions after the NGT (MNGT-options = 28.667); using repeatedmeasures shows a within-subject contrast of F = 22.883 andp 6 0.001. The conversations after prototyping did not show a sig-nificant difference for the number of criteria (Mprot-crit = 95.167)compared to the conversations after NGT (MNGT-crit = 94.917); usingrepeated measures shows a within-subject contrast of F = 0.001

Fig. 3. The average number of Options and Criteria generated per group of children in a fobars in the figure show the 0.05 confidence interval, using repeated measures of the genecontrast, the confidence interval is divided by

ffiffiffi

2p

(Loftus and Masson, 1994).

and p 6 0.980. Fig. 3 shows the means and the respective confi-dence intervals for the Options and Criteria per group.

The difference between the two methods is large enough to re-sult in a significant difference for the options. For the criteria, how-ever, we did not find a significant difference. As explained, thegroup interactions could average out this potential difference be-tween the two methods. However, the analysis on a per child basisshows similar results. The conversations after prototyping showeda significantly larger number of options (mprot-options = 7.789) thanthe conversations after the NGT (mNGT-options = 6.035); using

cus group session, either after a NGT session or after a prototyping session. The errorral linear model. To display a representative confidence interval for a within-subject


repeated measures shows a within-subject contrast of F = 12.002and p 6 0.001. The conversations after prototyping did not show asignificant difference for the number of criteria (mprot-crit = 20.035)compared to the conversations after NGT (mNGT-crit = 19.982);using repeated measures shows a within-subject contrast ofF = 0.002 and p 6 0.967. Fig. 4 shows the means and the respectiveconfidence intervals for the Options and Criteria per child.

6.2. No bias from interaction – or order effects

The setup of the study controlled for interaction and ordereffects. The analysis on the groups did not show any significantdifferences. However, the number of groups was probably toosmall to find an order effect. Visual inspection of the data showedthat there is a possible order effect for the method applied.Therefore the data was further analyzed on a per child basis.To analyze for an order effect, we removed the children thatgenerated a number of Options and/or Criteria that fell outsidethe two standard deviations interval around the mean. Figs. 5and 6 show the means and the respective confidence intervalsfor the Options and Criteria per method per child. Each pair ofbars compares the data from the same design activity and thesame design case, the only difference is whether the design activ-ity was performed first (left bar) or second (right bar). Althoughfrom the graphs it seems that more differences are signifi-cant, after applying a correction with the False Discovery Rate(Benjamini and Hochberg, 1995), there are only significantlymore criteria generated when NGT was applied first, than whenNGT was applied second in the teaching-design case. Using uni-variate analysis shows a between-subject contrast of F = 15.379and p 6 0.001.

The significant effect we found concerns only an order effect forthe number of criteria in the teaching-design case. Visual inspec-tion of the graphs shows that in any session when NGT was appliedfirst, NGT resulted on average in more options and more criteriathan when NGT was applied second. Furthermore, also regardlessof design case, when prototyping was applied second the group

Fig. 4. The average number of Options and Criteria generated per child in a focus groep sfigure show the 0.05 confidence interval. To display a representative confidence intervaMasson, 1994).

discussion resulted in more options and more criteria than whenprototyping was applied first. Because we controlled for order inthe study setup, this finding does not affect our conclusions regard-ing the hypothesis of this study.

7. Discussion

In this section we will first discuss the results of the study. Thenwe will discuss the extent to which the study was successful inbeing an example study on exploring hypotheses about designmethods for children based on the Theory of Multiple Intelligences.Finally we will discuss how the framework can be used to initiatemore research in optimizing design methods (for children).

7.1. Comparing design methods: prototyping vs. Nominal GroupTechnique

The results of the study show that, as expected, the discussionsafter the prototyping activity generated more options than afterthe NGT activity. However, unexpectedly, for the criteria we didnot find a difference. The number of criteria generated is non-dis-criminative for these two methods. We could argue that optionsgenerated in the NGT activity require more explanation than crite-ria, since the options do not have a physical representation thatcomplements the verbal explanation, as is the case in the prototyp-ing activity. This compensating behaviour biases the differences incriteria between the two methods. Another explanation is that thenumber of criteria is not dependent of the method, but rather ofthe product domain. Criteria do not only further specify the options(e.g. number and size of options), but also explain more about thecontext in which the options make sense (locations and valuesassociated with the options). A thorough qualitative analysis couldshed light on potential qualitative differences between criteriafrom both methods.

In one case there was a significant order effect. When NGT wasperformed first, the children generated on average more Optionsand Criteria during the prototyping session than when NGT was

ession, either after a NGT session or after a prototyping session. The error bars in thel for a within-subject contrast, the confidence interval is divided by

ffiffiffi

2p

(Loftus and

Fig. 5. The average number of options generated per child in a focus group session, either after a NGT session or after a prototyping session, split by method, design case andorder. The error bars in the figure show the 0.05 confidence interval.

Fig. 6. The average number of criteria generated per child in a focus group session, either after a NGT session or after a prototyping session, split by method, design case andorder. The error bars in the figure show the 0.05 confidence interval.


performed second. When prototyping was performed first thechildren generated on average less Options and Criteria than whenprototyping was performed second. These order effects did not af-fect our results, as we controlled for an order effect (see Figs. 5 and6) in the design of the study. The effect turned out to be small butconsistent; visual inspection revealed that differences in Optionsand Criteria between the sessions point in the same direction.Apparently, the most productive order of these two methods isto perform an NGT session first and then a prototyping session.This finding gives quantitative support to the general idea thatNGT is very suitable as a warming-up activity as suggested in(e.g. Langford and McDonagh, 2003). A new finding is that

prototyping seems counter-productive for the NGT session follow-ing that prototyping session.

In adopting MacLean’s observation model, we conduct a quanti-tative study on design conversations. That is the line of research wewant to pursue in this article. A complementary approach ispursued in (Thang et al., 2008), that study looks for evidence onqualitative differences between Options and Criteria from bothmethods in terms of creative value for designers. Additionally itwould be interesting to perform a discourse analysis to learnwhether there are structural differences between the Optionsand Criteria from both methods. For example whether the Optionsand Criteria from both methods are equally informative in


linguistic terms. Quantitative, qualitative and structural evidencecan support a designer in choosing a design method for designingwith children.

The session length could have been a confounding factor sincewe allowed more time for the children to develop their low-fi pro-totype. However, as explained we preferred to have the majority ofthe children to finish their design session. It appeared that for bothdesign sessions this period of time was sufficient. By using a differ-ent amount of time, the children finished the description (NGT) ormanufacturing (prototyping) of their idea within the time given. Apilot study showed that using 5 min for both methods, the childrenwould have felt that they had not developed their prototype wellenough. By using 10 min for both methods, the children wouldhave become rather bored after 5 min in the brainstorming condi-tion. What seemed more decisive in the measurements is howskilled the children were in explaining their options and discussingthe criteria with the other children. Therefore, it was good to in-clude in the protocol a maximum of two questions to ask the chil-dren to elaborate on their ideas. Two questions were sufficient toprompt the less talkative children without making them feeluncomfortable, and the number was small enough to limit the talk-ative ones. In doing so, we think we achieved a fair distribution ofspeaking opportunities for each participant and each method.

The intelligence profiles were slightly different for grade five inDutch children than the profile for children in grade five ofAmerican children. The difference in the number of participatingchildren in this study (76 children in 13 groups) and in theAmerican study (>6000 children), could account for this minor shiftin the profile for a child in grade five. For the hypothesis studied inthis study, the Dutch profile was not sufficiently different to affectthe outcomes: there is only a difference in order of dominance; thefour dominant skills as a whole are the same. The difference couldbe of influence on hypotheses that zoom in on particular intelli-gences, for example when comparing the success of methodsbetween two grades. Therefore it is recommended that in furtherstudies on the use of this framework the use of the TIMI (Teele,1992) should be included (or an alternative with higher internalconsistency), to control for possible differences in the profiles asdescribed by Teele (1996) and the profiles in the children partici-pating in the study.

7.2. Comparing methods requiring more design skills with methodsrequiring fewer design skills

Another point for discussion is whether the two methods com-pared in this study are really representative of the clusters, i.e.whether based on this study we can also conclude that dramas(a many-skill method) will result in more options than storytellingor questionnaires (both one-skill methods). Further research couldvalidate this assumption by making a comparison between othermethods from the clusters used in this study.

The difference in options supports our hypothesis that proto-typing results in more information than NGT. As prototyping in-volves more intelligences than NGT, the results also support theidea that the involvement of more intelligences leads to more de-sign solutions than the involvement of less intelligences. Hence thesupport we found for this hypothesis encourages to use the frame-work for further exploration of the suitability of design methods onthe basis of multiple intelligences. This study was a comparison inbandwidth of design skills. Further studies should provide insightson whether the framework can predict the performance of childrenon tailor-made design methods. According to the TIMI, children inthe first grade have a dominant spatial-visual intelligence, andchildren in grade six have the bodily-kinaesthetic intelligence asa dominant intelligence (see Table 2). That would suggest that bet-ter results are expected from a spatial-visual method in the first

grade than a bodily-kinaesthetic method, and vice versa in gradesix. Such a study will be a next challenge for this framework toprove its value. We have planned such a study to explore differ-ences in the output between 6-year-olds and 10-year-olds.

7.3. Using the framework for future comparison studies

In creating the framework we focused on the design skills. Inour framework we stated that communicative skills and skills inthe product domain are also important aspects of a design activity.For the product domain of our design problem, dominant interper-sonal skills could have been beneficial in generating ideas aboutonline interaction with peers. Another example would be a musicalproduct for children. To explore the musical space of design solu-tions it would make sense to select a group of musically gifted chil-dren (in terms of the TIMI) to conduct a design session. Futureresearch could explore the framework in terms of multiple intelli-gences for hypotheses about the product domain or the communi-cative skills of children.

The proposed framework is based on Gardner’s Theory of Multi-ple Intelligences. In this study we found four intelligences that areassociated with design activities. As indicated, the proposed frame-work is not restricted to the Theory of Multiple Intelligences. Fur-ther research on psychology and design could provide evidencethat other differentiating design skills should be added to theframework.

8. Conclusion

With this work we make three contributions to the researcharea of interaction design with children. One contribution is anoverview of design activities in terms of skills, inspired by theTheory of Multiple Intelligences. To conduct a design method wedistinguish between communicative skills, design skills and skillsconcerning the product domain. We focused on design skillsand we found four types of skills that are characteristic for a designactivity: linguistic, interpersonal, spatial-visual and bodily-kinaesthetic skills. Such overview of how types of skills couldrelate to design activities did not exist yet.

The second contribution is the framework, which can be appliedto optimize design methods for young children. The proposalframework can be useful in two ways. First, the use of psychologicalconstructs in this tool provides a rationale for choosing a designmethod for children in a particular educational stage. Our focus ison the developmental stage of children in terms of the MultipleIntelligences, however, this can be applied to other user groups aswell (e.g. adults). If the behaviour of a particular user group canbe defined in terms of design skills, a method can be chosen onthe basis of the comparison between the user group profile andthe profile of the design method. Second, similar to the practice ineducation to adjust educational exercises (Anderson, 2000), theframework can be used to adjust existing design methods to bettersuit the profile of the target user group. For example, the dominantintelligence for children up to grade five is spatial-visual. Thereforea designer can pick a spatial-visual method, but also adjust amethod from another cluster to include a spatial-visual technique,for example, by adding a drawing task to an NGT session.

The third contribution is the application of the framework tocompare design methods. The influence of the bandwidth of skillsinvolved in a design method results in a difference in the numberof generated Options and Criteria. Involving four skills in a methodlike prototyping results in more options than a method involvingpredominantly only one skill (NGT). Apart from the immediate re-sults, the study is a methodological example of how to conductquantitative design-integrated research with children, applying


measurements that are known to work with adults, but that havenot yet been used with children. Furthermore, the study showshow to compare the output of an NGT session with the output ofa prototyping session. The fruitful results encourage further explo-ration of the usefulness of the framework as a means to optimizedesign methods for young children.

References

Acuff, D.S., Reiher, R.H., 1997. What Kids Buy and Why: The Psychology ofMarketing to Kids. Free Press.

Anderson, J., 2000. Cognitive Psychology and Its Implications, fifth ed. WorthPublishers, New York.

Antle, A.N., 2007. Designing tangibles for children: what designers need to know. In:CHI ’07: CHI ’07 Extended Abstracts on Human Factors in Computing Systems.ACM, New York, NY, USA, pp. 2243–2248.

Audacity, Retrieved: July 30, 2006. A Free Audio Editor and Recorder. <http://audacity.sourceforge.net>.

Barendregt, W., Bekker, M.M., 2005. Developing a coding scheme for detectingusability and fun problems in computer games for young children. In:Proceedings of Measuring Behaviour, pp. 245–248.

Becker, K., 2003a. A multiple-intelligences approach to teaching number systems.Journal of Computing Sciences in Colleges 19.

Becker, K.A., 2003b. History of the Stanford-Binet intelligence scales: Content andpsychometrics. In: Stanford-Binet Intelligence Scales, Assessment ServiceBulletin, fifth ed., vol. 1, Riverside Publishing, Itasca, IL, pp. 1–16.

Bekker, M., Beusmans, J., Keyson, D., Lloyd, P., 2003. Kid reporter: a userrequirements gathering technique for designing with children. Interactingwith Computers 15, 187–202.

Benjamini, Y., Hochberg, J., 1995. Controlling the false discovery rate: a practical andpowerful approach to multiple testing. Journal of the Royal Statistics Society 57,289–300.

Bono, E.d., 1985. Six Thinking Hats. Key Porter Books.Breeuwsma, G., 1994. De constructie van de levensloop. Boom, Amsterdam.Brody, N., 1992. Intelligence. Academic Press Inc.Calvo, P., Gomila, T. (Eds.), 2008. Handbook of Cognitive Science. Elsevier Science.Caraux, G., Pinloche, S., 2005. Permutmatrix: a graphical environment to arrange

gene expression profiles in optimal linear order. Bioinformatics 21, 1280–1281.Carroll, J.B., 1993. Human Cognitive Abilities: A Survey of Factor-analytic Studies.

Cambridge University Press.Cohen, J., 1960. A coefficient of agreement for nominal scales. Educational and

Psychological Measurement 20 (1), 37–46.Cuthbertson, A., Hatton, S., Minyard, G., Piver, H., Todd, C., Birchfield, D., 2007.

Mediated education in a creative arts context: research and practice at whittierelementary school. In: Proceedings of the 6th International Conference oninteraction Design and Children, pp. 65–72.

Diaz-Lefebvre, R., 1999. Coloring Outside the Lines: Applying Multiple Intelligencesand Creativity in Learning. John Wiley and Sons, Inc.

Dillen, F.M.A.V., 2001. KidsMarketing: The Empowerment of the Kids. Kluwer,Alphen aan de Rijn, NL.

Dindler, C., Eriksson, E., Iversen, O.S., Lykke-Olesen, A., Ludvigsen, M., 2005. Missionfrom mars: a method for exploring user requirements for children in a narrativespace. In: IDC ’05: Proceedings of the 2005 Conference on Interaction Designand Children. ACM, New York, NY, USA, pp. 40–47.

Donker, A., Reitsma, P., 2004. Usability testing with young children. In: IDC ’04:Proceedings of the 2004 Conference on Interaction Design and Children. ACM,New York, NY, USA, pp. 43–48.

Dorst, K., 2003. Understanding Design. BIS Publishers.Druin, A., 1999a. The Design of Children’s Technology. Morgan Kaufmann, San

Francisco.Druin, A., 1999b. The role of children in the design of new technology. Report.Eggen, B., Feijs, L., Peters, P., 2003. Childrens participation in the design of physical

and on screen-intervention strategies to prevent excessive game play. In:Proceedings of the 6th ADC Conference.

Everitt, B., Landau, S., Leese, M., 2001. Cluster Analysis. Oxford University Press.Fern, E.F., 2001. Advanced Focus Group Research. Sage Publications, Thousand Oaks,

California.Gardner, H., 1983. Frames of Mind, 2004th ed. Basic Books, New York.Gardner, H., 1999. Intelligence Reframed. Basic Books.Gelderblom, H., 2004. Designing software for young children: theoretically

grounded guidelines. In: IDC ’04: Proceedings of the 2004 Conference onInteraction Design and Children. ACM, New York, NY, USA, pp. 121–122.

Greenbaum, T., 1988. The Practical Handbook and Guide to Focus Group Research.Lexington Books, Lexington, MA.

Heary, C.M., Hennessy, E., 2002. The use of focus group interviews in pediatrichealth care research. Journal of Pediatric Psychology 27 (1), 47–57. <http://jpepsy.oxfordjournals.org/cgi/content/abstract/27/1/47>.

Hennessy, E., Heary, C., 2004. Exploring children’s views through focus groups. In:Greene, S.M., Hogan, D.M. (Eds.), Researching Children’s Experiences:Approaches and Methods. Saga, pp. 236–252 (Chapter 13).

Hertzum, M., Jacobsen, N.E., 2001. The evaluator effect: a chilling fact aboutusability evaluation methods. International Journal of Human–ComputerInteraction 13, 421–443.

Hou, W., Komlodi, A., Lutters, W., Boot, L., Cotton, S., 2006. Girls don’t waste time:pre-adolescent attitudes toward ict. In: Proceedings of the Conference onHuman Factors in Computing Systems 2006, pp. 875–880.

Hummels, C., Frens, J., 2008. Designing for the unknown: a design process for thefuture generation of highly interactive systems and products. In: InternationalConference on Engineering and Product Design Education.

Isomursu, M.P.I., Still, K., 2004. Capturing tacit knowledge from young girls.Interacting with Computers.

Iversen, O.S., Kortbek, K.J., Nielsen, K.R., Aagaard, L., 2007. Stepstone: an interactivefloor application for hearing impaired children with a cochlear implant. In:Proceedings of the 6th International Conference on Interaction Design andChildren, pp. 117–124.

Jensen, J.J., Skov, M., 2005. A review of research methods in children’s technologydesign. In: Proceedings of the 2005 Conference on Interaction Design andChildren.

Kang, H.W., Zentall, S.S., Burton, T.L., 2007. Use of images in instructionaltechnology for children with attentional difficulties. In: Proceedings of the6th International Conference on Interaction Design and Children, pp. 129–132.

Kelley, T., Littman, J., 2006. The Ten Faces of Innovation: Strategies for HeighteningCreativity. Profile Books LTD.

Kelly, D., Tangney, B., 2006. Adapting to intelligence profile in an adaptiveeducational system. Interacting with Computers 18.

Kelly, S.R., Mazzone, E., Horton, M., Read, J.C., 2006. Bluebells: a design method forchild-centred product development. In: NordiCHI ’06: Proceedings of the 4thNordic Conference on Human–Computer Interaction. ACM, New York, NY, USA,pp. 361–368.

Langford, J., McDonagh, D. (Eds.), 2003. Focus Groups: Supporting Effective ProductDevelopment. Taylor and Francis.

Loftus, G.R., Masson, M., 1994. Using confidence intervals in within-subject designs.Psychonomic Bulletin and Review 1, 476–490.

Loori, A., 2005. Multiple intelligences: a comparative study between the preferencesof males and females. Social Behaviour and Personality 33 (1), 77–88.

MacLean, A., Young, R.M., Bellotti, V.M.E., 1996. Design Rationale: Concepts,Techniques and Use. Lawrence Erlbaum, Ch. Questions, Options, and Criteria:Elements of Design Space Analysis, pp. 21–51.

Markopoulos, P., Bekker, M.M., 2003. On the assessment of usability testingmethods for children. Interacting with Computers 15 (3), 227–243.

McMahon, S.D., Rose, D.S., Parks, M., 2004. Multiple intelligences and readingachievement: an examination of the Teele inventory of multiple intelligences.The Journal of Experimental Education 73 (1), 41–52.

Moraveji, N., Li, J., Ding, J., O’Kelley, P., Woolf, S., 2007. Comicboarding: using comicsas proxies for participatory design with children. In: CHI ’07: Proceedings of theSIGCHI Conference on Human Factors in Computing Systems. ACM, New York,NY, USA, pp. 1371–1374.

Muller, M., 2003. Handbook of Human–Computer Interaction. Erlbaum, Ch.Participatory Design: The Third Space in HCI, pp. 1051–1069.

Olson, G.M., Olson, J.S., Carter, M.R., Storrosten, M., 1992. Small group designmeetings: an analysis of collaboration. Report.

Osborn, A.F., 1942. How to Think Up. McGraw-Hill.Pardo, S., Vetere, F., Howard, S., 2005. Broadening stakeholder involvement in ucd:

designers’ perspectives on child-centred design. In: Proceedings of OZCHI 2005.CHISIG, ACM Press, Canberra, Austrialia, pp. 1–9.

Raffle, H., Ishii, H., Yip, L., 2007. Remix and robo: sampling, sequencing andreal-time control of a tangible robotic construction system. In: Proceedingsof the 6th International Conference on Interaction Design and Children,pp. 89–96.

Ruland, C.M., Slaughter, L., Starren, J., Vatne, T.M., 2006. Children as design partnersin the development of a support system. In: Park, H.-A., Murray, P., Delaney, C.(Eds.), Consumer-Centered Computer-Supported Care for Healthy People:Proceedings of the Ninth International Congress of Nursing Informatics,Studies in Health Technology and Informatics, vol. 122. IOS Press, Amsterdam,pp. 80–85.

Sanders, E., March 2005. Information, inspiration and co-creation. In: 6th EuropeanAcademy of Design, Bremen, Germany.

Sanders, Elizabeth B.-N., Stappers, P.J., 2008. Co-creation and the new landscapes ofdesign. CoDesign 4 (1), 5–18.

Sas, C., Dix, A., June 2006. Exploring the design space. In: DIS 2006 Workshop:Exploring Design as a Research Activity, Penn State, USA.

Sawyer, R.K., John-Steiner, V., Moran, S., Sternberg, R., Feldman, D., Nakamura, J.,Csikszentmihalyi, M., 2003. Creativity and Development. Oxford UniversityPress.

Scaife, M., Rogers, Y., Aldrich, F., Davies, M., 1997. Designing for or designing with?informant design for interactive learning environments. In: Proceedings of theSIGCHI Conference on Human Factors in Computing Systems, pp. 343–350.

Scarr, S., 1985. An author’s frame of mind [review of frames of mind: the theory ofmultiple intelligences]. New Ideas in Psychology 3 (1), 95–100.

Shah, J., Vargaz-Hernandez, N., Smith, S., 2003. Metrics for measuring ideationeffectiveness. Design Studies 24, 111–134.

Shearer, C.B., August 1997. Development and validation of a multiple intelligencesassessment scale for children. In: Proceedings of the 105th Annual Meeting ofthe American Psychology Association. Chicago IL, p. 16.

Siegel, S., Castellan, N.J., 1988. Nonparametric Statistics for the Behavioral Sciences.McGraw-Hill.

Sluis-Thiescheffer, W., Bekker, T., Eggen, B., 2007. Comparing early design methodsfor children. In: IDC ’07: Proceedings of the 6th International Conference onInteraction Design and Children. ACM, New York, NY, USA, pp. 17–24.

http://audacity.sourceforge.net

http://audacity.sourceforge.net

http://jpepsy.oxfordjournals.org/cgi/content/abstract/27/1/47

http://jpepsy.oxfordjournals.org/cgi/content/abstract/27/1/47


Sternberg, R.J., 2003. Wisdom, Creativity and Intelligence Synthesized. CambridgeUniversity Press.

Stienstra, M., November 2003. Is every kid having fun? Ph.D. thesis, University ofTwente, Enschede.

Teele, S., 1992. Teele inventory for multiple intelligences. Tech. rep., Sue Teele andAssociates, Redlands, California.

Teele, S., 1996. Redesigning the educational system to enable all students tosucceed. Tech. rep., NASSP Bulletin, v80.

Thang, B., Sluis-Thiescheffer, W., Bekker, T., Eggen, B., Vermeeren, A., de Ridder, H.,2008. Comparing the creativity of children’s design solutions based on expertassessment. In: IDC ’08: Proceedings of the 7th International Conference onInteraction Design and Children. ACM, New York, NY, USA, pp. 266–273.

Vasta, R., Haith, M., Miller, S., 1995. Child Psychology. John Wiley and Sons, Inc.Verhaegh, J., Soute, I., Kessels, A., Markopoulos, P., 2006. On the design of

camelot, an outdoor game for children. In: IDC ’06: Proceedings of the 2006Conference on Interaction Design and Children. ACM, New York, NY, USA,pp. 9–16.

Vickers, A., 2003. How many repeated measures in repeated measures designs?statistical issues for comparative trials. BMC Medical Research Methodology 3(1), 22. <http://www.biomedcentral.com/1471-2288/3/22>.

Wyeth, P., Purchase, H., 2003. Using developmental theories to inform the design oftechnology for children. In: Proceedings of the 2003 Conference on InteractionDesign and Children. ACM, ACM Press, pp. 93–100.

http://www.biomedcentral.com/1471-2288/3/22

Development and application of a framework for comparing early design methods for young children

Documents

Transcript of Development and application of a framework for comparing early design methods for young children