1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

1

A Methodology for Eliciting, Modelling, and Evaluating Expert Knowledge for an

Adaptive Work-integrated Learning System

Tobias Ley a, c

(Corresponding Author)

Barbara Kump b

Dietrich Albert c

a Know-Center Graz

Inffeldgasse 21a, A-8010 Graz, Austria

Telephone: +43 316 873 9273

Fax: +43 316 873 109273

tley@know-center.at

b Graz University of Technology

Knowledge Management Institute

Inffeldgasse 21a, A-8010 Graz, Austria

bkump@tugraz.at

c University of Graz

Cognitive Science Section

Universitätsplatz 2, A-8010 Graz, Austria

dietrich.albert@uni-graz.at

Running Title: Expert Knowledge Elicitation, Modelling and Evaluation

*ManuscriptClick here to view linked References

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

2

ABSTRACT

We present a methodology for constructing and evaluating models for adaptive

informal technology-enhanced workplace learning. It is designed for knowledge intensive

work domains which are not pre structured according to a fixed curriculum. We extend

research on Competence-based Knowledge Space Theory which has been mainly applied in

educational settings. Our approach employs systematic knowledge elicitation, and

practically feasible evaluation techniques performed as part of the modelling process for

iterative refinement of the models. A case study was performed in the Requirements

Engineering domain to apply and test the developed methodology. We discuss lessons

learned and several implications for knowledge engineering for adaptive workplace

learning.

Keywords: competence-based knowledge space theory; work-integrated learning;

domain modelling; knowledge elicitation; model evaluation

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

3

1 INTRODUCTION

Recent advances in information and communication technologies have had a

significant impact on learning at work by facilitating integration of learning into working

processes directly at the workplace. Work-integrated learning as it is seen here is a form of

informal workplace learning as conceptualized by Eraut (2004) or Garavan et al. (2002) and

it shares with this the informal, interpersonal and experiential character and draws out the

place and process of work as the main setting of learning activities. Learning activities are

not determined by a course structure but by the working context, i.e. by individual

characteristics of the learner, the task at hand, the business process context, the

organizational environment, or several of those (Farmer et al., 2004; Lindstaedt et al., 2008;

Schmidt, 2004). Work-integrated learning systems1 can support a learner in her search by

exploiting available context information and pre-selecting and structuring available content

without restricting her self-regulation. Such approaches seek to make use of available

knowledge artefacts in an organization and reuse them for learning purposes, thereby

creating learning opportunities “on the fly”.

Imagine, for example, a Requirements Engineer who develops a context model for a

flight control system. Further imagine, the Requirements Engineer is a recent graduate and

has never developed a context model for a system in a practical setting. Hence, in this task,

the person may benefit greatly from available documents, like context model templates or

previous applications of these templates in similar projects, or from advice from

knowledgeable colleagues. In the specific situation, our Requirements Engineer may not be

aware that all these resources are available, or that they could be helpful to her. A learning

1 We use the term “work-integrated learning system” to refer to the technical system which supports the

learning process of an individual in the context and in the process of his or her daily work.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

4

system could provide guidance on what content is available, and could use information

about the Requirements Engineer, in order to select the most relevant content for her.

The issue of using context information for selecting appropriate learning resources has

been addressed in research focussing on the development of adaptive eLearning systems

(e.g., Albert et al., 2002; Brusilovsky and Nijhavan, 2002; De Bra et al., 2004; Eklund and

Brusilovsky, 1999; Rich, 1999). Here, adaptivity means that learners are provided with

information that is neither redundant (i.e. already known), nor irrelevant (i.e. not useful for

performing the task) nor incomprehensible. While a human teacher easily, intuitively, and in

a flexible manner is able to respond to the learner‟s (work related) needs, establishing

adaptivity constitutes a complex and demanding matter within the construction of an

eLearning system.

In order to realise adaptivity with a learning system, different types of formal models

need to be constructed. For this purpose, traditional intelligent tutoring systems take as a

starting point either a well structured task domain with algorithmic tasks, such as

performing tasks with a computer software (e.g., Rich, 1999), or a well structured

curriculum (e.g., Billett, 2006). However, this is rare in the case of work-integrated

learning. Very often, neither the tasks are algorithmic, nor is the learning domain

sufficiently well structured in advance. Therefore, modelling is especially difficult in the

case of work-integrated learning.

Even though a variety of adaptive learning systems have been presented in the

literature, the process of how the necessary formal models were built is hardly ever

described in a way that it can be replicated. In most cases, formal models are created by a

domain expert and then considered valid. We would argue that this assumption does not

necessarily hold. Valid formal models, however, constitute a necessary pre-condition for the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

5

success of an adaptive learning system. Obviously, if the models are not valid, this impairs

the quality of the adaptive learning support.

With this article, we suggest a methodology for constructing and evaluating formal

models for an adaptive work-integrated learning system. The modelling methodology is

designed for application domains which are knowledge-intensive (i.e. dealing

predominantly with the acquisition, application and transfer of knowledge) and less

structured than those of traditional tutoring systems (i.e. allowing for considerable amount

of leeway in the execution of tasks). It employs systematic elicitation and modelling of

expert knowledge and a rigorous evaluation technique.

Consequently, the methodology we present improves the process of building models

in two ways. First, it constitutes a theoretically founded guideline for creating different

types of formal models. Second, it includes as an inherent part of the modelling

methodology several validation checks (i.e. checking whether the underlying assumptions

are fulfilled) from which it is possible to judge the current quality of the models and

improve them in an iterative way. The latter point is especially important. Because of the

complexities and irregularities encountered in practical settings, we feel that it is impossible

to devise a methodology which in itself is guarantee for valid and reliable models. Instead,

reliability and validity can only be guaranteed by making quality checks and refining

accordingly.

The remainder of this paper is organized as follows: In section 2, we propose a

number of models for adaptive work-integrated learning. In section 3, we then present a

variation of Doignon & Falmagne‟s Theory of Knowledge Spaces as the theoretical basis.

We then present our modelling methodology in section 4. At each step of the methodology,

we report lessons learned we obtained in a case study, where the methodology was applied

for the requirements engineering domain. We discuss the applicability of the methodology

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

6

and several critical issues we came across during the case study in section 5 and conclude

with an outlook on future work in section 6.

2 ELICITING AND EVALUATING MODELS FOR ADAPTIVE TECHNOLOGY-

ENHANCED WORK-INTEGRATED LEARNING

The first step in building a knowledge model for adaptive technology-enhanced work-

integrated learning is to collect and structure the expert knowledge that is available in the

domain of interest (e.g. the learning domain of Requirements Engineering). Subsequently,

and in line with e.g. Benyon and Murray (1993), we refer to the structured expert

knowledge of a domain as the Domain Model. In our conception, the Domain Model for a

work-integrated learning system consists of two components: the tasks that need to be

performed in the domain, such as creating a context model in the domain of Requirements

Engineering, and the knowledge and skills needed to perform these tasks, such as an

understanding of the concept of system boundaries which constitutes one important element

of a context model in Requirements Engineering. For this second component we adopt the

term competency, which relates to a collection of knowledge and skills that is used to

perform the tasks in question.

In line with the knowledge-intensive domains we are targeting, the use of

competencies as more holistic latent constructs should enable the system to go beyond a

mere performance support system where information about exact task executions or defined

problem solving procedures is the only input variable. Taking into account human

competencies and their role in the production of performance should allow learners to

develop capacities to perform in a broad range of situations. Competencies as related to

learning and work performance have also received great attention within skill based content

personalisation (Cardinali, 2006), or in organisational eLearning (Sicilia, 2007). In brief,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

7

these approaches seek to tailor content presentation for learning purposes to the learner‟s

current learning needs which can be derived from (a) demands of the current task and (b)

available knowledge, skills and prior experience of the learner. This kind of analysis – the

comparison of task demands with a learner‟s abilities – has also been termed competency

gap analysis (see Sicilia, 2005).

In order to perform a competency gap analysis, a model of the learner‟s available

knowledge (competency state) is required. In line with e.g. Benyon and Murray (1993) or

Jameson (2003), we refer to that kind of model as the User Model. As a component of the

learning system, the User Model depicts the learner‟s abilities either in terms of

(observable) tasks he or she is able to achieve, or in terms of (latent) competencies (e.g. by

inferring skills and knowledge from the performance of observable tasks), or in terms of

both.

The User Model constitutes the rationale for presenting individualised learning

opportunities and has to be updated according to the learning progress. Ideally, this update

happens to a large extent automatically, as the learner‟s use of the system is monitored.

Such automatic updating certainly presents a major challenge. However, in the case of

work-integrated learning, where learning happens directly in the task context, there is a

potential for updating the User Model according to past task executions (Ley et al., 2006). It

is for this reason that the Domain Model in our case reflects the connections between tasks

and competencies needed to perform those tasks, and we refer to the updating of the User

Model as a task based competency assessment.

For building both, the Domain Model and the User Model as components of a learning

system, eliciting knowledge from experts is necessary. Knowledge acquisition and

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

8

elicitation methods2 have a long tradition in the area of constructing expert systems. For

example Cooke (1994), or Shadbolt and Burton (1989) present and discuss diverse

knowledge elicitation techniques for knowledge based systems. With the growing demand

for adaptive systems to support workplace learning and development, the topic has recently

received more attention once again.

Knowledge elicitation constitutes a critical phase in building the Domain Model and

the User Model of a learning system and the application of appropriate knowledge

elicitation techniques is crucial to the success of work-integrated learning systems. The

main difficulty associated with knowledge elicitation has been termed the knowledge

specification and acquisition gap, denominating the gap that exists between the actual

knowledge of experts and the knowledge they might be able to articulate (Castro-Schez et

al., 2004, see also Studer et al., 1998). This holds especially for situations where learning

prerequisites need to be modelled: There is still a lack of methods to support domain experts

in deciding which elements are prerequisite for learning some other elements.

With these considerations in mind, it becomes clear that before a competency gap

analysis can be conducted to devise individualized learning opportunities, it has to be

assured that Domain and User Model are empirically valid, i.e. that they correspond to the

true state of affairs. This question of evaluating and validating models has been discussed

both in psychological conceptions of modelling (e.g., Brewer, 2000; Sohn and Doane,

2002), in the domain of eLearning (e.g., Millán and Perez-De-La-Cruz, 2002) as well as in

knowledge engineering (e.g., Cooke, 1994; Studer et al., 1998; Sure et al., 2004), and

Human Computer Interaction (e.g., Chin, 2001; Fischer, 2001).

2 In line with the general convention, we refer to the task of gathering information from any source as

knowledge acquisition, and to the sub-task of gathering knowledge from the expert as knowledge elicitation

(see e.g., Shadbolt, Burton, 1989).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

9

In practice, the evaluation of formal models, especially for adaptive work-integrated

learning systems, is very often neglected for several reasons. On the one hand, the activity

of building formal models is costly in itself (Cooke, 1994), and expert resources required

for building, evaluating and revising the models are scarce. On the other hand, most

modelling approaches are inherently difficult to validate.

In the next section, we present a theoretical approach, Competence-based Knowledge

Space Theory, which can serve as a reference framework for building formal models, and

which clearly formalizes all assumptions so that they can be checked with different

validation techniques.

3 COMPETENCE-BASED KNOWLEDGE SPACE THEORY FOR MODELLING

WORK-INTEGRATED LEARNING

Comptence-based Knowledge Space Theory (based on the Competence-Performance

Approach by Korossy, 1997, 19993) is a mathematical psychological theory that originated

from research into the Theory of Knowledge Spaces 4 (see, e.g., Doignon and Falmagne,

1985, 1999; Falmagne et al., 1990). Knowledge Space Theory structures knowledge

elements in terms of their learning prerequisites which makes it especially suited for

adaptive learning.

The practical convenience of Knowledge Space Theory and its extensions has been

demonstrated in several areas. Among others, the approaches were integrated in a large

scale and commercially successful environment for adaptive testing and teaching in high-

school Mathematics (ALEKS; see ALEKS Corp., 2003) and in a demonstration system to

learn and teach elementary probability theory (RATH; see, e.g. Hockemeyer and Albert,

3 For similar conceptions see Düntsch and Gediga (1995), or Doignon (1994)

4 Our presentation of Competence-based Knowledge Space Theory constitutes a simplification, as the approach

originally takes into account the case of more than one possible “learning history” (Doignon and Falmagne,

1999, p. 62) for one task. Within the present work, only one such learning history is considered.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

10

1999). The application of Knowledge Space Theory was also suggested for modelling less

formalised fields such as workflow-processes (Stefanutti and Albert, 2002), educational and

cross-cultural values (Albert et al., 2003) or child philosophy (Pilgerstorfer et al., 2006).

Ley and Albert (2003a) proposed the use of Competence-based Knowledge Space Theory in

knowledge management to establish worker profiles and derive individual educational

needs. Ley (2006) examined the applicability of the approach in four case studies and

obtained promising results in terms of construct validity.

3.1 Basic assumptions of Competence-based Knowledge Space Theory

The Theory of Knowledge Spaces by Doignon and Falmagne (1985, 1999)

presupposes that a field of knowledge can be parsed into a set A of problems (or tasks)

Ax each of which has a correct response. Instances of tasks from the Requirements

Engineering domain are Plan and Prepare Acquisition Sessions, Identify a Basic List of

Stakeholders, or Build a First-Cut Context Model.

According to Competence-based Knowledge Space Theory, every person is able to

accomplish a particular subset Z of tasks of A , and accordingly he or she is in a particular

performance state AZ . For instance, the Requirements Engineer introduced in section 1

may be able to identify stakeholders and to build a first-cut Context Model but may not be

able to develop an extended Context Model. The pattern of mastery and failure of the

Requirements Engineer in all tasks of the domain constitutes her solution pattern, and the

set of tasks she is able to accomplish represents her performance state.

One of the central ideas of Knowledge Space Theory is that solution dependencies

exist among problems of a domain. These solution dependencies are formally denoted by a

prerequisite relation AA that is interpreted as follows: When ba holds for two

tasks Aba, , one can say that a is a prerequisite for b . For example, a prerequisite

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

11

relationship might be given between the two tasks ( a ) Build a First-Cut Context Model and

( b ) Develop an Extended Context Model. The relationship ba means that being able to

develop a first-cut context model ( a ) is a prerequisite for being able to build an extended

Context Model (b ). Accordingly, every performance state in the domain that contains the

task ( b ) would also contain the task ( a ). For our example, this means that because of the

prerequisite relation, there is no Requirements Engineer who is able to build an extended

Context Model but is not able to build a first-cut Context Model. Because of these

dependencies among tasks, not every conceivable subset of the set of tasks Z constitutes a

feasible performance state. The set of all feasible performance states in a set A is called

performance structure P on A , denoted by the pair )( PA, . If PA, and P is stable

under union, the pair )( PA, is called performance space on a domain.5

This closure under union property is sensible from the learning and teaching

perspective: It is reasonable to assume that any union of two performance states can be

reached by learning, and, hence, this union of the two sets should also be a performance

state and be part of the performance space. The closure under union property also leads to a

computational simplification as it allows for storing the space as a base. The base of a

performance space is the set of performance states which can not be generated by taking

unions of other performance states. Thus each performance state in P can be written as

union of base elements.

By extending Knowledge Space Theory, Korossy and others describe a domain not

only by a set A of tasks Ax but, in addition, the domain is conceived by a set of

elementary competencies for each of which it can be assumed whether a person has

this competency or not.

5 The mathematical properties of performance structures and the relationship to corresponding prerequisite

relations are formulated in Doignon & Falmagne (1999).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

12

Competencies are knowledge and skills which a person has available to perform a

task, and which can be acquired through learning. Examples for Requirements Engineering

competencies are data-gathering skills, or knowledge about adjacent systems. Analogous to

the performance structure, a prerequisite relation is assumed on the set of elementary

competencies. The collection of all knowledge and skills of a person form his or her

competence state K . The set of all feasible competence states can be modelled through a

finite, non-empty family of competence states, the competence structure ),( K . The pair

),( K is a competence space if K, and K is stable under union.

The formal linkage between competencies and tasks happens by defining a

relationship between tasks and competence states. For each task Ax and each

competence state KK it is uniquely determined whether or not x can be successfully

performed in K . Formally spoken, the set of tasks A is interpreted in K by the mapping

)(Ak , which is called interpretation function. The interpretation function assigns to each

task Ax a set xk that contains all elementary competencies which are necessary

and sufficient for successfully performing the task6. For example the task Build a First-Cut

Context Model may require two competencies: Knowledge about Context Models (i.e.

knowledge about the elements and relations of these models) and Ability to produce a

Context Model (i.e. knowledge about the notation and rules and how to apply them). The

task can therefore be solved by every person whose competence state contains at least these

two competencies, i.e. obviously also by persons whose competence states include

additional competencies.

6In Korossy (1997), the set of competencies KK that are necessary and sufficient for

solving the problem was referred to as minimal interpretation, k xMin , of a task Ax

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

13

Another mapping, the representation function )(Kp assigns to each competence state

KK (and each person respectively) a unique (possibly empty) set of tasks which can be

solved in that competence state (and by that person). Note that the representation function is

fully determined by the interpretation function.

Within the present work, a model consisting of a competence space, an interpretation

function and a representation function and a resulting performance structure is referred to as

competence-performance structure.

3.2 Constructing a Competence-Performance Structure for Work-integrated

Learning

As mentioned previously, a central aspect of work-integrated learning is the close

connection between task performance and learning which makes it a natural area of

application for Competence-based Knowledge Space Theory. At this stage, we will derive

an approach for constructing a competence-performance structure for a work-integrated

learning application. In the next section, we will then present an illustrative example.

Korossy (1997, see also Korossy and Held, 2001) describes the rough steps in the

process of developing a competence-performance structure as a knowledge acquisition and

modelling procedure with experts. Typical tasks have to be identified and a set of

competencies has to be established which are necessary for performing the tasks. The set of

competencies has to be structured with regard to “diagnostically relevant” prerequisite

relationships (Korossy, 1999). As suggested above, establishing a prerequisite relation on

the set of elementary competencies constitutes the hardest challenge for ill-structured

learning domains which we are typically facing in the case of work-integrated learning: One

can imagine that experts may be overstrained when they are asked to decide whether, e.g.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

14

analytical skills are prerequisites for the ability of abstraction, whether it is right the other

way round or whether the two are independent from each other.

In order to overcome the difficulties of directly establishing a prerequisite relation on

the set of elementary competencies, we have introduced the simplified method of a task-

competency matrix (Ley and Albert, 2003a, 2003b; see also Ley, 2006). With this matrix,

experts are asked to assign to each task the set of competencies necessary for performing the

task (the minimal interpretation of that task). The idea is to identify feasible competence

states by means of the interpretation function: if a person is able to perform a certain task,

he or she has to have all required competencies. Hence, determining the minimal

interpretation of a task leads to a feasible competence state. By closing the set of minimal

interpretations for a set of tasks under union, the set of all feasible competence states in a

domain, i.e. the whole competence space, is obtained. The corresponding performance

structure is then obtained by assigning to each competence state the set of tasks that are

solvable in the respective state.

3.3 An Illustrative Example

To illustrate the mechanics of our approach and the potential for application, we will

give a small illustrative example taken from the case study to be presented later in more

detail (in section 4).

Assume the set of ten tasks and the set of six competencies from the Requirements

Engineering domain given in Table 1.

======== INSERT TABLE 1 HERE ========

Also assume the interpretation function given as a task competency matrix in Table 2.

In each line of the table, crosses indicate that a task requires the set of competencies as a

minimum. The rightmost column of the table gives the minimal interpretation for each task.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

15

In case the minimal interpretation can not be generated by the union of two or more other

elements, it is an element of the base. For this reason, the minimal interpretation of task

5_3, which is 13, 15, 20, is not part of the base. In total, the base of the competence space

in the example comprises seven competence states.

======== INSERT TABLE 2 HERE ========

From the minimal interpretation given in Table 2, a prerequisite relation on the set of

competencies can be derived, as there is a formal relationship between the set of

competence states and the prerequisite relation on the set of competencies (as described in

section 3.1). As all minimal interpretations constitute competence states, we can

approximate the prerequisite relation on the set of competencies using the following rule: A

prerequisite relationship is always assumed between two competencies when one only is

present when the other is also present. More formally, this means, a competency a is

prerequisite for a competency b ( ba ) if whenever b is assigned to a task then a is also

assigned to that task for all tasks of the domain. For the example, we arrive at the

prerequisite relation given in Figure 1. For instance, a person who has Competency 16 is

assumed to also have Competencies 13 and 15. In other words, Competencies 13 and 15 are

prerequisites for Competency 16. This is because there is no base state (and hence no

competence state) that includes 16 without at the same time including 13 and 15.

======== INSERT FIGURE 1 HERE ========

The whole competence space can then be spanned by closing the base states under

union. In the example, we obtain 18 feasible competence states which is a small subset of

the 64 possible combinations in the power set (26, where 6 is the number of competencies).

Figure 2 shows the Hasse-diagram of the competence space and the performance

representation for the example. Performance states were generated by assigning to each

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

16

competence state the set of tasks that can be performed by a person being in the respective

state.

======== INSERT FIGURE 2 HERE ========

Figure 2 also shows a prerequisite relation on the set of tasks which is due to the

relation on the set of competencies. This is illustrated by the following example: Any person

who is able to accomplish Task 4_3 is assumed to be able to perform well Task 4_6. In

other words, Task 4_6 is considered as a prerequisite of Task 4_3 since the minimal

interpretation of Task 4_6 (13, 20) is a subset of the minimal interpretation of Task 4_3

(3, 13, 20).

In continuing with the example, we now show how to assess a learner‟s competence

state from the tasks performed in the past (task-based competency assessment), and how to

compute a competency gap for a task at hand. For task-based competency assessment, the

learner‟s task performance is reviewed. Imagine a Requirements Engineer, who is able to

perform all tasks except for 5_4_Check that each individual SD Model is complete and

correct with stakeholder goals, soft goals, tasks and resources and 5_5 Validate the i* SR

Model against the SD Model (cross-check), in other words the set 3_1, 4_2, 4_3, 4_5, 4_6,

5_1, 5_2, 5_3 constitutes the performance state of that person according to the example

introduced previously. The competence state of the person would then be inferred by taking

the union of the well performed tasks‟ minimal interpretations. Consequently, the

competence state of the Requirements Engineer (3, 12, 13, 15, 20) would comprise all

competencies except for competency 16 Knowledge of validating the SR Model.

Conducting a competency gap analysis happens by building the difference of a

learner‟s competence state and the minimal interpretation of a task or a set of tasks the

learner should perform. For example, if a person only had the competency 15 Knowledge

about the Strategic Rationale Model (SR-Model), and should perform task 4_2 Model the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

17

system’s hard and soft goals, the person would need to acquire the competency 20 Ability to

produce an i* Model. In other words, the set 20 constitutes the competency gap of this

person.

This deterministic approach to competency assessment is of course limited in the face

of uncertainty. Stochastic knowledge assessment routines, originally proposed by Falmagne

and Doignon (1988), take into account uncertainties in updating the user model. Falmagne

et al. (2006) show how such knowledge assessment happens in a large scale adaptive

learning system that makes use of Knowledge Spaces by endowing the knowledge structure

with a likelihood distribution and updating this distribution using Bayesian inference rules.

Because in knowledge space theory theses stochastic methods do not have to be built into

the domain model (see e.g. Villano, 1992), they will not play a major role in the following

presentation of our modelling methodology. We will extend this discussion in section 5.5

when we talk about the application of the domain model for user modelling.

4 A METHODOLOGY FOR BUILDING AND EVALUATING COMPETENCE-

PERFORMANCE STRUCTURES FOR WORK-INTEGRATED LEARNING

After we have established Competence-based Knowledge Space Theory as a

promising approach for adaptive work-integrated learning, we will now present a

methodology for building a competence-performance structure for this purpose.

Because the application of the methodology has to be feasible in practical settings, the

following constraints have to be taken into account. First, in order to minimise the involved

domain experts‟ efforts, we are relying mainly on document analyses, and on reusing

existing sources (e.g., competency catalogues) during knowledge acquisition. Only at

critical stages of the modelling process (e.g., the review of task lists, or the task-competency

assignment), we are involving experts. Second, we are attempting to embed model

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

18

evaluation into the process of model development, and thereby again minimizing experts‟

efforts. Finally, the methodology does not require spatial proximity of the modelling team,

and hence it takes into account limited opportunities for face to face meetings. As can be

seen from Figure 3, the methodology we are suggesting consists of five steps initially being

performed in sequence. As we have indicated previously, it is an important feature of the

methodology that it is iterative: with the information obtained during model evaluation,

further elicitation of tasks and competencies or a change in the model needs to take place

(see the broken arrows in the Figure).

======== INSERT FIGURE 3 HERE ========

A case study7 was performed to apply and test the developed methodology for

establishing and evaluating competence-performance structures in the Requirements

Engineering work domain. With the case study it was to be examined whether the

methodology was appropriate in practical settings. For this purpose, we decided to involve

two independent experts and to establish the models in two separate sub domains in order to

be able to make comparisons later on.

Table 3 gives an overview of each step of the modelling methodology, its purpose, the

suggested methods to be employed and the roles responsible for carrying out the activities.

In the following sections, we will describe each step in more detail together with lessons

learned from our case study.

======== INSERT TABLE 3 HERE ========

4.1 Defining the Scope and Purpose of the Models

In our view, defining the scope and purpose is an important first step in knowledge

engineering, while surprisingly it is often neglected in other methodologies (see Schreiber et

7 This case study was part of the APOSDLE project (see http://www.aposdle.org and Lindstaedt and Mayer,

2006) in which a work-integrated learning system for prospective Requirements Engineering professionals is

currently under development.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

19

al. (1999) and Uschold et al. (1998) for notable exceptions). It is generally employed to

generate requirements on the models which will be heavily influenced by their intended use

in the adaptive system. The better the intended purpose is known, the easier it will be to

formulate these requirements in advance. The intended purpose also determines many of the

subsequent model design decisions including granularity and the degree of model precision

and validity that is needed.

A second important question in this step is the learning domain that is the intended

target, and its intended scope. At this stage, the decision has to be made who is the target

group, and what knowledge shall be conveyed. Partly, scoping of the models will also

depend on the amount of documented information sources about the domain and the

availability of experts.

4.1.1 Procedure and Outcomes

In our case study, the purpose of the models had been specified in advance by an

intensive requirements analysis with stakeholders in the application domain and by

describing the intended use cases of the adaptive learning system8. Use cases, such as the

accessibility of documents, learning material and experts in a work-integrated fashion, and

the tailoring of these materials to the current user‟s task context and prior knowledge, gave

us a good basis for specifying the intended use of the models. Also it became apparent that

competency assessment would have to exploit information on task executions, since a test-

like assessment scenario was not felt to be adequate for the intended user groups.

The domain selected for testing the developed modelling methodology was

Requirements Engineering, more specifically the RESCUE process (Requirements

Engineering with Scenarios in User-Centred Environments, Maiden and Jones, 2004). We

8 An in-depth presentation of the requirements gathering activities in the project is out of scope of this article.

For an extensive documentation of these activities, see APOSDLE Consortium (2006).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

20

selected the RESCUE process for our case study, because it is a well-tried Requirements

Engineering method that has been successfully applied in practise. RESCUE was developed

to standardise the process of requirements elicitation and requirements management. The

designated outcome of the RESCUE process is a catalogue of consistent and valid

requirements for a future system. We think of RESCUE as a typical domain for work-

integrated learning as it is on the one hand clearly structured into separable tasks and

processes. On the other hand, it offers considerable leeway for performing the tasks and

requires significant amounts of skills and knowledge to be acquired.

Throughout the RESCUE process, different modelling and analysis activities run in

parallel. These sub processes strongly inform each other and are aligned at various

synchronisation points. We chose the first two sub-processes of RESCUE, namely Activity

Modelling and System Goal Modelling, as the domains for the case study.

Activity Modelling is done to provide an understanding of how people work in order to

baseline possible changes to it. Data on human activities are gathered using observations or

interviews, structured into activity descriptions and built into the Activity Model. The aim

of the second stream, System Goal Modelling, is to model the future system boundaries and

to identify dependencies between actors for goals to be achieved. This is done by building

and iteratively refining a graphical Context Model using the i* notation that shows what

actors can achieve on their own, and what they depend on others for.

The reason we chose these two sub-domains was twofold. First, it was found during

the scoping phase that sufficient documented material and expertise was available for the

two, both for modelling purposes and for learning with the realized system. A more detailed

description of the available materials and the experts is given in section 4.2.

The second reason for choosing these two sub-processes was that they were

sufficiently distinct in terms of some characteristics that we deemed to be important for the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

21

applicability of our methodology: The amount of practical expertise with the two streams

differed, as experts had more practical experience with System Goal Modelling. The

amount of structuring in terms of a clearer separation and sequencing of tasks was also more

pronounced in System Goal Modelling than in Activity Modelling. This would enable us to

compare the impact of different domain characteristics for the applicability of our

methodology. After having presented the results of the case study, we will return to this

discussion of domain applicability in section 5.1. We will also discuss the effects of this

phase for granularity of the models (in section 5.3) and for the amount of model precision

and validity needed (in section 5.5).

4.2 Eliciting Knowledge: Tasks and Elementary Competencies

For constructing a competence-performance structure, tasks and competencies need to be

collected that cover the selected domain (see section 3.2). For doing so, suitable experts as

well as existing documentations have to be identified. Complying with the requirements of

work-integrated learning, the chosen documents and experts need to cover two types of foci.

First, a performance focus requires that experts have performed the tasks that are being

modelled, and the material needs to come from actual task executions. Second, a learning

focus has to be present in which experts have to make inferences about knowledge needed

for a task and the documents ideally have some relevance for learning in the tasks.

This means that experts ideally have both, practical experience in the domain as well

as experience in supervising others or in teaching. Documents need to be both learning

content (e.g. a description of how to perform a context analysis) but also results from prior

executions of tasks (e.g. context models from prior projects). The documents can later be

used as a seed for the actual learning material used by the system.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

22

Next, sets of tasks and competencies have to be formulated. In our experience, it is a

good idea to start with the tasks as experts find it more natural to describe a domain in terms

of what they do. Also, this way the model of learning is more directly related to actual task

performance. The selected tasks form the operationalisation of the domain. Hence, it is

important to ensure that they cover the whole domain to be modelled. For this it is helpful to

sequence tasks, as it may help experts to come up with a consistent list. Task granularity is

important, as it will determine the number of assigned competencies and the granularity of

the whole model. At the outset, we decided to rely mainly on experts‟ intuition and on the

use cases from the scoping phase when deciding about the granularity of tasks.

In a next step, a set of elementary competencies has to be identified. Clearly, this is

the more difficult task as it involves making inferences and is therefore more error prone

(Lievens et al., 2004). Formal techniques which we have employed to support this process

include the Critical Incidents Technique (Ley and Albert, 2003a) and the Repertory Grid

Technique (Ley and Albert, 2003b, see also Gaines and Shaw, 1993). Especially the latter

helps to draw out underlying assumption that experts have and is also perceived to be

helpful by the experts themselves.

As in the present case, our experts were experienced teachers, we decided against

these formal techniques. Instead, we relied on extensive existing documentation on the

RESCUE process, as well as on existing competency catalogues. We also made an attempt

to ground the elicitation process by referring the experts to concrete situations and persons

they had worked with in the past.

4.2.1 Procedure and Results of the Case Study

First of all, one set of tasks to be achieved in the Activity Modelling stream and one set

of tasks for System Goal Modelling were derived. To enable a later comparison of the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

23

resulting models, the two streams were modelled separately. Since a number of detailed

descriptions and tutorials for the RESCUE process were available, a content analytical

approach was chosen to derive candidate lists of tasks and elementary competencies. These

were later validated with two requirements engineering experts with considerable expertise in

the RESCUE methodology.

4.2.1.1 Deriving Tasks and Initial Competency Phrases: Summarising Qualitative Content

Analysis

The main part of tasks were elicited from the extensive RESCUE process

documentation (Maiden and Jones, 2004), a very detailed description of the four streams

that includes theory, worked examples (e.g. from a requirements elicitation project for a

transportation management system) and practical guidance.

We employed a summarising qualitative content analysis (e.g. Mayring, 2003) to

derive initial task and competency descriptions. For the tasks, the whole document was

walked through phrase by phrase. All phrases in the document that described activity (i.e.

no explanations, no theory) were collected and paraphrased, i.e. reworded in a simple

grammatical form without elaborations. To give an example, the text passage “The analysis

of the activity should identify constraints of the domain in order for the design to support

practitioners in complying with them” was transformed into “Identify constraints of the

domain”.

The list of activities was further reduced by removing activities that did not fit the

intended granularity for tasks, i.e. too specific and too general activities. Similar tasks were

accumulated into one task. For example, the phrases “Identify external resources” and

“Identify resources available to achieve goals (observable)” were aggregated into “Identify

external resources that are available to achieve goals”. In contrast, the phrases “Identify the

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

24

system‟s goals and high level functional goals” and “Identify local goals” were not merged,

since they were assumed to be performed in a different manner (and thus to require different

sets of competencies). In so doing, a first list of 85 tasks was identified, 60 for Activity

Modelling and 25 for System Goal Modelling.

To derive competency descriptions, the RESCUE documentation was analyzed in a

similar manner. Phrases that potentially indicated knowledge or skills required for Activity

Modelling and System Goal Modelling were extracted and summarized. Moreover, existing

competency catalogues, e.g., from van den Berg (1998) or the Occupational Information

Network (“O*NET”, National O*NET Consortium, 2005), were used to complement the list

of competency phrases. The latter, “O*NET”, is a database of worker attributes that was

created to support, e.g., the development of job descriptions. From all these sources, a list of

98 phrases describing competencies was produced.

4.2.1.2 Validation of Tasks and Competencies with RESCUE Experts

In order to ensure that the two task lists were complete and correct, and in order to

complement the list of competency phrases, two RESCUE experts were involved.

The first, Expert 1, was professor of Systems Engineering who had been principal

investigator on numerous projects in the fields of Requirements Engineering and socio-

technical systems design. The other, Expert 2, a research assistant of Expert 1, had done

significant work in the areas of human-computer interaction and Requirements Engineering.

She also had taught various courses and published a significant amount of papers in related

areas. According to their own judgement, both domain experts had considerable practical

experience in System Goal Modelling, and to a lesser degree in Activity Modelling.

As the experts and the modelling team were spatially separated (the Experts were in

England, the modelling team in Austria), the review of tasks and elementary competencies

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

25

happened via email and telephone. Prior to the phone call, the lists of tasks were e-mailed to

the RESCUE experts and they were asked to run through the task lists and to make

suggestions for modifications. Also, the experts were asked to jointly think of five

Requirements Engineers they had both worked with in the past and describe them in a short

paragraph. We will refer to these as personas. The personas were used in the current phase

for drawing on expert‟s practical experience, as well as in the validation phase (see section

4.5.1.2).

During the phone call, the meanings of ambiguous tasks were clarified or removed.

After this review of the task lists, in total 47 tasks remained, 29 of which belonged to

Activity Modelling, and 18 belonged to the System Goal Modelling stream. This list was

later accepted as the final version (see Appendix A).

After that, the RESCUE experts were asked to think of the personas and some

concrete situations in which tasks were performed and to brainstorm competencies

necessary for performing the tasks. During the interview, the experts came up with 36

additional phrases describing competencies, which were added to the list of 98 phrases that

had been derived from document analyses. When considering the intended granularity of

competencies as defined in the scoping phase, it became clear that the list of competency

phrases would have to be reduced. As a more specific guideline and in line with the purpose

of the system, it was useful to think of learning content that could be consumed in 1-2 hours

working time. Also, if two separate elements of knowledge or skill would always be used

together in all of the tasks, and additionally would most likely appear together in any

learning material that was available, this constituted a good argument that they could be

combined.

Hence, competency phrases referring to such low-level competencies (e.g., Knowledge

about strengths and weaknesses of every acquisition technique) were categorised into more

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

26

general competencies (e.g. Knowledge about acquisition techniques available). A catalogue

of competencies was derived (see Appendix B) in which the original phrases were preserved

under the merged competencies so as to retain the meaning of the merged competencies.

After validation with the RESCUE experts, the final competency catalogue consisted of 33

competencies (20 knowledge, 13 skills).

4.2.2 Lessons Learned

The method of a Qualitative Content Analysis led to useful first lists of activity

descriptions and competency phrases, and thereby significantly facilitated subsequent

interactions with the experts. As to the experts involved, we certainly were fortunate to have

available experts both with practical experience and with experience of supervising and

teaching others. The latter clearly helped to identify competencies which turned out to be

the more difficult part. We used several hints when interacting with the experts: We asked

them to think about concrete persons they had supervised in the past and why these persons

did a task well or needed support. We asked them to think of how people had developed

expertise in a certain task, or about what they were teaching. Although we did not use any

formal methods in this stage, we would recommend them in a case where only experts with

less of a learning focus are available.

One of the main issues in this phase and a regular topic of discussion was the

granularity of the model. We found that in general, experts had a good intuition about the

granularity as they could draw on their own experience. We often reminded the experts of

how the models would be used in a work-integrated learning fashion. The experts found it

helpful to think of the possible future learning interventions and the available content to

decide whether the granularity was in line with the overall purpose. We will be giving some

more thought to granularity issues as well as some formal considerations in section 5.3.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

27

4.3 Linking Tasks and Competencies

To build the competence-performance structure, it is necessary to link tasks and

competencies obtained in the previous step to establish the interpretation function. We

applied the method of a task-competency matrix as described in section 3.2 which asks

experts to assign to each task the set of all competencies necessary for an adequate

performance. We decided to let experts make these judgements in increments of “absolutely

necessary” (2), “somewhat necessary” (1), and “not necessary” (0). First, our experience

had shown that raters feel more comfortable when they can rate in increments, and second

we were intending to use only the extreme value of (2) in order to get more stable results for

the structures. We also decided to obtain these assignments independently from both experts

in order to check for inter-rater reliability later on and, hence, to obtain an impression of the

agreement of experts‟ understanding of the domain.

4.3.1 Procedure and Results of the Case Study

In the case study, the task-competency matrix consisted of an Excel spreadsheet with

all 47 tasks in the rows and all 33 competencies in the columns. This spreadsheet was sent

to the RESCUE experts via e-mail before we called each expert separately. During the calls,

the experts were instructed to start with the first Activity Modelling task, and to assign the

value 2 to every competency in the catalogue that they regarded „indispensable‟ for

performing the task well. Every competency they regarded as „somehow important, nice to

have‟, should be given the value 1. Once the procedure was clear, the experts were asked to

finish the assignments on their own and to return the results via e-mail (see Appendix C for

an example of such an assignment). Table 4 reports the descriptive results in terms of the

number of assignments for each expert and for each stream.

======== INSERT TABLE 4 HERE ========

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

28

4.3.2 Lessons Learned

In both streams there was a large discrepancy of the amount of times value 1 was

assigned by each of the two experts. Most likely, this was due to the fact that the description

of “somewhat necessary” was not interpreted in the same way by the two experts. Using a

scale for rating has proven to be a good strategy, as this helped to uncover these rating

tendencies. As we later dichotomized the scale for building the models (see section 4.4),

these different rating tendencies did not cause any problems.

We also informally asked the experts about their experiences with assigning

competencies in the matrix. Both experts were in agreement that the task had been

extremely demanding requiring them to make over one thousand judgements. Also, they

criticised the presentation format (Excel spreadsheet), and considered this to be very error

prone. We are taking this feedback into consideration when we discuss the task competency

matrix and its application in section 5.4.

4.4 Building Competence-Performance Structures

The task competency matrices derived in the previous step form the basis for spanning

competence-performance structures. The procedure for doing so has been presented in

section 3.3. It is important to note that this procedure does not involve any more modelling

effort, but it is merely a computational procedure.

In our case study, we had derived independent assignments from the two experts. A

natural next step would be to find ways of merging the two to reduce errors. This can be

done by the two experts agreeing on common assignments, the drawback being that it is

time consuming and that the experts may not be available. So we decided to check for the

effects of computational merging and apply the validation procedures to both the separate

structures as well as a computationally merged structure. One method for merging two (or

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

29

more) performance spaces computationally was suggested by Dowling (1994) who built a

common space by taking the union of the different spaces. This procedure leads to a large

number of performance states, and in turn to a reduction of prerequisite relationships (and

hence reduction of the adaptive potential of the structures). Alternatively, one can unite the

two minimal interpretations for each task, and this was what we did in the case study.

4.4.1 Procedure and Results of the Case Study

Four competence-performance structures were built on the basis of the task-competency

assignments obtained from the two RESCUE experts in the previous modelling step: From

each expert‟s task-competency assignment, a separate structure was derived for Activity

Modelling and System Goal Modelling applying the procedure described in section 3.3. The

rating scales were dichotomized, that is only competencies with an assigned „2 (mandatory)‟

for a task were used to build the structures. Additionally, a merged structure was produced

for System Goal Modelling by uniting each task‟s minimal interpretation. In other words,

each competency at least one of the experts had considered „mandatory‟ for the

accomplishment of a certain task was also considered „mandatory‟ in the merged

assignment. A competence-performance structure was then derived for the merged

assignment. The assignments for Activity Modelling were not merged because of low inter-

rater agreement (see section 4.5.1.1).

Table 5 summarises the most important characteristics of the five structures derived

from the RESCUE experts‟ task-competency assignments. The second column shows the

numbers of competencies considered mandatory in the respective assignment, i.e. the

cardinalities of the sets of elementary competencies. Furthermore, the numbers of all

possible combinations of competencies, i.e. the cardinalities of the power sets on the sets of

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

30

elementary competencies, are given, as well as the cardinalities of the bases and the

competence spaces.

======== INSERT TABLE 5 HERE ========

Figure 4 gives a picture of the prerequisite relation on the set of elementary

competencies derived from the merged task-competency assignment for System Goal

Modelling9. According to the prerequisite relation, a Requirements Engineer who is

performing System Goal Modelling and who has Knowledge about the Context Model

(Competency 12) is assumed to also have at least Knowledge about different types of

adjacent systems (Competency 11) and Knowledge of different types of system stakeholders

(Competency 9) because these two competencies constitute prerequisites of Competency 12.

======== INSERT FIGURE 4 HERE ========

It should be noted that the procedure of deriving prerequisite relations rests on the

assumption that the modelled set of tasks is complete with regard to the domain to be

modelled. This means that the tasks should include all tasks that a person operating in the

domain is usually required to perform to be considered competent. One way of verifying

that the assumption has been met is by performing validity checks which will be described

in the next section.

4.5 Evaluating Competence-Performance Structures: Checking Reliability and

Validity

Before the structures are employed in an adaptive learning system, it needs to be

shown that the resulting structures correspond to the true state of affairs. Our approach here

is to employ standard measures of reliability and validity which can be established in the

9

For ease of inspection, the prerequisite relation is depicted separately for elementary competencies referring

to knowledge and for competencies referring to skills.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

31

modelling process. The purpose is to give experts and modellers early feedback about the

quality of the structures for an iterative refinement.

Evaluating the reliability in establishing a competence-performance structure means to

examine the degree to which different modellers (or experts) give consistent estimates of the

same phenomenon, i.e. examining inter-rater reliability (e.g., Kline, 2005). For the present

work, nonparametric measures of contingency were considered the most appropriate for

comparing the two RESCUE experts‟ task-competency matrices. In cases where only one

expert is involved, consistency can be checked by a variation of test-retest reliability, e.g. by

having the expert repeat some of the assignments at a later point. We have employed the

latter method to compare assignments from a repertory grid interview to those obtained

from task competency matrices and found close to 80% agreement (Ley et al., 2006).

The validity of a competence-performance structure concerns the model‟s usefulness

for accurately assessing competencies and predicting performance in the context of the

model‟s application. For competence-performance structures, empirical validity can be

demonstrated by examining whether observed solution patterns coincide with predicted

performance states. This validation method has been carried out for several domains, e.g.,

Chess, Word Problems and Mathematics (e.g., Albert and Lukas, 1999; Doignon and

Falmagne, 1999; Falmagne et al., 2006).

In knowledge intensive work domain such as Requirements Engineering, a direct

empirical validation by comparing observed solution patterns with predicted performance

states is difficult or even impossible for the following reasons. A sufficient number of

workers is required, who are neither able to achieve every task nor fail at every task. And, in

order to obtain accurate solution patterns, every person under consideration must have tried

to perform every task of the domain. Both these conditions may be hard to meet. Alternative

validation methods focussing on construct validity have been employed for competence-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

32

performance structures in knowledge management (e.g., Ley, 2006; Ley and Albert, 2003b).

Another possibility to overcome the problem of a small validation sample is suggested by

the Leave One Out cross-validation method (e.g., Ley et al., 2006; Witten and Frank, 2005).

Instead of a large empirical sample, the application of Leave One Out for validating a

competence-performance structure only requires a small number of complete solution

patterns (at least one), where complete means the assessment of one worker‟s performance

in every task of the domain under consideration.

The Leave One Out method considers the solution pattern of one person for n-1 tasks,

that is one of the tasks is “left out”. His or her competence state is inferred from the solution

pattern in the remaining n-1 tasks by taking the union of all mastered tasks‟ minimal

interpretations. From the obtained competence state, the person‟s performance in the nth

task, the left-out task, is predicted: If the minimal interpretation of the nth

task is a subset of

the competence state, the person should perform the task well. Otherwise, the person should

not be able to perform the task. This procedure is repeated for each task and each person

under consideration. A contingency coefficient can be computed in order to determine

whether a solution pattern of a person coincides with the performance (mastery or failure) in

a task as predicted by the Leave One Out method.

It should be noted that this procedure corresponds to a task based competency

assessment procedure in which a person‟s competency state is predicted from an analysis of

her previous task performance. We therefore consider this validation procedure to be

especially applicable in our case.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

33

4.5.1 Procedure and Outcomes of the Case Study

4.5.1.1 Checking Reliabilty

To assess reliability in terms of inter-rater agreement between two experts‟ task-

competency assignments, contingency-tables were produced and a contingency coefficient

( b ) was computed. Inter-rater agreement was examined across both streams as well as for

each stream separately. Significance was tested two-tailed on a 5%-level.

The contingency coefficient of the raters‟ assignments was moderately positive ( b =

.32, z = 8.60, p < .01) across both streams. Looking at the two streams separately, the

coefficient was .25 for Activity Modelling (z = 5.90, p < .01) and .45 for System Goal

Modelling (z = 6.13, p <.01). Obviously, especially for Activity Modelling, the two

RESCUE experts had a rather different understanding of which competencies were required

for performing the Activity Modelling tasks.

4.5.1.2 Checking Validity

To employ the Leave One Out method, estimations for complete solution patterns of

persons performing the tasks are needed. For this, we used the personas the RESCUE

experts had come up with in the elicitation phase (see section 4.2.1.2). Notional solution

patterns for these five personas were collected by means of a validation questionnaire that

was filled out by the two RESCUE experts.

The validation questionnaire was an excel spreadsheet with the descriptions of all five

personas and the lists of tasks for Activity Modelling and System Goal Modelling. Experts

were requested to go through the task lists for each persona and decide for each task,

whether the person would be able to accomplish the task without assistance (value 2) with

assistance (value 1), or not at all (value 0). This way, complete solution patterns for all five

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

34

personas were generated. Only tasks that the experts had allocated value „2‟ in the

validation questionnaire were regarded as mastered by a person under consideration.

First, we investigated the agreement between the two RESCUE experts‟ assessments

of personas. There was a positive correlation between the assessments of Expert 1 and

Expert 2 across all five personas and both streams with an overall contingency of b = .57

(z = 9.15, p < .01), b = .59 for Activity Modelling (z = 6.01, p < .01), and b = .49 for

System Goal Modelling (z = 5.39, p < .01). Overall, the agreement between the experts‟

assessment of personas was of satisfactory magnitude from which we conclude that the

assessments were done reliably.

As a next step, the validity of each of the five competence-performance structures was

investigated by means of the Leave One Out method. For examining validity, a contingency

coefficient was computed for comparing the predicted performance according to each of the

five competence-performance structures with the solution patterns from the validation

questionnaires.

For Activity Modelling, the assessment of one persona had to be excluded since both

experts did not assume the person to be able to accomplish any task without help (i.e. no

task got value 2). In System Goal Modelling, the validation with the assessments of Expert

1 is based on only four solution patterns for the same reason.

Table 6 shows contingency coefficients of the predictions of each of the five

competence-performance structures with the solution patterns generated by the two

RESCUE experts10

. The values of all coefficients were significantly positive.

======== INSERT TABLE 6 HERE ========

10

In Table 6, there is no merged structure for Activity Modelling because of low inter-rater agreement

between the RESCUE experts‟ task-competency assignments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

35

Agreement coefficients vary between b = .77 (z = 7.72, p < .01) for the assessment of

Expert 2 and the predictions for System Goal Modelling derived from her own assignment,

and b = .26 (z = 3.01, p < .01) for the predictions of Expert 2 for Activity Modelling and

the assessment by Expert 1. In 3 of 4 cases, predictions of the model of Expert 2 were more

consistent with both experts‟ appraisals than the predictions of Expert 1 and of the merged

structure.

As expected, the solution patterns generated by both experts for Activity Modelling

were higher correlated with the predictions of their own model than with the model of the

other expert. This was not the case for System Goal Modelling, where the solution patterns

of both experts were higher correlated with the predictions of the structure of Expert 2. We

considered this as an evidence for the model of Expert 2 being the more valid of the two

structures for the System Goal Modelling stream. Predictions from the merged structure did

not yield higher contingency coefficients than the unmerged structures.

4.5.2 Lessons Learned

The results pertaining to reliability are in line with Ley (2006), who found rather high

agreement within (stability), but low agreement between raters for task-competency

matrices in knowledge-intensive work domains. Obviously, the two RESCUE experts were

having different views about what competencies were required for performing certain tasks.

A reason for this may have been that our task descriptions had been rather short so that

some tasks were possibly interpreted differently by the two experts. More comprehensive

documentation of results would have been advantageous.

Despite the rather low inter-rater reliability, the validity coefficients we obtained were

moderate to high. We take this result as a first indication for the structures being robust for

the purpose they were conceived for, namely to diagnose a competence state from task-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

36

based information. Obviously, merging competence-performance structures by uniting each

task‟s minimal interpretation did not enhance validity. Consequently, other ways to improve

validity by integrating competence-performance models should be sought. We will return to

a more thorough discussion of reliability and validity in the overall discussion in section

4.5.

Summarizing from the above, the procedures employed here (inter-rater reliability and

Leave One Out method) would suggest the following: In the case of the Activity Modelling

stream, it would be necessary that the experts come to agree about some of the modelled

tasks. A starting point could be those tasks that have especially low degrees of agreement

and in which performance is poorly predicted by the Leave One Out method. For System

Goal Modelling, we would recommend that the structure of Expert 2 is taken as a basis and

subsequently refined by taking into account inconsistencies obtained from the analyses. For

example, when looking at Figure 5 which shows reliability coefficients for all tasks

modelled in System Goal Modelling, it appears that task 4_1, 4_2 and 4_4 would be

candidates for such refinement, as they obtained especially low scores.

======== INSERT FIGURE 5 HERE =======

5 OVERALL DISCUSSION AND FUTURE WORK

The modelling methodology presented in this paper is based on a formal model of the

relationships between tasks and competencies needed to perform the tasks. Hence, the

methodology follows a theory-based and systematic approach for eliciting expert

knowledge. In doing so, it complies with several demands that have been brought forward

for establishing explicit linkages between the manifest behavioural level (task performance)

and the latent level (competencies needed) in competency modelling (Lievens et al., 2004;

Schmitt and Chan, 1998).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

37

The developed modelling methodology was found to be generally feasible in

organisational settings. Modelling the Domain Model and the User Model of a domain by

starting from the tasks seems to be well suited for a work-integrated approach as it focuses

the competencies very much on the actual tasks that need to be performed. Nonetheless,

several improvements are conceivable in different modelling stages, and some of them have

been identified in the previous sections. At this stage, we have identified five critical issues

pertaining to the application of the methodology which we will now discuss in more detail.

In each section, we will also present ongoing and future work to address the research

challenges.

5.1 Applicability across domains

As stated in the introduction, applicability of our approach in different domains is still

a matter of ongoing research. In this paper, we have gathered evidence about differences in

two subdomains of the Requirements Engineering domain, Activity Modelling and System

Goal Modelling. It turned out that inter-rater reliability was especially low for the first

subdomain. This is also the one that is less formally structured in terms of a clear separation

into tasks. This may have resulted in a poorer understanding between the two experts of

what a particular task entails. For these less structured domains, we feel it is therefore

important to establish higher agreement between the experts. This can be accomplished by

more interactive methods, such as workshops or collaborative modelling environments (see

also next section).

Practical feasibility is another important concern. Work in four other knowledge-

intensive domains is currently being undertaken to check for more generalisable findings.

Furthermore, we are continuing to seek ways for exploiting existing data sources (e.g.

existing document collections or competency catalogues) to reduce manual modelling

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

38

effort. For example, Pammer et al. (2007) show how term extraction from a collection of

documents can support domain modelling.

5.2 Distributed and iterative modelling

When modelling in practical settings, one is faced with a number of challenges that

threaten the applicability of rigorous and systematic procedures prescribed in theory. As a

case in point, shortly after we had performed modelling of the two subdomains, one of the

two experts was assigned to a different project and was no longer available to us. Besides

availability, there are also issues around the distributed nature of modelling teams where it

is often impractical or impossible to gather the team for extended periods (e.g. in a

workshop setting). It is for these pragmatic reasons that our methodology seeks to support

development of valid structures in a distributed setting, to allow for fine-tuning the

applicability of the model to its intended purpose, and for repetitive refinement in those

parts of the models where there are major discrepancies. We have gathered promising

evidence that in spite of having abstained from using more interactive methods, the validity

of the models has been rather high. Also, we have identified techniques to overcome a

model‟s deficits by looking at single tasks with low inter-rater agreement. Information about

which tasks are problematic and need further improvement may be used as an input for

more interactive and costly methods like modelling workshops to make them more

effective.

Work is currently being undertaken to provide modelling tool support for the

methodology we have presented here. In particular, we are currently designing a web-based

modelling and collaboration tool. An extension of a Semantic Media Wiki will allow for

communicating about tasks and competency descriptions and collaborative modelling more

easily (Rospocher et al., 2008).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

39

5.3 Granularity of the models

A critical issue, and one of frequent debate in our case study, was the question of the

right level of granularity for the models. In general, this is not a one-off decision. Clearly, to

answer the question of the right level of granularity the focus should be on the intended use

of the models. In our case, we did well with introducing some heuristics in the modelling

process, such as asking experts to consider the length of the resulting learning episodes.

More formally, the method of the task competency matrix allows one to generate

indicators that check the coverage of the models. For example, an index showing that

several tasks have a very low number of competencies assigned (e.g. 0-2), or a very high

number (e.g. more that 15) may indicate a mismatch in the granularity of the task and

competency models.

While the issue of granularity will always require some sort of trading off in the

process of modelling, there are conceptualizations that more formally describe tasks and

competencies and which give hints for an appropriate level of granularity. For tasks that

describe knowledge intensive activities, one can rely on the Common KADS methodology

(Schreiber et al., 1999) which has suggested a typology of knowledge intensive tasks. In a

similar manner Anderson and Krathwohl (2001) have introduced an extension of Bloom‟s

taxonomy of learning goals which may be taken as the basis for a competency model (this

idea has been briefly sketched in Ley et al., 2008).

5.4 Use of the task competency matrix

The informal feedback we gathered about the use of task competency matrix led us to

consider several alternative strategies. The alternative suggested by research into

Knowledge Space Theory would be to directly model the prerequisite relation by an expert

query procedure. The drawback from this would be that this is usually perceived to be even

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

40

more demanding because it is difficult to decide on the prerequisite relation without having

a curriculum in mind.

A different alternative would be to assign metadata to available learning material

which describes the kinds of learning goals the material teaches without considering the task

demands (e.g. Conlan et al., 2002). Here, the drawback would be that no automated

competency assessment would be possible, and that the learning goals would not be focused

on actual demands (such as is the case when a competency gap analysis is employed), but

rather on available learning material.

One decisive advantage in using task-competency matrices is the possibility to simply

add or remove tasks or competencies from the structures without the need for a complete

revision of the structures. Albert and Kaluscha (1997) have shown how this procedure

affects the underlying structures.

We are currently designing a mapping tool that supports the use of task competency

mappings in the modelling process. It is intended that this tool will also integrate some basic

measures of model coverage, reliability and validity such as those presented here. And,

finally, our plan is also to integrate some graphical representation (e.g. of the prerequisite

structures) into this tool (e.g. Nussbaumer et al., 2008), so that experts can be shown the

resulting structures in the modelling process and make changes as they see fit.11

5.5 Validity and Use of the Models

We have presented an approach and several methods to embed model evaluation into

the modelling process for iterative refinement. The application of these evaluation methods

is dependent on a formal model of the correspondence between the competence and the

11

We thank one of the anonymous reviewers of an earlier draft for pointing this out.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

41

performance level. To the best of our knowledge, this is a unique feature of our

methodology within the field of knowledge engineering for adaptive learning systems.

The question of the validity of a model in general comes back to the question of how

the model will be used. In our view, iterative validation should be checked against and

tailored towards measures that relate to the effectiveness of model use. This is exemplified

by our use of the Leave One Out validation method which simulates use of the models for

task based competency assessment. As this method did make rather robust and valid

judgements, we take this as an indication that the models are “fit for purpose”.

Actual use of our models in work-integrated learning systems follows a “scruffy”

approach (Lindstaedt et al., 2008), meaning that a set of deliberately simple models is

complemented by the use of probabilistic methods during runtime. In section 3.2, we have

indicated such stochastic procedures which could be used to deal with uncertainties in the

user model. Intelligent Tutoring approaches have often employed Bayesian Nets in

modelling uncertainties between evidence and hypotheses about latent knowledge states

(e.g. Conati et al., 2002; Jameson, 1996). These approaches usually employ a more fine-

grained modelling of problem solving processes, and therefore allow for more exact

diagnoses. Also, instead of prerequisites relations, they model causal inferences.

Nevertheless, Desmarais and colleagues have shown how procedures that operate on

Bayesian Net structures (directed acyclic graphs) can be readily applied in the context of

Knowledge Spaces for building item to item structures (Desmarais and Gagnon, 2006), and

for user modelling (Desmarais et al., 2005). A promising direction of research appears to be

a variation of the multiplicative updating rule originally proposed by Falmagne and

Doignon (1988) which does not operate on the Knowledge States, but rather exploits the

prerequisites structures of competencies12

.

12

Thomas Augustin, Personal Communication, 22 April 2009.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

42

Finally, we would like to reiterate the formative character of the reliability and validity

checks. These should be useful for iterative refinement of the models. They are designed to

allow modellers or experts themselves to revisit parts of the models to improve them until

they are valid to a degree that is consistent with the intended purpose of the models. This

approach coincides with more iterative modelling methodologies (Angele et al., 1998) and

with the fact that in our experience models need constant adaptation anyway.

6 CONCLUSIONS AND OUTLOOK

With this work, we presented a psychological theory, the Competence-based

Knowledge Space Theory, and an iterative methodology for modelling and validating the

Domain Model and the User Model of an adaptive learning system. A valid competence-

performance structure integrates both, the Domain and the User Model for a given domain

and hence allows for directly deriving the most appropriate learning opportunity in a theory-

driven manner. Moreover, it incorporates hypotheses (e.g. in order to perform a task, a

certain set of skills is required) which can be directly exposed to different validation

techniques.

The next step for our research will be to devise methods and measures to empirically

evaluate a model once established in a work setting, i.e. to explore its fit to real data instead

of notional solution patterns. This may be achieved with analysing documents produced by

workers or by questioning supervisors about their employees‟ actual performance in the

modelled tasks. We are currently designing evaluation measures and strategies that may be

employed in such a workplace setting when the models are operational.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

43

7 ACKNOWLEDGEMENTS

APOSDLE (www.aposdle.org) is partially funded under the FP 6 of the European

Community within the IST Workprogramme (Project Number 027023). The Know-Center

is funded within the Austrian COMET Program - Competence Centers for Excellent

Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation

and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the

State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG. We

thank the editor and three anonymous reviewers for the constructive remarks on an earlier

draft of this manuscript.

8 REFERENCES

Albert, D., Hockemeyer, C., Wesiak, G., 2002. Current Trends in eLearning based on

Knowledge Space Theory and Cognitive Psychology. Psychologische Beiträge 4 (44), 478-

494.

Albert, D., Kaluscha, R., 1997. Adapting Knowledge Structures in Dynamic Domains,

in: C. Herzog (Ed.), Beiträge zum Achten Arbeitstreffen der GI–Fachgruppe 1.1.5/7.0.1

“Intelligente Lehr–/Lernsysteme”, September 1997, Duisburg, Germany [Contributions of

the 8th Workshop of the GI SIG “Intelligent Tutoring Systems”]. TU München, Munich, pp.

89–100.

Albert, D., Lukas, J., (Eds.), 1999. Knowledge Spaces: Theories, Empirical Research,

and Applications. Lawrence Erlbaum Associates, Mahwah, New Jersey.

Albert, D., Pivec, M., Spörk-Fasching, T., Maurer, H., 2003. Adaptive intercultural

competence testing: A computerized approach based on knowledge space theory, in:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

44

Proceedings of the UNESCO Conference on Intercultural Education, Jyväskylä, Finland.

June 15-18, 2003.

ALEKS Corp., 2003. ALEKS - A Better State of Knowledge. Consulted March 4,

2008. <http://www.aleks.com>.

Anderson, L. W., Krathwohl, D. A., 2001. A taxonomy for learning, teaching, and

assessing: A revision of bloom's taxonomy of educational objectives. Longman, New York.

Angele, J., Fensel, D., Landes, D., Studer, R., 1998. Developing Knowledge-Based

Systems with MIKE. Automated Software Engineering 5 (4), 389-418.

APOSDLE Consortium, 2006. Use Cases and Application Requirements 1 (First

Prototype), Project Deliverable D6.2, Consulted March 4, 2008

http://www.aposdle.org/media/multimedia/files/use_cases_application_requirements_1_first

_prototype

Benyon, D. R., Murray, D. M., 1993. Adaptive systems; from intelligent tutoring to

autonomous agents. Knowledge-Based Systems 6 (4), 197-219.

Billett, S., 2006. Constituting the Workplace Curriculum. Journal of Curriculum

Studies 38 (1), 31-48.

Brewer, M. B., 2000. Research Design and Issues of Validity, in: H. T. Reis, C. M.

Judd (Eds.), Handbook of Research Methods in Social and Personality Psychology.

Cambridge University Press, Cambridge, pp. 3-16.

Brusilovsky, P., Nijhavan, H., 2002. A Framework for Adaptive E-Learning Based on

Distributed Re-usable Learning Activities, in: M. Driscoll, T. C. Reeves (Eds.), Proceedings

of World Conference on E-Learning, E-Learn 2002. AACE, Montreal, pp. 154-161.

Cardinali, F., 2006. Innovating eLearning and Mobile Learning Technologies for

Europe's Future Educational Challenges, Theory and Case Studies, in: Neidl, W.,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

45

Tochtermann, K. (Eds.), Proceedings of 1st European Conference on Technology-enhanced

Learning (LNCS, Vol. 4227), Springer, Heidelberg, pp. 2-7.

Castro-Schez, J. J., Jennings, N. R., Xudong, L., Shadbolt, N. R., 2004. Acquiring

domain knowledge for negotiating agents: a case of study. International Journal of Human-

Computer Studies 61, 3-31.

Chin, D. N., 2001. Empirical Evaluation of User Models and User-Adapted systems.

User Modeling and User-Adapted Interaction 11 (1-2), 181-194.

Conati, C., Gertner, A. S., VanLehn, K., 2002. Using Bayesian Networks to Manage

Uncertainty in Student Modeling. User Modeling and User-Adapted Interaction 12 (4), 371-

417.

Conlan, O., Hockemeyer, C., Wade, V., Albert, D., 2002. Metadata driven approaches

to facilitate adaptivity in personalized eLearning systems. The Journal of Information and

Systems in Education 1, 38-44.

Cooke, N. J., 1994. Varieties of knowledge elicitation techniques. International

Journal of Human-Computer Studies 41, 801-849.

De Bra, P., Aroyo, L., Chepegin, V., 2004. The next big thing: Adaptive web-based

systems. Journal of Digital Information 5 (1).

Desmarais, M., Fu, S., Pu, X., 2005. Tradeoff analysis between knowledge assessment

approaches, in: Looi, C., McCalla, G. I., Bredeweg, B., Breuker, J. (Eds.), Artificial

Intelligence in Education: Supporting Learning through Intelligent and Socially Informed

Technology, IOS Press, Amsterdam, pp. 209-216.

Desmarais, M., Gagnon, M., 2006. Bayesian Student Models Based on Item to Item

Knowledge Structures, in: Neidl, W., Tochtermann, K. (Eds.), Proceedings of 1st European

Conference on Technology-enhanced Learning (LNCS, Vol. 4227), Springer, Heidelberg,

pp. 111-124.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

46

Doignon, J., 1994. Knowledge Spaces and Skill Assignments, in: Fischer, D. H.,

Laming, D. (Eds.), Contributions to Mathematical Psychology, Psychometrics and

Methodology. Springer, New York, pp. 111-121.

Doignon, J., Falmagne, J., 1985. Spaces for the assessment of knowledge.

International Journal of Man-Machine Studies 23, 175-196.

Doignon, J., Falmagne, J., 1999. Knowledge Spaces. Springer, Heidelberg.

Dowling, C., 1994. Integrating different knowledge spaces, in: Fischer, G. H.,

Laming, D. (Eds.), Contributions to mathematical psychology. Springer, New York, pp.

149-158.

Düntsch, I., Gediga, G., 1995. Skills and knowledge structures. British Journal of

Mathematical and Statistical Psychology 48, 9-27.

Eklund, J., Brusilovsky, P., 1999. InterBook: An Adaptive tutoring System. Uniserve

Science News 12 (3), 8-13.

Eraut, M., 2004. Informal learning in the workplace. Studies in Continuing Education

26 (2), 247-273.

Falmagne, J., Cosyn, E., Doignon, J., Thiéry, N., 2006. The Assessment of

Knowledge, in Theory and in Practice, in: Missaoui, R., Schmid, J. (Eds.), Formal Concept

Analysis, Springer, Heidelberg, pp. 61-79.

Falmagne, J.-C., Doignon, J.-P., 1988. A class of Stochastic Procedures for the

Assessment of Knowledge. British Journal of Mathematical and Statistical Psychology 41,

1-23.

Falmagne, J., Doignon, J., Koppen, M., Villano, M., Johannesen, L., 1990.

Introduction to Knowledge Spaces: How to Build, Test, and Search Them. Psychological

Review 97 (2), 201-224.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

47

Farmer, J., Lindstaedt, S. N., Droschl, G., Luttenberger, P., 2004. AD-HOC - Work-

integrated Technology-supported Teaching and Learning, in: Proceedings of the 5th

European Conference on Organisational Knowledge, Learning and Capabilities, Innsbruck,

Austria, April 2-3, 2005, Innsbruck. Consulted October 12, 2009.

http://www2.warwick.ac.uk/fac/soc/wbs/conf/olkc/archive/oklc5/

Fischer, G., 2001. User Modeling in Human-Computer Interaction. User Modeling

and User-Adapted Interaction 11 (1-2), 65-86.

Gaines, B. R., Shaw, M. L. G., 1993. Knowledge Acquisition Tools Based on Personal

Construct Psychology. Knowledge Engineering Review 8 (1), 49-85.

Garavan, T. N., Morley, M., Gunnigle, P., McGuire, D., 2002. Human resource

development and workplace learning: emerging theoretical perspectives and organisational

practices. Journal of European Industrial Training 26 (2, 3, 4), 60-71.

Hockemeyer, C., Albert, D., 1999. The Adaptive Tutoring System RATH, in: Auer,

M., Ressler, U. (Eds.), ICL99 Workshop Interactive Computer aided Learning: Tools and

Applications, Technikum Kärnten, Villach/Austria.

Jameson, A., 1996. Numerical Uncertainty Management in User and Student

Modeling: An Overview of Systems and Issues. User Modeling and User-Adapted

Interaction 5 (4), 193-251.

Jameson, A., 2003. Adaptive interfaces and agents, in: Jacko, J.A., Sears, A. (Eds.),

Human-computer interaction handbook. Erlbaum, Mahwah, NJ, pp. 305-330.

Kline, T., 2005. Psychological testing. A practical approach to design and evaluation.

Sage, Thousand Oaks.

Korossy, K., 1997. Extending the theory of knowledge spaces: A competence-

performance approach. Zeitschrift für Psychologie 205 (1), 53-82.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

48

Korossy, K., 1999. Modeling Knowledge as Competence and Performance, in: Albert,

D., Lukas, J. (Eds.), Knowledge Spaces: Theories, Empirical Research, and Applications.

Lawrence Erlbaum Associates, Mahwah, New Jersey, pp. 103-132.

Korossy, K., Held, T., 2001. Theory-based knowledge modeling in a subdomain of

elementary algebra. Zeitschrift für Psychologie 209 (3), 277-315.

Ley, T., 2006. Organizational Competency Management - A Competence

Performance Approach. Shaker, Aachen.

Ley, T., Albert, D., 2003a. Kompetenzmanagement als formalisierbare Abbildung von

Wissen und Handeln für das Personalwesen. Wirtschaftspsychologie 5 (3), 86-93.

Ley, T., Albert, D., 2003b. Identifying Employee Competencies in Dynamic Work

Domains: Methodological Considerations and a Case Study. Journal of Universal Computer

Science 9 (12), 1500-1518.

Ley, T., Albert, D., Lindstaedt, S. N., 2006. Competency Management using the

Competence Performance Approach: Modeling, Assessment, Validation and Use, in: Sicilia,

M. A. (Ed.), Competencies in Organizational E-Learning: Concepts and Tools. Idea Group,

Hersey, pp. 83-119.

Ley, T., Kump, B., Ulbrich, A., Scheir, P., Lindstaedt, S. N., 2008. A Competence-

based Approach for Formalizing Learning Goals in Work-integrated Learning, in:

Proceedings of Ed Media 2008 World Conference on Educational Multimedia, Hypermedia

& Telecommunications, Vienna, Austria, June 30-July 4, AACE, Chesapeake, VA, pp.

2099-2108.

Lievens, F., Sanchez, J. I., De Corte, W., 2004. Easing the Inferential Leap in

Competency Modeling: The effects of Task-related Information and Subject Matter

Expertise. Personnel Psychology 57, 881-904.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

49

Lindstaedt, S. N., Ley, T., Scheir, P., Ulbrich, A., 2008. Applying „Scruffy‟ Methods

to Enable Work-integrated Learning. Upgrade: The European Journal of the Informatics

Professional 9 (3), 44-50.

Lindstaedt, S. N., Mayer, H., 2006. A Storyboard of the APOSDLE Vision, in: Neidl,

W., Tochtermann, K. (Eds.), Proceedings of 1st European Conference on Technology-

enhanced Learning (LNCS, Vol. 4227), Springer, Heidelberg, pp. 628-633.

Maiden, N. A., Jones, S. V., 2004. The RESCUE Requirements Engineering Process -

An Integrated User-centered Requirements Engineering Process, Version 4.1. Centre for

HCI Design, The City University, London/UK. Consulted March 4, 2008

<http://hcid.soi.city.ac.uk/research/Rescuedocs/RESCUE_Process_Doc_v4_1.pdf>.

Mayring, P., 2003. Qualitative Inhaltsanalyse. Grundlagen und Techniken. Beltz,

Weinheim.

Millán, E., Pérez-De-La-Cruz, J., 2002. A Bayesian Diagnostic Algorithm for Student

Modeling and its Evaluation. User Modeling and User-Adapted Interaction 12 (2-3), 281-

330.

National O*NET Consortium, 2005. O*NET OnLine. Consulted March 4, 2008.

<http://online.onetcenter.org>.

Nussbaumer, A., Steiner, C., Albert, D., 2008. Visualisation Tools for Supporting

Self-Regulated Learning through Exploiting Competence Structures, in: Tochtermann, K.,

Maurer, H. (Eds.), Proceedings of I-Know 2008 - International Conference on Knowledge

Management, 3-5 September 2008, Know-Center, Graz/Austria, pp. 288-296.

Pammer, V., Scheir, P., Lindstaedt, S., 2007. Two Protégé plug-ins for supporting

document-based ontology engineering and ontological annotation at document-level, Paper

presented at the 10th International Protégé Conference, Budapest, Hungary, July 15-18,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

50

2007, Consulted April 24, 2009

http://protege.stanford.edu/conference/2007/presentations/05.01_Pammer.pdf

Pilgerstorfer, M., Albert, D., Camhy, D. G., 2006. Considerations on Personalized

Training in Philosophy for Children, in: D. G. Camhy, R. Born (Eds.), Encouraging

Philosophical Thinking: Proceedings of the International Conference on Philosophy for

Children in Graz, Austria. Academia Verlag, pp. 85-89.

Rich, E., 1999. Users are individuals: individualizing user models. International

Journal of Human-Computer Studies 51 (2), 323-338.

Rospocher, M., Ghidini, C., Serafini, L., Kump, B., Pammer, V., Lindstaedt, S., Faatz,

A., Ley, T., 2008. Collaborative enterprise integrated modelling, in: Gangemi, A., Keizer,

J., Presutti, V., Stoermer, H. (Eds.), 5th International Workshop on Semantic Web

Applications and Perspectives (SWAP 2008), December 15th to 17th, 2008, Rome, Italy,

Consulted April 24, 2009 http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-

WS/Vol-426/

Schmidt, A., 2004. Context-steered learning: The learning in process approach, in:

Proceeding of the IEEE Conference on Advanced Learning Technologies (ICALT 04) (pp.

684 - 686). Washington, DC: IEEE Computer Society.

Schmitt, N., Chan, D., 1998. Personnel Selection. Sage, Thousand Oaks.

Schreiber, G., Akkermans, H., Anjewierden, A., Hoog, R. d., Shadbolt, N., Velde, W.

V. d., Wielinga, B. J., 1999. Knowledge Engineering and Management: The

CommonKADS Methodology. MIT Press, Cambridge, Mass.

Shadbolt, N., Burton, M. A., 1989. The Empirical Study of Knowledge Elicitation

Techniques. SIGART Newsletter 108, 15-18.

Sicilia, M., (Ed.), 2007. Competencies in Organizational E-Learning: Concepts and

Tools. Information Science Publishing, Hershey.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

51

Sicilia, M., 2005. Ontology-Based Competency Management: Infrastructures for the

Knowledge-intensive Learning Organization, in: M. Lytras, A. Naeve (Eds.), Intelligent

Learning Infrastructures in Knowledge Intensive Organizations: A Semantic Web

perspective. Idea Group, Hershey, pp. 302-324.

Sohn, Y. W., Doane, 2002. Evaluating Comprehension-Based User Models:

Predicting Individual User Planning and Action. User Modeling and User-Adapted

Interaction 12 (2-3), 171-205.

Stefanutti, L., Albert, D., 2002. Efficient assessment of organizational action based on

knowledge space theory, in: Tochtermann, K., Maurer, H. (Eds.), Proceedings of I-KNOW

'02, 2nd International Conference on Knowledge Management, Know-Center, Graz, pp.

183-190.

Studer, R., Benjamins, V. R., Fensel, D., 1998. Knowledge Engineering: Principles

and methods. Data & Knowledge Engineering 25, 161-197.

Sure, Y., Gómez-Pérez, A., Daelemans, W., Reinberger, M., Guarino, N., Noy, N. F.,

2004. Why Evaluate Ontology Technologies? Because It Works! In: IEEE Intelligent

Systems, 19 (4), pp. 74-81.

Uschold, M., King, M., House, R., Moralee, S., Zorgios, Y., 1998. The Enterprise

Ontology. The Knowledge Engineering Review 13, 31-89.

van den Berg, P. T., 1998. Competencies for Work Domains in Business Computer

Science. European Journal of Work and Organisational Psychology 7 (4), 517-531.

Villano, M., 1992. Probabilistic Student Models: Bayesian Belief Networks and

Knowledge Space Theory, in: Frasson, C., Gauthier, G., McCalla, G. I. (Eds.), Intelligent

Tutoring Systems (LNCS, Vol.608). Springer, Heidelberg, pp. 491-498.

Witten, I. H., Frank, E., 2005, Data mining: practical machine learning tools and

techniques, Elsevier, Morgan Kaufman, Amsterdam.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

52

Appendix A

Reviewed List of Tasks for Activity Modelling and System Goal Modelling

ACTIVITY MODELLING

Gather Data on Human Activity

1_1 Plan and prepare acquisition sessions, decide on acquisition methods

Analyse the functional structure on the work domain

1_2 Analyse the scope/validity domain of the new tool

1_3 Identify properties of the domain and the environment that constrain current and

future performance

1_4 Identify properties of the system that constrain performance

1_5 Identify degrees of freedom allowing for that variability that makes the system

flexible

1_6 Identify the system‟s goals and high level functional goals

1_7 Identify external resources that are available for practitioners to achieve their goals

Identify tasks/goals to be achieved (independently of how they are to be achieved or by

whom they are to be achieved)

1_8 Identify the workers‟ prescribed goals as defined by norms and regulations

1_9 Identify the prescribed tasks to be executed to achieve the prescribed goals

Identify strategies how tasks can be achieved independently of who is executing them

1_10 Identify non-prescribed goals set up by the practitioners to achieve prescribed goals

1_11 Identify non-prescribed tasks that operators put in place to achieve goals while

coping with internal and external constraints

1_12 Identify different strategies of the workers to acquire and process information

1_13 Identify information needs for the new tool

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

53

1_14 Identify inefficiencies of the existing system experienced by the users

1_15 Identify resources available to achieve goals

1_16 Analyse the different action sequences envisaged in certain situations

1_17 Identify and record inter- and intra- subjective variability to highlight work

processes

1_18 Analyse contextual features that are sources of variability

1_19 Analyse the relevance of an aiding tool in relation to certain actions

Identify social organisation and co-operation between actors (both human and

artificial)

1_20 Identify prescribed goals that are to be achieved by a team of workers

1_21 Identify prescribed tasks that are to be achieved by a team of workers

1_22 Identify different forms of communication and the exchange of information among

workers

1_23 Analyse the mutual knowledge of teams or team members

Identify worker competencies such as knowledge, rules or skills that workers need to

effectively accomplish their tasks

1_24 Identify semantic knowledge the workers need for effectively accomplishing the task

Model Human Activity

2_1 Identify the major types of activity in the current system the automation will affect

2_2 Model activities in a functional matter with respect to a purpose they serve

2_3 Break down actions in the normal course into their physical, cognitive and

communicative components

2_4 Structure the particular activities into activity descriptions by using the Activity

Modelling templates

2_5 Review the Activity Model

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

54

SYSTEM GOAL MODELLING

Determine System Boundaries

3_1 Use the findings of the Activity Model to identify system boundaries

3_2 Build a first cut Context Model to identify system boundaries

3_3 Identify a basic list of stakeholders

3_4 Carry out an initial stakeholder analysis to determine the major categories of system

stakeholder

3_5 Consider for each stakeholder whether he corresponds to an adjacent actor

3_6 Develop an extended Context Model to refine the first cut model of system

boundaries

Determine system dependencies, goals and rationale

4_1 Allocate functions between actors according to boundaries

4_2 Model the system‟s hard and soft goals

4_3 Interpret the Activity Model and integrate the identified actors and goals into the

SD-Model

4_4 Identify the intentional strategic actors

4_5 Model dependencies between strategic actors for goals to be achieved and tasks to be

performed

4_6 Model dependencies between strategic actors for availability or entity of resources

4_7 Write different forms of dependency descriptions

Refine system dependencies, goals and rationale

5_1 Refine the Strategic Dependency Model

5_2 Refine the Strategic Rationale Models

5_3 Produce a single, integrated SR model using dependencies in the SD model

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

55

5_4 Check that each individual SD model is complete and correct with stakeholder goals,

soft-goals, tasks and resources

5_5 Validate the i* SR model against the SD model by cross-checking the two models

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

56

Appendix B

Reviewed Competence Catalogue. Knowledge and Skills

KNOWLEDGE

1. Knowledge about acquisition techniques available

Knowledge about strengths and weaknesses of every technique

Knowledge about limits on use of each method

Knowledge about minimum conditions for method use

Knowledge about method interdependencies

2. Knowledge about how to organise an acquisition programme

Knowledge about guidelines for selecting from a broad range of different methods

Knowledge about practical constraints on each session

3. Knowledge about the Activity Model and the activity descriptions

Knowledge about the content of activity descriptions

Knowledge about the properties and the content of an Activity-Model

4. Knowledge about actors, tasks, goals and resources

Knowledge about what is an actor, what is an ‟intentional strategic actor‟

Knowledge about different kinds of goals (prescribed, non prescribed, local, soft-g., …)

Knowledge about what is a task

Knowledge about resources (means available to achieve goals and sub-goals)

Knowledge about the connection between actors, tasks and goals

Knowledge about the differentiation between these concepts

5. Knowledge about possible physical/environmental factors influencing human

activity

Knowledge about preconditions of actions,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

57

Knowledge about contextual features (distinctive features of the working context)

Knowledge about constraints (properties of the environment that need to be taken into

account when deciding about an action)

Knowledge about the differentiation between these concepts

6. Basic understanding of human cognition, background in cognitive psychology

Knowledge about different types of knowledge (tacit, semi-tacit, non-tacit)

Knowledge about properties of non-tacit, semi-tacit and tacit knowledge

Basic understanding of selection and evaluation of information

Knowledge about individual and collective information processing strategies

Knowledge about individual and collective problem solving strategies

7. Basic understanding of social and organisational behaviour

Basic knowledge of social psychology or sociology

Knowledge about the appreciation of social networks

Knowledge about different forms of team cooperation

8. Knowledge about the domain and the environment of the system

Domain knowledge

Knowledge of the environment

9. Knowledge of different types of system stakeholders

Knowledge about the responsibilities and roles of different types of system stakeholders

(USTM)

10. Basic technical understanding

Knowledge about automation available

Knowledge about possible technical solutions

Knowledge about technical possibilities

11. Knowledge about different types of adjacent systems

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

58

Knowledge about the taxonomy of adjacent system types to consider

Knowledge about the properties of different types of adjacent systems

12. Knowledge about the Context Model

Ability to read and understand a Context Model

Knowledge about the properties and the content of a Context model

Knowledge about guidelines to develop an extended Context model

13. Knowledge about the Strategic Dependency Model (SD-Model)

Ability to read and understand an SD model

Knowledge about the properties and the content of an SD-Model

Knowledge of different types of the dependency link

Knowledge about guidelines for the different possibilities of wording for links between

elements of the SD-Model

14. Removed: Knowledge about the different possibilities of notation of the

elements of the SD-Model

15. Knowledge about the Strategic Rationale Model (SR-Model)

Ability to read and understand an SR model

Knowledge about the properties and the content of an SR-Model

Knowledge about different types of the dependency links

Knowledge about different types of the task decomposition link

Knowledge about different types of the means end link

Knowledge about different types of the contribute to soft-goal link

Knowledge about what dependencies between what concepts are allowed

16. Knowledge of validating the SR Model

Knowledge of guidelines and techniques to cross-check the SD- and SR-Model

17. Knowledge of brainstorming techniques

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

59

18. Knowledge of creativity techniques

19. Ability to produce a Context Model

20. Ability to produce an i* Model

SKILLS

21. Standard project management skills

Ability of creating and maintaining an environment that guides a project to its successful

completion

Understanding the procedures and methods that define a project while confronting and

overcoming the problems encountered over the project lifespan.

22. Listening Skills

Ability to carefully listen in a way that you don‟t prejudge information

Ability to clearly understand spoken, partly expressed, and unspoken messages from others

(workers, stakeholders)

Understand the message being conveyed and identify the key ideas the speaker is sending

by paying close attention to what is being communicated both verbally and non-verbally

Ability of giving full attention to what other people are saying, taking time to understand

the points being made, asking questions as appropriate, and not interrupting at inappropriate

times

23. Communication skills

Ability of sending clear and convincing messages

Ability to clearly articulate your ideas

24. Collaboration skills and Empathy

Ability to balance a focus on tasks with attention to relationships

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

60

Ability of sensing others‟ feelings and perspective and taking an active interest in their

concerns

25. Writing skills

Ability to write well

Ability of non-biased writing

Ability to relay a message or represent one‟s thoughts in written format with clarity and

accuracy

26. Learning skills

Ability to acquire and integrate new information by gathering data

Ability to understand the implications of new information for both current and future

problem solving and decision making

27. On-line data gathering skills

Skills of collecting data while practitioners are working with the system

Ability to make non-biased observations

Knowledge of on-line data gathering techniques such as Protocol Analysis, Ethnography, …

28. Off-line data-gathering skills

Skills of eliciting information from practitioners while they are not involved in any activity

Structured and unstructured interviewing skills, non-prejudging interviewing, asking the

right questions

Knowledge of direct elicitation techniques, off-line data gathering techniques such as Card

sorting, Laddering, …

29. Judgement and decision making

Ability to consider the relative costs and benefits of potential actions to choose the most

appropriate one

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

61

Ability of using logic and reasoning to identify the strengths and weaknesses of alternative

solutions, conclusions or approaches to problems

Ability to make effective decisions quickly, based on a careful and balanced consideration

of all available facts

Ability to identify complex problems and review related information to develop and

evaluate options and implement solutions

30. Ability of thinking in processes and analysing processes

Ability to regard a particular course of action as intended to achieve a goal or result

Ability of analysing actors and concepts with regard to their mutual dependencies

31. Ability of structuring data and organising information

Ability to arrange things or actions in a certain order or pattern according to a specific rule

or set of rules

Ability to organise the material you collect

32. Ability of functional decomposition

Ability to break down concepts (goals, tasks, …) into sub-elements

Ability to decompose processes into sub-elements

33. Analytical skills

Ability to provide a logical, in-depth analysis of a problem or situation

Ability to decompose and structure the problem space

Ability to examine the situation in a systematic way

Ability of filtering out the important information

Ability to extract the most important thing of what somebody says

Ability to select and understand the most important facets of some peace of work

34. Ability of abstraction

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

62

Ability to abstract from literal objects or instances and to extract concepts such as actors,

goals, etc. from them

Ability to understand a general rule of examples people are telling you

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

63

Appendix C

Task-competency matrix of Expert 1 (Extract)

======== INSERT FIGURE 6 HERE ========

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

64

Figure Captions

Figure 1: Hasse-diagram of the prerequisite relation on the set of elementary competencies.

(Example). Edges in the diagram indicate transitive prerequisite relationships among

competencies, with lower vertices being prerequisites for upper vertices.

Figure 2: Hasse-diagram of competence space and performance representation (Example).

Edges indicate transitive sub-set relationships among competence states and performance

states respectively.

Figure 3: Overview of the methodology

Figure 4: Hasse-diagram of the prerequisite relation on the set of elementary competencies

(merged competence-performance structure for System Goal Modelling) separated into

knowledge and skills. Numbers refer to competencies given in Appendix B. Edges indicate

transitive prerequisite relationships among competencies, with lower vertices being

prerequisites for upper vertices.

Figure 5: Inter-rater reliability for competency assignments in 18 System Goal Modelling

tasks

Figure 6: Task-competency matrix of Expert 1 (Extract)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

65

Author Biographies

Tobias Ley is a senior researcher and deputy division manager at the Know Center Graz,

and a university assistant at the Cognitive Science Section at the University of Graz. Since

2001, he has been leading industry-based research projects in the areas of knowledge

management, technology enhanced learning and competency management. Tobias holds a

PhD in Cognitive Psychology from the University of Graz in Austria, and an MSc in

Psychology from Darmstadt University of Technology in Germany. During his studies, he

spent a semester at Melbourne University in Australia and a year at the Krannert Graduate

School of Management at Purdue University as a Fulbright scholar.

Barbara Kump is a research assistant at the Knowledge Management Institute at Graz

University of Technology. In 2006, she graduated with an MSc in Psychology from the

University of Graz. She is currently completing her PhD where she applies principles of

cognitive psychology to user modeling in technology enhanced learning.

Dietrich Albert graduated from the University of Göttingen, in 1966 with a Diploma (MS)

in psychology. In 1972 he received his Dr.rer.nat. (D.Sc.) and in 1975 his postdoctoral

degree (Habilitation in Psychology) by the University of Marburg/Lahn. He was Professor

of general experimental psychology at the University of Heidelberg from 1976 to 1993.

Since 1993, he is professor and head of the Cognitive Science Section at the University of

Graz. He is member of several scientific societies (e.g, EARLI, APC of AACE, and the

SMP) and advisory boards (e.g. Leibniz-Institute for Psychology Information (ZPID);

Know-Center; Academy of New Media and Knowledge Transfer).

Table 1

Set of tasks and set of elementary competencies (example)

Tasks and Elementary Competencies

Tasks

3_1 Use the findings of the Activity Model (AM) to identify system boundaries

4_2 Model the system's hard and soft goals

4_3 Interpret the AM and integrate the identified actors and goals into the Strategic

Dependency (SD) Model

4_5 Model dependencies between strategic actors for goals to be achieved and tasks to

be performed

4_6 Model dependencies between strategic actors for availability of resources

5_1 Refine the Strategic Dependency Model

5_2 Refine the Strategic Rationale (SR) Models

5_3 Produce an integrated SR Model using dependencies in the SD Model

5_4 Check that each individual SD Model is complete and correct with stakeholder

goals, soft goals, tasks and resources

5_5 Validate the i* SR Model against the SD Model (cross-check)

Competencies

3 Knowledge about the Activity Model and the activity descriptions

12 Knowledge about the Context Model

13 Knowledge about the Strategic Dependency Model (SD-Model)

15 Knowledge about the Strategic Rationale Model (SR-Model)

16 Knowledge of validating the SR Model

20 Ability to produce an i* Model

Table 1

Table 2

Task-competency assignment, minimal interpretation and base states (example)

Task-Competency Assignment

Competencies Minimal Interpretations

Tasks 3 12 13 15 16 20 and Base States

3_1 X X X 3, 12, 13*

4_2 X X 15, 20*

4_3 X X X 3, 13, 20*

4_5 X X 13, 20*

4_6 X X 13, 20

5_1 X 13*

5_2 X 15*

5_3 X X X 13, 15, 20

5_4 X X X 13, 15, 16*

5_5 X X X 13, 15, 16

* Base State

Table 2

Table 3

Overview of the Steps in the Methodology: Purposes, Methods and Responsibilities

Defining the Scope and Purpose of the Model

Purpose of this Step:

Generating requirements on the resulting models

Scoping of the domain

Suggested Methods and Activities:

Requirements analysis techniques, e.g. Use Case methods, stakeholder analysis

Roles and Responsibilitiesa:

Facilitating Process (RE), Providing Input (St)

(2) Eliciting Knowledge

Purpose of this Step:

Generating lists and descriptions of tasks and competencies

Suggested Methods and Activities:

Gather experts and documents with performance and learning focus, existing task

and competency catalogues

Perform content analysis techniques

Generate personas and concrete instances of task performance with experts

Complement Knowledge Elicitation with formal techniques, like Repertory Grid

or Critical Incidents Technique

Validate lists with experts informally in structured interviews

Roles and Responsibilitiesa:

Performing Analysis (KE), Providing Input and Validation (DEP)

Performing Analysis (KE), Providing Input and Validation (DEL)

Table 3

Linking Tasks and Competencies

Purpose of this Step:

Generating the interpretation function, i.e. sets of competencies needed to

perform in tasks

Suggested Methods and Activities:

Gather expert ratings using the task-competency matrix

Roles and Responsibilitiesa:

Facilitating Process (KE), Providing Input (DEP, DEL)

Building Competence-Performance Structures

Purpose of this Step:

Generating the Knowledge Base from elicited knowledge

Suggested Methods and Activities:

Employ Computational Procedures from Knowledge Space Theory, i.e. deriving

base, competence space, and prerequisite relation on competencies

Roles and Responsibilitiesa:

Performing Analysis (KE)

Evaluating Competence-Performance Structures

Purpose of this Step:

Evaluating the Knowledge Base in terms of reliability and validity

Suggesting ways to improve the Knowledge Base

Suggested Methods and Activities:

Check reliability using inter-rater or test-retest consistency measures

Check for validity using actual or notional solution patterns or assessments (e.g.

by employing Leave One Out method)

Generate suggestions for revisiting the Knowledge Base where reliability or

validity is low

Roles and Responsibilitiesa:

Performing Analysis (KE), Validating and Providing Input for Model

Adaptations (DEP, DEL)

a Roles: Stakeholders (St), Requirements Engineer (RE), Knowledge Engineer (KE), Domain

Experts with Performance Focus (DEP), Domain Experts with Learning and Teaching Focus

(DEL)

Table 4

Task-competency assignments of the two experts for the two RESCUE streams (“1”

competency is somewhat required, “2” competency is definitely required)

Task Competency Assignments

Total number of

cells

Number of “1”

Assignments

Number of “2”

Assignments

Expert 1

Activity Modelling

System Goal Modelling

Total

Expert 2

Activity Modelling

System Goal Modelling

Total

957

594

1551

957

594

1551

316 (33.0%)

111 (18.7%)

427 (27.5%)

48 (5.0%)

0 (0%)

48 (3.1%)

128 (13.4%)

63 (10.6%)

191 (12.3%)

195 (20.4%)

80 (13.5%)

275 (17.7%)

Table 4

Table 5

Number of elementary competencies (NComp), cardinality of power set (Power Set),

cardinality of bases (Base) and cardinality of competence spaces (C. Space) obtained by

task-competency assignments

Competence Spaces

Assignment NComp Power Set a Base C. Space

Activity Modelling

Expert 1

Expert 2

System Goal Modelling

Expert 1

Expert 2

Merged Structure

22

21

19

15

22

4 194 304

2 097 152

524 288

32 768

4 194 304

25

23

15

12

14

3 338

849

756

166

420

a Cardinality of the power set was calculated as 2

n where n is the number of competencies

Table 5

Table 6

Rank correlation between predicted performance and assessed performance of personas,

Leave One Out method

Notional Solution Patterns

(Assessment) a

Competence-Performance (CP) Structure

(Prediction) b

Expert 1 Expert 2

Activity Modelling c

CP Structure of Expert 1

CP Structure of Expert 2

System Goal Modelling

CP Structure of Expert 1 c

CP Structure of Expert 2

Merged CP Structure

.33**

.26**

.54**

.76**

.54**

.45**

.77**

.32**

.42**

.41**

a Notional solution patterns obtained by the validation questionnaire Appraisal of Personas

b Competence-performance structure used for the prediction of task performance by means of

Leave One Out

c Only four solution patterns were considered

** p<.01

Table 6

1315

20

312

16

... E

lem

enta

ry C

ompe

tenc

y

... P

rere

quis

ite R

elat

ions

hip

(t

rans

itive

)

Figure 1

3 1

2 1

3 1

5 1

6 2

0

.

.

.

.

.

.

.

.

.

. .

.

. .

.

3_1

4_2

4_3

4_5

4_6

5_1

5_2

5_3

5_4

5_5

3_1

4_2

4_3

4_5

4_6

5_1

5_2

5_3

.

.

3 1

2 1

3 1

5 1

6 .

3_1

. .

. .

5_1

5_2

. 5

_4 5

_5.

4_2

4_3

4_5

4_6

5_1

5_2

5_3

5_4

5_

5

3 .

13

15

16

20

3_1

. 4

_3 4

_5 4

_6 5

_1 .

. .

.

3 1

2 1

3 .

.

20

3 1

2 1

3 1

5 .

.

3_1

. .

. .

5_1

5_2

. .

..

4_2

4_3

4_5

4_6

5_1

5_2

5_3

.

.

3 .

13

15

.

20

. 4_

2 .

4_5

4_6

5_1

5_2

5_3

5_4

5_

5

. .

13

15

16

20

3 1

2 1

3 1

5 .

20

. .

4_3

4_5

4_6

5_1

. .

. .

3 .

13

.

. 2

03

12

13

.

. .

3_1

. .

. .

5_1

. .

. .

. .

. .

. 5_

1 5

_2 .

5_4

5_

5

. .

13

15

16

.

. 4_

2 .

4_5

4_6

5_1

5_2

5_3

.

.

. .

13

15

.

20

. .

. 4_

5 4

_6 5

_1 .

. .

.

. .

13

.

. 2

0

. .

. .

. 5_

1 5

_2 .

.

.

. .

13

15

.

.

. 4_

2 .

. .

. 5

_2 .

.

.

. .

.

15

. 2

0

. .

. .

. 5_

1 .

. .

.

. .

13

.

. .

. .

. .

. .

5_2

. .

.

. .

.

15

. .

... C

ompe

tenc

e S

tate

... P

erfo

rman

ce S

tate

... P

rere

quis

ite R

elat

ions

hip

(t

rans

itive

)

Figure 2

Defining the Scope and Purpose of the Model

Linking Tasks and Competencies

Building Competence-Performance Structures

Eliciting Knowledge

Eliciting Tasks Eliciting Competencies

Evaluating Competence-Performance Structures

Checking Reliability Checking Validity

Figure 3

4 15 20

3

12

16

... Elementary Competency

... Prerequisite Relationship (transitive)

139

1011

8

19

7

34 25

32

29

33

31

30

22, 23

Elementary Competencies 1 to 20: Knowledge

Elementary Competencies 21 to 34: Skills

Figure 4

-0.300

-0.200

-0.100

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

4_1

4_4

4_2

3_1

5_3

5_5

3_2

3_6

3_4

4_7

5_2

3_5

3_3

5_4

4_3

5_1

4_5

4_6

System Goal Modelling Tasks

Inte

r-ra

ter R

elia

bilit

y

Figure 5

1.Knowledge about acquisition techniques available

2.Knowledge about how to organise an acquisition programme

3.Knowledge about the Activity Model and the activity descriptions

4.Knowledge about actors, tasks, goals and resources

5.Knowledge about possible physical factors influencing human activity

6.Basic understanding of human cognition, background in cogn. psychology

7.Basic understanding of social and organisational behaviour

8.Knowledge about the domain and the environment of the system

9.Knowledge of different types of system stakeholders

10. Basic technical understanding

11. Knowledge about different types of adjacent systems

12. Knowledge about the Context Model

13. Knowledge about the Strategic Dependency Model (SD-Model)

15. Knowledge about the Strategic Rationale Model (SR-Model)

16. Knowledge of validating the SR Model

17. Knowledge of brainstorming techniques

18. Knowledge of creativity techniques

19. Ability to apply the data flow diagram notation (DFD)

20. Ability to apply the i* notation

21. Standard project management skills

22. Listening Skills

23. Communication skills

24. Collaboration skills and Empathy

25. Writing skills

26. Learning skills

27. On-line data gathering skills

28. Off-line data-gathering skills

29. Judgement and decision making

30. Ability of thinking in processes and analysing processes

31. Ability of structuring data and organising information

32. Ability of functional decomposition

33. Analytical skills

34. Ability of abstraction

1_1P

lan and prepare acquisition sessions, decide on acquisition methods

22

11

11

11

11

11_2

Analyse the scope/validity dom

ain of the new tool

11

11

21

11

12

11_3

Identify properties of the domain and the environm

ent that constrain current and future performance

12

21

21

12

11

22

12

1_4Identify properties of the system

that constrain performance

12

21

21

12

11

22

12

1_5Identify degrees of freedom

allowing for that variability that m

akes the system flexible

11

11

21

12

11

12

11

11_6

Identify the system’s goals and high level functional goals

11

11

11

11

21

22

22

1_7Identify external resources that are available for practitioners to achieve their goals

12

11

11

11

11

11

11

12

21_8

Identify the workers’ prescribed goals as defined by norm

s and regulations1

11

11

12

11

12

12

11

11_9

Identify the prescribed tasks to be executed to achieve the prescribed goals1

11

11

11

21

11

21

21

11

1_10Identify non-prescribed goals set up by the practitioners to achieve prescribed goals

11

12

12

11

21

11

21

22

11

1_11Identify non-prescribed tasks that operators put in place to achieve goals w

hile coping with internal and external constraints

11

12

12

11

21

11

21

22

11

1_12Identify different strategies of the w

orkers to acquire and process information

11

11

11

11

11

11

12

21

11

1_13Identify inform

ation needs for the new tool

11

11

11

11

11

11

11

11

21

1_14Identify inefficiencies of the existing system

experienced by the users2

11

11

12

21

11

11

11

11_15

Identify resources available to achieve goals 1

11

11

21

11

11

11

11

11

1_16A

nalyse the different action sequences envisaged in certain situations 1

11

11

11

11

21

21

1_17Identify and record inter- and intra- subjective variability to highlight w

ork processes2

21

12

22

22

12

12

12

12

11_18

Analyse contextual features that are sources of variability

22

11

22

22

21

21

21

21

21

1_19A

nalyse the relevance of an aiding tool in relation to certain actions1

11

11

11

21

11

12

11_20

Identify prescribed goals that are to be achieved by a team of w

orkers1

11

11

12

11

11

11

11

21_21

Identify prescribed tasks that are to be achieved by a team of w

orkers1

11

11

11

21

11

11

21

12

1_22Identify different form

s of comm

unication and the exchange of information am

ong workers

12

12

11

11

11

11

22

11

21

11

11

1_23A

nalyse the mutual know

ledge of teams or team

mem

bers2

21

22

22

11

21

12

22

12

22

1_24Identify sem

antic knowledge the w

orkers need for effectively accomplishing the task

11

11

11

11

11

21

11

21

11

12_1

Identify the major types of activity in the current system

the automation w

ill affect2

12

11

11

11

11

21

2_2M

odel activities in a functional matter w

ith respect to a purpose they serve1

22

22

21

11

11

12

22

22

2_3B

reak down actions in the norm

al course into their physical, cognitive and comm

unicative components

11

21

11

21

22

2_4S

tructure the particular activities into activity descriptions by using the Activity M

odelling templates

21

22

2_5R

eview the A

ctivity Model

22

11

11

11

21

3_1U

se the findings of the Activity M

odel to identify system boundaries

21

12

21

11

21

3_2B

uild a first cut Context M

odel to identify system boundaries

11

11

11

12

12

22

11

11

11

11

23_3

Identify a basic list of stakeholders2

11

13_4

Carry out an initial stakeholder analysis to determ

ine the major categories of system

stakeholder 2

11

11

11

11

11

3_5C

onsider for each stakeholder whether he corresponds to an adjacent actor

22

22

11

11

13_6

Develop an extended C

ontext Model to refine the first cut m

odel of system boundaries

21

22

21

22

11

21

22

4_1A

llocate functions between actors according to boundaries

12

11

11

11

11

11

14_2

Model the system

’s hard and soft goals1

21

21

12

22

4_3Interpret the A

ctivity Model and integrate the identified actors and goals into the S

D-M

odel2

11

21

21

12

22

4_4Identify the intentional strategic actors

21

21

11

11

11

4_5M

odel dependencies between strategic actors for goals to be achieved and tasks to be perform

ed1

11

21

12

12

24_6

Model dependencies betw

een strategic actors for availability or entity of resources1

11

21

12

12

24_7

Write different form

s of dependency descriptions 2

21

15_1

Refine the S

trategic Dependency M

odel1

21

11

11

12

25_2

Refine the S

trategic Rationale M

odels1

12

11

11

12

25_3

Produce a single, integrated S

R m

odel using dependencies in the SD

model

22

21

11

5_4C

heck that each individual SD

model is com

plete and correct with stakeholder goals, soft-goals, tasks and resources

22

21

11

5_5V

alidate the i* SR

model against the S

D m

odel by cross-checking the two m

odels2

22

11

1

5. Refine System Dependencies 1. Gather Data on Human Activity2. Model Human

Activity3. Determine

System Boundaries4. Determine

System Dependencies

Figure 6