Collaborative modelling of reflection to inform the development and evaluation of work-based...

© MIRROR Project - Co-Funded by EU IST FP7 – www.mirror-project.eu

Collaborative Modelling of Reflection to Inform the Development and Evaluation of Work-Based Learning Technologies

Birgit R. Krogstie and John Krogstie (NTNU, Trondheim, Norway)

Neil Maiden and James Lockerbie (City University, London)

Daniel Wessel and Kristin Knipfer (KMRC, Tübingen, Germany)


Outline of the presentation

▪ The MIRROR project and the objectives for the model of Computer Supported Reflective Learning (CSRL)

▪ The process of creating the CSRL model v0.2 ▪ Use of the i* framework

▪ Insights gained

▪ What happened next? The CSRL model v1.0


MIRROR – Reflective Learning at Work

▪ Informal learning in the workplace through reflection on work experience

▪ EU 7FP Integrated Project with 15 Partners

▪ Austria (1), UK (2), Germany (7), Italy (2), Netherlands (2), Norway (1)

▪ http://www.mirror-project.eu/

▪ 5 test beds with diverse learning needs

▪ health/care, emergency work, technology sales/consultancy

▪ 8 scientific partners with different research foci

▪ data capturing, individual reflection, collaborative knowledge construction, organizational learning, serious games, creative problem solving,..

http://www.mirror-project.eu/






The authors of this paper

MIRROR participants:

▪ Birgit Krogstie – responsible for development of the CSRL model.

▪ Neil Maiden – i* expert/facilitator

▪ James Lockerbie – i* modeller

▪ Daniel Wessel – working on the evaluation framework

▪ Kristin Knipfer – working on the evaluation framework

In addition:

▪ John Krogstie – expert on quality of models

Not all project partners equally involved -> impact on internal validity of the study


Main objective of the MIRROR Model of Computer Supported Reflective Learning

From our Description of Work:

“To provide a reference framework for the development of MIRROR Apps. The framework includes a model accounting for the role of technology in reflective learning processes and a set of conceptual tools supporting App development and their use in the test-beds.”


Intended roles of the CSRL model

▪ In MIRROR, serve as a reference framework to:

▪ Provide insights about reflective learning and its support across test-beds

▪ Promote communication among MIRROR partners and externally by providing an explicit and clear terminology and conceptualization

▪ Support analysis of reflective learning in specific situations at the individual, collaborative, and organizational level

▪ Support the design and deployment of applications supporting reflection.

▪ Become a tool to support development of CSRL solutions outside MIRROR


Roles of the CSRL model

▪ In MIRROR, serve as a reference framework to:

▪ Provide insights about reflective learning and its support across test-beds

▪ Promote communication among MIRROR partners and externally by providing an explicit and clear terminology and conceptualization

▪ Support analysis of reflective learning in specific situations at the individual, collaborative, and organizational level

▪ Support the design and deployment of applications supporting reflection.

▪ Become a tool to support development of CSRL solutions outside MIRROR


Status of model development September 2011

▪ Shared conceptual understanding of reflective learning had been formulated in a theory White paper

▪ Based on theory/literature review + discussion and input from the partners throughout project year 1

▪ Key concepts, but limited model

▪ Lack of a visualization

▪ Project reviewers asking us to integrate, and provide preliminary deliverables on:

▪ The reflection model

▪ The framework for evaluating the tools (MIRROR Apps) developed in the project


Collaborative work

Process of applying i*

Shared conceptual understanding of reflective learning

(White paper)

Empirical data from the test beds

(User studies + scenarios)

Other conceptual models in the

project

Iteratively working on the

i* model (in selected groups

of partners)

Partners providing their perspectives on

reflective learning

CSRL Model draft

CSRL Model v0.2

i* Model Evaluation Framework


Why use a goal-modelling technique to aid the development of a model of reflective learning (1)

▪ Goal modelling techniques provide goals as first-order entities.

▪ Goal modelling techniques enable explicit modelling of tasks and means-end links to goals achieved by their completion.

▪ Many goal-modelling techniques provide precise semantics .

▪ Many goal-modelling techniques provide a graphical notation with which to describe the precise semantics, communicate it to other stakeholders and obtain agreement





▪ Many goal-modelling techniques provide precise semantics.



Why i* in our case? Additional considerations

▪ An agreed-upon process starting with (semi)blank sheets

▪ Addressing multiple, current issues, e.g. of integration

▪ Drawing on current findings (theoretical and empirical)

▪ Focus on key aspects: What processes/tasks is it that we support; what goals is it that the learners want to achieve

▪ Collaboration among scientific partners

▪ Visualizing for improved communication and joint sense-making

▪ i* competence in the project

▪ Part of a more comprehensive evaluation approach

▪ While some partners worked top-down with i* other partners were collecting data (KPIs) bottom-up from the test beds


i* workshops

▪ 3 workshops (October-November 2011), covering the individual, collaborative and organizational perspective

▪ Original intention: one workshop

▪ Partial models combined into one model

▪ 4-5 active participants in each workshop

▪ Covered the relevant perspectives


The elements of i* (1)

Actor Goal Soft goal

Task Resource


The elements of i*: Examples

Actor Goal Soft goal

Task Resource

Emotions better

understood Individual

reflector

Work task

completed

Reconstruct

work

experience(s)

Work data


The elements of i* (2)

Dependency link

Soft goal link

A goal, a task, a resource or a soft

goal can be positively or negatively

contributing to the attainment of a

soft goal, without ensuring the

attainment.

Means-end link

When there are different ways to

achieve a goal. Multiple links =

‘OR’.

Task decomposition link used

to decompose a task into sub-

components. Multiple links =

’AND’

Means End

(goal or task) (goal, task

or resource)

Means End

(any type) (soft goal)

Means End

(any type) (soft goal)

+

-

Sub- Task

component

(any type)


Strategic rationale (SR) model in i*

▪ An intentional description of how each actor achieves its goals and soft goals, conducting tasks and using/producing resources.


Collaborative work on the Individual Reflector i* SR model


The Individual reflector i* model in more detail (1)


Final i* model for the Individual reflector


Complete i* model

Individual

worker

Re-evaluate

work

experience(s)

Undertake work

tasks

Do reflection

session

Reach a

resolution

Reconstruct

work

experience(s)

Attend to

emotions

Frame the

reflection

Unpack the

experience

Critique

experience

Make reflection

outcome

applicable

Work data

Monitor work

tasks

Decide to reflect

Work procedures

(static)

[Triggers]

Knowledge

(behaviour,

ideas, feelings)

Find & create

space for

reflectionSpace/time

Energy

MotivationExperience(s)

better

understood

+

Work data

Clearer mental

model of

experience(s)

+

Reflection

frames

Desirable best

practices

(organisational)

Selected

reflection

frame(s)

Partial structured

mental model of

experience(s)

Interpreted

model of

experience(s)

Tentative

reflection

outcome

Applicable

reflection

outcome

Individual

expected norm

Individual

desirable best

practices

Quality of own

work experience

improved

Satisfaction

increased

Work mastery

increased

+

Self-efficacy

increased

Quality of work

performance

improved

Monitoring

effortless

Monitoring

accurate

Monitoring

relevant

Monitoring timely

+Decision more or

less correct

Reflection

session efficient

Reflection

session effective

Reflection session

achievable in work

constraints

+

General reflection

capability

increased

+

Reflection

session more

successful

Reflection

outcome

achieved

Reflection

outcome

applicable

+Suitable space

found

+

Motivated to

reflect on work in

general+

Motivated to

reflect in session+

Capability to

reflect improved+

+

+

Emotions better

understood

Emotions

suitably attended

to

+

+

Useful mental

model+

Frame the

reflection

suitably

Unpacked

experience

relevant

Reflection

objective

Reflection

objective met

+

Useful

understanding

achieved

+

Satisficing

reflection outcome

achieved

Work

Knowledge

(behaviour,

ideas, feelings)

+

Make reflection

outcome

available

,,,,,,,,,,,,

Applicable

reflection

outcome

Reflect

collaboratively

within team

Make individual

reflections

available

Refined personal

understanding of

work practice

Available related

experiences

Make related

experiences

available

Individual

reflective

thoughts

Assigned

meanings

Knowledge

shared in team

Available

information

about individual

reflections

Make sense of

available

information

Individually

understand

meaning

Articulate

meaning

Reconstruct

work

experience(s)

Re-evaluate

work

experience(s)

Reach a

resolution

Shared team

knowledge/

insights

Proposals for

action (e.g. plans

for change)

Tentative shared

reflection

outcome

Find & create

space for

reflectionSpace/time

EnergyMotivation

Organise

collaborative

reflection

session

Initiate

spontaneous

collaborative

reflection

OR

Reflection

objective

Schedule

Attend to

emotions

Knowledge

(behaviour,

ideas, feelings)

Work data

Shared model of

work

experience(s)

Roles / people

for refection

session

Frame the

reflection

Critique

experience

Reflection

frames

Selected

reflection

frame(s)Critique outcome

Team expected

norm

Plan to apply

reflection

outcome

Desirable best

practices

(organisational)

Applicable

reflection

outcome

Individual

desirable best

practices

Reflections

socially

acceptable

Individual

reflections

relevant

Privacy

maintained

Individual

reflections

valued

All team

members heard+

Reconstructed

comprehensively

Reconstructed

experience

agreed

Make sense with

confidence

Reflection

session efficient

Reflection

session effective

Reflection session

achievable in work

constraints

Reflection

session more

successful

Reflection

objective met

+

Make sense to a

sufficient depth

?

Emotions

attended to

constructively

The more

important aspects

considered

Re-evaluated

with strong

rationale

Appropriate

frame selected

Frame applied

correctly

Critique criteria

appropriate

Critique fairness

achieved

Critique outcome

valuable

Solution fairness

achieved

+

Resolution

applicable

Team satisfied

with resolution

Resolution

avoids negative

wider impacts

Resolution

provenance

known

Resolution

meets work

goals

Reflection

outcome

operationalized

Reflection

outcome

acceptable

Resolution

transferable

Reflection

outcome feasible

Organisational

model

General reflection

capability

increased

Reflection

outcome

achieved

+

+

+

Individual

team

reflector

Reflect

individually

within team

Re-assess own

experience

Relate own

experience to

others’Work data

Set of individual

experiences

Articulate

outcomes

Norms (feeling

rules)

Attend to

emotions

+

Emotions better

understood

Emotions

suitably attended

to

+

Emotional

outcomes

Personal pre-

understanding of

work practice

Desirable best

practices

(organisational)

Individual

desirable best

practices

Time and space

sufficient

Re-assessed

confidently

Articulated

clearly to self

Articulated

clearly to others

Knowledge

about the team

Team work

practice

improved

+

Scaffolds for

group reflection

and articulation

Scaffolds for

changing between

group/individual

Individual

reflector

,, ,, ,, , ,, ,, ,

Available related

experiences

Available

information

about individual

reflections

,,,,, ,,,

,, ,,

Refined personal

understanding of

work practice

Individual

reflective

thoughts

,,,,,,, ,,,,,

,,,,,,,,,,,,

Understanding of

work practice

,,,,

,,,,

,,,,

, , , , ,, , ,, , ,,

,,,,

,,,,

,,,,

Knowledge

(behaviour,

ideas, feelings)

Suitable space

found

+

Motivated to

reflect in session

+

Capability to

reflect improved

+

+

+

, , , , ,, , , , ,, ,

Collaborative

team reflector

,,,,,,,,,,,,

+

General reflection

capability

increased

+ +

+

+ +

+

+

+

Scaffolds for

individual

reflection

Scaffolds for

individual

reflection

Decision agreed

+

+

+


Extracting key steps of the reflection process from the i* model: The CSRL model

Examples:

▪ From the Individual reflector i* model: Frame the reflection session (A),

▪ Attend to emotions (B), and

▪ Re-evaluate work experience (D)

▪ From the Team reflector i* model: Articulate meaning (C)


Getting agreement on the use of the CSRL model in the project: General Assembly in MIRROR November 2011

▪ Workshop with app developers: Exercise of instantiating categories of tool use associated with the process steps

▪ Presentation of how the CSRL model aligned with other perspectives/models in the project (e.g. input from the scientific partners)

▪ Instantiation of the i* model with test bed cases


Learning cycle view of the CSRL model


Linking tool use to the CSRL process steps


Status of the CSRL model November 2011

▪ Reference model containing a set of steps generic to different cases of reflective learning

▪ Instantiation with real cases, or types of cases, of reflective learning should be based on selection of relevant steps

▪ Basis for further refinement through instantiation with MIRROR cases


The CSRL model v1.0 (Status September 2012)

▪ Refined model; same core

▪ Procedure for applying the model to a case (analysis and design)

▪ Evaluated with MIRROR test beds

▪ Published to the wider research community

▪ Refinement will continue over the next two years (-> v2.0)


The CSRL model: quality aspects and tradeoffs (1)

▪ Social quality (agreement and acceptance among participants) was very important, e.g. getting an accepted common basis. ▪ Feedback from the project partners at the general assembly indicated that this

was achieved.

▪ Genuine agreement mandates pragmatic quality, e.g. that the participants understand the model. ▪ Using a simple language and being able to represent intensions explicitly are

particularly useful.

▪ (Perceived) semantic quality is also a mean for achieving agreement. ▪ With regard to the i* model in MIRROR, the model instantiation evoked some

comments about completeness, for instance regarding the representation of organizational structure.


The CSRL model: quality aspects and tradeoffs (2)

▪ Syntactic quality (that i* is used correctly) is less important when using a model for sense-making and communication than if the model is to be activated as part of an application.

▪ In MIRROR, several example of non-standard use of i* can be found (e.g. ‘energy’ as a resource). “Bastardising” i*!

▪ We used a subset of the language, based on our needs.

▪ The main trade-off in our process of applying the i* methodology was simplicity (to support understanding and enable participation and through this perceived agreement and acceptance) vs. completeness.


Lessons learned

▪ Use of i* to model a learning domain ▪ ..allows the combining of theoretical/generic process models with

characteristics specific to different cases

▪ ..allows a focus on users (actors, goals) – e.g. grounding in real-life needs

▪ ..utilizes a well-known approach for which many modelling tools exist

▪ ..facilitates communication

▪ ..allows a ‘divide and conquer’ (or divide and combine) approach to allow different groups of stakeholders develop different parts of the model

▪ Insights about i* modelling workshops: ▪ Having a dedicated ‘i* modeller’ is very useful, both during the workshop

and for later improvement of the layout

▪ Include relevant stakeholders, but stick to 4-5 people for each collaborative session


Key factors in making the process successful

▪ Project management agreement to the process -> legitimacy/mandate for a limited group of people to work on the models

▪ The extracting of elements for the CSRL model from the i* model involved some choices and decisions.

▪ Showing alignment of perspectives + doing initial validation within the project (at the General Assembly)

▪ The model represents a sufficiently shared understanding and a sufficient level of detail to serve as a starting point for further work.

▪ This is science but also politics!

Collaborative modelling of reflection to inform the development and evaluation of work-based...

Documents

Transcript of Collaborative modelling of reflection to inform the development and evaluation of work-based...