1

PARTICIPATORY RESEARCH: A DELPHI STUDY ON THE METHODOLOGICAL

CRITERIA FOR THE QUALITY OF THE RESEARCH PROCESS

Inge Bleijenbergh, Hubert Korzilius, Piet Verschuren

Keywords: participatory research, Delphi Study, Validity, Reliability, Methodological Criteria

2

INTRO

In this paper we discuss the development of methodological criteria to assess the quality of a

participatory research process. Participatory research is of growing importance in organizational

research. Examples are the use of Delphi methods, gaming, facilitated modeling and cognitive

mapping. In contrast to traditional quantitative and qualitative research methods, participatory

research involves the people under investigation directly in the process of data collection and/or

analysis, giving them bigger responsibility in influencing and defining the results of the research.

The relation between researcher and respondent becomes less hierarchical, while there is an

increasing need for reporting about the quality of the research process. Since traditional

methodological quality criteria, like validity and reliability, are more suitable for assessing the

results rather than the process of research, this paper aims at addressing this gap by contributing

to the development of methodological criteria for the participatory research process. Based on a

Delphi study involving scholars who are involved in participatory research projects in five

countries, a provisional list of methodological criteria is proposed.

3

INTRODUCTION

A relatively new type of practice-oriented research is what we call participatory research.

This is a research strategy where a group of stakeholders tackles an organizational problem by

means of confrontation and discussion of their knowledge, perceptions, evaluations and attitudes

concerning the problem. Besides this problem solving variant there is also a knowledge

generating type of participatory research, where new knowledge is produced by confrontation of

the ideas of relevant experts (Bleijenbergh, Korzilius & Verschuren 2011). However, in this

paper we will concentrate on the problem solving variant. The reason for this is that a group of

experts that we consulted in a Delphi, almost unanimously defined participatory research in this

sense.

The question that will be answered in this paper is what methodological criteria should be

used to assess the quality of the process of this type of research. As to the quality of theory-

oriented research, methodological criteria that are generally known are internal and external

validity, reliability and controllability (Adèr & Mellenbergh 1999, Babbie 1998, Bernard 2002,

Kidder & Judd 1986, Swanborn 2009). In the qualitative research paradigm alternative criteria

have been develop, like thick description, trustworthiness, reflexivity and triangulation (Scharz,

Shea, 2006). For practice-oriented research in general these are no longer the sole criteria to be

met. The criterion of utility of the results is of growing importance (Bickman & Rog 1998,

Bleijenbergh et al 2011, Bryman 2007, Shadish et al 1995, Verschuren 2011). The results of

practice oriented research, need to guarantee an adequate resolution of the problem to be solved,

the construction of the material or immaterial artifact to be realized, or the decision to be made.

Moreover, we argue that we not only need methodological criteria to evaluate the results

of participatory research, but also to evaluate the quality of the research process.

4

First: define what we consider participatory research: we focus on strategies rather than

techniques. Strategies are coherent… (later uitgewerkt in beschrijving verschillende strategieen).

Second: We assume that participatory research can be performed in a variety of research

paradigm, varying from a positivist to an interpretative or constructivist approach (overzicht via

Guba and Lincoln, 2011).

Third: we contribute to the debate about criteria for assessing the quality of research,

without assuming that a definite list is possible. However, we agree with Shea-Schwarz (2006)

that proposing a list of criteria is a first step in the debate.

Fourth, when we focus on participatory research, we refer to participatory as cooperation

between researchers and members of organizations in terms of insider outsider collaboration

(Bartunek & Resi Louis, 1996). So, more radical interpretation of participatory than in

participatory observation, where the researcher participates in the organization or community he

or she examines, without necessary involving the insiders in the analysis and collection of data.

Fifth: we assume participatory research may be directed at both knowledge generation

and problem solving, although it is often primary initiated with a practical purpose. Participatory

research is not necessarily action research. Action research has the perspective of critically

assessing power relations and being directed at empowerment of organizational members

(Bartunek, Reis Louis, 1996). Participatory research may contribute to this aim, but not

necessarily does so. Participatory research is also called Participatory Action Research, because

the knowledge generated may help to solve organizational problems (Whyte, 1982; Cancian &

Armsted, 1992).

5

To explain this focus, we make a distinction between two types of practice-oriented

research: (a) the knowledge generation variant and (b) the problem solving variant (see Table1).

The first is based on either empirical data obtained by means of sensory observation (i.e. data

based research), or on the confrontation of the ideas of experts (i.e. participatory knowledge

generating research), both aiming at the production of new knowledge. This knowledge can be

used by those who have to solve a problem, to construct an artifact or to make complex

decisions, once the research is finished.

The problem solving variant of practice-oriented research coincides with participatory

problem solving research. It is primarily based on human resources, such as the knowledge,

opinions, attitudes, evaluations and such like, of participants. It is directly aimed at the resolution

of a practical problem, such as a policy problem of public authorities, or a strategic management

problem of a private or public organization. As Table 1 shows these differences in inputs and

objectives of the two models of practice-oriented research are linked with several other

characteristics. These regard the division of roles between researcher, respondent and

stakeholders, several methodological traits such as the question if the research is empirically or

theoretically-driven, an open or pre-structured research design is used, an iterative-parallel

versus a linear-serial approach, cross-sectional or diachronic data collection is performed, and

the primacy of sensory observation or of human mental capacities such as human rationality, and

the knowledge, opinions and attitudes of participants. As a consequence the researcher either

needs primarily knowledge of and skills in research methodology, or also needs skills in

facilitating group processes and knowledge of the problem to be solved, the decision to be made

or the artifact to be produced.

6

-----------------

Insert Table 1 about here

-----------------

On the basis of these distinctions we emphasize the need for developing methodological

criteria. In case of the problem solving variant of practice-oriented research we not only need

product criteria as is the case in theory-oriented and practice-oriented research in general. We

also need criteria that assess the way the research is carried out, i.e. the research process. There

are three arguments for this importance. Firstly, contradictory to traditional theory- and practice-

oriented research, in the problem solving variant an intensive collaboration between the

researcher(s) and the research subjects is taking place. Their roles may even totally or partly

coincide (see Table 1). Thus, the value of the research results strongly depends on the way the

research is carried out. Secondly, and closely related to the first criterion, in a problem solving

type of practice-oriented research group processes play a crucial role (see Table 1). So, the

quality of the research as such depends in large part on the quality of these group processes. And

a third, perhaps most fundamental, reason for the primacy of the research process as a

determinant of the methodological quality is, that in this type of research the learning process of

the participants is an important aim. For instance, they may learn from each other as to the way

they communicate, articulate hopes and fears, their values and priorities and their mental models.

From this follows our main research question to be answered in this paper. Which

methodologically relevant process criteria can be used by researchers and users of problem

solving practice-oriented research in order to improve the quality of the research process.

7

In order to answer this question we first try to develop a general definition of

participatory research strategy, and to find a list of strategies that fit this definition. This is

needed, as there is a lot of disagreement between scholars and practitioners about this

phenomenon (section 1). Next we try to find a set of criteria that may be used to assess the

quality of the research process of these strategies (section 2). We end up with a discussion

(section 3).

PARTICIPATORY RESEARCH METHODS IN ORGANIZATIONS

Academic literature recognizes a range of different participatory research techniques and

strategies. In order to develop criteria for assessing the quality of the research process, we first

need to describe the goals of each particular strategy and the approaches they use to reach them.

1 Cognitive mapping

Cognitive mapping is a method for structuring organizational problems by representing

them in networks of arrows and concepts (Eden, 2003). Goals of cognitive mapping may be

problem solving or a reflective device (Eden, 1988). Cognitive maps are characterized by a graph

that shows the hierarchical structure of means and ends. The formal basis for cognitive maps

derives from personal construct theory, which relates to understanding how people make sense of

the world. Cognitive mapping is a group technique, when cognitive maps of members of a team

are merged in a group map. The maps depict cognitive structures of members of organizations

concerning complex issues

2 Delphi study

The Delphi study is directed at confronting, analyzing and assembling expert knowledge

on a specific problem or theme. The Delphi method was developed by Helmer, Dalkey, and

Rescher (Rescher, 1998). Original, Delphi studies focus achieving consensus between experts

8

on future predictions. At present they are used for a broader set of tasks such as decision making

and knowledge production. Some eight to fifteen participating experts do not interact face-to-

face, but usually by means of e-mail, and via a facilitator. In a first round, the participants work

individually on a few tasks and send their results to the facilitator who makes an inventory,

identifies common and conflicting viewpoints. In a second round participants receive this

feedback and are asked to change or elaborate on their initial response. The facilitator checks if

there are many changes and elaborations. If there are substantial changes and/or elaborations a

third run is started. Generally, viewpoints converge after two or three rounds and then the

process is stopped.

3. Gaming

Gaming was developed by Duke (1974). The goal is to increase participants

understanding of how complex social-technological systems work and how decisions can be

made about them (Duke, 1998, in Mayer & De Jong, 2004, p. 223); the outcome provides

guidelines for future action (Duke, 1974: 171). A group of stakeholders acts in a simulated

environment through role-playing and mutual communication. The game is based on a simulated

model derived from a real referent system. The processes of the referent system are simulated by

actual imitation performed by real stakeholders or experts. Throughout the game, the

stakeholders interactively experience how the system works and how it could be changed by

looking ‘forward into the future” (Geurts & Joldersma, 2001, p. 305).

4. Group Decision Room

A Group Decision room is a computer supported face-to-face interaction between

stakeholders or experts. The goal may be the development of a model, an idea for problem

solving, the evaluation of an action or of an artifact, and the like. A group of ten to fifteen

9

participants work individually behind their computer on tasks, and send their results to the

central computer, which is operated by the researcher or facilitator. The results are shown on a

central computer screen and discussed by the participants. Next a new task for the individual

participants may be to combine different models to one single model, to eliminate doubles in a

list of ideas, to find more abstract categories or to detect contradictions in propositions of the

participants.

5. Group model building

Group model building is a technique whereby a facilitator supports a group of people

with building a causal loop diagram on the causes of or the solutions for highly complex

problems. Group model building is based on the theory of System Dynamics of Forrester (1961)

and was further developed by Anderson and Richardson (1987) and Vennix (1996). Group

Model building consists of a number of distinct phases that are often divided over a number of

sessions. first, the facilitator uses a nominal group technique to collect opinions of all

participants on the causes of a problem; second, the facilitator collects these opinions and takes

them as the base for a group discussion on the problem at stake; third, the facilitator supports a

group discussion on the causes of the problem and put them in a causal loop diagram; fourth, and

finally, feedback processes are identified that support the group to understand the dynamics of

the particular problem at stake and that help the group to identify directions for solving the issue.

6. Nominal Group Technique

The Nominal Group Technique is a technique in which a facilitator aids a group of

people by listing the opinions of the different participants on the causes of a problem or the

solutions for it. Nominal Group Technique was developed by Delbecq and VandeVen (Delbecq

et al., 1975)to prevent, among other things, that some participants dominate a debate. Moreover,

10

it encourages all group members to participate in the discussion. By giving all group members a

turn in articulating their opinion, everyone’s opinion is taken into consideration. Therefore,

Nominal Group Technique is suitable for heterogeneous groups. A common approach is that a

facilitator asks the participants to write down their opinion and gives them opportunity to

articulate this opinion in turn. Next, the facilitator lists the opinions and facilitates a group

discussion on the issues. Finally, the facilitator asks the participants to order them in rank.

--------------

Insert Table 2 here

-------------

RESEARCH METHODS

For the aim of formulating a set of methodological criteria to assess the quality of the

research process, we choose a Delphi study amongst experts in the field of participatory research.

Alternatives like a survey or a case-study seemed not to be suitable for the purpose of our study.

A survey was not applicable since we did not have detailed ideas about the definitions and

criteria to examine and the operationalization had to be developed during the research process. A

case-study was not suitable since we did not want to describe the current use of participatory

research strategies, but rather wanted to reach consensus between experts using them. And since

we were looking for consensus on the definition of participatory research and on the criteria to

assess the quality of the research process, we needed a research strategy that supported both the

exchange of expert knowledge and the fostering of consensus between them. Moreover, since

experts are dispersed over the world, we needed a research strategy that supported exchange of

11

knowledge over a long distance. In addition, the participants, being experts in the field, were not

only giving information, but were also closely watching the research process.

We developed questionnaires containing open and closed questions, which were

distributed to the participants via e-mail (first round) and by means of the program Qualtrics

(second round). The combination of qualitative and quantitative data collection allowed us to

triangulate the information. The qualitative data (texts from the open questions) were coded and

interpreted by the three researchers; the quantitative data were analyzed using SPSS

(frequencies, descriptive statistics).

Selection of participants

We selected participants for our Delphi study via the snowball method. Key figures in

three fields of participatory research such as Delphi, Gaming and Modeling were selected on the

basis of our professional network We asked them for participation, as well as to recommend

others with broad experience in their field for participation in the study. Finally of the 16

researchers who were invited ten agreed. After all nine of these ten actually participated in both

rounds. Of them three were working in the field of Delphi studies, three in the field of gaming

and four in the domain of modeling. The participants were living in five countries, i.e. Australia,

Belgium, the Netherlands, the United Kingdom and the United States. All of them have an

academic affiliation or formerly had one and are now working in business.

RESULTS

A definition of participatory research strategies

Definition

12

The first issue we addressed in the Delphi was finding a definition of participatory

research. In the first round we asked participants to give their own definition of a participatory

research strategy. All participants gave a definition, which varied in length from fifteen words

A participatory research strategy involves the stakeholders of the research question in

the research process (r. 1)

to 68 words:

An approach to gathering date in which a single participant or, more usually, a group of

participants does not only supply the researcher with data on predefined subjects, but is

[also] involved in an iterative process of codetermining the questions to be studied and

providing qualitative and or quantitative data to answer these questions. In a group

setting this involves building a shared representation of the issue at hand (r. 3).

A common feature of the presented definitions is that a group of people is actively

involved in the research process. Terms to address this group vary from stakeholders and

participants to ‘examined’. Disagreement appears to exist on the participants’ involvement;

either the research process (r. 1, 2, 3, 4) or the intervention cycle (r. 6, 8, 10). The definitions

also vary regarding the role of the researcher, especially regarding the level of involvement of

the researcher. The role of the researcher varies from being an ‘objective moderator (r. 4) to

‘addressing an issue of genuine interest …to both researchers and researched’ (r. 10).

In the second round we tried to derive a common definition from the definitions presented by the

participants and we asked them to what extent they agreed or not, and why. The definition we

gave was:

In a participatory research strategy the researcher actively involves stakeholders and/or

participants: in defining a research question or a problem to be solved; in exchanging

13

and/or analyzing data, knowledge, and information; in generating answers to the

question and/or designing solutions to the problem; and/or in implementing and

evaluating actions for problem solving.

The results show that consensus on a common definition is not yet reached, since only six

participants agreed or strongly agreed (measured by means of a 5-point scale ranging from

strongly disagree (1) to strongly agree (5)), and three participants disagreed or strongly disagreed

to the definition (M = 3.44; SD = 1.42) Elements that following the participants should be left out

of the definition are for example the formulation ‘stakeholders and/or participants’. Different

participants propose to use these terms not as mutually exclusive, but to talk about ‘participants’

in general (r. 5) or about ‘stakeholders as participants’ (r. 1). Another point of criticism was that

the communication between the participants did not receive enough attention, in comparison to

the communication between the researcher and the participants. In the words of one participant:

The essential element of participatory research in my opinion is, that it is not just a 'one

way communication' (the participants feed the researcher with information - knowledge,

judgment, evaluations, interpretations), but it is also an explicit objective of this

approach to exchange the vision of the participants among each other, trying to build a

shared vision or to get a shared frame of reference. So in the definition 'this two-way

communication' aspect should be present.

We concluded that a third round of the Delphi study is needed to develop a definition that fits all

the criteria of all the participants.

Participatory research strategies

The second issue we addressed was answering the question what possible research

strategies fit this definition. In the first round we gave the participants a preliminary list of

14

strategies we exstracted from the above mentioned literature: Cognitive Mapping, Delphi,

Gaming, Group Decision Room, Group Model Building and Nominal Group Technique.

Participants were invited to add other strategies that in their view fit the definition, resulting in

the following list: Complexity Theory, Interactive Workshops, Knowledge Cafes, Large Group

Interventions, Learning Histories and Naturalistic Inquiry. No single added strategy was

mentioned twice. The result was a considerable extension of the number of participatory research

strategies and we wondered if all strategies are relevant and if any groupings of strategies are

possible.

In the second round of the Delphi we asked participants to indicate if they were familiar

with the added strategies or not. It appeared that all participants are familiar with the Delphi

Study (classical), Gaming, and Interactive Workshops, while six out of nine are familiar with

Cognitive Mapping, Group Decision Room, Group Model Building, Nominal Group Technique,

Open Spaces, Policy Delphi, Serious Computer Games and Soft Systems Methodology.

Participants are less familiar with Complexity Theory, Knowledge Cafes, Large Group

Intervention, Learning Histories, Naturalistic Inquiry, Strategic Choice Approach, and Value

Focused Thinking.

On the basis of the Delphi we can conclude that the differentiation between the Classical

Delphi and the Policy Delphi seems relevant, just as the addition of Serious Games Computer

Based to Gaming. Moreover, the addition of Interactive Workshops is relevant to all participants,

while we did not find references to this method in the academic literature. Also the research

strategy of Open Spaces appears to be familiar to the participants. On the basis of the second

round, it seems not necessary to add participatory research strategies to the list.

15

Methodological criteria

The third issue we addressed was the development of criteria for assessing the quality of

the research process. In the first round, we asked participants in an open question which criteria

they considered important for this. In total, the respondents mentioned 25 criteria, of which ten

were mentioned more than once. In order of frequency these criteria were Representation of all

(relevant) stakeholders (7), Commitment of participants (5), Diversity of (type of) input (4),

Convergence of participants’ opinions (3), Power of participants to influence the results (3),

Transparency of the (research)process (3), Participant satisfaction (2), Diversity of participants

(persons or groups) (2), Balance between time investment of stakeholders and external

researchers (2) and Implementation of conclusions/results (2).

We wanted to find out if these ten criteria were suitable to assess the process of

participatory research. In order to test this, we asked participants to what extent they agreed that

the successive criteria were suitable by means of a 5-point scale (1 = strongly disagree, 5 =

strongly agree; see Table 3). From Table 3 it appeared that the following three criteria were

considered most suitable (having a mean score above 4); representation of all (relevant)

stakeholders, transparency of the (research) process and commitment of participants. In contrast,

the criterion of balance between time investment of stakeholders and external researchers was

regarded as least suitable. Further participants in the second round, had heterogeneous opinions

on the criteria of convergence of participants' opinions, power of participants to influence the

results and diversity of participants. On the basis of the results, we consider representation of all

(relevant) stakeholders, transparency of the (research) process and commitment of participants as

suitable criteria for assessing the process of participatory research.

-------------

16

Insert Table 3

-------------

CONCLUSION AND DISCUSSION

This paper aims at developing methodological criteria for assessing the quality of the

process of participatory research, in particular for problem solving in organizations. In order to

achieve this we performed two rounds of a Delphi study involving experts in participatory

research strategies in five countries. This resulted in a list of quality criteria, of which

representation of all relevant stakeholders, transparency of the research process and commitment

of participants were considered the most suitable. As we focused on criteria for assessing

participatory research directed at problem solving, the criteria we found do not refer to the value

of the results for the academic community. We assume that criteria for this last purpose are

already better developed. They consist, amongst others, of internal and external validity,

reliability, and verifiability, although this depends on your epistemological position (Heron &

Reason, 1997; Schwarz-Shea, 2006). The current study intends to contribute to the discussion

about the quality criteria of the research process.

We did not reach consensus about a definition of participatory research strategies.

Consequently there is no unanimous consensus about the question which types of research may

be regarded as variants of participatory research. However, some agreement can be identified.

Group Model Building, Policy Delphi and Soft Systems Methodology are relatively often

regarded as representatives of participatory research strategies, and to a lesser extent also

17

Cognitive Mapping. The experts in our Delphi appeared to be most familiar with Delphi,

Gaming, Group Model Building, Interactive Workshops, and Serious Games Computer Based.

We are aiming to have a third round of the Delphi, in which we want to reach consensus

on the definition of participatory research strategy and also on the measurement of the quality

criteria of the research process we identified in this paper. We want to ask the experts about their

opinion on the appropriate way of measuring quality criteria: would a quantitative measurement

suffice or is qualitative assessment more appropriate? By means of a pretest and posttest

questionnaire changes in the opinions of participants can be measured. An alternative would be

to use open interviews or semi-structured questionnaires to measure the quality of the research

process.

On the basis of our analysis we assumed that participatory research strategies, not only

have to fulfill product criteria such as validity and reliability as is the case in theory-oriented

research, and utility as in practice-oriented research in general, but also and especially process

criteria. This Delphi study supports this assumption and shows that consensus seems to exist on

at least three criteria: the representation of all (relevant) stakeholders, transparency of the

research process and commitment of participants.

Finally, although this was not envisaged beforehand, there is one last significant and

surprising consequence attached to the process criterion of consensus. This is that the use of

participatory research strategies may result in a technically sub optimal resolution of the

problem at hand. The reason is that sometimes several parts of (an optimal) resolution must be

abandoned in order to achieve consensus. This price for consensus should be kept in mind by

every stakeholder, not only those who use participatory strategies. The disadvantage of this has

18

to be balanced with the advantages of the combined effect of consensus and anchoring of a

resolution in the group of stakeholders.

19

REFERENCES

Adèr, Herman and Don Mellenbergh, 1999, Research Methodology in the Social, Behavioural

and Life Sciences. Thousand Oaks (CA): Sage.

Andersen, David and George Richardson, 1997, “Scripts for Group Model Building”. System

Dynamics Review, 13: 107-129.

Babbie, Earl, 1998, The practice of social research. Belmont (CA): Wadsworth.

Bernard, H. Russel, 2002, Research methods in anthropology. Qualitative and quantitative

approaches. Oxford: Altamira.

Bickman, Leonard and Debra Rog, 1998, Handbook of Applied Social Research Methods.

Thousand Oaks (CA): Sage.

Bleijenbergh, Inge, Hubert Korzilius and Piet Verschuren, 2011, “Methodological criteria for

the internal validity and utility of practice-oriented research”. Quality & Quality, 45: 145-

156.

Bryman, Allan and Emma Bell, 2007, Business Research Methods. Oxford: Oxford University

Press.

Delbecq, André, Andrew van de Ven and David Gustafson, 1975, Group techniques for

program planning: A guide to nominal group and delphi processes. Glenview (IL): Scott,

Foresman and Co.

Duke, Richard, 1974, Gaming: The future’s language. Beverly Hills (CA): Sage.

Eden, Collin, 1988, “Cognitive mapping”. European Journal of Operational Research, 128: 1-

13.

Eden, Collin, 2003, “Analyzing cognitive maps to help structure issues or problems”. European

Journal of Operational Research, 159: 673-686.

20

Forrester, Jay, 1961, Industrial Dynamics. Cambridge (MS): MIT Press.

Geurts, Jac and Cisca Joldersma, 2001, “Methodology for participatory policy analysis”.

European Journal of Operational Research, 128: 300-310.

Gibbons, Michael, Camille Limoges, Helga Nowotny, Simon Schwartzman, Peter Scott and

Martin Trow, 1994, The New Production of Knowledge. London: Sage.

Heron, John and Peter Reason, 1997, “A participatory inquiry paradigm”. Qualitative Inquiry,

3: 274-294.

Kidder, Louise and Charles Judd, 1986, Research Methods in Social Relations. New York,

Sydney, Tokio: CBS College Publishing.

Mayer, Igor and Marten de Jong, 2004, “Combining GDDS and Gaming for Decision

Support”. Group Decision and Negotiation, 13: 223-241.

Rescher, Nicholas, 1998, Predicting the future. Albany (NY): State University of New York

Press.

Schwartz-Shea, Peregrine, 2006, “Judging quality: Evaluative criteria and epistemic

communities”. In: Dvora Yanow and Peregrine Schwartz-Shea (Eds.) Interpretation and

Method. Empirical Research Methods and the Interpretive Turn. Amonk (NY); M.E.

Sharpe: 89-113.

Shadish, William, Thomas Cook and Laura Leviton, 1995, Foundations of Program

Evaluation. Theories of Practice. Newbury Park, London, New Delhi: Sage.

Swanborn, Piet, 2009, Research methods. The basics. The Hague: Boom.

Vennix, Jac, 1996, Group Model Building: Facilitating team learning using System Dynamics.

Chichester: Wiley.

21

Verschuren, Piet, 2011, Why a methodology for practice-oriented research is a necessary

heresy. The Hague: Eleven International Publishing.

22

Table 1 Characteristics of two models of practice-oriented research

Characteristic Knowledge generation model Problem solving model

Input - empirical data - knowledge, opinions, attitudes

- empirical data

Objective - knowledge, insights,

recommendations

- problem resolution (+ knowledge)

Division of

roles

- separation of roles of researcher,

respondent and stakeholders

- coincidence of the roles of

researcher, respondent and

stakeholders

Methodological

traits

- empirically or theoretically-driven

research

- open or pre-structured approach

- cross-sectional or diachronic data

collection

- primacy of sensory observation

(data)

- empirically-driven research

- open approach

- iterative-parallel approach

- diachronic data collection

- primacy of human resources

Qualities of the

researcher

- methodological knowledge and

skills

- skills in group processes

- know-how of the problem at hand

- methodological knowledge and

skills

23

24

Table 2 Outline of participatory research strategies.

Strategy Goals Group composition Approach

1. Cognitive

mapping

- solving organizational

problems

- reflecting on goals and

values

- alternating individual

and group

- heterogeneous

- hierarchically

structuring goals and

ends

- merging the concept

maps of individuals

2. Delphi

Method

- collecting adequate

and actual expert

knowledge

- traditional: consensus

building; later

making expert

opinions explicit and

clear

- individual

- long distance

- identifying common

and conflicting

viewpoints

- two or three rounds

3. Gaming - safe and realistic

learning environment

- experiment with

decisions

- increase

understanding of

complex systems and

decision making

- mutual

communication

- actual imitation by

real stakeholders

- role-playing

- simulated model

derived from a real

referent system

25

Strategy Goals Group composition Approach

4. Group

Decision

Room

- model development

- ideas for problem

solving

- evaluation of action or

phenomenon

- partly individual,

partly group

- room with computers

5. Group

Model

Building

- explicating role of

feedback processes in

decision making

- preventing cognitive

bias

- finding causes or

solutions for complex

problems

- achieving

commitment, shared

- alternating individual

and group

- causal loop diagram,

feedback process

- five phases divided

over more sessions

6. Nominal

Group

Technique

- collecting

opinions/ideas

- encouraging

participation of all

members

- preventing dominance

- heterogeneous

groups

- writing down

opinions/ideas

- articulating

opinions/ideas

- listing opinions/

ideas

26

Table 3 The suitability of process criteria Process criterion M S D Rank*

Representation of all (relevant) stakeholders

4.67 0.50 1

Commitment of participants 4.11 0.78 3

Diversity of (type of) input 3.88 0.84 5

Convergence of participants' opinions 3.44 1.24 8

Power of participants to influence the results

3.67 1.32 6

Transparency of the (research) process 4.44 0.73 2

Participant satisfaction 3.33 0.71 9

Diversity of participants (persons or groups)

3.89 1.05 4

Balance between time investment of stakeholders and external researchers

2.56 0.73 10

Implementation of conclusions/results 3.56 0.88 7

*Means ranked: 1 = highest; 10 = lowest.