Ecological Economics 69 (2010) 579–591

Contents lists available at ScienceDirect

Ecological Economics

j ourna l homepage: www.e lsev ie r.com/ locate /eco lecon

Analysis

Q methodology to select participants for a stakeholder dialogue on energy optionsfrom biomass in the Netherlands

Eefje Cuppen a,⁎, Sylvia Breukers b, Matthijs Hisschemöller a, Emmy Bergsma a

a Institute for Environmental Studies, Vrije Universiteit Amsterdam, De Boelelaan 1085, 1081 HV Amsterdam, The Netherlandsb Energy Research Centre of the Netherlands (ECN), PO Box 56890, 1040 AW Amsterdam, The Netherlands

⁎ Corresponding author.E-mail address: eefje.cuppen@ivm.vu.nl (E. Cuppen)

0921-8009/$ – see front matter © 2009 Elsevier B.V. Aldoi:10.1016/j.ecolecon.2009.09.005

a b s t r a c t

a r t i c l e i n f o

Article history:Received 1 December 2008Received in revised form 26 May 2009Accepted 10 September 2009Available online 12 October 2009

Keywords:BiomassDeliberationParticipationStakeholder dialogueStakeholder selectionQ methodology

Stakeholder dialogues are proposed as a method for assessing complex ecological and environmentalproblems. Stakeholder dialogues aim to enhance mutual learning by generating and evaluating divergentknowledge claims and viewpoints, i.e. problem structuring. Problem structuring requires that the diversity ofperspectives is represented in the dialogue. We argue therefore that stakeholder dialogues should involveprocedures for stakeholder selection that allow for the identification of the diversity of perspectives as wellas of stakeholders identifying with those perspectives. We explore the use of Q methodology as a tool forpursuing this goal. Q methodology can be used to uncover perspectives that exist within a particular (policy)field. We applied Q methodology to a stakeholder dialogue on energy options from biomass in theNetherlands to identify stakeholder perspectives on energy from biomass and to select stakeholders for thedialogue. In order to discuss the use of Q methodology for stakeholder selection, we compare thisstakeholder selection with a hypothetical selection based on actor type (NGOs, knowledge institutes,industry, etcetera). Our analysis shows that, although often implicitly assumed, actor type is not a goodproxy for perspective: the actor types appeared to be heterogeneous in terms of perspectives. This meansthat a stakeholder selection procedure based on perspectives will very likely result in a different groupcomposition than a selection based on affiliation or actor type. The analysis however also shows that someactor types were more heterogeneous than others and that some perspectives are dominated by particularactor types. We conclude that Q methodology is a useful method for stakeholder selection in stakeholderdialogues and wrap up with a discussion of the results and the implications for the design and preparation ofa dialogue process.

.

l rights reserved.

© 2009 Elsevier B.V. All rights reserved.

1. Introduction

Complex ecological and environmental problems are character-ized by (scientific) uncertainties, and a diversity of (conflicting)values at stake. Actors often disagree on the question what the goal ofpolicy should be, as well as what the relevant means are for attainingthat goal (e.g. which policy measures). This type of policy problemshas also been referred to as ‘wicked’ (Rittel and Webber, 1973), ‘ill-structured’ (Dunn, 1988; Mitroff and Sagasti, 1973; Simon, 1973),‘messy’ (Ackhoff, 1974) and ‘unstructured’ (Hisschemöller, 1993;Hisschemöller and Hoppe, 2001). Stakeholder participation can beused as a method to structure the problem, i.e. to improve theunderstanding of how different stakeholders perceive and define theproblem and its potential solutions. A stakeholder is defined as anactor involved in, affected by, knowledgeable of, or having relevantexpertise or experience on the issue at stake (based on Van Asselt andRijkens-Klomp, 2002). This definition encompasses different types of

actors, such as academia, government, policy makers, business andNGOs. We argue that, in order to deal with complex environmentalissues, structured stakeholder dialogues are needed that map out andarticulate the various perspectives–values, interests, knowledgeclaims and underlying assumptions–that exist with regard to theissue. This should help to make actors more aware of their ownperspectives and the perspectives of others. In order to include abroad range of perspectives in the dialogue, stakeholder dialoguesshould involve procedures for stakeholder selection that allow for theidentification of the diversity of perspectives and stakeholdersidentifying with those perspectives. We will explore the use of Qmethodology for doing this.

Others have written already on the use of Q Methodology forpolicy analysis, i.e. giving insight in the different positions andopinions in a debate (see e.g. Breukers, 2006; Ellis et al., 2007; VanEeten, 2001). In line with this type of Q research, we show that Qmethodology is a useful method to map the different perspectives onenergy from biomass in the Netherlands, in a more nuanced way thanonly mapping pro- and con viewpoints. Although the idea to use Qmethodology for participant selection has been advanced before, mostnotably by Davies et al. (2005) and Dryzek and Niemeyer (2008),

2 Marginality refers to the newness of perspectives for stakeholders: a marginalperspective is not frequently mentioned in the dominant debate about the issue understudy. This does not necessarily mean that there are only a small number ofstakeholders adhering to the perspective; neither does it mean that a perspective thatonly a small number of stakeholders adhere to cannot be dominant.

3 This reflects the assumption that small entrepreneurs represent a marginal

580 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

there are to our knowledge no empirical applications of Q method-ology for this purpose. We used Q methodology for the selection ofparticipants for a stakeholder dialogue on energy options frombiomass in the Netherlands. We discuss the use of Q methodologyfor stakeholder selection and reflect on this by comparing thestakeholder selection based on Q perspectives with a hypothetical,alternative selection based on affiliation (actor type), an approachcommonly used for the selection of participants for stakeholderdialogues.

Section 2 discusses stakeholder dialogues as problem structuringprocesses and the implications for stakeholder selection. In Section 3we introduce the empirical case study: the Biomass Dialogue.1

Section 4 explains what Q methodology is and Section 5 discusseshow it was applied in the Biomass Dialogue. The results of the Qanalysis are presented in Section 6: six stakeholder perspectives onenergy options from biomass in the Netherlands. In Section 7 weexplain how we used these results to select stakeholders for theBiomass Dialogue. In order to reflect on the added value of using Qmethodology for stakeholder selection, we compare our stakeholderselection with a hypothetical stakeholder selection based on actortype in Section 8 (knowledge institutes and academia, energycompanies and sector/branch organizations; small and mediumsized enterprises, NGOs, national, regional and local government). InSection 9 we discuss how the six perspectives were used to structurethe dialogue. Finally, Section 10 wraps up with conclusions anddiscusses the results and the use of Q methodology for stakeholderdialogues.

2. Stakeholder dialogue as problem structuring and theimplications for stakeholder selection

Participation can take many forms. Also the degree of participationvaries widely, e.g. from informing the public to co-producingknowledge or policy plans (see Arnstein, 1969; Mayer, 1997 citedin: Van de Kerkhof, 2004). Not all forms and degrees of participationare relevant for complex environmental problems. When both theproblem and its potential solutions are unclear, stakeholder partic-ipation can be used to gain an improved understanding of both. Thishas been referred to as problem structuring (Hisschemöller andHoppe, 2001; Hisschemöller, 2005). Participation as problem struc-turing involves learning about the different perspectives on theproblem and its solutions. This requires a relatively high degree ofparticipation; a process in which stakeholders with different back-grounds, knowledge, values and expertise interact and exchangetheir knowledge and ideas. This kind of participatory processes shouldfacilitate mutual learning by generating, articulating and evaluatingdivergent knowledge claims and viewpoints. Hence, they shouldprovide ample opportunity to scrutinize conflicting viewpoints andknowledge claims, rather than for instance negotiating or compro-mising preferences. After all, negotiation or compromising is onlypossible when people know what their own, and other people'spreferences are.

A stakeholder dialogue is an organized meeting of stakeholderswith different perspectives, knowledge and backgrounds, who wouldotherwise not meet (or not all together), structured to a greater orlesser extent by means of specific methods, tools or techniques.Stakeholders deliberate on a specific issue in order to producenew insights. From research in the field of social psychology andmanagement science we know that heterogeneous groups in generalproduce higher quality decisions (e.g. Hoffman, 1959; Hoffman andMaier, 1961; Schweiger et al., 1986; Jehn et al., 1999), as they showmore divergent thinking, consider multiple perspectives, and share

1 For a full description of the Biomass Dialogue we refer to: Cuppen, E. (forthcoming)Puttingperspectives into participation: Constructive ConflictMethodology for stakeholderdialogues.

more unique information than homogeneous groups do (Brodbeck etal., 2002). Including divergent perspectives also means that marginalperspectives are included.2 This is important as marginal viewpoints(or rarely mentioned hypotheses) are more likely to provide newinsights than dominant viewpoints (hypotheses mentioned morefrequently) (Dunn, 2001). Highly probable or predictable hypothesesdo not challenge accepted knowledge claims (see also Brodbeck et al.,2002). The notion that it is important to involvemarginal perspectivesin problem solving can also be found in innovation systems literatureand institutional analyses, which argue that because incumbentsystem players are usually not the first to initiate successful systeminnovations, it is important to involve e.g. small innovative entrepre-neurs3 (Agterbosch, 2006; Breukers andWolsink, 2007; Hekkert et al.,2007).

The next question is how the diversity of perspectives should bereflected in the dialogue. We argue that it is important to include thevariety of perspectives on an equal footing in the dialogue. This meansthat each of the perspectives is represented by an equal number ofparticipants in the dialogue. Hence, the balanced distribution in thedialogue (i.e. the sample) does not reflect the distribution in thestakeholder population. Balanced groups are more likely to discussunshared information than unbalanced groups (Brodbeck et al.,2002). Furthermore, it can be argued that balanced groups are lesssusceptible to groupthink (Dryzek and Niemeyer, 2008) and biasedinformation processing (Cuppen, forthcoming). The balanced inclu-sion of perspectives can be interpreted as a specific form of ‘fairness’,in that the broad range of perspectives, regardless of how manypeople share that perspective, are represented in the stakeholderdialogue and get an equal opportunity of contributing to the dialogue.

In consequence, rather than focusing on consensus, stakeholderdialogues should allow for diversity of perspectives, preferences,(policy) options and goals. This means that participants are not forcedto reach a consensus, as this may impede the creation of usefulinsights for policy makers and stakeholders and may lead to theadoption of invalid assumptions and/or inferior (policy) choices(Gregory et al., 2001; Janis, 1972; Mason and Mitroff, 1981). Groupprocesses that are aimed at reaching a consensus often fail incapturing the diversity of perspectives (Stasser and Titus, 1985). Incontrast, confrontation between competing viewpoints preventsshifting away from reaching a quality decision and moving it towardsreaching an agreeable one (Coglianese, 1999; Van de Kerkhof, 2006).

Much attention is paid in literature to the representativeness ofparticipatory processes, “offering criticism of their frequent failure toreflect the characteristics of wider society in the groups used” (Martin,2008). It is however rarely clear which criteria are used to judgewhether a selection procedure is representative (Crawford et al.,2003: 46, cited in Martin, 2008). Furthermore, the number ofparticipants in stakeholder dialogues is too small to be representativein the statistical sense. This is why we argue against a commonly usedselection method for participatory processes: random sampling.Random sampling is based on the implicit assumption that relevantexpertise (in terms of perspectives, i.e. knowledge, values, interests)is evenly distributed over the whole population. However, thisassumption does not reflect the fact that particular knowledge, valuesand interests can be found among particular people, working andliving in particular groups, institutions or part of the population (see

perspective. Although this is in fact an empirical question, it is certainly so that theideas of small entrepreneurs are not often mentioned in the dominant debates on e.g.sustainable energy issues in the Netherlands. This is what Agterbosch (2006) andBreukers (2006) showed for the case of wind energy.

4 2% of their production in terms of energy.

581E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

also Collins and Evans, 2002; Hisschemöller, 2005; Schön and Rein,1994). Another commonly used approach to stakeholder selection isstratified (random) sampling based on demographic variables. Thisshould ensure that the invited group of people represents the widerpublic in terms of demographic variables, such as socio-economicclass, ethnicity, age, and gender. There is however no fundamentalreason to assume that representation in terms of demographicvariables will translate into representation in terms of perspectives(Martin, 2008), neither does it guarantee representation in theelectoral sense (Goodin and Dryzek, 2006; Huitema et al., 2007).

This argument also goes for stakeholder selection based onaffiliation, which is also a commonly used approach for stakeholderdialogues. When stakeholder selection is based on affiliation,participants represent different actor types. Using a classification ofactor types for stakeholder selection reflects a top–down approach foraddressing the diversity of perspectives. Although there may verywell be high correlations between specific actor types and specificperspectives, it is an empirical question whether actor type is a goodproxy for perspective. Therefore, we emphasize the importance ofusing bottom–up approaches for the identification of the diversity ofperspectives, which means that diversity is not assumed throughsome kind of classification (such as type of organization, demographicvariables, or predefined value orientations), but rather the outcome ofan empirical analysis. There is no fundamental reason to assume thatrepresentation of different actor types results in representation ofdiverse perspectives. In fact, this study shows that this assumption isincorrect.

In conclusion, stakeholder dialogues on complex environmentalissues should include stakeholders who reflect the variety ofperspectives in a balanced way. This has been referred to as discursiverepresentation (Dryzek and Niemeyer, 2008). A stakeholder selectionprocedure based on perspectives requires methods that enable theidentification of the diversity of perspectives and the stakeholderswho adhere to those perspectives. In this article, we explore as towhether Q Methodology is able to do so.

3. The Biomass Dialogue

We applied Q methodology to a stakeholder dialogue on energyoptions from biomass in the Netherlands, the Biomass Dialogue. Theaim of the Biomass Dialogue was to develop ideas about sustainablebiomass chains for the Netherlands, and to identify what is needed inorder to realize these.

Biomass is all organic material: wood, plants, but also organicwaste and offal. When converted to energy, it is an option formitigating climate change, because, in the ideal case, it is CO2-neutral(in the non-ideal but more realistic case, it reduces CO2 as comparedto a fossil alternative). The CO2 releasedwas earlier absorbed from theatmosphere by plants. Wood, for example, incorporates absorbed CO2

from the air. This CO2 is released when the wood is combusted, whichmakes the net CO2-emission zero. In reality, the story is morecomplex. Firstly, there are many different elements and optionswithin a biomass-system, and they all influence the CO2-balancedifferently. Energy can be made from various sources of biomass, e.g.waste, or energy crops. Energy crops can be oily, woody, or sugary.Biomass can be converted to biofuels (bio-ethanol or biodiesel), or itcan be used to make electricity or heat. Many different conversiontechniques exist that all impact efficiency differently. Biomass can beapplied in large-scale chains (e.g. import of energy crops), but it canalso be applied in small-scale chains (on the level of a village or anumber of households). It can involve centralized energy production,or decentralized energy production. All these elements influence theefficiency (CO2 reduction) of biomass applications differently.Secondly, even if there is agreement on the CO2-balance of a biomassoption, it is not the only aspect that impacts the sustainability of thatoption. Other aspects, such as (socio-) economic impacts and impacts

on biodiversity, can be taken into account as well. Many uncertaintiesexist with regard to the impacts on the sustainability of biomassoptions. Thirdly, there are many different actors involved (energycompanies, farmers, NGOs, science, etcetera), with many differentstakes and interests. They often have divergent perspectives on whichaspects and impacts of biomass options should be consideredimportant.

From the 1990s onwards biomass has been put forward as anoption for addressing the climate change problem. However, not onlyis it a complex issue, it also appears to be a controversial issue. In2003, the EU commissioned a directive, which states that in 20105.75% of the volume of transport fuels should be biofuels, with theprospect that this share should increase up to 10% in 2020(Energietransitie, 2008). In the Netherlands, as from January 1st2007, oil companies are legally bound to make sure that 2% of theirproduction4 consists of biofuels (www.energieportal.nl). In the lastyears however, the use of biomass for energy has been criticizedincreasingly. Discussions revolve mainly around the use of food cropsfor the production of biofuels, such as the use of palm oil andrapeseed oil for biodiesel and the use of corn and sugarcane for bio-ethanol. These types of biomass applications are controversial,because they may lead to an increase of food prices and a decreaseof access to food for the poor. As a result of the controversies and thedifficulties in determining sustainability of biofuel applications, theDutch Minister of Environment decided in October 2008 to lower theDutch efforts with regard to biofuels from 5.75 to 4% in 2010 (ANP,2008).

4. What is Q methodology?

Q methodology was developed originally in the 1930s as aninnovative way to study people's subjectivity (Stephenson, 1935;Brown, 1980). Since then, it has been applied in various fields of socialscience, in attempts to uncover patterns of perspectives that aresituated within people's subjectivity (Clarke, 2002; Dryzek andBerejikian, 1993; Ellis et al., 2007; Van Eeten, 2001; Webler et al.,2001). For instance, it has been employed to identify views regardingcitizenship, the public interest, environmental policy, and the qualityof participation processes (Barry and Proops, 1999; Davies and Hodge,2007; Swedeen, 2006;Webler et al., 2001;Wolsink, 2004). Recently, ithas also been adopted in studies that particularly address policy andplanning of renewable energy sources (Ellis et al., 2007; Breukers,2006).

Q methodology can uncover perspectives or positions in a debate,without imposing predefined categories. The merit of Q methodologyis that “by allowing the categories of the analysis to bemanipulated byrespondents, the researcher loses the exclusive power to signify thereality of the researched” (Robbins and Krueger, 2000: 645). Qmethodology differs from R methodology (surveys and question-naires) in that the latter asks respondents to express views on isolatedstatements, whereas Q methodology identifies respondents' views onstatements in the context of the valuation of all statements presented(see e.g. Dryzek & Berejikian, 1993). Furthermore, as opposed to Rmethodology, Q Methodology traditionally intends to give a picture ofthe perspectives that exist (the variety of perspectives) among thepopulation, rather than analyzing the level of support for thoseperspectives among the population (the balance of perspectives). Thisimplies that the procedure for sampling respondents is usuallydifferent from that in R methodology. Rather than random sampling(and large sample sizes), Qmethodology relies on purposive sampling(and smaller sample sizes): the fact that there is a person who isassumed to have a different point of view is enough reason to includehim or her in the sample.

582 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

Q methodology comprises six stages (Davies and Hodge, 2007).The first stage is the definition of the ‘concourse’: the full range ofdiscussions and discourses on the particular issue under study.Defining the concourse means identifying sources, either written orspoken, which contain ideas, opinions, values, preferences andknowledge claims on the issue under study. From the concourse, alarge set of statements is derived in the second stage of Qmethodology. These statements should reflect the diversity of theconcourse. This set has to be reduced to a manageable number(usually no more than sixty), while still reflecting the full diversity ofviewpoints, claims and ideas. Preferably, the wording of the state-ments stays as close as possible to the original wording (and thus theoriginal meaning) of that idea or opinion as found in the concourse.The set of statements is referred to as the Q set. The third stageconcerns the identification of a group of respondents, referred to asthe P sample. As noted above, Q studies use purposive sampling,which means that the P sample needs to comprise as many differentideas, preferences and opinions on the issue under study as possible.In the fourth stage, respondents do the Q sort, which involves rankingthe statements on a scale that represents significance or salience forrespondents (Brown, 1980: p198), such as most agree to mostdisagree (usually normally distributed). Data from the Q sorts arefactor analyzed in the fifth stage, resulting in clusters of Q sorts thatare highly similar in their rankings of the statements (highcorrelation). These clusters can be interpreted as perspectives ordiscourses in the final (sixth) stage. The typical way to interpret afactor in Q methodology is to look at the statements that receive thehighest and the lowest scores respectively (strongest agreement anddisagreement) for that factor. In addition, the statements thatdistinguish most between one factor and the other factors are usefulin interpreting a factor because these indicate how a factor is differentfrom the other factors. Furthermore, the Q sorting task is oftenaccompanied by an interview, in which respondents are asked toexplain their sort — this helps to interpret the factors.

5. Q methodology in the Biomass Dialogue

5.1. Concourse definition and selection of Q statements

In an attempt to reflect the wide range of ideas and opinions aboutbiomass, we collected about two hundred statements. Statementswere taken from transcripts from discussions in the Costa Duestakeholder dialogue on biomass in which one of the authors wasinvolved one year earlier.5 Furthermore, statements were taken frompublic debates, reports, newspaper articles, etcetera. Definition of theconcourse was relatively easy, as the issue of biomass was heavilydiscussed in the media and at several public debates at the time of thisstudy. As a consequence, the range of ideas and opinions wasrelatively well articulated. The transcripts from the Costa Duedialogue were furthermore very useful for identifying marginalperspectives, as it included people and ideas that were not all verywell represented in the dominant debate on biomass.6

In order to reduce the Q set to a manageable number, threemembers of the project team individually categorized the statements,and then identified unique statements within categories. The threecategorizations and sets of unique statements were compared anddiscussed, and iteratively, this process continued until sixty-twostatements remained. The large size of the Q set points to the wide

5 In the Costa Due (Concrete Steps to a sustainable Eemsmond), coordinated byHisschemöller (Vrije Universiteit Amsterdam) and Stokman (Groningen University),stakeholders from the northern region in the Netherlands interactively exploredinvestment opportunities for energy options, in particular from biomass.

6 As sufficient material was available, interviews (a common technique forconcourse definition) would probably not have added to this broad range ofstatements. Interviews may be more relevant when the researcher is unfamiliar withthe issue under study, or when there is no material (i.e. transcripts) available.

variety of ideas and opinions that exist with regard to biomass in theNetherlands. The Q statements were piloted with five people (whowere all well acquainted with the biomass debate) to check whetherwe had not missed any relevant statements. This resulted in a definitelist of sixty statements (see Appendix A). Furthermore, to check therepresentativeness of the statements, we asked during the Qinterview whether the respondent thought important statementsweremissing. This was in general not the case. Of the few respondentswho chose to add statements, it concerned stressing an issue that wasalready covered by other statements, for instance by rephrasing astatement into its positive or negative counterpart.

5.2. Identification of the P sample

We used newspaper articles, and news-websites to identifystakeholders. As there was a lot of public debate on biomass,identifying stakeholders was relatively easy. We also used our earliercontacts to identify relevant stakeholders. Furthermore, we used thesnowball-sampling technique to identify stakeholders: we askedthe respondent to mention someone with a different, and someonewith a similar perspective on energy from biomass. This resultedin a group of seventy-five respondents from different sectors andorganizations: knowledge institutes and academia, (energy) compa-nies, branch/sector organizations (e.g. sector organization for oil andfat), small/medium sized enterprises (including energy consultants,e.g. working on cultivation or treatment of biomass, or on energy/heat/fuel production), NGOs, national government, regional and localgovernment. But more importantly, we expected this sample to reflectthe broadest possible range of knowledge, expertise, interests, andvalues–in short–perspectives.

5.3. Q interview

The interviews took place between August 2007 and October 2007.Interviews typically lasted 60–90min. The central task in the inter-views was the Q sort, added upon by a number of open questions togather qualitative data for interpretation of the factors. Before the sortwe asked: “Can you explain briefly what your ideas on biomass are inrelation to a sustainable energy supply for the Netherlands?” After thesort we asked three questions: “Why are these statements at theextremes?”, “Do you miss specific statements?” and “Would you liketo come back to, or add something to your answer on my earlierquestion about your ideas on biomass?”

In the interview, the sixty statements were presented to therespondents. Respondents were asked to rank-order the statementsaccording to a forced normal distribution with eleven positions frommost to least ‘according to my point of view’ (see Fig. 1). Thestatements were printed on small cards. Respondents put the cards onthe normal distribution that was printed on a sheet of paper.

Fig. 1. Q sort distribution.

Table 1Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 1.

Agree (11): 60: We should concentrate on the use of residuals for biofuel production.34: The distinction between 1st and 2nd generation is not as black-and-white as is often posed.

Agree (10): 53: First, try to make high-quality products from a biomass source, and make energy from what is left.44: The Netherlands is strong in knowledge development in the area of biomass technology.20: The issue of unsustainable land-use, for example in South America, Africa and South-east Asia also exist without biomass production.

Disagree (1): 7: Biomass is a temporary solution; in the end, solar and wind should be the main energy sources.2: If biofuels are being stimulated in the EU, this will definitively result in negative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.

Disagree (2): 23: The cultivation of energy crops in the Netherlands will make the landscape monotonous, attract harmful insects and spread a dirty smell.22: The production of biomass should be restricted to within the EU to be able to control sustainable preconditions with regard to society,economy and the environment.27: Cultivation of energy crops is not favorable because manure and irrigation are needed.

Disagree: Other distinguishingstatements

17: The Dutch government mainly has an eye for large companies; there is not enough attention and support for small, innovative players.

Relevant quotes from interviews “It is not one particular chain, but each biomass type needs it's own chain, that makes it complicated.”“It is not either first or second generation biomass. It depends on the place and time.”“So much energy is being used that if you would bet on one option, either biomass, solar or wind, you won't make it. You just needeverything.”“We have just started with it, [biorefinery] is still in its infancy. But if you know how to develop it in a good way, you can produce in additionto transport fuels bio-ethanol and biodiesel also all kinds of other applications, and combine that in such a way that you create rawmaterialsfor white chemistry”.

Most distinguishing statements printed in italics.

7 Factor loadings above 0.34 were accepted as statistically significant at the 0.01level (calculated as: 2.58*standard error (SE); SE=1/√(number of statements)) (seeMcKeown and Thomas, 1988).

Table 2Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 2.

Agree (11): 40: In the formulation of criteria for certification of biomass also stakeholders from the South should be involved.2: If biofuels are being stimulated in the EU, this will definitively result in negative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.

Agree (10): 1: If the complete lifecycle is taken into account in the analysis, biofuels do not reduce as much greenhouse gas emissions as was hoped for.29: There is a need for generic policy aimed at all clean and efficient vehicles, instead of a policy that is aimed specifically on biofuels.21: The production of biomass is only sustainable if it contributes to the social–economic development of the local community.

Agree: Other distinguishingstatements

7: Biomass is a temporary solution; in the end, solar and wind should be the main energy sources.

Disagree (1): 4: If we want, we can drive clean and fly clean now with biofuels.5: Within the Netherlands and the EU, the production of rapeseed oil should be taken seriously.

Disagree (2): 18: The Dutch government should give tax exemption for biofuels.35: There is no use to develop niche markets; in the end we need large-scale biomass applications, and that niche won't help to reach that.24: The cultivation of energy crops contributes to a colorful landscape and to the bee-population.

Disagree: Other distinguishingstatements

31: Too much money goes to research, and too little to implementation in the market.19: The potential of degraded and marginal grounds is so large that it can mean an economic impulse in rural areas.32: Biomass delivers an important contribution to the security of supply, namely less dependency on geopolitical sensitive areas, and a higher degree ofself-support for the EU.51: In the long run, biofuels will compete on the world market with fossil fuels.3: If there are no entrepreneurs who want to experiment with biofuel applications, nothing will happen.

Relevant quotes frominterviews

“My fear is that projects on a large scale, with large investments, take place with insufficient eye for the local community”“[Large-scale cultivation] has a negative influence on local prosperity and employment. Production is overtaken by large companies, as a resultof which there is no place for the normal population. Employment for local population is used as an argument, but in practice it is disappointing.(…) Without biomass production for energy the cutting of rainforest is already taking place on a large scale (…). We need to know for sure thatwe can guarantee sustainable biomass crops first, and only then can we start with it”“It should not be a new sort of colonialism, in which we use their natural resources”

Most distinguishing statements printed in italics.

583E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

5.4. Q analysis

When all interviews were finalized, analysis of the Q sorts wasdone with help of PQMETHOD 2.11 (Schmolck, 2002). The correla-tions between the Q sorts of all seventy-five respondents werecalculated, creating a seventy-five by seventy-five correlation matrix.The average correlation between Q sorts is .22, which indicates thatthe seventy-five respondents are rather heterogeneous in terms oftheir ideas and opinions about biomass. Due to the high variation inQ sorts, factor analysis of the correlation matrix was not straight-forward. In order to be able to identify meaningful factors, we tookan iterative approach. Going back and forth from different types offactor extraction and rotation to the qualitative interview data,meaningful factors were eventually identified using the centroidanalysis method (with Q sorts/respondents as variables and thestatements as cases), and rotated using Varimax (both are standardprocedures in Q methodology). Factors were identified that had at

least one significant loading. Six factors could clearly be identifiedand explained with help of the qualitative interview data. Of theseventy-five respondents, forty-two loaded significantly7 on onefactor and not on others. Of these forty-two respondents, sevenrespondents loaded significantly on factor 1, fourteen on factor 2,three on factor 3, seven on factor 4, seven on factor 5 and four onfactor 6. One respondent did not load significantly on any of thefactors, and all of the remaining respondents loaded significantly onmore than one factor.

The number of factors extracted is quite high for a Q study (usuallytwo to four factors are extracted). This is mainly due to the highvariation in respondents' Q sorts. Total variance explained of the

Table 3Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 3.

Agree (11): 17: The Dutch government mainly has an eye for large companies; there is not enough attention and support for small, innovative players.31: Too much money goes to research, and too little to implementation in the market.

Agree (10): 42: Small-scale energy production with biomass can give a huge impulse to security of supply in developing countries.18: The Dutch government should give tax exemption for biofuels.60: We should concentrate on the use of residuals for biofuel production.

Disagree (1): 35: It doesn't make sense to develop niche markets; in the end we need large-scale biomass applications, and niches won't help to reach that.33: Given the pace of the development towards more efficient cars, maybe we don't need biofuels for transport.

Disagree (2): 23: The cultivation of energy crops in the Netherlands will make the landscape monotonous, attract harmful insects and spread a dirty smell.49: Because, as a small country with limited means, we are forced to make choices, the government should support only the most favorabletrajectories.55: Technology development is the key to large-scale use of biomass, not an active subsidy policy.

Disagree: Other distinguishingstatements

1: If the complete lifecycle is taken into account in the analysis, biofuels do not reduce as much greenhouse gas emissions as was hoped for.7: Biomass is a temporary solution; in the end, solar and wind should be the main energy sources.

Relevant quotes frominterviews

“Government officials have a risk aversive attitude. They rather don't do business with small companies that can come and leave from one dayto another”“..The government does not have an idea about business. (…) She is too often on the chair of deciding which trajectory is best in terms ofenvironmental efficiency, and also there major mistakes are made. The government does not have sufficient knowledge about this, so sheshould not do this”“At a certain point one should do it. And by doing it we have broken through barriers that we otherwise [only on the basis of knowledgedevelopment] would not have broken through.”“I think that we have in the Netherlands all the solutions to process [residuals and waste]. But we have a history of research and reporting. Wekeep on driveling, but we don't do anything. So that would be my first approach: make sure that we actually use everything we have at thismoment”

Most distinguishing statements printed in italics.

Table 4Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 4.

Agree (11): 52: Over the whole biomass chain, there should be a positive score as regards economic profit, energy- and CO2-balance.32: Biomass delivers an important contribution to the security of supply, namely less dependency on geopolitical sensitive areas, and a higher degree ofself-support for the EU.

Agree (10): 46: Political pressure, at minimum on the EU level, is needed to make sustainability criteria function.51: In the long run, biofuels will compete on the world market with fossil fuels.15: The competition between food, feed, and fuel will have a negative financial impact on people

Agree: Other distinguishingstatements

35: There is no use to develop niche markets; in the end we need large-scale biomass applications, and that niche won't help to reach that.

Disagree (1): 33: Given the pace of the development towards more efficient cars, maybe we don't need biofuels for transport.38: Biofuels can only succeed if the government subsidizes until the end of times.

Disagree (2): 45: Dutch farmers will not benefit from a growing use of biomass.18: The Dutch government should give tax exemption for biofuels.5: Within the Netherlands and the EU, the production of rapeseed oil should be taken seriously.

Disagree: Other distinguishingstatements

56: 2nd generation biofuels benefit from stimulating the 1st generation (E85 and diesel variant E95) now.2: If biofuels are being stimulated in the EU, this will definitively result in negative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.11: Criteria will not prevent that in the future there will be a number of large agro-companies, that supply biomass without taking social andenvironmental interests sufficiently into account.

Relevant quotes frominterviews

“[The discussion is about] the scarcity of resources. Not in the sense that resources are run out, but it is on places where it is difficult to get andwhere it becomes at a certain point too expensive to take it out”“Biomass actually opens the door to [sustainability] criteria”“There are also small companies that show bad practices. Large companies can more easily deal with [sustainability] criteria”“[Positive impacts on the local community] is what you want to achieve, but you need frameworks [sustainability criteria] for that”“..The idea that first generation infrastructure and network forms a basis for the second generation, I think that that is meanwhile rejected, thatit strongly causes problems, and no acceleration”

Most distinguishing statements printed in italics.

584 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

six factors is 46%. Although variance explained is not considered arelevant measure in Q methodology,8 the relatively low explainedvariance also points to the high variation of ideas and viewpoints with

8 In regularR (notQ) factor analysis a total varianceexplainedof46% is considered low.Therelatively low total variance explained is probably a consequence of the large number ofrespondents (seventy-five compared to usually about thirty in Q studies), as this means thatthere are more Q sorts (seventy-five) than statements (sixty). In Q methodology however,variance explained is not considered a relevant measure, since one is not interested in thequestion what the percentage of a perspective in the population is, but Q methodology isdeveloped to show that various factors exist, and what the similarities and differencesbetween these factors are. If the variance explainedof factorA is higher than that of factor B, itonlymeans that there aremore people of factor A in the sample. Contrary to Rmethodology,the sample is not randomly selected. The 6 perspectives are salient perspectives that can berecognized in the policy discourse. They can be interpreted as ideal types. Reality is of coursemore nuanced.

regard to biomass, which underlines the complexity and uncertaintywith regard to the issue.

6. Q methodology: results

The six factors that resulted from the factor analysis wereinterpreted as six different perspectives on biomass. As noted inSection 4, the statements that receive the highest and the lowestscores respectively and the statements that distinguish mostbetween one factor and the other factors are useful in interpretinga factor. We identified the respondents that loaded significantly on afactor, i.e. that define a factor. The qualitative interview data of theserespondents were used to interpret the factor. We used quotes fromthe interviews to describe and explain the factors. Below we show

Table 6Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 6.

Agree (11): 3: If there are no entrepreneurs who want to experiment with biofuel applications, nothing will happen.57: 2nd generation biofuels are for the time being not ready for large-scale application.

Agree (10): 34: The distinction between 1st and 2nd generation is not as black-and-white as is often posed.51: In the long run, biofuels will compete on the world market with fossil fuels.41: In the Netherlands, biofuels are discriminated compared to fossil fuels when it concerns the calculation of CO2-emissions.

Agree: Other distinguishingstatements

19: The potential of degraded and marginal grounds is so large that it can mean an economic impulse in rural areas.56: 2nd generation biofuels profit from stimulating the 1st generation (E85 and diesel variant E95) now.

Disagree (1): 23: The cultivation of energy crops in the Netherlands will make the landscape monotonous, attract harmful insects and spread a dirty smell.22: The production of biomass should be restricted to within the EU to be able to control sustainable preconditions with regard to society,economy and the environment.

Disagree (2): 35: There is no use to develop niche markets; in the end we need large-scale biomass applications, and that niche won't help to reach that.33: Given the pace of the development towards more efficient cars, maybe we don't need biofuels for transport.9: Biomass should be used only for electricity production and heat supply, not for transport fuels.

Disagree: Other distinguishingstatements

44: The Netherlands are strong in knowledge development in the area of biomass technology.2: If biofuels are being stimulated in the EU, this will definitively result in negative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.52: Over the whole biomass chain, there should be a positive score as regards economic profit, energy- and CO2-balance.

Relevant quotes frominterviews

“Do today what you can do today. Do not postpone it because you are just winding around the plans and trying to achieve the optimal”“One person favors natural gas, the other bio-ethanol. Factories that make diesel engines ever cleaner, more efficient, and better. In which sootfilters are installed. In which NOx filters are installed. You see that sort of things increasing. So I think it will be a mix of different types”

Most distinguishing statements printed in italics.

Table 5Statements that received the highest positive (agree score (11) and (10)) and highest negative scores (disagree score (1) and (2)) for perspective 5.

Agree (11): 52: Over the whole biomass chain, there should be a positive score as regards economic profit, energy- and CO2-balance.7: Biomass is a temporary solution; in the end, solar and wind should be the main energy sources.

Agree (10): 51: In the long run, biofuels will compete on the world market with fossil fuels.60: We should concentrate on the use of residuals for biofuel production.37: The precarious Dutch energy policy has lead to stagnation of the market development concerning biomass.

Agree: Other distinguishingstatements

55: Technology development is the key to large-scale use of biomass, not an active subsidy policy.

Disagree (1): 9: Biomass should be used only for electricity production and heat supply, not for transport fuels.54: Public resistance is an obstacle for local biomass applications.

Disagree (2): 33: Given the pace of the development towards more efficient cars, maybe we don't need biofuels for transport.23: The cultivation of energy crops in the Netherlands will make the landscape monotonous, attract harmful insects and spread a dirty smell.45: Dutch farmers will not benefit from a growing use of biomass.

Disagree: Other distinguishingstatements

2: If biofuels are being stimulated in the EU, this will definitively result in negative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.31: Too much money goes to research, and too little to implementation in the market.

Relevant quotes frominterviews

“The input that we use for the process to make biomass is about just as high as what comes out as energy. The gain is nil. Moreover, it issubsidized. If you realize this, it is actually money thrown away, the subsidy”“Number 1 is efficiency, dealing more efficiently with energy. (…) So if we focus only on CO2 neutral things are not going well. Then companieswill just buy biomass, but therefore use twice as much energy”“I see that there aremany organic waste flows in the Netherlands that we all can use as a source of energy. That is amongst others dung. (…).Wehave much [household] waste, from vegetables, fruit and garden, that is composted at the moment because we cannot get rid of it. There is anoverproduction of compost. There are all kinds of industrial organic waste flows that you can use very easily as a source of energy.”“[The fossil energy industry] exists already for such a long time, as a result of which their operational costs are much lower than for biomass.They commercialized all their by-products. If biomass wants to compete economically in a healthy way, technology development should takeplace.”

Most distinguishing statements printed in italics.

585E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

for each factor a table that lists the ten statements that score highestand lowest on that factor, the most distinguishing statements(printed in italics),9 and some relevant quotes from the interviews.We briefly discuss each perspective. The section wraps up with ashort discussion of similarities and differences between theperspectives.

6.1. Perspective 1: “Keep all options open"

This perspective focuses on knowledge development. Genericclaims about the sustainability of biomass applications are notpossible, because the sustainability of an application is very muchdependent on the specific situation. Therefore, it does not make senseto exclude specific options in advance, or to embrace others.Biorefinery (refining biomass in order to use all valuable elements

9 Arrays of differences between z-scores of statements on factors above |2.5| wereaccepted as most distinguishing.

from the biomass) is seen as a promising development. Table 1 showsthe statements with the highest positive and negative scores for thisperspective, the distinguishing statements, and some relevant quotesfrom the interviews.

6.2. Perspective 2: “Hit the brakes"

This perspective is very skeptical about the possibilities ofsustainable biomass applications and calls for a pause. A growinginternational biomass-market increases the risks for developingcountries, with regard to environment, socio–economic situation,human rights and food supply. At the moment, there is no biomassthat is sustainable for people, planet and profit. As long as we cannotguarantee sustainable biomass, we should halt the development ofnew applications. Table 2 shows the statements with the highestpositive and negative scores for this perspective, the distinguishingstatements, and some relevant quotes from the interviews.

586 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

6.3. Perspective 3: “Support small-scale innovative initiatives"

The third perspective focuses mainly on small-scale, decentralizedapplications in the Netherlands. Initiatives by small innovativeentrepreneurs are hard to get off the ground, because the Dutchgovernment mainly has an eye for the large energy companies.However, we should not expect innovations from these companies,because they benefit from maintaining the existing fossil fuel-basedsystem. We should not keep putting money in research, but inimplementation. Table 3 shows the statements with the highestpositive and negative scores for this perspective, the distinguishingstatements, and some relevant quotes from the interviews.

6.4. Perspective 4: “Security of supply with global, certified, 2ndgeneration biomass"

This perspective has a strong market orientation. The mostimportant incentive for the development of biomass applications isthe replacement of fossil fuels (security of supply). This perspective isoptimistic about the potential of biomass, especially the 2ndgeneration biomass,10 and states as a condition that the sustainabilityof biomass is guaranteed by means of a certification system. Table 4shows the statements with the highest positive and negative scoresfor this perspective, the distinguishing statements, and some relevantquotes from the interviews.

6.5. Perspective 5: “Efficiency the goal, biomass a means?"

According to this perspective, we should not overestimate thepotential of biomass. In the future, other renewable sources (e.g. solar,wind) will be better suited for our energy supply, because theavailability of those sources is larger. We should be critical about thesustainability of biomass applications: the whole chain should betaken into account when determining whether there is a positiveenergy balance. Energy-efficiency is key. Technology andmarket havenot sufficiently been developed. Table 5 shows the statements withthe highest positive and negative scores for this perspective, thedistinguishing statements, and some relevant quotes from theinterviews.

6.6. Perspective 6: “Just do it, step by step"

This perspective is pragmatic. It underlines that we cannot know atthis moment what will be the best option in the future. This meansthat we should act now with the knowledge that we have, instead ofpostponing actions. All options should be kept open; there should be abroad range of applications. The role of entrepreneurs is veryimportant in this perspective. Table 6 shows the statements withthe highest positive and negative scores for this perspective, thedistinguishing statements, and some relevant quotes from theinterviews.

6.7. Similarities and differences between perspectives

To enhance our understanding of the perspectives and the relationbetween perspectives, we also calculated correlations between the

10 There is no commonly shared definition of 2nd generation biomass. An often-useddefinition in the Netherlands originates from the Dutch GAVE program (support fortechnological development of 2nd generation technologies) is based on the amount ofCO2 emission reduction: 1st generation biofuels achieve CO2 emission reductions ofaround 30–50% compared to fossil fuels and 2nd generation biofuels achieve a CO2

reduction of 80% or more. However, others define 2nd generation on the basis of thebiomass source that is being used: 2nd generation biofuels are made from residuals,and 1st generation from cultivated biomass. Others define 2nd generation biomass aslignocellulose (e.g. woody material), whether or not combined with a specific type oftechnology to convert the biomass into a fuel.

factor scores (with the software program PQMethod). The higher thecorrelation between two factor scores, the more similarities there arebetween two perspectives. Based on these correlations we identifiedwhich perspectives are most similar, and we interpreted thesimilarities based on the descriptions of the perspectives and therelevant statements (see Section 4). Table 7 shows the correlationsbetween the factor scores.

The correlations show that perspectives 4 (‘Security of supply withglobal, certified, 2nd generation biomass’) and 5 (‘Efficiency the goal,biomass a means?’) are most alike (r=0.55). Both see biomass as acommodity in a market in which it will eventually compete with fossilfuels, but on the condition that biomass applications have a positiveenergy balance. Perspective 4 is however more positive about thequestion as to whether this will be feasible. This perspective seesthe solution in 2nd generation, certified biomass. Perspective 5 doubtsthe feasibility of a positive energy balance, as well as the potentialavailability of biomass. According to perspective 5, technology andmarket have not been sufficiently developed, which has negativeimplications on the efficiency of biomass applications.

Both perspectives correlate with perspective 1 (‘Keep all optionsopen’). Perspectives 1 and 4 are both optimistic about the potential ofbiomass to contribute to a sustainable energy-system (perspective 5 ismore critical about this), but perspective 1 is more knowledge-oriented, and perspective 4moremarket-oriented. Another differenceis that perspective 1 calls for keeping all options open, whereasperspective 4 focuses only on 2nd generation, certified biomass.Perspectives 1 and 5 share a focus on residuals as a biomass sourceand the attention for technology development. However, perspective5 is more critical about the efficiency of biomass applications andfocusesmainly on knowledge development in themarket (rather thanin science).

Perspective 2 (‘Hit the brakes’) shows the lowest correlations withall other perspectives. It is the most critical perspective.

Lastly, perspective 3 (‘Support innovative initiatives’) and 6 (‘Justdo it, step by step’) are correlated. These perspectives are similar inthat they focus on entrepreneurship, and putting things into practicenow. Perspective 6 takes a more pragmatic, and less ideological stancethan perspective 3. Perspective 3 focuses on small-scale, decentralizedapplications in the Netherlands, whereas perspective 6 does not wantto make a choice for a specific scale and type of application. Inaddition, perspective 3 is very critical about the role of the Dutchgovernment and policy; this does not seem to play a very importantrole for perspective 6.

Based on the correlations displayed in Table 7 it seems thatperspective 2 is most different from the other perspectives, as it haslow correlations to the other perspectives. At the same time, it is aperspective that is well represented in the dominant policy debate onbiomass in the Netherlands; it is not an unfamiliar or marginalperspective. Perspectives 3 and 6 however–although they may appearstronger in a statistical sense as expressed by their correlation–reflectmarginal perspectives. These perspectives are not frequentlyexpressed in the dominant policy debate in the Netherlands and assuch they present relatively unknown, or unfamiliar ideas. We wouldlike to emphasize that it still remains an empirical question whetherand how stakeholders with divergent perspectives actually differ in

Table 7Correlations between factor scores; the higher the correlation between two factors, themore similarity between two perspectives (factors).

Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6

Factor 1 1.00 0.11 0.37 0.50 0.46 0.43Factor 2 0.11 1.00 0.17 0.26 0.39 −0.02Factor 3 0.37 0.17 1.00 0.34 0.40 0.45Factor 4 0.50 0.26 0.34 1.00 0.55 0.26Factor 5 0.46 0.39 0.40 0.55 1.00 0.44Factor 6 0.43 −0.02 0.45 0.26 0.44 1.00

Table 8Number of defining sorts per actor type and per perspective.

P1 P2 P3 P4 P5 P6 Total peractor type

Keep alloptions open

Hit thebrakes

Support small-scaleinnovative initiatives

Security of supply with global,certified, 2nd generation biomass

Efficiency the aim:biomass a means?

Just do it,step by step

Knowledge institutes and academia 3 2 2 2 9(Energy) companies, sector organizations 2 2 2 3 9Small/medium sized enterprises(incl energy consultants)

1 2 2 1 2 8

NGOs 7 2 9National government 1 2 3Regional/local government 1 1 3 5Total per perspective 7 14 3 7 7 5 43

587E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

terms of their preferences, or whether they can agree on a particularcourse of action. This question can only be resolved in the dialogueitself.

7. Q methodology to select participants

The perspectives that resulted from the Q analysis were used toselect participants for the dialogue. For practical reasons, we couldhave only thirty from seventy-five respondents taking part in theBiomass Dialogue. Therefore, the second step in our approach (afterthe Q sort analysis) was to select forty out of seventy-five people(taking about 10 cancellations into account). For this second step weused the results of Q methodology. We calculated factor loadings foreach respondent. The higher the factor loading of a person on a factor(perspective), the more that person's Q sort resembles the archetypalsort for that factor, and hence, the more that person shares thisparticular perspective. This resulted in an overview of the respon-dents and the perspectives they most and least adhered to. Then, foreach perspective, we identified the persons that loaded most stronglyon that perspective. Furthermore, we identified the respondents thatshowed large similarities with each of the perspectives, on the basis ofthe interview data. This resulted in a list of forty people whorepresented the six perspectives in a balanced way. These were thepersons that we invited to take part in the Biomass Dialogue.11

8. Comparison of selection based on Q perspectives and onactor type

In Section 2 we argued in favor of sampling based on perspectives,and against random sampling, stratified sampling based on demo-graphic variables, and based on affiliation. In order to assess the valueof Q methodology for stakeholder selection, we empirically comparethe stakeholder selection discussed above with a hypotheticalselection based on affiliation. This is possible as we know from eachrespondent what his or her affiliation is.12 Stakeholder selection basedon affiliation is a commonly used approach for stakeholder dialogues.In this section, we will investigate the (implicit) assumptionunderlying this selection procedure, i.e. whether stakeholder affilia-tion (actor type) is a good proxy for perspective.

The six perspectives described above can be interpreted as sixsalient perspectives in the discourse on biomass. Respondents' Q sortscorrelate to a greater or lesser extent with these perspectives. Table 8lists the number of respondents with a ‘defining’ sort on each of the

11 In the interviews we asked the respondents if they would be willing to take part inthe Biomass Dialogue. Self-evidently, their answer on this question was taken intoaccount.12 We do not have socio-demographic information of the respondents. If we had hadthis, also a comparison with a stakeholder selection procedure based on demographicvariables would have been possible.

perspectives. A person's sort is defining for a perspective when thatperson scores significantly on that perspective andnot on others.13 Therespondents are grouped by actor type, i.e. general categories ofaffiliation: knowledge institutes and academia, energy companies andbranch/sector organizations, small and medium sized enterprises,NGOs, national government, and regional and local government.

Table 8 shows that within one actor type different perspectives canbe identified. This means that actor types are not homogeneous withregard to perspectives. For example, respondents from knowledgeinstitutes and academia load on perspectives 1, 2, 4 and 5. In addition,on each perspective respondents from different types of actors appear.For example, on perspective 1 there are respondents from knowledgeinstitutes and academia, (energy) companies and sector organizations,small/medium sized enterprises, and national government.

Table 8 shows only the defining sorts, i.e. the respondents who arein significant agreement with a perspective. However, even when arespondent is not defining a perspective, he or she can still supportthe perspective to a certain extent, which is indicated by the factorloading. To further investigate to what extent each of the perspectivesis represented within actor types, we calculated the average factorloadings per actor type on each of the six perspectives. Factor loadingscan be interpreted as a measure of agreement. The higher a factorloading, the more a respondent agrees with (shares) the particularperspective. For this analysis all data (seventy-five factor loadings onsix perspectives) can be taken into account, and it therefore gives abetter image of how heterogeneous or homogeneous actor types arein terms of perspectives on energy from biomass than Table 8, whichis based only on defining sorts (and hence includes low frequencies).The results of the analysis are displayed in Fig. 2.

The six axes represent the six perspectives. Each line represents anactor type. The further from the centre a data point (representing anactor type) is positioned on an axis, the higher the average factorloading on that factor for that actor type is. The results displayed inFig. 2 underline the findings displayed in Table 8: actor types arerather heterogeneous in terms of perspectives. After all, the coloredlines are pretty much scattered over the web.

Although our results do not allow for making any definitestatements on the degree of support for the perspectives amongactor types, Fig. 2 allows for making some tentative observations asregards the congruence of actor types and perspectives. The mostnotable observation concerns NGOs. NGOs have a high averageloading on perspective 2 (it is so high that it is even positioned outsidethe figure (0.49)), and low loadings on all the other perspectives,especially on perspective 6 (‘Just do it, step by step’). This latter resultis not surprising, since perspective 2 calls for a moratorium on the useof biomass for energy as long as sustainability cannot be guaranteed.Perspective 6 however calls for learning-by-doing, and argues that we

13 Thirty-two respondents do not load significantly on one factor, but load (non-significantly) on more than one factor. Hence, they are ‘a mix of perspectives’.

Fig. 2. Distribution of average factor loadings per actor type across the six perspectives; 1: Keep all options open, 2: Hit the brakes, 3: Support small-scale innovative initiatives,4: Security of supply with global, certified, 2nd generation biomass, 5: Efficiency the goal: biomass a means?, 6: Just do it, step by step.

588 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

will not be able to develop sustainable applications as long as we donot act. These are opposite calls.14 So, NGOs appear to be homoge-neous with regard to their perspective, in that they are stronglydominated by perspective 2 (‘Hit the brakes’).

Also for the other actor types some observations can be made.Regional and local governments load mainly on perspectives 1, 3 and5, and notably low on perspective 4. As the focus of perspective 4(‘Security of supply with global, certified, 2nd generation biomass’) ison large-scale, global biomass chains, it is not surprising that thisperspective does not appeal to government officials active at theregional and local level. As regards the results for (energy) companiesand sector organizations, the low average factor loading on perspec-tive 2 is notable. Not surprisingly, a critical perspective that stressesnegative impacts of biomass applications for developing countriesdoes not prevail among actors of this type.

Then, for knowledge institutes and academia, there seems to be arelative low loading on perspective 3. This is interesting, as it mightindicate that knowledge institutions in theNetherlands do not interact ona regular basis with small and medium sized innovative firms. It ishowever not so surprising that actors from research institutes do not loadstrongly on a perspective that argues that there is enough knowledgealready. According to perspective 3, it is time to put the knowledge intopractice, rather than promoting more academic research.

Finally, the results for small and medium enterprises (includingenergy consultants) show a rather heterogeneous picture. Loadingsare relatively low for perspectives 2 and (to a lesser extent) for

14 In general, this seems to be reflected by the ordering of the organization-types onperspectives 2 and 6: on perspective 2 NGOs load most strongly, then nationalgovernment, knowledge institutes and academia, regional and local government, smalland medium sized enterprises, and the lowest loading is for (energy) companies andsector organizations. For perspective 6, this ordering is reversed, except for the rank ofsmall and medium sized enterprises.

perspective 6. Actors from small and medium enterprises probablyhave a low loading on perspective 2 for similar reasons as the (energy)companies and sector organizations. In addition, these actors areoptimistic about small-scale, decentralized biomass applications(perspective 3). As perspective 2 is focused on large-scale, globalbiomass chains, the skepticism of perspective 2 may not apply tothe small-scale, decentralized biomass applications. The low loadingon perspective 6 for actors from small and medium enterprises(including energy consultants) is actually surprising, as two of thesorts that defined this perspective were from people from this actortype (see Table 8), and because perspective 6 is an entrepreneurialperspective. This underlines the heterogeneity of this actor type.

In conclusion, the analysis shows that a stakeholder selectionbased on perspectives is likely to result in a different groupcomposition than a selection based on affiliation. Actor types arerather heterogeneous in terms of their perspectives on biomass. Thisimplies that actor type is not a good proxy for perspective. However,the analysis also shows that some actor types aremore heterogeneousthan others (most notably NGOs are rather homogeneous). Someperspectives (in particular perspective 2) are dominated by specifictypes of organizations (NGOs). The finding that particular actor typesare ‘over-represented’ on particular perspectives, and ‘under-repre-sented’ on others might indicate that these actors are unfamiliar withsome of the perspectives. This underlines the importance oforganizing stakeholder dialogues that facilitate mutual learning, i.e.the interaction between stakeholders with different perspectives andfrom different organizational networks.

9. Further use of the Q results in the Biomass Dialogue

Setting-up a dialogue process obviously requires more than only agood stakeholder selection procedure. In order to facilitate problemstructuring, the six perspectiveswere, apart from stakeholder selection,

589E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

also used for structuring the Biomass Dialogue. For this purpose, andalso tovalidate ourQanalysis, the six perspectiveswerepresented to theparticipants in the first workshop. The presentation also included eachparticipant's position with respect to the six perspectives. In general,participants recognized the perspectives. They indicated that thepresentation of perspectives clarified the complexity of opinions, claimsand arguments. It helped them to better understand other people'sstatements and ideas. Most participants also recognized their positionwith respect to the different perspectives; some of them indicated thatthey felt being a mix of two perspectives. They indicated in theevaluation after theworkshops that theway the perspectiveswere usedthroughout the dialogue contributed to a constructive and opendialogue. The perspectives were for instance used to form subgroupsaround the perspectives: subgroups of ‘like-minded’ people for specifictasks and exercises in the dialogue, e.g. to elaborate upon argumenta-tions for the sustainability of specific biomass chains. Hence, diversity ofperspectives was emphasized throughout the dialogue, as a conse-quence of which participants felt that they were ‘allowed’ to disagree.This underlines the importance of not forcing a consensus in deli-berative processes on issues that are so complex and controversial asbiomass. Furthermore, it shows the importance of the articulation ofperspectives for problem structuring processes. In that sense, this studyprovides an answer on the question posed by Davies et al. (2005: p612)whether the intervention with Q methodology before the dialogue willactually assist or impede the subsequent process by emphasizingparticular issues in the minds of participants. Of course, by applying Qmethodology we imposed some frame on the participants. Ourimpression is however that participants openly engaged in the Q sort,as well as in the dialogue, and that the dialogue only benefited from theidentification of the Q perspectives and their use in structuring thedialogue. This impression is supported by the evaluations from theparticipants.

10. Conclusion and discussion

For the Biomass Dialogue, Qmethodology has proved to be a usefulmethod for identifying and selecting stakeholders with salientperspectives, i.e. a high loading on one of the six perspectives. Thisenabled a balanced representation of the variety of perspectives in theBiomass Dialogue.

Importantly, this study shows that stakeholder selection based onQ methodology is likely to result in a group composition that doesmore justice to the diversity of views, claims and interests asexperienced by the respondents themselves than selection based onactor type. Hence, in order to include the diversity of perspectives in adialogue, a selection procedure based on Q methodology is moreadequate than selection on the basis of actor type.

The biomass issue is a very controversial and complex issue thatinvolves a huge variety of ideas, preferences, opinions, knowledge claimsand values. This is reflected by the relatively high number of identifiedperspectives (6, compared to 2 to 4 inmost Q studies) and the relativelylow total variance explained. Factor analysis was as a result not astraightforward task, as it was difficult to identify definitive clusters of Qsorts. Davies et al. (2005) also point to this difficulty for complex anduncertain issues. They argue that this should be considered as reflectingthe difficulty of identifying different perspectives, rather than as a directlimitation of Q methodology for identifying perspectives and selectingstakeholders. Ideas and opinions on complex and uncertain issues aredynamic and can change over time. Interestingly, Q methodology canalso be used to analyze changes in perspectives (see Cuppen,forthcoming). The observation that factor analysis was not a straight-forward task raises some new questions however with regard to theprocedure for identifying and selecting factors. We argued that thediversity of perspectives should be reflected in the dialogue. Whendifferent procedures for selecting factors (e.g. based on Eigenvalues,scree plot, or on number of significant loadings) result in different

numbers of factors, this has consequences for thewaydiversity is treatedand covered in the dialogue. This issue deserves attention in furtherwork.

Yet if we adopt the articulation of diversity as the starting point forthe design of stakeholder dialogues on complex issues, there are someserious implications. An extensive preparation phase is necessary toidentify stakeholders, to identify the diversity of perspectives, and toconduct the stakeholder selection process (e.g. by means of Qmethodology). In the dialogue processes that we have organizedusually twenty to thirty stakeholders participate, which means that atleast forty to fifty respondents are needed for the Q sorts. This requiresa time (and money) investment. In the Biomass Dialogue forexample, the preparation phase (including stakeholder identification,Q interviews, Q analysis, stakeholder selection) lasted 7 months,whereas the actual dialogue lasted 5 months. As this study showed, anextensive preparation phase, which enables the inclusion and articu-lation of diverse perspectives, is worthwhile.

Stakeholder selection is a necessary condition, but not a guaranteefor a good dialogue process. Besides the design of the dialogue, thereare other things that can impede or facilitate the articulationand evaluation of the diversity of perspectives in the process. Animportant issue here is that methods can be used to determine whoshould be invited to take part in the dialogue, but that they obviouslyprovide no guarantee as to whether people will show up and, if so, asto whether they are willing to engage in an open exploration ofdivergent knowledge claims and viewpoints. Our findings from theBiomass Dialogue showed that the focus on articulation of diversity inthe design of the dialogue contributed to an open and constructiveatmosphere. Rather than forcing participants to reach consensus, theywere allowed to disagree. The use of the Q perspectives in thedialogue helped participants to understand the complexity of thebiomass issue, and to engage in an open exploration of views,arguments and ideas (see also Cuppen, forthcoming).

Acknowledgments

We want to thank the Research Program ‘Investing in KnowledgeInfrastructure’ (BSIK), Climate Changes Spatial Planning, IC8 and ME4(http://www.klimaatvoorruimte.nl), and NWO (project title “Strategiesfor implementing sustainable transition trajectories in the transportsector”) for supporting the research on which this paper is based. Inaddition, we thank Joop de Boer (IVM-VU), Cees Midden (EindhovenUniversity of Technology), Frans Berkhout (Institute for EnvironmentalStudies, VU), Rik Leemans (Wageningen University and ResearchCentre) and Mattijs Taanman (DRIFT, Erasmus University Rotterdam)and two anonymous reviewers for their valuable comments to earlierversions of this paper.

Appendix A. Q statements

1

If the complete lifecycle is taken into account in the analysis, biofuels do notreduce as much greenhouse gas emissions as was hoped for.

2

If biofuels are being stimulated in the EU, this will definitively result innegative impacts on environment, socio-economic conditions, violation ofhuman rights and food shortages in developing countries.

3

If there are no entrepreneurs who want to experiment with biofuelapplications, nothing will happen.

4

If we want, we can drive clean and fly clean now with biofuels. 5 Within the Netherlands and the EU, the production of rapeseed oil should be

taken seriously.

6 Bio-ethanol is more promising the biodiesel, because there are more

possibilities for improving the process and the efficiency with ethanol.

7 Biomass is a temporary solution; in the end, solar and wind should be themain

energy sources.

8 Algae are the biomass source of the future. 9 Biomass should be used only for electricity production and heat supply, not for

transport fuels.

(continued on next page)

Appendix A (continued) Appendix A (continued)

590 E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

10

Biorefinery offers huge opportunities for small-scale and regional sustainabledevelopments.

11

Criteria will not prevent that in the future there will be a number of large agro-companies, that supply biomass without taking social and environmentalinterests sufficiently into account.

12

Tax on fossil fuels should be increased. 13 The most important obstacle for biofuels is not the conversion, but the

uncertainty in the future supply of biomass.

14 The availability of private capital is at the moment not a limiting factor for the

development of a large-scale bio-based energy supply.

15 The competition between food, feed, and fuel will have a negative financial

impact on people.

16 The European blending-targets, such as 5.75% in 2010 and 10% in 2020, require

significant import volumes from countries outside the EU.

17 The Dutch government mainly has an eye for large companies; there is not

enough attention and support for small, innovative players.

18 The Dutch government should give tax exemption for biofuels. 19 The potential of degraded and marginal grounds is so large that it can mean an

economic impulse in rural areas.

20 The issue of unsustainable land-use, for example in South America, Africa and

South-east Asia also exist without biomass production.

21 The production of biomass is only sustainable if it contributes to the social–

economic development of the local community.

22 The production of biomass should be restricted to within the EU to be able to

control sustainable preconditions with regard to society, economy and theenvironment.

23

The cultivation of energy crops in the Netherlands will make the landscapemonotonous, attract harmful insects and spread a dirty smell.

24

The cultivation of energy crops contributes to a colorful landscape and to thebee-population.

25

The time of large-scale is over; we need flexible, decentralized energy systems. 26 Every form of subsidy on imported biomass should be stopped. 27 Cultivation of energy crops is not favorable because manure and irrigation are

needed.

28 Cultivation of energy crops for the 2nd generation biofuels will cause much

less problems in developing countries than for the 1st generation crops.

29 There is a need for generic policy aimed at all clean and efficient vehicles,

instead of a policy that is aimed specifically on biofuels.

30 If the European fuel has to meet higher standards than the American, this

results in unfair competition.

31 Too much money goes to research, and too little to implementation in the

market.

32 Biomass delivers an important contribution to the security of supply, namely

less dependency on geopolitical sensitive areas, and a higher degree of self-support for the EU.

33

Given the pace of the development towards more efficient cars, maybe wedon't need biofuels for transport.

34

The distinction between 1st and 2nd generation is not as black-and-white as isoften posed.

35

There is no use to develop niche markets; in the end we need large-scalebiomass applications, and that niche won't help to reach that.

36

Stimulating biofuels has more to do with agricultural policy than withenvironmental policy.

37

The precarious Dutch energy policy has lead to stagnation of the marketdevelopment concerning biomass.

38

Biofuels can only succeed if the government subsidizes until the end of times. 39 Import of end- or half products is preferable to import raw biomass. 40 In the formulation of criteria for certification of biomass also stakeholders from

the South should be involved.

41 In the Netherlands, biofuels are discriminated compared to fossil fuels when it

concerns the calculation of CO2-emissions.

42 Small-scale energy production with biomass can give a huge impulse to

security of supply in developing countries.

43 The Netherlands can supply an important part of the houses with sustainable

energy by means of local residuals.

44 The Netherlands is strong in knowledge development in the area of biomass

technology.

45 Dutch farmers will not benefit from a growing use of biomass. 46 Political pressure, at minimum on the EU level, is needed to make

sustainability criteria function.

47 Because stakeholders did not succeed in forming a successful lobby, there is

insufficient support for the development of biomass technologies.

48 Because the CO2-reduction potential of 1st generation biofuels is limited, we

should not invest in 1st generation, but in 2nd generation.

49 Because, as a small country with limited means, we are forced tomake choices,

the government should support only the most favorable trajectories.

50 Entrepreneurs are not only competitors: cooperation is required to learn from,

and support each other.

51

In the long run, biofuels will compete on the world market with fossil fuels. 52 Over the whole biomass chain, there should be a positive score as regards

economic profit, energy- and CO2-balance.

53 First, try to make high-quality products from a biomass source, and make

energy from what is left.

54 Public resistance is an obstacle for local biomass applications. 55 Technology development is the key to large-scale use of biomass, not an active

subsidy policy.

56 2nd generation biofuels profit from stimulating the 1st generation (E85 and

diesel variant E95) now.

57 2nd generation biofuels are for the time being not ready for large-scale

application.

58 Consumers and end-users are increasingly interested in biofuels. 59 We can never compete on price with biofuel that are made in developing

countries.

60 We should concentrate on the use of residuals for biofuel production.

References

Ackhoff, R., 1974. Redesigning the Future. John Wiley & Sons, New York.Agterbosch, S., 2006. Empowering Wind Power. On Social and Institutional Conditions

Affecting the Performance of Entrepreneurs in the Wind Power Supply Market inthe Netherlands. Utrecht University, Utrecht.

ANP. Kabinet beteugelt inzet biobrandstoffen. Algemeen Nederlands Persbureau . 9-10-2008. (Newspaper).

Arnstein, S.R., 1969. A ladder of citizen participation. Journal of the American Instituteof Planners 35, 216–224.

Barry, J., Proops, J., 1999. Seeking sustainability discourses with Q methodology.Ecological Economics 28, 337–345.

Breukers, S. 2006. Changing institutional landscapes for wind power implementation. Ageographical comparison of institutional capacity building: The Netherlands,England and North-Rhine Westphalia. Amsterdam: University of Amsterdam.

Breukers, S., Wolsink, M., 2007. Wind power in changing institutional landscapes: aninternational comparison. Energy Policy 35, 2737–2750.

Brodbeck, F.C., Kerschreiter, R., Mojzisch, A., Frey, D., Schulz-Hardt, S., 2002. Thedissemination of critical, unshared information in decision-making groups: theeffects of pre-discussion dissent. European Journal of Social Psychology 32, 35–56.

Brown, S.R., 1980. Political Subjectivity: Applications of Q Methodology in PoliticalScience. Yale University Press, New Haven and London.

Clarke, A.H., 2002. Understanding sustainable development in the context of otheremergent environmental perspectives. Policy Sciences 35, 69–90.

Coglianese, C., 1999. The limits of consensus. The environmental protection system intransition: toward a more desirable future. Environment 41, 1–6.

Collins, H.M., Evans, R., 2002. The third wave of science studies: studies of expertise andexperience. Social Studies of Science 32, 235–296.

Crawford, M., Rutter, D., Thelwall, S., 2003. User Involvement in Change Management.NCCSDO, London.

Cuppen, E. Putting perspectives into participation: Constructive Conflict Methodologyfor stakeholder dialogues. Forthcoming.

Davies, B.B., Hodge, I.D., 2007. Exploring environmental perspectives in lowlandagriculture: a Q methodology study in East Anglia, UK. Ecological Economics 61,323–333.

Davies, B.B., Blackstock, K.L., Rauschmayer, F., 2005. ‘Recruitment’, ‘composition’, and‘mandate’ issues in deliberative processes: shouldwe focus on arguments rather thanindividuals? Environment and Planning C: Government and Policy 23, 599–615.

Dryzek, J.S., Berejikian, A., 1993. Reconstitutive democratic theory. American PoliticalScience Review 87, 48–60.

Dryzek, J.S., Niemeyer, S., 2008. Discursive representation. American Political ScienceReview 102, 481–492.

Dunn, W.N., 1988. Methods of the second type: coping with the wilderness ofconventional policy analysis. Policy Studies Review 7, 720–737.

Dunn, W.N., 2001. Using the method of context validation to mitigate type III error inenvironmental policy analysis. In: Hisschemöller, M., Hoppe, R., Dunn,W.N., Ravetz,J.R. (Eds.), Knowledge, Power and Participation in Environmental Policy Analysis.Transaction Publishers, New Brunswick and London, pp. 417–436.

Ellis, G., Barry, J., Robinson, C., 2007. Many ways to say ‘no’, different ways to say ‘yes’:applying Q-methodology to understand public acceptance of wind farm proposals.Journal of Environmental Planning and Management 50, 517–551.

Energietransitie, 2008. Biomassa, hot issue. Slimme keuzes in moeilijke tijden.SenterNovem, Utrecht.

Goodin, R.E., Dryzek, J.S., 2006. Deliberative impacts: the macro-political uptake ofmini-publics. Politics & Society 34, 219–244.

Gregory, R., MacDaniels, T.L., Fields, D., 2001. Decision aiding, not dispute resolution:creating insights through structured environmental decisions. Journal of PolicyAnalysis and Management 20, 415–432.

Hekkert, M.P., Suurs, R.A.A., Negro, S.O., Kuhlman, S., Smits, R.E.H.M., 2007. Functions ofinnovation systems: a new approach for analysing technological change.Technological Forecasting and Social Change 74.

Hisschemöller, M., 1993. De democratie van problemen. VU Uitgeverij, Amsterdam.Hisschemöller, M., 2005. Participation as knowledge production and the limits of

democracy. In: Maasen, S., Weingart, P. (Eds.), Democratization of Expertise?

591E. Cuppen et al. / Ecological Economics 69 (2010) 579–591

Exploring New Forms of Scientific Advice in Political Decision-Making. KluwerAcademic Publishers, Dordrecht, pp. 189–207.

Martin, G.P., 2008. ‘Ordinary people only’: knowledge, representativeness, and thepublics of public participation in healthcare. Sociology of Health & Illness 30, 35–54.

Hisschemöller, M., Hoppe, R., 2001. Coping with intractable controversies: the casefor problem structuring in policy design and analysis. In: Hisschemöller,M., Hoppe, R.,Dunn,W.N., Ravetz, J.R. (Eds.), Knowledge, Power and Participation in EnvironmentalPolicy Analysis. Transaction Publishers, New Brunswick and London, pp. 47–72.

Hoffman, L.R., 1959. Homogeneity of member personality and its effect on groupproblem-solving. Journal of Abnormal and Social Psychology 58, 27–32.

Hoffman, L.R., Maier, N.R., 1961. Quality and acceptance of problem solutions bymembers of homogeneous and heterogeneous groups. Journal of Abnormal andSocial Psychology 62, 401–407.

Huitema, D., Van de Kerkhof, M., Pesch, U., 2007. The nature of the beast: are citizens'juries deliberative or pluralist? Policy Sciences 40, 287–311.

Janis, I.L., 1972. Victims of Groupthink: Psychological Studies of Foreign Policy Decisionsand Fiascoes. Houghton-Mifflin, Boston.

Jehn, K.A., Northcraft, G.B., Neale, M.A., 1999. Why differences make a difference: a fieldstudy of diversity, conflict and performance in workgroups. Administrative ScienceQuarterly 44, 741–763.

Mason, R.O., Mitroff, I.I., 1981. Challenging Strategic Planning Assumptions: Theory,Cases and Techniques. Wiley, New York.

Mayer, I.S., 1997. Debating Technologies: A Methodological Contribution to the Designand Evaluation of Participatory Policy Analysis. Tilburg University Press, Tilburg.

McKeown, B., Thomas, D., 1988. Q Methodology. Sage Publications, Newbury Park,Beverly Hills, London, New Delhi.

Mitroff, I.I., Sagasti, F., 1973. Epistemology as general systems theory: an approach tothe design of complex decision-making experiments. Philosophy of the SocialSciences 3, 117–134.

Rittel, H., Webber, M., 1973. Dilemmas in general theory of planning. Policy Sciences 4,155–169.

Robbins, P., Krueger, R., 2000. Beyond bias? The promise and limits of Q method inhuman geography. Professional Geographer 52, 636–648.

Schmolck, P., 2002. Pqmethod. www.rz.unibw-muenchen.de/~p41bsmk/pqmethod/.Schön, D.A., Rein, M., 1994. Frame Reflection: Towards the Resolution of Intractable

Policy Controversies. Basic Books, New York.Schweiger, D.M., Sandberg, W.R., Ragan, J.W., 1986. Group approaches for improving

strategic decision making: a comparative analysis of dialectical inquiry, devil'sadvocacy and consensus. The Academy of Management Journal 29, 51–71.

Simon, H., 1973. The structure of ill-structured problems. Artifical Intelligence 4,181–201.

Stasser, G., Titus, W., 1985. Pooling of unshared information in group decision making:biased information sampling during discussion. Journal of Personality and SocialPsychology 48, 1467–1478.

Stephenson, W., 1935. Correlating persons instead of tests. Character and Personality 4,17–24.

Swedeen, P., 2006. Post-normal science in practice: a Q study of the potential forsustainable forestry in Washington State, USA. Ecological Economics 57, 190–208.

Van Asselt, M., Rijkens-Klomp, N., 2002. A look in the mirror: reflection on participationin Integrated Assessment from a methodological perspective. Global Environmen-tal Change 12, 167–184.

Van de Kerkhof, M., 2004. Debating Climate Change: A Study of Stakeholder Partici-pation in a Integrated Assessment of Long-Term Climate Policy in the Netherlands.Lemma, Utrecht.

Van de Kerkhof, M., 2006. Making a difference: on the constraints of consensus buildingand the relevance of deliberation in stakeholder dialogues. Policy Sciences 39,279–299.

Van Eeten, M., 2001. Recasting intractable policy issues: the wider implications of TheNetherlands civil aviation controversy. Journal of Policy Analysis and Management20, 391–414.

Webler, T., Tuler, S., Krueger, R., 2001. What is a good public participation process? Fiveperspectives from the public. Environmental Management 27, 435–450.

Wolsink, M., 2004. Policy beliefs in spatial decisions: contrasting core beliefs con-cerning space making for waste infrastructure. Urban Studies 41, 2669–2690.