Classifying Alternatives in SMAA Using Cross Confidence Factors

Risto Lahdelma University of Turku, Department of Information Technology, Lemminkäisenkatu 14, FIN-20520 Turku, Finland

Pekka Salminen University of Joensuu, Department of Business and Economics, P.O. Box 111, FIN-80101 Joensuu, Finland

Turku Centre for Computer Science TUCS Technical Report No 472 September 2002 ISBN 952-12-1037-0 ISSN 1239-1891

Abstract

Stochastic multicriteria acceptability analysis (SMAA) is a family of methods for aiding multicriteria group decision making. These methods are based on exploring the weight space in order to describe the preferences that make each alternative the most preferred one, or that give a certain rank for a specific alternative. The main results of the analysis are rank acceptability indices, central weight vectors and confidence factors for different alternatives. The rank acceptability indices describe the variety of different preferences resulting in a certain rank for an alternative; the central weight vectors describe the typical preferences favoring each alternative; and the confidence factors measure whether the criteria data is sufficiently accurate for making an informed decision.

In some cases, when the problem involves a large number of efficient alternatives, the analysis may fail to discriminate between them. This situation is revealed by low confidence factors. In this paper we develop so-called cross confidence factors, which are based on computing confidence factors for alternatives using each other’s central weight vectors. The cross confidence factors can be used for classifying efficient alternatives into sets of similar and competing alternatives. These sets are related to the concept of reference sets in Data Envelopment Analysis (DEA), but generalized for stochastic models. Forming these sets is useful when trying to identify one or more most preferred alternatives, or suitable compromise alternatives. The reference sets can also be used for evaluating whether criteria need to be measured more accurately, and at which alternatives the measurements should be focused. This may cause considerable savings in measurement costs. We demonstrate the use of the cross confidence factors and reference sets using a real-life example. Keywords: multicriteria decision support, stochastic multicriteria

acceptability analysis, cross confidence factor, Helsinki harbor

1. Introduction

In discrete multicriteria group decision-making problems the decision makers (DMs) want to choose one or a few ‘best’ alternatives. The choice depends on the DMs’ different valuations, which are normally represented through weights for the criteria. Often in real-life problems, neither the criteria measurements nor the DMs’ preferences are known precisely. Sometimes no preference information is available.

Stochastic Multicriteria Acceptability Analysis (SMAA) methods have been developed for aiding decision making in such problems. In SMAA, uncertain or imprecise criteria and preference information is represented through probability distributions. Instead of choosing the best alternative, the SMAA methods provide descriptive information about the acceptability of each alternative subject to different valuations. The analysis is based on exploring the weight space in order to describe the preferences that would make each alternative the most preferred one, or that would give a certain rank for a specific alternative. The main results of the analysis are rank acceptability indices, central weight vectors and confidence factors for different alternatives. The rank acceptability indices describe the variety of different preferences resulting in a certain rank for an alternative; the central weight vectors describe the typical preferences favoring each alternative; and the confidence factors measure whether the criteria data is sufficiently accurate for making an informed decision.

A number of different variants of the SMAA methods exist. In the original SMAA method by Lahdelma et al. (1998) the weight space analysis is performed based on an additive utility or value function and stochastic criteria data. SMAA-2 by Lahdelma and Salminen (2001) generalizes the analysis to apply a general utility or value function, to include various kinds of preference information and to consider holistically all ranks. SMAA-3 (Lahdelma and Salminen, 2002) is based on so-called pseudocriteria as in the ELECTRE III decision-aid (see e.g., Vincke 1992, Maystre et al. 1994). SMAA-D applies instead of a value function the efficiency score of Data Envelopment Analysis (DEA) in the analysis (Lahdelma et al. 1999). The SMAA-O method by Lahdelma et al. (2002a) extends SMAA-2 for treating mixed ordinal and cardinal criteria in a comparable manner. In SMAA-A the DMs’ preferences are modeled through so-called achievement scalarizing functions based on reference points (Lahdelma et al. 2002b). For real-life applications of different SMAA methods, see e.g., Lahdelma et al. (2000, 2002b), Makkonen et al. (2002), and Kangas & Kangas (2002).

An important advantage of the SMAA methods is that they can be used with both low and high quality information in the same problem. In particular, the information may be a mixture of precise information, uncertain cardinal information, ordinal information, and absent information. Obviously, in problems where the information is highly uncertain or imprecise, the method cannot always provide a clear recommendation. In such situations, multiple alternatives may appear clearly acceptable as indicated by the acceptability indices. Furthermore, multiple alternatives may be favored by similar preferences, as indicated by the central weight vectors. Finally, the most favorable

1

preferences of an alternative may fail to make it most preferred with sufficient certainty, which is indicated by a low confidence factor. Weak discrimination between the alternatives may be caused not only by overly imprecise information, but the fact that some of the alternatives indeed are too similar with respect to the different preferences. There is a clear need for more detailed analysis in situations where the discrimination between the alternatives is weak.

In this paper we introduce so-called cross confidence factors, which are based on computing confidence factors for alternatives using each other’s central weight vectors. The cross confidence factors can be used for identifying so-called reference sets for alternatives. The reference set for an efficient alternative consists of the similar and competing alternatives that obtain high cross confidence factors. The concept of reference sets is similar to that in Data Envelopment Analysis (DEA) (Charnes et al. 1978), but generalized for stochastic models. We present the new concepts as an extension of the SMAA-2 method. However, the new concepts apply directly to other variants of SMAA, with the exception of SMAA-3.

This paper is organized as follows. In section 2, we describe briefly the SMAA-2 MCDA method. In section 3, we introduce the cross confidence factors for classifying the efficient alternatives into sets of competing alternatives. In section 4, we demonstrate the use of the cross confidence factors and reference sets using real-life data from the Helsinki Harbor application (Hokkanen et al, 1999). The last section contains a discussion and conclusions.

2. The SMAA-2 method

The SMAA-2 method (Lahdelma and Salminen 2001) has been developed for discrete multicriteria group decision problems where neither criteria measurements nor weights are precisely known. SMAA-2 applies inverse weight space analysis to describe for each alternative what kind of preferences make it the most preferred one, or place it on any particular rank.

The decision problem is represented as a set of m alternatives {x1, x2, …, xm} to be evaluated in terms of n criteria. The DMs’ preference structure is represented by a real-valued utility or value function u(xi,w). The value function maps the different alternatives to real values by using a weight vector w to quantify DMs’ subjective preferences. Uncertain or imprecise criteria are represented by stochastic variables ξij with joint density function f(ξ) in the space X⊆Rm×n. The DMs’ unknown or partially known preferences are represented by a weight distribution with joint density function f(w) in the feasible weight space W. Total lack of preference information is represented in ‘Bayesian’ spirit by a uniform weight distribution in W, i.e., f(w) = 1/vol(W). The weight space can be defined according to needs, but typically, the weights are non-negative and normalized, i.e.,

W = {w∈Rn | w≥0 and }. (1) 11 =∑ =nj jw

2

The value function is then used to map the stochastic criteria and weight distributions into value distributions u(ξi,w). Based on the value distributions, the rank of each alternative is defined as an integer from the best rank (=1) to the worst rank (=m) by means of a ranking function

rank(i,ξ,w) = 1 + ∑ =

m

k 1ρ(u(ξk,w)>u(ξi,w)), (2)

where ρ(true) = 1 and ρ(false) = 0. Note that SMAA-2 does not require a cardinal utility function. An ordinal value function is sufficient for ranking the alternatives in formula (2). The analysis is then based on the properties of the stochastic sets of favourable rank weights

= {w∈W | rank(i,ξ,w) = r}. (3) )(ξWri

Any weight w∈ results in such values for different alternatives, that alternative x)(ξWri i

obtains rank r.

The first descriptive measure of SMAA-2 is the rank acceptability index , which measures the variety of different preferences that grant alternative x

rib

i rank r. It is the share of all feasible weights that make the alternative acceptable for a particular rank, and it is most conveniently expressed in percent. is computed numerically as a multidimensional integral over the criteria distributions and the favourable rank weights using

rib

. (4) ∫ ∫=X ξW

ξwwξ)(

)()(ri

ddffbri

The most acceptable (best) alternatives are those with high acceptabilities for the best ranks. Evidently, the rank acceptability indices are in the range [0,1] where 0 indicates that the alternative will never obtain a given rank and 1 indicates that it will obtain the given rank always with any choice of weights. Graphical examination of the rank acceptability indices is useful for comparing how different varieties of weights support each rank for each alternative. Alternatives with high acceptabilities for the best ranks are taken as candidates for the most acceptable solution. On the other hand, alternatives with large acceptabilities for the worst ranks should be avoided when searching for compromises - even if they would have fairly high acceptabilities for the best ranks.

The first rank acceptability index is called the acceptability index ai. The acceptability index is particularly interesting, because it is nonzero for efficient alternatives and zero for inefficient alternatives. The acceptability index not only identifies the efficient alternatives, but also measures the strength of the efficiency considering the uncertainty in criteria and DMs’ preferences. The acceptability index can thus be used for classifying the alternatives into stochastically efficient ones (ai>0) and inefficient or “weakly efficient” ones (ai zero or near-zero). The acceptability index can also be interpreted as the number of DMs voting for an alternative, assuming that the applied weight distribution represents the DMs’ preferences accurately. However, in practice,

3

this assumption may not be valid. The acceptability index should thus not in general be used for ranking the alternatives.

SMAA-2 defines also for each alternative a holistic acceptability index which aims to measure the overall acceptability of the alternative. The holistic acceptability index is defined as a weighted sum of the rank acceptabilities

∑ ==

m

rrir

hi ba

1α , (5)

where suitable meta-weights 1=α1≥α2≥…≥αm≥0 are used to model that the best ranks are preferred over the worst ranks. The holistic acceptability index is thus in the interval [0,1] and it is always greater than or equal to the acceptability index. Different ways to choose the meta-weights are discussed in Lahdelma and Salminen (2001).

The central weight vector is the expected centre of gravity (centroid) of the favourable first rank weights of an alternative. The central weight vector represents the preferences of a ‘typical’ DM supporting this alternative. The central weight vectors of different alternatives can be presented to the DMs in order to help them understand how different weights correspond to different choices with the assumed preference model.

is computed numerically as a multidimensional integral over the criteria distributions and the favourable first rank weights using

ciw

ciw

iX Wci addwwwffw

i∫ ∫= )(1 )()( ξ ξξ . (6)

The confidence factor is the probability for an alternative to obtain the first rank when the central weight vector is chosen. The confidence factor is computed as a multidimensional integral over the criteria distributions using

cip

(7) .)()(: 1

ξξξWwXξ

dfpi

ci

ci ∫ ∈∈=

The confidence factors measure whether the criteria data are accurate enough to discern the efficient alternatives. A low confidence factor indicates that even when applying the central weight vector, the alternative cannot with certainty be considered the most preferred one. The confidence factor can also be used together with the acceptability index for eliminating weakly efficient alternatives. If the acceptability index is very low (near-zero, <<1/m) and the confidence factor is low (less than, say, 5%), we can argue that such an alternative is very unlikely the most preferred by any DM. In contrast, a very high confidence factor (over 95%) indicates that with suitable preferences, the alternative is almost certainly the most preferred one.

Observe that confidence factors can be calculated similarly for any given weight vector. The confidence factor can be interpreted as the proportion of the stochastic criterion space that makes the alternative most preferred with the given weight vector. Figure 1 illustrates this in a two-criteria case with a linear value function: the criteria measurements for x1 and x3 are deterministic, the stochastic criterion values for x2 are

4

represented by a uniform distribution in a rectangular area, and wc is some arbitrary weight vector.

wc

x 1

p c2

x 2

x 3

Criterion 1

Crit

erio

n 2

Figure 1. Confidence factor as a proportion of the stochastic criterion space. The dotted line is the indifference curve of the linear value function corresponding to weight vector wc and passing through x3. Alternative x2 is the most preferred alternative in the dark area above the indifference curve.

3. Cross confidence factors

The SMAA methods do not always provide a clear recommendation in problems where the information is highly uncertain or imprecise, or where several alternatives are very similar with respect to the preferences. In such cases, several alternatives may obtain reasonably high acceptability for the best ranks, and no single alternative appears to be superior. Also, multiple alternatives may be favored by similar preferences, as indicated by the central weight vectors. Finally, the central weight vector of an alternative may fail to make it the most preferred one with sufficient certainty, as indicated by a low confidence factor. The failure to discriminate between the alternatives may be caused not only by too imprecise information, but the fact that some of the alternatives indeed are too similar with respect to the different preferences. There is a clear need for more detailed analysis in situations where the discrimination of the alternatives is too weak. For this reason we introduce so-called cross confidence factors and reference sets helping the DMs and analysts to understand the problem better.

The cross confidence factors are based on computing confidence factors for alternatives using each other’s central weight vectors. In particular, the cross confidence factor for alternative xi with respect to target alternative xk is computed as

(8) .)()(: 1

ξξξWwXξ

dfpi

ck

ki ∫ ∈∈=

5

The cross confidence factor is the probability for an alternative to obtain the first rank (considering the uncertainty in the criteria measurements) when central weight vector pf the target alternative is chosen. Therefore, the nonzero cross confidence factors identify which alternatives xi compete about the first rank with a given alternative’s xk central weight vector and how strongly they do it. Observe that the target alternative has to be efficient, or at least weakly efficient; otherwise its central weight vector is undefined. In principle, the cross confidence factors can of course be computed based on arbitrary weight vectors, e.g. weight vectors specified by the DMs.

The cross confidence factors have certain properties:

1. From (8) we can immediately see that the cross confidence factor equal to the original confidence factor .

iip

c

c

ip

2. The cross confidence factors are equal to the acceptability indices akip i

computed using precise preference information W = . kw

3. The sets of favorable first rank weights cover the entire weight space. Therefore, each central weight vector must belong to the set of favorable first rank weights of at least one alternative. We have thus

.11

≥∑ =

m

kkip (9)

Formula (9) holds as equality if no shared first ranks between alternatives occur. Ranks are almost never shared if the value function u(·) and the cumulative criteria distribution F(·) are continuous.

The cross confidence factors augment the basic analysis with useful new information in situations where SMAA discriminates the alternatives weakly. The original confidence factors can be used to detect that the discrimination is weak; the cross confidence factors pinpoint why the discrimination is weak. The alternatives that obtain high cross confidence factors based on the central weight vector of a target alternative are exactly those that cause the weak discrimination of the target alternative.

To alleviate the interpretation of the cross confidence factors, we define for each target alternative xk a reference set Rk. The reference sets are ordered stochastic sets of pairs

kipi,

)}(,...,1|),,({ ),( kmrprkiR krki

k == (10)

where m(k)≤m determines how many elements are included in each reference set and the index function i(k,r) orders the elements by their cross confidence factor into descending order

. (11) )(,...,2,:),( ),()1,( kmrpprki krki

krki =≥−

6

The ordering of the reference set makes it easy to quickly identify which alternatives are most severely competing about first rank with a given weight vector. The reference sets can also be visualized as column charts, where the height of each column corresponds to the cross confidence factor.

In principle, the reference sets could include all alternatives with a non-zero cross confidence factor. In practice it is convenient to restrict the reference sets to alternatives with the highest cross confidence factors. The suitable size of a reference set is context-specific. We suggest two alternative ways to determine how many alternatives to include.

We may include to a reference set all alternatives whose cross confidence factor exceed a given threshold δk and exclude the alternatives with too small cross confidence factors, i.e.

. (12) }|max{)( ),(kk

rkiprkm δ≥=

An appropriate value for the threshold could be e.g. 1% or 5%, corresponding to a 99% or 95% confidence level for each excluded alternative not to be the most preferred one. Still, it may be a good idea to choose a large enough value for m(k) (small enough for

) to include the target alternative itself in the reference set. Sometimes it might be more appropriate to consider the probability for any of the excluded alternatives not to be the most preferred one. In that case we would define the size of the reference set by

kδ

}|'min{)(1' ),(

km

rrk

rkiprkm δ<= ∑ +=. (13)

This formula is conservative if shared ranks between the alternatives occur, i.e. if (9) holds as a strict inequality.

The reference set for an efficient alternative consists of the similar and competing alternatives that obtain high cross confidence factors. Forming the reference sets is useful when trying to identify one or more most preferred alternatives, or suitable compromise alternatives. In particular, when the DMs want to consider the choice of an alternative that cannot be identified as the most preferred one with confidence, they can restrict additional evaluations to the reference set. This can, of course, save a significant amount of effort and money because more precise additional measurements can be focused to a smaller set of alternatives.

The reference set may also be useful when trying to create new and better alternatives during the decision process. The concept of reference sets is similar to that in Data Envelopment Analysis (DEA) (Charnes et al. 1978), but generalized for stochastic models. In DEA, reference sets are defined for inefficient alternatives (decision making units, DMUs). The DEA reference set consists of alternatives that are similar to an inefficient alternative, but efficient. In order to improve its performance, the inefficient alternative could try approach the alternatives in its reference set. DEA also defines for each inefficient alternative a goal alternative. The goal alternative is formed either by projecting the inefficient alternative to the efficient frontier, or equivalently, by composing it as a linear (convex) combination of the reference alternatives.

7

In analogy with DEA, it is also possible to form goal alternatives in SMAA. The goal alternative is formed as a convex combination of the reference alternatives using the cross confidence factors as weights

∑∑ ===

)(*

1 ),()(*

1 ),(),(* km

rk

rkikm

r rkik

rkik pp xx . (14)

The goal alternative should denote an improvement with respect to the target alternative. Therefore, goal alternatives are composed of the m*(k) ‘best’ alternatives that have a higher cross confidence factor than the target alternative. This means that goal alternatives are defined only for weak alternatives that are outperformed by others in terms of the cross confidence factors. There are two requirements for the composite goal alternatives to be meaningful.

1) The feasible criterion space should be convex. Unless the space is convex, the composite alternatives may be infeasible. This implies that the problem should be inherently continuous, not discrete.

2) The value function should be linear (or such that it can be transformed to a linear value function: e.g., additive, multiplicative, or fractional like the DEA model). Unless the preference model is linear, the composite alternative is not necessarily better than the target alternative.

4. Example

4.1 The Helsinki Harbor decision problem

As an example we consider the data from the decision problem of constructing a new general cargo harbor in Helsinki, Finland (Hokkanen et al., 1999). In addition to the harbor itself, the problem involved the construction of navigation channels (I, II), roads (A, B, C) and railway connections (1, 2, 3, 4) leading to the harbor. As required by the Finnish Act on Environmental Impact Assessment Procedure, a ‘ZERO’ alternative consisting of developing the existing harbors instead was included in the analysis. The original problem consisted thus of 25 alternatives and n = 11 criteria. Because alternatives associated with navigation channel I were excluded early in the process, m = 13 alternatives remained. The alternatives and the mean criteria measurements are shown in Table 1. For simplicity, we represent in this example all measurements ξij by independent normal distributions with mean xij and standard deviation σij. We choose the standard deviation so that the 90% confidence interval for criteria measurements is 10% of the interval between the largest and smallest mean for each criterion, that is, σij = 0.1 (xj

max-xjmin)/1.64. In the original analysis, a detailed model was applied for the

uncertainties of measurements and their dependencies. Therefore, the data in the original analysis was somewhat more precise than in this example.

8

Table 1. The mean criteria measurements xij.

Alternative 1 (min) 2 (min) 3 (min) 4 (max) 5 (max) 6 (min) 7 (max) 8 (max) 9 (max) 10 (min) 11 (max)IIA1 4 1 985 30 166 705 25000 4.5 4.2 15.1 1.75 IIA2 4 2.5 985 30 166 765 25000 4.5 4.1 15.3 1.69 IIA3 4 1.5 985 30 166 705 25000 4.5 4.3 12.7 1.75 IIA4 4 1.5 985 30 166 705 25000 4.5 4.3 12.2 1.65 IIB1 4 1.5 985 35 177 705 25000 4.5 4.4 15.1 1.68 IIB2 4 2.5 985 35 177 765 25000 4.5 4.3 15.3 1.62 IIB3 4 2 985 35 177 705 25000 4.5 4.5 12.7 1.68 IIB4 4 2 985 35 177 705 25000 4.5 4.5 12.2 1.58 IIC1 4 1 985 35 166 705 25000 4.5 4.6 14.8 1.72 IIC2 4 2.5 985 35 166 765 25000 4.5 4.5 15.0 1.66 IIC3 4 1.5 985 35 166 705 25000 4.5 4.7 12.4 1.72 IIC4 4 2 985 35 166 705 25000 4.5 4.7 11.9 1.62 ZERO 1 0 1300 50 266 4200 0 2 1 18.8 1

4.2 Basic SMAA-2 analysis

We next analyze the problem using SMAA-2. We represent the DMs’ preference structure by a linear value function, where the criteria measurements are first scaled to the interval [0,1] and then combined using normalized weights. The scaling is based on best and worst criteria measurements, that is, uij = (xij-xij

worst)/(xijbest-xij

worst). We use a uniform weight distribution to represent missing preference information. Table 2 presents the holistic acceptability index, the confidence factor, and the rank acceptability indices for the alternatives (sorted by their holistic acceptability index). Figure 2 shows the rank acceptability indices as a 3-dimensional chart.

Table 2. The holistic acceptability index, confidence factor, and rank acceptability indices (%) by SMAA-2. Alternatives are sorted by their holistic acceptability index. Weakly efficient alternatives appear in boldface.

Alt. ah pc b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13

IIC3 60 58 26 21 17 12 9 6 4 2 2 1 0.5 0.2 0.05IIC1 48 57 18 15 11 10 9 9 11 7 5 3 1 1 0.1 IIB3 41 20 9 12 13 13 13 12 10 8 6 3 1 0 0.03IIC4 40 26 10 12 11 11 10 10 10 9 8 5 2 1 0.2 IIB4 37 20 8 10 11 11 11 11 10 10 9 5 2 1 0.1 IIA3 33 16 5 8 9 11 12 13 11 9 8 6 5 2 1 IIB1 31 13 4 7 9 10 11 11 12 13 12 6 3 1 0.1 IIA4 29 12 4 6 8 10 12 12 11 10 9 8 6 4 1 IIA1 27 21 5 7 7 7 7 7 9 11 13 11 10 6 1 ZERO 12 100 9 1 1 1 1 1 1 1 1 2 2 3 76 IIC2 12 1 0.4 1 1 2 3 4 6 9 12 22 22 15 3 IIB2 12 2 0.4 1 1 2 3 4 5 8 11 19 26 19 2 IIA2 5 0.3 0.1 0.1 0.2 0.3 0.5 1 1 2 4 10 19 45 16

9

b1

b6

b11

IIC3

IIC1

IIB3

IIC4

IIB4

IIA3

IIB1

IIA4

IIA1

ZER

OIIC

2IIB

2IIA

2

0

20

40

60

80

Acceptability

Rank Alternative

Figure 2. SMAA-2 rank acceptabilities b (%). Alternatives are sorted by their holistic acceptability index.

ri

The results show that all alternatives receive at least some acceptability for all ranks. This is not surprising, because the criteria measurements were modeled by normal distributions, which will include arbitrarily high and low values with some (small) probability. Based on the first rank acceptability index, alternatives IIA2, IIB2 and IIC2 can be considered weakly efficient. The extremely low confidence factors (0.3-2%) support eliminating them from further analysis.

The remaining alternatives must be considered efficient. However, it is difficult to discriminate between the efficient alternatives, because each of them obtains reasonably high first rank acceptability. The rank acceptability indices (and the low holistic acceptability index) show that the ZERO-alternative differs considerably from the other efficient alternatives. Roughly speaking, depending on the preferences, the ZERO-alternative is almost always either the best or the worst alternative. It is thus not a potential compromise, but will be the obvious choice under suitable preferences. The full 100% confidence factor also supports this conclusion. In contrast, all other efficient alternatives IIC3, IIC1, IIB3, IIC4, IIA4, IIB1, IIA4, and IIA1 obtain fairly high acceptability for the best ranks, which is also indicated by their holistic acceptability indices ranging from 60% to 27%.

The decision process would normally continue by examining the central weight vectors, which are presented in Table 3 and Figure 3. The DMs can judge what kinds of weights best match their preferences. For example, the ZERO-alternative differs considerably from the remaining alternatives by much higher weights for criteria 1, 4, and 5, and

10

much lower weights for criteria 3, 6, 7, and 8. It is, of course, also possible to apply some specific procedure for collecting explicit preference information and then to repeat the analysis. However, in the current problem, the confidence factors for all other efficient alternatives except the ZERO-alternative are markedly low, in the range of 58% to 12%. This means, that even with precise preference information conforming to some central weight vector, none of these alternatives can be claimed to be the most preferred one with reasonable confidence (over 90%, say). Thus, we know that some additional analysis or more precise criteria information is required to discern between these alternatives.

Table 3. The central weight vectors (%) for the alternatives. Alternatives are sorted by their holistic acceptability index.

Alternative w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 w11 IIA1 8 16 10 4 8 10 10 10 7 4 13IIA2 13 2 11 3 9 13 14 15 6 2 12IIA3 9 8 10 4 8 10 10 10 7 10 14IIA4 9 10 10 4 7 10 10 10 7 15 7IIB1 8 7 10 10 12 11 10 11 8 3 9IIB2 9 2 12 10 14 10 12 13 7 3 7IIB3 8 4 10 10 12 10 10 10 9 9 9IIB4 8 5 10 9 11 10 10 10 9 13 5IIC1 8 14 9 9 7 9 9 9 10 5 10IIC2 9 2 12 12 7 11 13 13 9 3 8IIC3 8 8 9 9 7 9 9 9 10 11 10IIC4 9 5 9 9 7 9 9 9 11 15 7ZERO 17 13 5 15 17 5 5 5 5 6 5

11

2

4

6

8

10

12

14

16

18

w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 w11

IIA1IIA2IIA3IIA4IIB1IIB2IIB3IIB4IIC1IIC2IIC3IIC4ZERO

Figure 3. Central weight vectors for alternatives (%).

4.3 Analysis using cross confidence factors

To obtain more in-depth understanding of what causes the weak discrimination between the alternatives, we next analyze the problem using cross confidence factors. Table 4 presents the nonzero (≥ 1%) cross confidence factors for each alternative xk

cip i and

central weight vector . Each row of the table lists the cross confidence factors corresponding to a particular central weight. The original confidence factors lie on the diagonal. The sum of each row is, within rounding precision, 100%.

kw

Table 4. Cross confidence factors (%). kip

ckw \ xi IIA1 IIA2 IIA3 IIA4 IIB1 IIB2 IIB3 IIB4 IIC1 IIC2 IIC3 IIC4 ZERO

IIA1 21 5 1 3 52 17 0 IIA2 7 0.3 11 3 10 1 16 5 16 1 24 5 IIA3 2 16 6 8 3 10 49 5 IIA4 10 12 7 8 2 47 13 IIB1 2 2 13 12 5 32 30 4 IIB2 1 15 2 24 14 13 1 21 8 IIB3 2 1 2 20 13 5 42 14 IIB4 1 2 17 20 1 36 22 IIC1 5 2 4 1 57 29 IIC2 9 1 17 9 18 1 31 11 IIC3 3 2 9 5 10 58 12 IIC4 1 2 12 14 44 26 ZERO 100

12

The cross confidence factors reveal why many alternatives are discriminated so weakly. The nonzero cross confidence factors on each row identify which alternatives compete about the first rank with a given alternative’s central weight vector and how strongly they do it. For example, the first row indicates that if the DMs preferences conform to the central weight of IIA1, that alternative has only a 21% probability of being the most preferred one. Instead, alternative IIC1 is then most preferred with 52% probability, and IIC3 with 17% probability. Therefore, if the DMs accept the central weight vector of IIA1, they should also consider the competing alternatives IIC1, IIC3 and maybe others with a reasonably high cross confidence factors (IIA3, IIB1, …). IIA1 along with the competing alternatives form the reference set for IIA1. In this case the decision process could continue in one of two directions. 1) If the DMs consider all alternatives in the reference similar enough, they can choose the alternative with the maximal cross confidence factor. 2) If the DMs consider that the alternatives have significant differences, the criteria must be measured more accurately to distinguish the alternatives. In this case it is sufficient to focus the measuring effort to the reference set. This may result in significant savings in measurement costs.

4.4 Analysis using reference sets

To better visualize the reference sets, we sort the cross confidence factors on each row of Table 4 in decreasing order, and draw them as column charts. In the following we include to the reference set for an alternative all alternatives with a cross confidence factor of at least 1%. Figure 4 shows the reference sets for all alternatives, except alternative ZERO. The reference set for ZERO consists only of ZERO itself.

The reference sets reveal sets of one or more most preferred alternatives corresponding to different preferences. This is useful for resolving whether it is possible to make a compromise decision based on the current information, or if the criteria should be measured more accurately. This is done by examining the first alternatives in a chosen reference set. For example, in the reference set of IIC1 the two first alternatives IIC1 and IIC3 together obtain 86% confidence. If the DMs consider 86% confidence sufficient, and they are willing to approve both alternatives, then they could without additional measurements choose either one. In practice they should probably choose the alternative with higher cross confidence factor (IIC1).

13

IIC3

0102030405060

IIC3

IIC4

IIC1

IIB3

IIB4

IIA3

IIA4

IIC1

0102030405060

IIC1

IIC3

IIA1

IIB1

IIA3

IIB3

IIB3

0102030405060

IIC3

IIB3

IIC4

IIB4

IIC1

IIA3

IIB1

IIA4

IIC4

0102030405060

IIC3

IIC4

IIB4

IIB3

IIA4

IIA3

IIB4

0102030405060

IIC3

IIC4

IIB4

IIB3

IIA4

IIA3

IIC1

IIA3

0102030405060

IIC3

IIA3

IIC1

IIB3

IIA4

IIC4

IIB4

IIA1

IIB1

0102030405060

IIC1

IIC3

IIB1

IIB3

IIB4

IIC4

IIA1

IIA3

IIA4

0102030405060

IIC3

IIC4

IIA4

IIA3

IIB4

IIB3

IIC1

IIA1

0102030405060

IIC1

IIA1

IIC3

IIA3

IIB1

IIA4

IIC2

0

10

20

30

IIC3

IIC1

IIB3

IIC4

IIB1

IIB4

IIC2

IIB2

IIB2

0

10

20

IIB3

IIC3

IIB1

IIB4

IIC1

IIC4

IIB2

IIC2

IIA3

IIA2

0

10

20

IIC3

IIC1

IIB3

IIA3

IIB1

IIA1

IIC4

IIB4

IIA4

IIC2

IIB2

IIA2

Figure 4. Reference sets for alternatives sorted by their cross confidence factors. The target alternatives are drawn as light columns.

We make three observations concerning the weakly efficient alternatives IIA2, IIB2, and IIC2:

1) Their reference sets are particularly large consisting of almost all alternatives except alternative ZERO.

2) They obtain relatively low cross confidence factors inside their reference sets.

3) They do not belong to the reference sets for any of the efficient alternatives.

The reference sets for the efficient alternatives are also fairly large, consisting of between 6 and 8 alternatives. If the DMs want to increase the confidence factor of a particular alternative, it is worth measuring more accurately alternatives with high cross confidence factors within the corresponding reference set. To strengthen the overall discrimination of the model, a good strategy is measure more accurately alternatives with high cross confidence factors in several reference sets. For example, alternative

14

IIC3 obtains a very high cross confidence factor within all reference sets (except that of alternative ZERO). Also, IIC1 obtains fairly high cross confidence factors in many reference sets. Therefore, more accurate measurements could be first targeted at these alternatives. In contrast, more accurate measurement of alternative ZERO will not significantly affect the discrimination between the alternatives.

Composite goal alternatives are not very useful in this example. Because the problem is inherently discrete, composite alternatives would not in general be feasible. Only such goal alternatives that consist of a single reference alternative are known to be feasible. Such singleton goals can be seen from Figure 4: the goal for IIA1 is IIC1 and alternatives IIA2, IIB3, and IIC4 have the same goal IIC3.

5. Conclusions

We have extended the SMAA-2 method by introducing cross confidence factors, which are based on computing confidence factors for alternatives using each other’s central weight vectors. The cross confidence factors can be used for classifying efficient alternatives into reference sets consisting of similar and competing alternatives. These sets are useful when trying to identify one or more most preferred alternatives, or suitable compromise alternatives. In a real-life decision problem the DMs can evaluate which of the central weights might represent their preferences, or choose target alternatives that they for other reasons are interested in. The cross confidence factors of the target alternative reveal then which alternatives are the most probable choices based on the corresponding preferences. If a target alternative is much less likely to be the best one than some other alternatives, its choice is difficult to justify based on the current criteria measurements. If none of the central weight vectors or alternatives appeals to a DM, the DM can modify the central weights, and re-analyze the problem with the modified weights.

The reference sets can also be used for evaluating whether criteria need to be measured more accurately, and at which alternatives new measurements should be focused. This may cause considerable savings in measurement costs. We have demonstrated the use of the cross confidence factors and reference sets using a real-life example. In practice, the reference sets may consist of the target alternative alone, or contain in addition many other alternatives. Our example was fairly challenging, because only one alternative formed its own reference set and all other reference sets were fairly large. Still, the cross confidence factor analysis was able to clearly identify candidates for a choice or for further evaluation.

A significant advantage of the proposed method in real-life problems is that the problem can be analyzed deeply without or prior to collecting preference information. The cross confidence factors reveal the need for additional information, which can then be collected by the analysts before presenting the problem with the DMs.

15

References Charnes A., Cooper W.W., Rhodes E., 1978. Measuring the efficiency of decision

making units. European Journal of Operational Research 2, 429-444.

Hokkanen J., Lahdelma R., Salminen P., 1999. A Multiple Criteria Decision Model for Analyzing and Choosing Among Different Development Patterns for the Helsinki Cargo Harbor. Socio-Economic Planning Sciences 33, 1-23.

Kangas J., Kangas A., 2002. Multiple Criteria Decision Support methods in forest management. Pukkala, T. (ed.) Multi-objective forest planning. Kluwer Academic Publishers. Managing Forest Ecosystems 4, (to appear).

Lahdelma R., Hokkanen J., Salminen P., 1998. SMAA - Stochastic multiobjective acceptability analysis. European Journal of Operational Research 106(1), 137-143.

Lahdelma R., Miettinen K., Salminen P., 2002a. Ordinal criteria in Stochastic Multicriteria Acceptability Analysis. European Journal of Operational Research, (to appear).

Lahdelma R., Miettinen K., Salminen P., 2002b. Stochastic Multicriteria Acceptability Analysis Using Achievement Functions. Technical Report 459, TUCS - Turku Centre for Computer Science, Turku, Finland.

Lahdelma R., Salminen P., 2001. SMAA-2: Stochastic multicriteria acceptability analysis for group decision making. Operations Research 49 (3), 444-454.

Lahdelma R., Salminen P., 2002. Pseudo-criteria versus linear utility function in Stochastic Multicriteria Acceptability Analysis. European Journal of Operational Research 141(2), 454-469.

Lahdelma R., Salminen P., Hokkanen J. 1999. Combining Stochastic Multiobjective Acceptability Analysis and DEA. In D.K. Despotis and C. Zopounidis (eds.) Integrating Technology & Human Decisions: Global Bridges into the 21st Century, New Technologies Publications, Athens, 629-632.

Lahdelma R., Salminen P., Hokkanen J., 2000. Using multicriteria methods in environmental planning and management. Environmental Management 26(6), 595-605.

Lahdelma R., Salminen P., Hokkanen J., 2002b. Locating a waste treatment facility by using Stochastic Multicriteria Acceptability Analysis. European Journal of Operational Research 142(2), 345-356.

Makkonen S., Lahdelma R., Asell A.-M., 2002. Multicriteria decision support in the liberated energy market. 16th MCDM World Conference, Semmering, Austria, February 18-22.

Maystre L.Y., Picted J., Simos J., 1994. Méthodes Multicritéres ELECTRE. Presses Polytechiques et Universitaires Romandes, Lausanne, Switzerland.

Vincke Ph., 1992. Multicriteria Decision-Aid. John Wiley & Sons, Chichester, UK.

16

Turku Centre for Computer Science Lemminkäisenkatu 14 FIN-20520 Turku Finland http://www.tucs.fi/

University of Turku • Department of Information Technology • Department of Mathematics

Åbo Akademi University • Department of Computer Science • Institute for Advanced Management Systems Research

Turku School of Economics and Business Administration • Institute of Information Systems Science