I

Em

Ja

b

c

d

e

f

F

a

A

R

R

1

A

A

K

I

A

R

M

S

E

T

1d

ARTICLE IN PRESSJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

journa l homepage: www. int l .e lsev ierhea l th .com/ journa ls / i jmi

lectre Tri-C, a multiple criteria decision aiding sortingodel applied to assisted reproduction

.R. Figueiraa,1, J. Almeida-Diasb,c,∗, S. Matiasd, B. Royb, M.J. Carvalhoe, C.E. Planchae,f

INPL, École des Mines de Nancy, Laboratoire LORIA, Nancy, FranceLAMSADE, Université Paris-Dauphine, Paris, FranceCEG-IST, Instituto Superior Técnico, Universidade Técnica de Lisboa, Lisbon, PortugalInstituto Superior Técnico, Universidade Técnica de Lisboa, Lisbon, PortugalCEMEARE, Centro Médico de Assistência à Reproducão, Lisbon, PortugalUnidade Biologia da Reproducão, Instituto Histologia e Biologia do Desenvolvimento,aculty of Medicine, University of Lisbon, Lisbon, Portugal

r t i c l e i n f o

rticle history:

eceived 9 May 2010

eceived in revised form

7 September 2010

ccepted 18 December 2010

vailable online xxx

eywords:

nfertility

ssisted reproductive technology

isk of multiple pregnancies

ultiple criteria decision aiding

orting problems

lectre Tri-C

a b s t r a c t

Objective: The aim of this paper is to apply an informatics tool for dealing with a medical

decision aiding problem to help infertile couples to become parents, when using assisted

reproduction.

Methods: A multiple criteria decision aiding method for sorting or ordinal classification prob-

lems, called Electre Tri-C, was chosen in order to assign each couple to an embryo-transfer

category. The set of categories puts in evidence a way for increasing the single pregnancy

rate, while minimizing multiple pregnancies. The decision aiding sorting model was co-

constructed through an interaction process between the decision aiding analysts and the

medical experts.

Results: According to the sample used in this study, the Electre Tri-C method provides a

unique category in 86% of the cases and it achieves a sorting accuracy of 61%. Retrospec-

tively, the medical experts do agree that some of their judgments concerning the number of

embryos to transfer back to the uterus of the woman could be different according to these

results. The current ART methodology achieves a single pregnancy rate of 47% and a twin

pregnancy rate of 14%. Thus, this informatics tools may play an important role for support-

ing ART medical decisions, aiming to increase the single pregnancy rate, while minimizing

multiple pregnancies.

Limitations: Building the set of criteria comprises a part of arbitrariness and imperfect knowl-

edge, which require time and expertise to be refined. Among them, three criteria are modeled

by means of a holistic classification procedure by the medical experts.

© 2010 Elsevier Ireland Ltd. All rights reserved.

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, a multiple criteria decision aiding sorting model applied to assistedreproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

∗ Corresponding author at: LAMSADE, Université Paris-Dauphine, Place du Maréchal De Lattre de Tassigny, F-75 775 Paris Cedex 16, France.el.: +33 1 44 05 42 87; fax: +33 1 44 05 40 91.

E-mail address: judiascv@gmail.com (J. Almeida-Dias).1 Part of this Work was accomplished when J.R. Figueira was a permanent member of CEG-IST, Lisbon.386-5056/$ – see front matter © 2010 Elsevier Ireland Ltd. All rights reserved.oi:10.1016/j.ijmedinf.2010.12.001

INc a l i

ARTICLEIJB-2709; No. of Pages 12

2 i n t e r n a t i o n a l j o u r n a l o f m e d i

1. Introduction

The application of informatics tools for medical decisionaiding problems is not new, and it has been increasing tremen-dously over the last years. There are applications in cancercare [1–5], decision tools for medical diagnosis [6–10], amongothers. For a comprehensive survey of the application of multi-attribute techniques and other decision aiding tools in healthcare and medical treatment, see [11–14]. However, to our bestknowledge there are no multiple criteria informatics toolsapplied to medical decision problems for dealing with AssistedReproductive Technology (ART).

When a couple fails to conceive, the source of the prob-lem must be investigated in both man and woman. Theproblem may be related only to one of the couple mem-bers, to both or, apparently to none of them. According toseveral multidisciplinary medical evaluations, a procedure ischosen to try to overcome the problem. The most effectiveprocedures in ART are in vitro fertilization (IVF) and intra-cytoplasmic sperm injection (ICSI). These techniques allowin most cases to obtain pre-implantation embryos in cul-ture.

A difficult task at this point is to decide on the adequatenumber of embryos to transfer back into the uterus. On theone hand, we want to obtain high pregnancy rates, but on theother hand, we want to minimize the risk of a multiple preg-nancy. Thus, this decision is usually based on the clinician’spoint of view, that includes the past history of the patientwith such information as indications for ART, previous cyclesof ART, prior pregnancies, use of donor gametes/embryos, aswell as couple’s values, and based on the point of view ofthe embryologist that includes the quality of sperm (basedon morphology, motility, and concentration of sperm in theejaculate), the quality of oocytes, and the quality of embryos.Thus, we are dealing with a difficult decision aiding problem[15].

An essential part of this approach is related to thestructuring process of building criteria, since each pointof view is composed of a large set of heterogeneous ele-mentary consequences (aspects, characteristics, attributes,exams, etc.), that should be associated in order to formhomogeneous groups of dimensions (elementary conse-quences, “measured” through the application of a descriptor,a metric, or a “primary scale”). The emergence of thesegroups will then allow the construction of a coherentset of criteria for operationalizing the several points ofview.

The aim of this study is thus to apply a sorting pro-cedure in order to reach a recommendation concerningthe number of embryos to be transferred back to theuterus of the woman, taking into account the prefer-ences of the embryologist when defining the parametersof the model. The choice of Electre Tri-C[16] methodamong the different tools available for sorting problems isrelated to the fact that this medical problem led to the

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

definition of the categories by characteristic reference cou-ples instead of boundary couples or other norms. ElectreTri-C was particularly designed for such a type of prob-lems.

PRESSn f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

2. Methods

This section presents the main concepts, definitions, andnotation with respect to the chosen sorting method, ElectreTri-C[16], including the main options of the ART applicationstructuring process.

2.1. Concepts, definition, and notation

As stated by [17], over the last 15 years, the need for well-founded methodologies is of the uttermost importance inmedical informatics. In what follows, we will describe ourmethodology in a comprehensive way. This means thatwe try to overcome the drawbacks frequently found inmedical informatics tools and decision support systems,where medical professionals have no knowledge about theunderlying methodology [18]. In our case, a co-constructiveinteractive process between the analysts and the medical expertsallowed to make the latter familiar with the methodologicalapproach.

Let A = {a1, a2, . . ., ai, . . . } be the set of potential actions,i.e. the set of couples (women/men) to be subjected to an ARTtreatment. This set can be completely known a priori or it mayappear progressively during the decision aiding process. Thecouples are evaluated on a coherent set of criteria [19], denotedF = {g1, g2, . . ., gj, . . ., gn}, with n ≥ 3. Therefore, gj(a) representsthe performance of the couple a according to the criteriongj, j = 1, . . ., n. When using the Electre Tri-C method [16], theobjective is to assign the couples to a set of completely orderedcategories, denoted {C1, C2, . . ., Ch, . . ., Cq}, with q ≥ 2. Assumethat C1 represents the worst category and Cq represents thebest category.

Each criterion gj must be either associated with an increas-ing preference direction or a decreasing preference direction.In the latter case, it means that the preferences increasewhen the performances decrease, and in the former case, itmeans that the preferences increase when the performancesincrease too. Moreover, each criterion is also associated witha completely ordered preference scale which contains all thepossible performances of a couple according to such a criteriontaking into account the decision aiding ART sorting context.

The criteria are also associated with two discriminatingthresholds (called indifference and preference thresholds,denoted qj and pj, respectively, j = 1, . . ., n). These thresholdsare relevant to take into account the imperfect character ofthe performances of each couple as well as some arbitrari-ness when building the set of criteria. The aggregation of theperformances of each couple is obtained by making use ofthe so-called “power of the criteria”, which is defined by therelative importance coefficients, or weights, denoted wj, j = 1,. . ., n, and, optionally, the veto thresholds, denoted vj, j = 1, . . .,n, which is used to manage critical values on a certain crite-rion, which appear risky when submitted to a certain medicaltreatment.

multiple criteria decision aiding sorting model applied to assisted

2.2. An overview of Electre Tri-C

Electre Tri-C[16] was designed to be used within theframework of a constructive approach. This decision aid-

ARTICLE IN PRESSIJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx 3

Table 1 – The set of criteria.

Code Description of the criteria Nature of the scale of the criteria Preference

g1 Age of the woman Years (integer) Decreasingg2 Duration of infertility Years (integer) Decreasingg3 Number of oocytes retrieved a (1) If Or = 0 Decreasing

(2) If Or = − 1 or Or = 1(3) If Or = − 2 or Or ≥ 2(4) If Or = − 3 or Or = − 4(5) If Or = − 5 or Or = − 6(6) If Or = − 7 or Or = − 8(7) If Or ≤ − 9

g4 Evaluation of the women Holistic classification model A b Increasingg5 Sperm origin (1) Testicular biopsy cryopreserved sperm Increasing

(2) Cryopreserved ejaculated sperm(3) Fresh ejaculated sperm

g6 Morphological quality of the four best embryos (in the transfer day) Holistic classification model B c Increasingg7 Developmental quality of the four best embryos Holistic classification model C d Increasing

a Or is the effective number of oocytes retrieved minus the ideal number (which is 12).b A holistic classification made by the gynaecologist/obstetrician according to seven categories, denoted {1, 2, 3, 4, 5, 6, 7}, where 1 is the worst

category and 7 is the best one.c An overall morphological classification of the four best embryos which is obtained by the sum of the individual score of each embryo previously

evaluated by the embryologist according to five points, denoted {1, 2, 3, 4, 5}, where 1 is the worst value and 5 is the best one.ryolo

iwwdwlbipe

oclcvaFtbo

edcwi

pdm

D�

t

((

((

d A holistic classification of the whole set of four embryos by the embworst category and 5 is the best one.

ng sorting method must be applied in the contextshere the categories are completely ordered (from theorst to the best, for instance). Each category must beefined a priori to receive actions (e.g. couples), whichill be or might be processed in the same way (at

east in a first step). The definition of each category isased on a unique characteristic reference couple, which

s the most representative taking into account the nextrocessing operations (e.g. transfer the same number ofmbryos).

The Electre Tri-C assignment results are based on theutranking credibility indices (see Appendix A) which areompared to a chosen credibility level, denoted �. Thisevel is a minimum degree of credibility, �(a,a′), which isonsidered or judged necessary by the medical experts toalidate or not the statement “a outranks a′” (meaning that

is at least as good as a′) taking all the criteria frominto account. In general, this minimum credibility level

akes a value within the range [0.5,1] and it can roughlye interpreted as a majority level as in the voting the-ry.

In order to preserve the role of the characteristic refer-nce couples, Electre Tri-C makes use of a selecting function,enoted �(a,a′). This function allows to choose between twoonsecutive selected categories and it can be defined in severalays. In this ART application the following selecting function

s going to be used: �(a,a′) = min {�(a,a′);�(a′,a)}.Definitions 1 and 2 present the Electre Tri-C assignment

rocedure, which is composed of two joint rules, called theescending rule and the ascending rule, respectively (whichust be used conjointly and not separately).

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

efinition 1 (Descending rule). Choose a credibility level,(1/2 ≤ � ≤ 1). Decrease h from (q + 1) until the first value, t, suchhat �(a,bt) ≥ �:

gist according to five categories, denoted {1, 2, 3, 4, 5}, where 1 is the

a) For t = q, select Cq as a possible category to assign action a.b) For 0 < t < q, if �(a,bt) > �(a,bt+1), then select Ct as a possible

category to assign a; otherwise, select Ct+1.(c) For t = 0, select C1 as a possible category to assign a.

Definition 2 (Ascending rule). Choose a credibility level,�(1/2 ≤ � ≤ 1). Increase h from zero until the first value, k, suchthat �(bk,a) ≥ �:

a) For k = 1, select C1 as a possible category to assign action a.b) For 1 < k < (q + 1), if �(a,bk) > �(a,bk−1), then select Ck as a pos-

sible category to assign a; otherwise, select Ck−1.(c) For k = (q + 1), select Cq as a possible category to assign a.

Each one of these rules selects only one category for a possi-ble assignment of a couple. They are used conjointly in order tohighlight the highest category and the lowest category, whichcan appear potentially appropriate to receive a couple. Thesetwo extreme categories can be the same. When they differ,this means that the assignment of such a couple remains ill-determined within a range of possible categories taking intoaccount the way that the set of characteristic couples definesthe categories.

2.3. Modeling the set of criteria

In this ART application, the couples (women/men) were eval-uated on a set of seven criteria (see Table 1) based on a set ofchosen medical exams, co-constructed through an interactionprocess between the analysts and the medical experts (gynae-cologists/obstetricians, embryologists, biologists, physicians,

multiple criteria decision aiding sorting model applied to assisted

psychologists). For modeling the set of criteria, only biologicaldata (the history of the couple and the quality of the embryos)are considered within the decision aiding assignment model.In addition to the main options presented in Table 1, according

ARTICLE IN PRESSIJB-2709; No. of Pages 12

4 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

Table 2 – Performance of the characteristic actions/couples.

Ch Nature of the category bh Criteria

g1 g2 g3 g4 g5 g6 g7

C1 Transfer of four embryos b1 42 10 7 1 1 6 1C2 Transfer of three embryos b2 38 6 5 2 2 10 2C3 Transfer of two embryos b3 33 3 4 5 3 16 4C4 Transfer of one embryo b4 25 1 1 6 3 19 5

Table 3 – Definition of the discriminating thresholds.

Thresholds Criteria

g3

01

g1 g2

Indifference q1(g1(bh)) q2(g2(bh))Preference p1(g1(bh)) p2(g2(bh))

to the set of criteria, we have [15]:

– g1: this criterion is relevant because the younger the womanis, the more favorable the infertility treatment prognostic;

– g2: this criterion is relevant because the longer the durationof infertility is, the worst the infertility treatment prognos-tic;

– g3: this criterion is relevant in order to take into account theOr (number of oocytes retrieved) differences. For the samedifference, an effective number of oocytes retrieved lowerthan 12 is worst than an effective number of oocytes greaterthan 12;

– g4: this is a very complex criterion which was modeledby a holistic classification procedure made by the gynae-cologist/obstetrician, when taking into account severaldimensions as well as the experience of the medical experts;

– g5: this criterion is related to the quality of the sperm and itrepresents the only male input of the decision aiding assign-ment model;

– g6 and g7: these two criteria are also very complex ones andthey were modeled through a holistic classification proce-dure made by the embryologist, when taking into accountseveral dimensions as well as the experience of the medicalexperts.

As stated above, the criteria g4, g6, and g7 were modeledthrough a holistic classification procedure (models A, B, andC) by simplifying a very complex task. Such a modeling simpli-fication was intuitively validated by the medical experts fromtheir large past medical experience.

2.4. Modeling the set of categories

The objective of the decision aiding sorting model is to give a“recommendation” to the medical experts about the numberof embryos to be transferred back to the uterus of the womanin order to obtain a pregnancy and, at the same time, to reducethe risk of multiple pregnancies. Of course the final decisionshould be made by the couples themselves, but the “medicalexperts’ recommendation” is very important.

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

The nature of the set of categories is, therefore, relatedto the transfer of embryos: four embryos (denoted C1), threeembryos (denoted C2), two embryos (denoted C3), and oneembryo (denoted C4). These categories are clearly ordered from

g4 g5 g6 g7

0 0 q6(g6(bh)) 01 0 p6(g6(bh)) 0

the least risky category, C4, to the most risky category, C1.Furthermore, the less the number of embryos to transfer is,the better the category is in terms of the risk of multiplepregnancies, but the possibility of achieving a pregnancy cansimultaneously decrease [15].

The set of categories was defined by characteristic ref-erence couples, denoted bh, h = 1, . . ., 4 (see Table 2). Thesecharacteristic couples were defined according to the maincharacteristics of reference couples through a co-constructionprocess between the analysts and the medical experts, whiletaking also into account the medical experience of the embry-ologists and the gynaecologists/obstetricians.

2.5. Modeling the imperfect knowledge andill-determination

As stated above, the set of criteria is built from a set of cho-sen medical exams and the characteristics of the couples. Inthis application variable discriminating thresholds are usedfor the criteria g1, g2, and g6, and constant ones for the remain-ing criteria (see Table 3). The indifference threshold, qj, is themaximal advantage of a couple over another couple accord-ing to the criterion gj, which is compatible with the statement“the two couples are indifferent”. The preference threshold,pj, is the minimal positive advantage of a couple over anothercouple according to the criterion gj to be passed for being con-vincing for the strict preference of the first couple.

When taking only the criterion g1 into account, we assumethat a 28 years old woman should be subjected to an ART treat-ment more similar to a 25 years old reference woman than toa 33 years old reference woman. In such a case, one embryoinstead of two embryos may be transferred back to the uterusof that woman. This means that the younger the woman is,the more favorable the ART treatment prognostic is, and lessembryos must be transferred to the uterus of such a woman[15]. The variable discriminating thresholds presented in Table4 must be interpreted as follows:

– Any woman and the younger 25 years old reference woman

multiple criteria decision aiding sorting model applied to assisted

are indifferent if the difference of their ages is at most fiveyears, q1(g1(b4)) = 5. On the contrary, any woman and theolder 25 years old reference woman are indifferent if thedifference of their ages is at most three years, q1(g1(b4)) = 3.

ARTICLE IN PRESSIJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx 5

Fig. 1 – Illustration of the variable thresholds on criterion g1. Indifference zone with respect to the characteristic couple bh.Strict preference zone (frontier and exterior) with respect to the characteristic couple bh.

Table 4 – Discriminating thresholds on the criterion g1.

g1(bh) − g1(a) ≤ 0 a g1(bh) − g1(a) ≥ 0 b

bh q1(g1(bh)) p1(g1(bh)) q1(g1(bh)) p1(g1(bh))

b4 5 6 3 4b3 3 4 1 3b2 0 1 0 0b1 0 0 0 0

–

wacum

Table 6 – Discriminating thresholds on the criterion g6.

bh g6(bh) − g6(a) ≥ 0 a g6(bh) − g6(a) ≤ 0 b

q6(g6(bh)) p6(g6(bh)) q6(g6(bh)) p6(g6(bh))

b4 1 2 0 1b3 1 2 0 1b2 0 0 0 0b1 0 0 0 0

a Inverse thresholds.b Direct thresholds.

Any woman is strictly preferred to the older 25 years oldreference woman if the difference of their ages is strictlygreater than four years, p1(g1(b4)) = 4. On the contrary, anywoman is strictly preferred by the younger 25 years old refer-ence woman if the difference of their ages is strictly greaterthan six years, p1(g1(b4)) = 6.

The above interpretation seems difficult to understandithout considering carefully the core objective of the decision

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

iding sorting model as well as the ART medical treatmentontext. Fig. 1 aims to clarify such an interpretation. This fig-re shows that small differences on the performances areore relevant for older women, where at least for a 40 years

Table 5 – Discriminating thresholds on the criterion g2.

bh g2(bh) − g2(a) ≤ 0 a g2(bh) − g2(a) ≥ 0 b

q2(g2(bh)) p2(g2(bh)) q2(g2(bh)) p2(g2(bh))

b4 1 2 1 2b3 1 2 1 2b2 0 1 0 1b1 0 1 0 1

a Inverse thresholds.b Direct thresholds.

a Inverse thresholds.b Direct thresholds.

old woman, any difference is significant of strict preferenceaccording to the criterion g1.

We have to stress that this is a new way of modeling thethresholds, which was not published before. The thresholdsfunctions must be consistent to the core objective of the mainanalysis provided in the decision aiding sorting context. Table5 presents the variable discriminating thresholds on the cri-terion g2. The longer the duration of infertility is, the moreurgent the ART treatment is. Table 6 presents the variable dis-criminating thresholds on the criterion g6. It was consideredthat under a classification of 16/20, any difference between theclassifications of two sets of embryos is important.

In the next section, the intrinsic weights of the criteria andthe veto thresholds are provided for achieving the modelingprocess of this ART application.

2.6. Modeling the role of the criteria

Within the Electre family of decision aiding methods sev-eral techniques can be used to assign the weights to thecriteria, wj, j = 1, . . ., n [20–22]. These intrinsic weights areinterpreted as the voting power of such criteria on a pair-

multiple criteria decision aiding sorting model applied to assisted

wise comparison between two different couples according totheir performances. One of the techniques which are used todetermine appropriate values for the weights of the criteriais the revised Simos’ procedure [20]. This procedure is usually

ARTICLE INIJB-2709; No. of Pages 12

6 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i

Table 7 – The weights according to the embryologist.

Code Rank a Whitecards b

Non-normalizedweights

Normalizedweights c

g5 1 1 22

g3 2 2.67 62

g2 3 4.33 102

g4 4 6 152

g7 5 7.67 191

g6 6 8.78 213

g1 7 11 27

Total 100

a 1 = worst rank and 7 = best rank;b Z value = 11;c Format = zero decimal places.

well accepted by the experts who are not familiarized withthis kind of decision adding methods. In what follows, onlythe weights obtained in interaction with the embryologist aretaken into account.

When using the revised Simos’ procedure, the names ofthe criteria, written in “white cards”, are ranked from theleast important criterion to the most important one accord-ing to the expert perspective (some criteria may have the sameimportance). Then, the difference of importance between suc-cessive levels of the rank obtained previously is expressedthrough a number of “white cards” introduced between thoselevels, and it is also recorded how many times the most impor-tant criterion is considered to be more important than theleast important one in the ranking (Z value). These inputsare introduced in the SRF (Simos-Roy-Figueira) software, whichwas developed with an algorithm for assigning a numericalvalue to the weights of the criteria by determining the non-normalized and the normalized weights (see Table 7).

The power of the criteria can also be reinforced by intro-ducing veto thresholds. A veto threshold, vj, is the minimaldifference of performance of a couple over another coupleto exceed on the criterion gj, for being incompatible with anoverall outranking statement. In this ART application, for thecriteria gj, j = 2, . . ., 7, constant veto thresholds were consid-

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

ered, such as v2 = 20, v3 = 7, v4 = 7, v5 = 3, v6 = 12, and v7 = 5.There are no desirable veto effects on the criterion g2, whichmeans that it is enough to consider a veto value greater thanany possible differences between any two couples [15].

Table 8 – Veto thresholds on the criterion g1.

bh g1(bh) − g1(a) ≤ 0 a g1(bh) − g1(a) ≥ 0 b

b4 15 15b3 12 10b2 10 6b1 10 6

a Inverse thresholds.b Direct thresholds.

PRESSn f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

As for the criterion g1, variable veto thresholds were alsodefined (see Table 8). Such thresholds were considered to bemore important for older women. When a woman is comparedto a younger 38 years old reference woman, there is a veto (notransfer of three embryos!) if the difference of ages is at leastten years, v1(g1(bh)) = 10. On the contrary, when a woman iscompared to an older 38 years old reference woman, thereis a veto if the difference of their ages is at least six years,v1(g1(bh)) = 6.

3. Results

The set of characteristic reference couples defined for this ARTapplication (see Table 2) achieves a minimum credibility levelequal to 0.007 (the best value is 0 and the worst value is 1). Thismeans that the characteristic couples are distinct enough toincrease the interpretation of the assignment results providedby the Electre Tri-C method [16].

A set of 51 couples were evaluated according to the set ofseven criteria to be assigned to the set of four embryo-transfercategories (see Table B.1). Then, the credibility indices of thecomprehensive outranking of the couples over the character-istic reference couples, and vice-versa, are computed. Theseindices are then compared to the chosen credibility level inorder to obtain the final assignment results.

The chosen credibility level validated by the medical expertwas � = 0.80 (see Table 9). This is the sum of the weights ofthe criteria which are favorable with each one of the assign-ment results. According to the weights provided in Table 7, thismeans that the “age of the woman” (g1) and the “morphologi-cal quality of the four best embryos” (g6) play the effective roleof the most important criteria because at least one of thesetwo criteria must “agree” with each assignment result.

The Electre Tri-C assignment results, the chosen medicaltreatment, and the success of the ART treatment (pregnancy)are presented in Table B.3. According to these results, 44 (86%)of the 51 studied couples are assigned to only one transfer-embryo category. A brief overall comparison to the medicalexpert choice is presented in Table 10. These results showthat the “recommendations” provided by both the methodand the medical expert go in the same direction. The med-ical experts’ team recognized that they could have taken adifferent medical decision if the sorting results of Electre Tri-C were available when such decisions were taken since theresults of the model were performed after such decisions.

According to the medical experts the results presented inTable B.3, the success of the ART methodology in terms of preg-nancy is 61%. This means that 31 pregnancies were obtainedfrom the 51 couples which were subjected to an ART treat-ment. Within these results, 24 single pregnancies (47%) and7 twin pregnancies (14%) were obtained. Such a methodologyhas achieved 30 live births (79%) of a total of 38 possible preg-nancies taking into account the level of success of the ARTmethodology.

According to the assignment results presented in Table B.3,

multiple criteria decision aiding sorting model applied to assisted

the Electre Tri-C method provides a unique category in 44(86%) of the 51 studied couples. The sorting accuracy of the typeI (when Electre Tri-C provides a unique category which isthe same as the one provided by the medical experts) is 61%

ARTICLE IN PRESSIJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx 7

Table 9 – Choice of the credibility level, �.

Version � UC ACR I a ACR II b

Unique Ch (m = 51) (m = UC) (m = 51)

1 0.55–0.70 51 (100%) 71% 71% 71%2 0.75 47 (92%) 65% 70% 73%3 0.80 44 (86%) 61% 70% 75%4 0.85 43 (84%) 59% 70% 75%5 0.90 40 (78%) 59% 75% 80%

a Sorting accuracy of the type I.b Sorting accuracy of the type II.

Table 10 – Summary of the assignment results.

Ch Nature of the category Electre Tri-C (� = 0.80) Medical expert

C1 Transfer of four embryos 0 (0%) 1 (2%)[C1, C2] Transfer of three or four embryos 2 (4%)C2 Transfer of three embryos 9 (18%) 9 (18%)[C2, C3] Transfer of two or three embryos 5 (10%)C3 Transfer of two embryos 31 (61%) 40 (78%)

wotw7

4

OcItts

tteork

pTtb�

cacotuat

C4 Transfer of one embryo

Total

hich is exactly the same as the success of the ART methodol-gy, given by the overall pregnancy rate. The sorting accuracy ofhe type II (when Electre Tri-C provides a range of categorieshich contains the one provided by the medical experts) is

5% (see also Table 9).

. Discussion

ur approach is an informatics tools to deal with a medi-al application. It is based on a theoretical sound basis [23].t is co-constructed through an interactive process betweenhe analyst and the medical experts. It is also an openool to incorporate other tools, in particular those related totatistics.

The key findings of this work were particularly impor-ant, with a special emphasis on the “confront” betweenhe “decisions-suggestions” provided by the tool and thembryologist current decisions. The fact that the embryol-gist agreed with the revision of some of his judgmentsepresents the major finding to add to the body of pertinentnowledge.

Some mechanisms related to this finding should be pin-ointed. As an example, consider the couple a50 (see alsoables B.2). Taking into account the set of parameters,his couple a50 strongly dominates the reference couples

1 and b2. This means that �(a50,b1) = �(a50,b2) = 1.00 and(b1,a50) = �(b2,a50) = 0.00. The assignment of a50 to C1 or C2

annot be justified! The reference couple b4 does not dominate

50 only because of the criteria g3 and g5, but such a referenceouple is strictly preferred to a50. Once again the assignmentf a to C is very difficult to be justified taking into account

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

50 4

he power of the criteria g3 and g5. The criteria g3 and g4 aresed for justifying a small difference in the judgment of thessignment of a50 in C3. But, since the power of each one ofhese criteria is at most 0.15, then the assignment of a50 to

4 (7%) 1 (2%)

51 (100%) 51 (100%)

C3 is well supported by 85% of the criteria, including the twomost important criteria, g1 and g6.

Despite the obtained results, we need to apply this model ina prospective way and step-by-step to allow fine tuning of thedifferent parameters of the model. This work allowed defin-ing a clear perspective of this research. As shown in Table 7,the subset of criteria {g4, g6, g7} has a relevant influence onthe results, since the cumulative sum of the weights is 55%.Therefore, further research is needed to design at least twospecific decision aiding assignment models: one of them mustprovide a “recommendation” concerning the evaluation of thewomen according to several criteria; and the other one mustprovide “recommendations” concerning the choice of the fourbest embryos as well as the evaluation of their morphologicaland developmental quality.

This work suggests a way to decreasing multiple pregnan-cies while maintaining the overall pregnancy rates. A relevantnext step will be to confirm these findings provided by themodel in a future prospective study. In addition, extendingthis type of analysis by incorporating detailed views of theclinicians will be another future avenue of research.

Author contributions

The idea for developing this informatics tool was firstly pro-posed by J.R.F. and C.E.P. Then S.M. has modeled a firstprototype of this decision aiding tool, based on a first ver-sion of the Electre Tri-C method proposed by J.A.D., J.R.F.,and B.R. Gathering data and structuring the problem throughthe set of criteria was done interactively by the decision ana-lysts (in this case J.R.F. and S.M.) and the decision makers

multiple criteria decision aiding sorting model applied to assisted

or medical experts (M.J.C. and C.E.P.). Some aspects relatedto the modeling process were done by all the authors.J.A.D., J.R.F., and B.R. applied a new version of Electre Tri-C.J.A.D. and S.M. made a draft of a first version of the paper,

INc a l i

ARTICLEIJB-2709; No. of Pages 12

8 i n t e r n a t i o n a l j o u r n a l o f m e d i

which was improved mainly by J.A.D. with the comments byJ.R.F., C.E.P., and B.R. All authors have approved this revisedfinal version.

Conflicts of interest statement

None declared.

Acknowledgements

Juscelino Almeida Dias acknowledges the financial supportfrom the Fundacão para a Ciência e a Tecnologia, Portugal(Grant SFRH/BD/22985/2005), the COST Action Number IC0602,and the Fundacão Calouste Gulbenkian, Portugal (Grant109475). The first two authors also acknowledge the financialsupport from the Luso-French bilateral cooperation agreementbetween LAMSADE and CEG-IST (FCT/CNRS 2009).

Summary pointsWhat was known before the study

• The successful application of multiple criteria decisionanalysis tools to health care problems and medicine,but not related to ART.

• A set of elementary consequences in ART (medicalexams, couple characteristics and attributes, etc.) thathave been further grouped into homogeneous con-sequences leading to the construction of the set ofcriteria.

• The basic concepts of the decision aiding informaticstool applied.

What the study has added to the knowledge

• Despite the variety of studies applied to health careproblems and medicine to our best knowledge no mul-tiple criteria informatics tool was applied to deal withART problems.

• The way of taking into account the preferences fromthe ART medical experts is new, in particular, the onerelated to the definition of some thresholds to take intoaccount the imperfect knowledge of the data regardingART.

• The fact that the informatics tool was well-accepted bythe ART medical experts. After observing the results,they agreed that the tool could provide improved deci-sions and they are looking forward to implement thereferred tool, with additional options for statistics, infuture ART decision processes.

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

Appendix A. The outranking credibility index

Let us assume, without loss of generality, that all the criteriagj ∈ F are to be maximized which means that the prefer-

PRESSn f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

ences increase when the criterion performances increase too.Consider two couples, denoted a and a′. When using the dis-criminating thresholds defined in Section 2, the followingbinary relations can be derived for each criterion [16]:

(1) If | gj(a) − gj(a′) |≤qj, then a is indifferent to a′ according togj, denoted a Ij a′. Let C(a I a′) be the subset of criteria suchthat a Ij a′.

(2) If gj(a) − gj(a′) > pj, then a is strictly preferred to a′ accordingto gj, denoted a Pj a′. Let C(a P a′) be the subset of criteriasuch that a Pj a′.

(3) If qj < gj(a) − gj(a′) ≤ pj, then a is weakly preferred to a′ (anhesitation between indifference and strict preference),denoted a Qj a′. Let C(a Q a′) be the subset of criteria suchthat a Qj a′.

The credibility of the comprehensive outranking of a overa′, denoted �(a,a′), which reflects the strength of the statement“a outranks a′” (a is at least as good as a′) when taking allthe criteria from F into account, is defined as follows (see alsoTable B.2):

�(a, a′) = c(a, a′)n∏

j=1

Tj(a, a′), (A.1)

where

Tj(a, a′) ={ 1 − dj(a, a′)

1 − c(a, a′)if dj(a, a′) > c(a, a′),

1 otherwise.(A.2)

dj(a, a′) =

⎧⎪⎨⎪⎩

1 if gj(a) − gj(a′) < −vj,gj(a) − gj(a′) + pj

pj − vjif − vj ≤ gj(a) − gj(a′) < −pj,

0 if gj(a) − gj(a′) ≥ −pj.

(A.3)

c(a, a′) =∑

j ∈ C(aPa′)

wj +∑

j ∈ C(aQa′)

wj +∑

j ∈ C(aIa′)

wj +∑

j ∈ C(a′Qa)

wjϕj,

(A.4)

ϕj = gj(a) − gj(a′) + pj

pj − qj∈ [0, 1[. (A.5)

where �(a,a′) is based on the aggregation of the partial con-cordance indices, expressed by formulas (A.4) and (A.5), withthe partial discordance indices, expressed by formula (A.3). Forthis aggregation issue, we assume without loss of generalitythat

∑n

j=1 = 1 and wj > 0, j = 1, . . ., n. All the thresholds havenon-negative values such that vj ≥ pj ≥ qj ≥ 0.

multiple criteria decision aiding sorting model applied to assisted

Appendix B. The comprehensive results

Tables B.1–B.3.

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, a multiple criteria decision aiding sorting model applied to assistedreproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

ARTICLE IN PRESSIJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx 9

Table B.1 – Performances of the couples according to the set of criteria.

Couples Criteria Couples Criteria

g1 g2 g3 g4 g5 g6 g7 g1 g2 g3 g4 g5 g6 g7

a1 36 1 6 4 3 16 3 a27 37 6 1 4 3 17 5a2 28 1 2 5 2 20 5 a28 34 3 3 2 3 13 5a3 38 1 6 1 3 14 4 a29 36 6 3 3 3 15 4a4 28 6 2 6 3 8 3 a30 33 5 5 4 3 9 4a5 44 3 2 3 1 11 3 a31 34 13 1 2 3 15 5a6 42 4 4 3 3 14 5 a32 33 5 4 3 3 14 5a7 30 1 6 4 2 7 3 a33 39 3 5 2 3 8 2a8 33 2 4 4 3 17 5 a34 37 3 5 4 3 16 4a9 36 3 5 2 3 9 2 a35 34 3 3 2 3 14 5a10 27 2 2 4 3 16 5 a36 38 4 4 5 3 13 5a11 39 3 4 3 3 5 1 a37 33 3 4 5 3 16 5a12 36 1 4 3 3 14 2 a38 34 2 4 3 3 16 4a13 37 2 4 5 3 15 3 a39 35 2 3 5 3 19 5a14 27 2 3 3 3 18 4 a40 34 3 2 4 3 13 5a15 30 3 3 2 3 16 5 a41 38 7 6 4 3 10 1a16 33 4 5 4 3 16 3 a42 26 5 3 2 1 15 5a17 38 6 3 4 3 13 4 a43 34 2 3 3 3 10 3a18 30 2 3 4 3 17 5 a44 28 3 4 3 3 14 4a19 32 3 1 3 3 14 4 a45 30 1 3 6 3 10 1a20 30 3 3 3 3 20 5 a46 31 2 3 2 3 16 5a21 33 12 5 3 3 14 4 a47 33 2 3 4 3 15 4a22 37 2 6 2 3 14 1 a48 41 2 3 3 3 19 5a23 40 3 1 2 3 12 3 a49 40 2 5 2 3 12 2a24 34 3 1 3 3 10 3 a50 32 3 1 3 3 16 4a25 28 2 3 3 3 12 4 a51 40 4 5 3 3 12 4a26 33 3 5 3 3 12 2

Table B.2 – Comprehensive outranking credibility indices.

Couples �(a,bh) �(bh, a)

b1 b2 b3 b4 b1 b2 b3 b4

a1 1.00 0.94 0.60 0.01 0.00 0.01 0.90 1.00a2 1.00 1.00 0.98 0.77 0.00 0.00 0.14 0.79a3 1.00 0.79 0.22 0.00 0.01 0.48 0.90 1.00a4 1.00 0.79 0.36 0.07 0.00 0.00 0.52 1.00a5 0.73 0.71 0.01 0.00 0.09 0.27 0.94 1.00a6 1.00 0.73 0.22 0.00 0.01 0.12 0.81 1.00a7 1.00 0.73 0.16 0.00 0.00 0.09 0.90 1.00a8 1.00 1.00 0.85 0.30 0.00 0.00 0.60 1.00a9 1.00 0.79 0.29 0.00 0.00 0.61 1.00 1.00a10 1.00 1.00 0.85 0.58 0.00 0.00 0.48 1.00a11 0.79 0.29 0.01 0.00 0.22 0.67 1.00 1.00a12 1.00 1.00 0.45 0.01 0.00 0.13 0.90 1.00a13 1.00 1.00 0.33 0.02 0.00 0.00 1.00 1.00a14 1.00 1.00 0.85 0.39 0.00 0.00 0.46 1.00a15 1.00 1.00 0.85 0.46 0.00 0.01 0.75 1.00a16 1.00 1.00 0.60 0.00 0.00 0.01 1.00 1.00a17 1.00 1.00 0.27 0.00 0.00 0.35 0.94 1.00a18 1.00 1.00 0.85 0.58 0.00 0.00 0.54 1.00a19 1.00 1.00 0.64 0.03 0.00 0.00 0.94 1.00a20 1.00 1.00 0.85 0.69 0.00 0.00 0.54 0.79a21 0.90 0.90 0.48 0.00 0.00 0.05 1.00 1.00a22 1.00 0.75 0.04 0.00 0.02 0.40 1.00 1.00a23 1.00 0.73 0.10 0.00 0.00 0.36 0.94 1.00a24 1.00 1.00 0.45 0.01 0.00 0.09 0.94 1.00a25 1.00 1.00 0.64 0.29 0.00 0.00 0.67 1.00a26 1.00 1.00 0.38 0.00 0.00 0.16 1.00 1.00a27 1.00 1.00 0.48 0.12 0.00 0.01 0.54 1.00a28 1.00 1.00 0.64 0.08 0.00 0.04 0.75 1.00a29 1.00 1.00 0.54 0.00 0.00 0.03 0.94 1.00a30 1.00 0.79 0.42 0.00 0.00 0.15 1.00 1.00

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, a multiple criteria decision aiding sorting model applied to assistedreproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

ARTICLE IN PRESSIJB-2709; No. of Pages 12

10 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

– Table B.2 (Continued)

Couples �(a,bh) �(bh, a)

b1 b2 b3 b4 b1 b2 b3 b4

a31 0.90 0.90 0.54 0.10 0.00 0.05 0.75 1.00a32 1.00 1.00 0.54 0.12 0.00 0.00 0.81 1.00a33 1.00 0.52 0.02 0.00 0.00 0.88 1.00 1.00a34 1.00 1.00 0.52 0.00 0.00 0.01 1.00 1.00a35 1.00 1.00 0.64 0.09 0.00 0.03 0.75 1.00a36 1.00 1.00 0.52 0.03 0.00 0.12 0.81 1.00a37 1.00 1.00 1.00 0.17 0.00 0.00 0.81 1.00a38 1.00 1.00 0.85 0.04 0.00 0.00 1.00 1.00a39 1.00 1.00 1.00 0.52 0.00 0.00 0.54 1.00a40 1.00 1.00 0.64 0.12 0.00 0.00 0.75 1.00a41 1.00 0.65 0.00 0.00 0.04 0.83 1.00 1.00a42 1.00 0.98 0.36 0.26 0.00 0.00 0.48 1.00a43 1.00 1.00 0.45 0.01 0.00 0.15 0.94 1.00a44 1.00 1.00 0.64 0.23 0.00 0.00 0.73 1.00a45 1.00 0.81 0.60 0.14 0.00 0.12 0.69 1.00a46 1.00 1.00 0.85 0.22 0.00 0.01 0.75 1.00a47 1.00 1.00 0.64 0.07 0.00 0.00 0.94 1.00a48 1.00 0.73 0.58 0.00 0.00 0.05 0.54 1.00a49 1.00 0.73 0.04 0.00 0.00 0.67 1.00 1.00a50 1.00 1.00 0.85 0.04 0.00 0.00 0.94 1.00a51 1.00 0.73 0.28 0.00 0.00 0.29 1.00 1.00

Table B.3 – Electre Tri-Cversus medical expert’s assignment results.

Electre Tri-C results a ART current results

Couples Lowest category Highest category Medical expert a Pregnancy Nr. of live birth b

a1 C3(2) C3(2) C2(3) Single 1 (1)a2 C4(1) C4(1) C3(2) No -a3 C2(3) C2(3) C3(2) No -a4 C2(3) C3(2) C3(2) No -a5 C1(4) C2(3) C2(3) No -a6 C2(3) C3(2) C2(3) No -a7 C2(3) C3(2) C3(2) No -a8 C3(2) C3(2) C3(2) No -a9 C2(3) C2(3) C3(2) Twin 1 (2)a10 C4(1) C4(1) C3(2) No -a11 C1(4) C2(3) C2(3) Single 1 (1)a12 C3(2) C3(2) C3(2) Twin 2 (2)a13 C3(2) C3(2) C3(2) Single 1 (1)a14 C3(2) C3(2) C3(2) Single 0 (1)a15 C3(2) C3(2) C3(2) No -a16 C3(2) C3(2) C3(2) No -a17 C2(3) C2(3) C2(3) No -a18 C4(1) C4(1) C3(2) No -a19 C3(2) C3(2) C3(2) No -a20 C4(1) C4(1) C3(2) No -a21 C3(2) C3(2) C3(2) No -a22 C2(3) C2(3) C3(2) No -a23 C2(3) C2(3) C1(4) Twin 1 (2)a24 C3(2) C3(2) C3(2) Twin 2 (2)a25 C3(2) C3(2) C3(2) No -a26 C3(2) C3(2) C3(2) Single 1 (1)a27 C3(2) C3(2) C3(2) No -a28 C3(2) C3(2) C4(1) No -a29 C3(2) C3(2) C3(2) No -a30 C2(3) C3(2) C2(3) Single 0 (1)a31 C3(2) C3(2) C3(2) Single 1 (1)a32 C3(2) C3(2) C3(2) Single 1 (1)a33 C2(3) C2(3) C2(3) Single 1 (1)a34 C3(2) C3(2) C3(2) Single 0 (1)a35 C3(2) C3(2) C3(2) Twin 2 (2)

ARTICLE IN PRESSIJB-2709; No. of Pages 12

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx 11

– Table B.3 (Continued)

Electre Tri-C results a ART current results

Couples Lowest category Highest category Medical expert a Pregnancy Nr. of live birth b

a36 C3(2) C3(2) C2(3) Single 1 (1)a37 C3(2) C3(2) C3(2) Twin 2 (2)a38 C3(2) C3(2) C3(2) Single 1 (1)a39 C3(2) C3(2) C3(2) Single 1 (1)a40 C3(2) C3(2) C3(2) Single 1 (1)a41 C2(3) C2(3) C3(2) Single 0 (1)a42 C3(2) C3(2) C3(2) Single 1 (1)a43 C3(2) C3(2) C3(2) Single 0 (1)a44 C3(2) C3(2) C3(2) Single 1 (1)a45 C3(2) C3(2) C3(2) Single 1 (1)a46 C3(2) C3(2) C3(2) Single 1 (1)a47 C3(2) C3(2) C3(2) Single 1 (1)a48 C2(3) C3(2) C3(2) Twin 2 (2)a49 C2(3) C2(3) C3(2) Single 0 (1)a50 C3(2) C3(2) C3(2) Single 1 (1)a51 C2(3) C2(3) C2(3) Single 1 (1)

a ( ) the corresponding number of embryos.b ( ) the possible number of live birth.

r

e f e r e n c e s

[1] A. Barnato, H. Llewellyn-Thomas, E. Peters, L. Siminoff, E.Collins, M. Barry, Communication and decision making incancer care: Setting research priorities for decisionsupport/patients’ decision aids, Medical Decision Making 27(2007) 626–634.

[2] D. West, P. Mangiameli, R. Rampal, V. West, Ensemblestrategies for a medical diagnostic decision support system:A breast cancer diagnosis application, European Journal ofOperational Research 162 (2005) 532–551.

[3] N. Belacel, Multicriteria assignment method proaftn:methodology and medical application, European Journal ofOperational Research 125 (2000) 175–183.

[4] J. Carter, N. Ritchey, F. Castro, L. Caccamo, E. Kessler, B.Erickson, Analysis of three decision-making methods: abreast cancer patient as a model, Medical Decision Making19 (1999) 49–57.

[5] J. Felli, R. Noel, P. Cavazzoni, A multiattribute model forevaluating the benefit–risk profiles of treatmentalternatives, Medical Decision Making 29 (2009) 104–115.

[6] P. Mangiameli, D. West, R. Rampal, Model selection formedical diagnosis decision support systems, DecisionSupport Systems 36 (2004) 247–259.

[7] N. Belacel, M. Boulassel, Multicriteria fuzzy classificationprocedure procftn: methodology and medical application,Fuzzy Sets and Systems 141 (2004) 203–217.

[8] N. Belacel, M. Boulassel, Multicriteria fuzzy assignmentmethod: a useful tool to assist medical diagnosis, ArtificialIntelligence in Medicine 21 (2001) 201–207.

[9] N. Belacel, P. Vincke, M. Scheiff, M. Boulassel, Acuteleukemia diagnosis aid software using multicriteria fuzzyassignment methodology, Computer Methods and Programsin Biomedicine 64 (2001) 145–151.

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

[10] P. Du Bois, J. Brans, F. Cantraine, B. Mareschal, Medicis: anexpert system for computer-aided diagnosis using thepromethee multicriteria method, European Journal ofOperational Research 39 (1989) 284–292.

[11] D. Sittig, A. Wright, L. Simonaitis, J. Carpenter, G. Allen, B.Doebbeling, A. Sirajuddin, J. Ash, B. Middleton, The state ofthe art in clinical knowledge management: an inventory oftools and techniques, International Journal of MedicalInformatics 79 (2010) 44–57.

[12] M. Liberatore, R. Nydick, The analytic hierarchy process inmedical and health care decision making: a literaturereview, European Journal of Operational Research 189 (2008)194–207.

[13] M. Bohanec, B. Zupan, V. Rajkovic, Applications of qualitativemulti-attribute decision models in health care, InternationalJournal of Medical Informatics 58–59 (2000) 191–205.

[14] L. Lhotska, V. Marik, T. Vlcek, Medical applications ofenhanced rule-based expert systems, International Journalof Medical Informatics 63 (2001) 61–75.

[15] S. Matias, A multicriteria decision aiding assignmentmethodology for assisted reproductive technology, MasterDissertation in Biomedicine, Instituto Superior Técnico,Technical University of Lisbon, Lisbon, Portugal, 2008.

[16] J. Almeida-Dias, J. Figueira, B. Roy, Electre Tri-C: a multiplecriteria sorting method based on characteristic referenceactions, European Journal of Operational Research 204 (2010)565–580.

[17] C. Goodman, R. Ahn, Methodological approaches of healthtechnology assessment, International Journal of MedicalInformatics 56 (1999) 97–105.

[18] R. Haux, Health information systems: past, present, future,International Journal of Medical Informatics 75 (2006)268–281.

[19] B. Roy, Méthodologie Multicritère d’Aide à la Décision,Economica, Paris, France, 1985 (English translation:Multicriteria Methodology for Decision Aiding, KluwerAcademic Publishers, Dordrecht, 1996).

[20] J. Figueira, B. Roy, Determining the weights of criteria in theelectre type methods with a revised Simos’ procedure,

multiple criteria decision aiding sorting model applied to assisted

European Journal of Operational Research 139 (2002)317–326.

[21] M. Rogers, M. Bruen, A new system for weightingenvironmental criteria for use within Electre III, EuropeanJournal of Operational Research 107 (1998) 552–563.

INc a l i

ARTICLEIJB-2709; No. of Pages 12

12 i n t e r n a t i o n a l j o u r n a l o f m e d i

Please cite this article in press as: J.R. Figueira, et al., Electre Tri-C, areproduction, Int. J. Med. Inform. (2011), doi:10.1016/j.ijmedinf.2010.12.001

[22] V. Mousseau, Eliciting information concerning the relativeimportance of criteria, in: P. Pardalos, Y. Siskos, C.Zopounidis (Eds.), Advances in Multicriteria Analysis, vol. 5,Nonconvex Optimization and its Applications, KluwerAcademic Publishers, Dordrecht, 1995, pp. 17–43.

PRESSn f o r m a t i c s x x x ( 2 0 1 1 ) xxx–xxx

multiple criteria decision aiding sorting model applied to assisted

[23] C. Green, P. Fortin, M. Maclure, A. Macgregor, S. Robinson,Information system support as a critical success factor forchronic disease management: necessary but not sufficient,International Journal of Medical Informatics 75 (2006)818–828.