Efficiency measurement of research groups using DataEnvelopment Analysis and Bayesian networks

Cristhian Fabian Ruiz • Ricardo Bonilla • Diego Chavarro •

Luis Antonio Orozco • Roberto Zarama • Xavier Polanco

Received: 17 November 2009 / Published online: 16 December 2009� Akademiai Kiado, Budapest, Hungary 2009

Abstract Applications of non-parametric frontier production methods such as Data

Envelopment Analysis (DEA) have gained popularity and recognition in scientometrics.

DEA seems to be a useful method to assess the efficiency of research units in different

fields and disciplines. However, DEA results give only a synthetic measurement that does

not expose the multiple relationships between scientific production variables by discipline.

Although some papers mention the need for studies by discipline, they do not show how to

take those differences into account in the analysis. Some studies tend to homogenize the

behaviour of different practice communities. In this paper we propose a framework to

make inferences about DEA efficiencies, recognizing the underlying relationships between

production variables and efficiency by discipline, using Bayesian Network (BN) analysis.

Two different DEA extensions are applied to calculate the efficiency of research groups:

one called CCRO and the other Cross Efficiency (CE). A BN model is proposed as a

method to analyze the results obtained from DEA. BNs allow us to recognize peculiarities

of each discipline in terms of scientific production and the efficiency frontier. Besides, BNs

provide the possibility for a manager to propose what-if scenarios based on the relations

found.

Keywords Data Envelopment Analysis � Bayesian Networks � Scientific production �Efficiency measurement � Research groups

C. F. Ruiz (&) � R. Bonilla � R. ZaramaCenter for Basic and Applied Interdisciplinary Studies on Complexity, CeiBA-Complejidad,Universidad de los Andes, Calle 19 No. 1 – 11, Bogota, DC, Colombiae-mail: cr-ruiz@uniandes.edu.co

D. ChavarroVice-Rectory for Research, Universidad de los Andes, Bogota, DC, Colombia

L. A. OrozcoSchool of Management, Universidad de los Andes, Bogota, DC, Colombia

X. PolancoCenter for Basic and Applied Interdisciplinary Studies on Complexity,CeiBA-Complejidad, Universidad del Rosario, Bogota, DC, Colombia

123

Scientometrics (2010) 83:711–721DOI 10.1007/s11192-009-0122-y

Introduction

In this paper we measure the efficiencies of research groups. Research groups are con-

sidered ‘‘decision making units’’ (DMUs). Our proposal combines two methods: Data

Envelopment Analysis (DEA) and Bayesian Networks (BNs). DEA gives an overview of

the relative efficiency of a research group in terms of its production. Complementarily,

BNs provide a useful representation of the influence between production variables and

DEA efficiency. Our major contribution is the use of BNs as a graphical probabilistic

reasoning tool to assess the influence of production variables on the efficiency of a group.

With this tool it is possible to propose dynamic scenarios to simulate changes in the

variables and obtain a reconfiguration of the influence network based on those changes.

A review of the literature shows that DEA has become useful to approximate the

efficiency of science and technology (S&T). Bonaccorsi and Daraio (2004) highlight the

use of non-parametric efficiency frontier methods (e.g. DEA) as the most useful to evaluate

efficiency in S&T systems. In scientometrics, the DEA method has been applied to

countries (Bonaccorsi and Daraio 2004; Rousseau and Rousseau 1997, 1998), scientific

fields (Garg et al. 2005; Meng et al. 2006), universities (Bonaccorsi et al. 2006), research

centres (Korhonen et al. 2001), and research groups (Guan and Wang 2004; Restrepo and

Villegas 2007).

Although it is frequently suggested to perform DEA analyses by scientific disciplines,

the differences between those disciplines are not shown empirically. Bonaccorsi and Da-

raio (2004, p. 53) suggest the introduction of ‘‘…assumptions about the structure of the

technology and the behaviour of producers’’. This suggestion points to the need for a

grouping criterion before applying DEA. Grouping by similar characteristics (disciplines,

in this case) is a way to produce comparable DMUs. However, once the results of DEA are

obtained, it is necessary to apply a method to go further and provide a mean for the

interpretation of the results. As Cherchye and Vanden Abeele (2005, p. 499) state:

It is important to emphasize that the computed efficiency values can have different

interpretations. (…) A related point concerns the possible interpretation of the

(in)efficiency results in terms of the unobservable characteristics of the research

inputs; this essentially means that efficiency differences between research programs

may at least partially reflect differences in the quality of the researchers who com-

pose the programs. In our opinion, such quality differences equally refer to diverging

efficiencies in the research production process

To map the internal configuration of efficiency results to comprehend those diverging

efficiencies we propose the use of BNs. BNs are widely used in data mining, machine

learning, and decision theory. BNs are used to construct influence diagrams, decision

making tools, and classifiers.

However, BNs are not widely used in scientometrics. BNs have been used to classify

authors according to citation patterns (Lehmann et al. 2008), to assess the impact of

variables in the innovation collaboration process (Kim and Park 2008) and webometrics

(Lamirel et al. 2004). Leydesdorff (1992) proposed a Bayesian approach to science studies

that can be understood as the analysis of the interaction between two dynamic principles:

hypotheses and evidences. In this paper we propose a BN analysis of DEA results. This

analysis is proposed to take into account both the underlying relationships of the variables

of the production of each discipline and the impact of production variables on the effi-

ciency. In that way it becomes possible to predict the network’s reconfiguration after

applying changes.

712 C. F. Ruiz et al.

123

Methodology

DEA is a method to make comparative analyses of the relative efficiencies of research

groups. The unit of analysis is called a Decision Making Unit (DMU). A DMU is the entity

that transforms a certain number of inputs into outputs through a specific process (Cooper

et al. 2004). The measurement of efficiency quantifies the distance between one DMU and

the efficiency frontier. Two DEA extensions were applied: CCRO, proposed by Charnes

et al. (1978) output oriented, and cross-efficiency (CE), proposed by Sexton et al. (1986).

CCRO constructs an efficiency frontier to compare all DMUs. The efficiency frontier is

delimited by the DMUs with the better ratio between outputs and inputs (O/I), i.e. those

that obtain a greater ratio. In this case a matrix of N rows (DMUs) and M columns (Input

and output variables) is transformed into a vector of N rows that contains the efficiency

values. An N-dimensional efficiency frontier is constructed based on the output values for

the transformation.

Geometrically, each DMU is a point in the M-dimensional space using the column

values as coordinates. Outstanding DMUs are the ones that reach greater production using

fewer resources. DMUs that have efficiency equal to 1 are the most efficient. The border is

delineated by them. To establish the relative efficiency of other research groups, these are

projected onto the hyperplane perpendicular to the border of the outputs (CCRO). Doyle

and Green (1994, p. 569) argue that CCRO high efficiency can be reached by having a high

value in one single output.

CE take not only the efficient DMUs but also the rest of them into account. It compares

each DMU with all other DMUs (Doyle and Green 1994, p. 570). Thus CE is a peer

appraisal approach. We use the implementation of Restrepo and Villegas (2007)1 in Matlab

for CCRO and CE. This implementation was programmed using the optimization linprogtoolbox, and it solves the linear program by means of the Simplex Method.

The results, obtained applying the CE extension by discipline, were incorporated into a

BN model. A BN is a probabilistic graphical model that represents a set of random

variables and their relationship (probabilistic independencies) by means of a Direct Acy-

clic Graph (DAG) and a set of probability tables (Heckerman 1999; Kjaerulff and Madsen

2008). BNs allow information analysis based on relationship dependence between vari-

ables. This approach is useful to figure out the effect of a particular intervention on a

variable (Heckerman 1999).

The BN graph shows dependent and independent relationships between variables. Those

relationships are built using the D-separation concept (Pearl 2000). The D-separation

criterion sets the conditions by which information is transferred between variables. We

speak of information transmission when an instantiated variable modifies the degree of

belief or plausibility of another. It is said that two variables are d-separated when infor-

mation cannot be transmitted between them. The graph in Fig. 1 shows three basic forms

of three nodes: serial, divergent and convergent.

In a serial structure the transfer of information between variables A and C occur as long

as there is no evidence of the intermediate variable B. The transfer is possible indepen-

dently of the direction of the links. The criterion shows how the degree of belief or

plausibility of the variable C is modified by evidence of variable A and vice versa. If there

is evidence of the intermediate variable B, then information transmission is not possible

between A and C, and it is said that the variables are d-separated. In a divergent structure

1 Retrieved February 2009, from http://industrial.udea.edu.co/jgvillegas/Pagina%20DEA/index.html.

Efficiency measurement of research groups 713

123

the criterion is applied in the same way as in serials. In the case of convergent structures

information transmission is only possible if there is evidence of the intermediate variable.

If not, information is not transmitted from A to C or C to A, and the variables are said to be

d-separated.

Information calculated by the DAG structure and the probability table enables the

assessment of the impact on efficiency of each variable (Pearl 2000; Spirtes et al. 2000).

The BN structure and conditional probability tables are constructed using software called

Hugin Lite.2 Two main algorithms are used: the NPC constrain based algorithm and the

maximization expectation (ME) algorithm. The NPC algorithm is used to construct the BN

structure by means of a statistical independence test using mutual information calculus.

The ME algorithm is used to construct probability tables based on likelihood maximization

over the data.

Data

The data for this study came from the ScienTI database.3 ScienTI is a Colombian repos-

itory of research groups and bibliographic references of research products. The sample is

composed of 553 research groups organized in 8 sets that correspond to the following

disciplines: Electrical Engineering (65), Civil Engineering (43), Physics (94), Chemistry

(78), Biology (53), Ecology (45), Law (89), and Economics (86).

To perform DEA we defined two input variables and five output variables. The input

variables are the number of researchers in the group and the group experience. The group

experience is measured by the number of years since its foundation. The output variables

are papers, books, chapters, proceedings, and theses. To perform BN analysis, all the

values of the output variables plus the efficiency calculated with the CE extension are

considered as random variables.

Results

Both DEA extensions (CCRO and CE) were applied to 553 Colombian research groups.

We obtained the following three main results: A comparison between CCRO and CE in the

whole sample, a comparison of CE between grouped and not grouped DMUs by disci-

plines, and the construction of BNs for each discipline using CE results.

Fig. 1 Basic connections:a serial, b divergent andc convergent

2 Retrieved February 2009, from http://www.hugin.com/.3 Retrieved July 2008, from http://thirina.colciencias.gov.co:8081/scienti/jsp/grupos.jsp.

714 C. F. Ruiz et al.

123

When comparing CCRO and CE we observe that the values of CCRO are always higher

than values of CE (Fig. 2). The difference could be explained by following Cooper et al.

(2004, p. 22). They state that CCRO assign ‘‘unreasonably low or excessively high values

to the multipliers in an attempt to drive the efficiency rating for a particular DMU as high

as possible’’. That means that a DMU can reach high values of efficiency with high

production values in only one output. In that sense Doyle and Green (1994, p. 569) argue

that there are two ways to be highly efficient: ‘‘in one way, an efficient DMU weights a

spread of inputs and a spread of outputs to achieve efficiency; in the other, a DMU

achieves efficiency by weighting a single input and a single output’’. CCRO could be

interpreted as an approximation of ‘‘self appraisal’’. It means the DMUs are evaluated by

their production without taking into account the production patterns of the other DMUs. On

the other hand, CE could be understood as ‘‘peer appraisal’’, since this extension not only

evaluates the DMUs separately but compares the efficiency value and production levels of

other DMUs (Doyle and Green 1994, p. 570).

The second result is the comparison between CE calculated for each DMU grouped by

disciplines and CE calculated for the 553 DMUs (Bulk). With this comparison we deter-

mined the difference between the evaluation by discipline and the whole sample. We

applied a statistical test based on the comparison between the Cumulative Distribution

Function (CDF) for the CE values obtained by discipline and the CE values in the whole

sample. This is the Kolmogorov–Smirnov test (KS-test) that is based on the maximum

distance between these two curves. The hypotheses are:

Ho : FdðxÞ ¼ FmðxÞ

H1 : FdðxÞ 6¼ FmðxÞ

Fd and Fm are the Cumulative Distribution Function (CDF) by discipline and the whole

sample, respectively. The k statistic is:ffiffiffiffiffiffiffiffiffi

2 � px

r

X

/

i¼1

e�ð2i�1Þ2p2=ð8x2Þ

From this calculation, we determine that the critical value of the k statistic is 1.36 with a

significance level a of 0.05. Figure 3 shows the CDF obtained for the 8 studied disciplines. The

dark line is the CDF for CE in the sample and the other one is the evaluation in the discipline. We

observe that efficiencies calculated by discipline are usually higher than efficiencies calculated

in the whole sample. Physics is the only discipline where the k-statistic is less than 1.36. It means

that Ho is only accepted in Physics. In the other disciplines Ho is rejected.

Fig. 2 Rank Plot of CCRO andCE values for 553 researchgroups in Colombia

Efficiency measurement of research groups 715

123

Figure 4 shows the relation of the standard deviation of each discipline (Y-axis) and the

value of the k-statistic (X-axis). A high value in the X-axis can be interpreted as a large

difference between a disciplinary evaluation and a general one. A high value in the Y-axis

Fig. 3 Cumulative Distribution Function of CE values calculated by discipline and CE values calculatedover the 553 DMUs

716 C. F. Ruiz et al.

123

indicates that the efficiency values are spread out over a large range of values. Physics is

the only discipline with the lowest value in these two axes. From this we can conclude that

the groups inside Physics have production patterns that permit them to be evaluated in the

same way inside the discipline and outside it. Ecology is the discipline with the highest

value of dispersion and Chemistry has the highest value of the k-statistic.

The last result is the BN construction for each discipline using efficiency calculated with

the CE extension and the DEA output variables as nodes. BNs represent the joint probability

distribution of selected variables. They show the dependency relationships between the

output variables and the efficiency. The construction of the BN was unsupervised which

means the structure and the probability tables reflect the underlying structure of the data. In

Fig. 5 we can observe different structures in the 8 BNs. This observation suggests that each

discipline has its own underlying relationships between the variables of production. For

example, the relation between Papers and CE has a different direction in Physics and in

Chemistry. In Civil Engineering the variable is not related with CE. It means that knowing

the state of this variable does not change the probability distribution of CE. In this case, we

cannot infer about the state of CE based on the number of theses.

These relational structures give the possibility to perform manipulations on the variables

and get the reconfiguration of a network based on a certain stimulus. We are going to

observe the response given by two disciplines to the same stimulus. In the first scenario

(see Fig. 6), both disciplines maintain their original network configuration. In the second

(see Fig. 7), both disciplines will increase the number of papers. Based on these two

scenarios, although there is an increase in efficiency in both disciplines, the internal

reconfiguration of the networks is different. In physics a greater production of papers also

implies a greater production of theses and conferences. In chemistry, a high production of

articles is related to an increase in the number of chapters and theses. This example

illustrates the use of BN in scientometrics and shows that BNs give us a better approxi-

mation that goes more in depth than the results produced by a DEA analysis.

To quantify this observation we calculated the impact of each variable over the effi-

ciency by means of a sensitivity analysis. For this, a calculus of mutual information was

done. It is a measurement of the information quantity shared by two variables X and Y. In

terms of uncertainty, it can be expressed as the fact that knowing the state of one variable

modifies the degree of uncertainty over the other. The test of independence is made from

the statistical likelihood test between two hypotheses:

Fig. 4 Standard Deviation and k-statistic value for all disciplines

Efficiency measurement of research groups 717

123

H0 : PðX; YÞ ¼ PðXÞPðYÞ

H1 : PðX; YÞ 6¼ PðXÞPðYÞ

The expression for G2 is:

G2 ¼ 2X

xy

NXY LogNXY

EXY

� �

Fig. 5 Representation of the relationship between production variables and CE using BNs

718 C. F. Ruiz et al.

123

where NXY is the number of cases in the simple in which X = x and Y = y. In the term

EXY ¼ NxNy

N , Nx is the number of cases in the sample in which X = x and N is the total

number of cases.

Fig. 6 Distribution of each variable in a Physics and b Chemistry Bayesian Network

Fig. 7 Distribution of each variable in a Physics and b Chemistry, increasing the number of papers

Efficiency measurement of research groups 719

123

The value G2 allows accepting or rejecting H0. If G2 is small enough (G2 \ c), the

hypothesis cannot be rejected. The level of significance a is used to set the value of the

threshold c. This value was manually adjusted in the software.

Table 1 shows the mutual information (MI) by discipline between CE and each pro-

duction variable. The MI quantifies information sharing between the two variables and can

be interpreted as the amount of information gained over the CE when the second variable is

known. A high value in the MI can be interpreted as a high impact from the product

variable over the CE. An MI equal to zero means that there is no statistical impact.

However, such a variable is required to calculate CE.

Conclusions

We have presented a combination of DEA and BN to calculate and analyze values of

efficiency in research groups. As a result we obtained a framework to perform comparisons

between disciplines in terms of efficiency. From the empirical results we can infer about

the impact of each production variable in efficiency and how it changes depending on the

discipline. When comparing CE results by discipline against CE in the whole sample, we

not only adopted the suggestion of Bonaccorsi and Daraio (2004), but we applied it. These

results are examples of an adequate application of efficiency analysis in science and

technology.

BN analysis provides a visual representation of the dependency relationship between

output variables and efficiency. This representation provides us with an interactive user

interface useful to refine intuition about the relation of the production variables and effi-

ciency. In this sense, BNs seem to be a good approach to take the differences between

disciplines into account.

Policy makers and science managers can use this approach to evaluate efficiency and

construct ‘‘What if’’ scenarios. Particularly, regarding the Colombian study case, we can

contribute to the discussion on the ranking methods of research groups. At this moment,

the Colombian Colciencias ranking is based on production. Here we present a procedure to

evaluate research groups by efficiency while taking their fields of knowledge into account.

Finally, as a perspective of this research we are considering three main points. The first

is the construction of a methodology based on Data Envelopment Analysis that permits

weighting outputs. In that sense, we could distinguish main products by disciplines. The

second point is to continue with the discussion on the DEA extensions in respect to

favoring specialization or favoring diversification. And the third point is to analyze the BN

structure and the in-depth inferences that can be made.

Table 1 Mutual information between cross efficiency and production variables by disciplines

Productionvariable

Physics Chemistry Biology Ecology Economics Law Civilengineering

Electricalengineering

Working papers 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Conferences 0.11 0.25 0.45 0.37 0.24 0.10 0.34 0.36

Chapters 0.00 0.02 0.01 0.00 0.14 0.06 0.00 0.11

Books 0.01 0.00 0.08 0.00 0.28 0.06 0.14 0.15

Papers 0.63 0.23 0.45 0.15 0.10 0.10 0.41 0.00

Thesis 0.52 0.22 0.38 0.19 0.00 0.08 0.00 0.22

720 C. F. Ruiz et al.

123

References

Bonaccorsi, A., & Daraio, C. (2004). Econometric approaches to the analysis of productivity of R&Dsystems. Production functions and production frontiers. In W. Glanzel, U. Schmoch, M. Zitt,E. Bassecoulard, & M. Luwel (Eds.), Handbook of quantitative science and technology research(pp. 51–74). Amsterdam, Netherlands: Springer.

Bonaccorsi, A., Daraio, C., & Simar, L. (2006). Advanced indicators of productivity of universities. Anapplication of robust nonparametric methods to Italian data. Scientometrics, 66(2), 389–410.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision-making units.European Journal of Operational Research, 2(6), 429–444.

Cherchye, L., & Vanden Abeele, P. (2005). Onresearch efficiency: A microanalysis of Dutch universityresearch in economic and business management. Research Policy, 34, 495–516.

Cooper, W. W., Seiford, L. M., & Zhu, J. (2004). In J. Zhu, W. W. Cooper, R. D. Banker, & L. M. Seiford(Eds.), Handbook on data envelopment analysis (pp. 1–39). Boston: Kluwer Academic.

Doyle, J., & Green, R. (1994). Efficiency and cross-efficiency in DEA: Derivations, meanings and uses.The Journal of the Operational Research Society, 45(5), 567–578.

Garg, K. C., Gupta, B. M., Jamal, T., Roy, S., & Kumar, S. (2005). Assessment of impact of AICTE fundingon R&D and educational development. Scientometrics, 65(2), 151–160.

Guan, J. C., & Wang, J. X. (2004). Evaluation and interpretation of knowledge production efficiency.Scientometrics, 59(1), 131–155.

Heckerman, D. (1999). A tutorial on learning with Bayesian networks. In M. Jordan (Ed.), Learning ingraphical models. Cambridge, MA: MIT Press.

Kim, H., & Park, Y. (2008). The impact of R&D collaboration on innovative performance in Korea: ABayesian network approach. Scientometrics, 75(3), 535–554.

Kjaerulff, U. B., & Madsen, A. L. (2008). Bayesian networks and influence diagrams. A guide to con-struction and analysis. New York: Springer.

Korhonen, P., Tainio, R., & Wallenius, J. (2001). Value efficiency analysis of academic research. EuropeanJournal of Operational Research, 130(1), 121–132.

Lamirel, J. C., Al Shehabi, S., Francois, C., & Polanco, X. (2004). Using a compound approach based onelaborated neural network for Webometrics: An example issued from the EICSTES project.Scientometrics, 61(3), 427–441.

Lehmann, S., Jackson, A. D., & Lautrup, B. E. (2008). A quantitative analysis of indicators of scientificperformance. Scientometrics, 76(2), 369–390.

Leydesdorff, L. (1992). Knowledge representations, Bayesian inferences, and empirical science studies.Social Science Information, 31(2), 213–237.

Meng, W., Hu, Z., & Liu, W. (2006). Efficiency evaluation of basic research in China. Scientometrics, 69(1),85–101.

Pearl, J. (Ed.). (2000). Causality: Models, reasoning, and inference (pp. 1–40). Cambridge: CambridgeUniversity Press.

Restrepo, M., & Villegas, J. (2007). Clasificacion de grupos de investigacion colombianos aplicando analisisenvolvente de datos. Revista Facultad de Ingenierıa Universidad de Antioquia, 42, 105–119.

Rousseau, S., & Rousseau, R. (1997). Data envelopment analysis as a tool for constructing scientometricindicators. Scientometrics, 40(1), 45–56.

Rousseau, S., & Rousseau, R. (1998). The scientific wealth of European nations: Taking effectiveness intoaccount. Scientometrics, 42(1), 75–87.

Sexton, T. R., Slinkman, R. H., & Hogan, A. (1986). Data envelopment analysis: Critique and extensions.In R. H. Silkman (Ed.), Measuring efficiency: An assessment of Data Envelopment Analysis (Vol. 32,pp. 73–105). San Francisco: Jossey-Bass.

Spirtes, P., Glymour, C. N., & Scheines, R. (2000). Discovery algorithms for causally sufficient structures.In P. Spirtes, C. N. Glymour, & R. Scheines (Eds.), Causation, prediction, and search (pp. 73–122).Cambridge, MA: MIT Press.

Efficiency measurement of research groups 721

123