The Evaluation of Studies and Training in

Information Technology: A Multicriteria

Approach(*)

Evangelia Krassadaki1, Evangelos Grigoroudis

1, Nikolaos Matsatsinis

1,

Yannis Siskos2

1 Technical University of Crete, Decision Support Systems Laboratory,

University Campus, 73100, Chania, dsslab@ergasya.tuc.gr 2 University of Piraeus, Department of Informatics, 80, Karaoli & Dimitriou, 18534,

Piraeus, ysiskos@unipi.gr

Abstract

The accreditation of information technology executive officers is being approached in this

article as a problem of multicriteria decision analysis of professional skills, studies and

vocational training. The proposed approach includes the analysis and modelling of the

qualitative criteria that fall into the problem as well as the implementation of a set of

multicriteria aggregation-disaggregation methods. In particular, it includes the presentation of

the evaluation system for Basic, Postgraduate and Doctoral studies and Non-formal

Vocational Training “with recognized” and “without recognized” title. For the evaluation of

the Basic Studies a table has been drawn up with all the possible combinations based on the

Greek model, and then the evaluation takes place through a multiplicative value function that

has been assessed by a disaggregation method. The proposed evaluation approach is being

applied to a particular example of candidate for accreditation.

Keywords: Accreditation of skills, Information Technology, Multicriteria analysis

1. Introduction

On a global level, the accreditation of skills and competences has troubled many

researchers, mainly the intergovernmental organizations such as the European

Council, the Cedefop organization, Unesco and OECD [3, 12, 4, 13]. The speculation

is focused mainly on the development of commonly accepted methods through which

a value will be attributed to the competences and skills acquired during the

professional life. New skills and competences are acquired every day through non-

formal or informal ways of learning, and particularly the learning acquired through

professional experience or team work, self-education and self-training, work field

training, etc. Especially in the last 5 to 10 years many European countries have

introduced and enacted methodologies and systems for the identification,

(*)

This research has been financially supported by the European Union, Leonardo Da Vinci

programme, no. EL/97/1/28132/PI/I.1.1.b/FPC

accreditation, certification of qualifications, skills and competences acquired through

non-formal vocational training. In particular, such initiatives have been taken in

France with the implementation of the “Bilan de Compétence” and the initiative of

the Accreditation of Professional Experience with a diploma of formal vocational

education, in England and Ireland with the implementation of the APL (Accreditation

of Prior Learning) system, and in Germany with the “Criteria Referenced Testing”

system [2]. The methods and systems proposed on a European level seek to

strengthen the value and provide clarity to the general skills and competences of

people, beyond the specialized and standardized knowledge that is closely connected

to a specific occupation or job, and subsequently aim to attribute value and recognize

various forms of non-formal learning. This tendency has led the majority of the

countries-members of the European Union to seek commonly accepted

methodologies, that will ensure with reliable procedures of diagnosis and evaluation,

the identification and accreditation of knowledge and skills acquired outside of formal

education and training.

More specifically, the fast development of information technology combined

with the new globalized environment and the rapid changes in all the fields of

business activity make the human factor the prevailing element. The need to update or

acquire new information technology knowledge has led more and more people to look

for new ways of learning. The significant demand results in the development of

several non-formal learning initiatives and many times in the offering of several

forms of information technology knowledge accreditation. The assessment of

information technology knowledge, suggested mainly by major manufacturing

companies of the field, is usually materialized through examinations accompanied by

the issuing of a certificate. The specific effort is focused on the knowledge of

candidates on a sector or product without taking into account other features of the

assessed persons, such as their working experience, their prior studies, their

participation in training programmes, the requirements of the jobs, etc. Moreover, it

requires the participation of the assessed person in an examination, which as a process

does not necessarily correspond to the profile of all ages, and mainly to those who

have been withdrawn from the educational procedure many years ago but are

nevertheless successful professionals in the field of information technology.

The approach proposed in this article for the accreditation of knowledge,

skills and competences in information technology concerns an original multicriteria

evaluation methodology. The accreditation process, as it is proposed, evaluates a

candidate in relation to the practice of a specific occupation or specialization, which

in our case is the field of information technology. The proposed methodology collects

facts for each person through all the possible combinations of learning and

acquirement of skills for an occupation, such as: professional experience, studies,

non-formal vocational training and personal activity. These facts result from personal

interviews with the candidate and the study of the personal candidature folder. The

target is to attribute through a unified way of evaluation, a kind of “identity” to the

interested parties for the accreditation per occupation/specialization, by classifying

them in four groups of information technology professionals. This procedure can lead

to the issuing of a formal document. Before the classification of the candidates in

groups of professionals, the evaluation in a set of qualitative criteria takes place in

order to use these values as input data in the applied multicriteria classification

method.

The analysis of the qualitative criteria which compose the total evaluation

problem as well as the presentation of the general methodological framework are

described in section 2. The evaluation system for the candidates‟ studies, applied in

the Greek educational system, is presented in section 3, while the evaluation system

for non-formal training programmes attended by the candidates is presented in section

4. Section 5 includes an implementation example, and finally section 6 deals with the

evaluation of the results and the suggestion of new research avenues.

2. The evaluation methodology

According to approaches that have been made by European bodies and by

educational-research institutions the views converge that in order for an employee to

succeed in the actual working conditions he/she must have a number of basic skills or

key skills. According to the English version of Skills Task Force [5], the six key skills

that the employees must have are: communication skills, knowledge of arithmetic and

information technology, problem solving skills, ability to work in a team and finally

ability to upgrade knowledge and performance. There are similar suggestions by USA

bodies, such as the National Skills Standard Board-NSBB and the Northwest Center

for Emerging Technologies-NWCET [10], which focus on the necessary basic skills

that each employee must have. According to the NWCET, a person in order to work

in the field of information technology must have, other than the basic skills required

for other occupations, certain “core computer software and hardware skills”, which

refer to the minimum technical skills or the entry skills. According to the NWCET

such skills are the following: knowledge of Windows environment, word processing,

spreadsheet applications, presentation software, database applications, principles of

programming, network technologies, hardware and software installation, use of e-

mail and internet. Certainly, beside the core skills, as they are proposed by the

NWCET, the practice of any occupation as well as any information technology

profession requires the existence of a number of specialized professional skills per

case, as it is suggested by the analytical proposals of the above bodies.

The proposed multicriteria methodology adopts the general principle that the

evaluation of a candidate should refer to the degree of his/her adequacy to practise a

specific occupation or speciality. By adopting the proposal of the NWCET, the

procedure for the evaluation of candidates for an information technology profession

consists of two phases (Fig. 1): in phase A all the candidates are evaluated on a set of

criteria-general skills and those who are judged sufficiently are afterwards evaluated

in phase B on another set of qualitative criteria composed of a number of specialized

skills, each time adjusted per occupation, studies and training. The successful

evaluation of a candidate in phase A does not necessarily mean that he/she can

become a professional in information technology, for it is known that most employees

nowadays have some information technology knowledge. For this reason, on the one

hand the evaluation of specialized skills is connected to the candidate‟s professional

experience and on the other hand the relevance of the studies and training with a

specific occupation is controlled. A complete presentation of the multicriteria

evaluation methodology and the proposed criteria of general and specialized skills is

given in [11].

It is commonly accepted that a person during his/her life span is in a constant

process of learning and acquiring new skills. If this period of time is delimited in the

period of a person‟s professional career, then it can be considered that the starting-

point of professional specialization is the education or training that he/she acquires by

attending a certain special school of the formal school system and/or outside of it,

which is usually referred as studies inside the formal educational system or/and the

system of non-formal vocational training. This view connects professional career to

the initial attendance in a certain educational organization without excluding the

possibility for each interested person to re-enter the educational or training system,

whenever it is thought necessary. This possibility is feasible and often imperative

mainly due the demands of the labour market. The case of employees who have learnt

an occupation through experience or practice means that they have to a certain degree

the skills for its performance but do not have the corresponding formal education

and/or non-formal training. All the above cases are examined and analyzed in relation

to the profile of each candidate for accreditation.

In particular, the following sections contain the analysis and presentation of

the studies and training criteria, in relation to the practice of a specific occupation.

The evaluation of the candidates‟ studies and training is part of the evaluations of

phase B, as it appears in Figure 1. The criteria, in which the general and specialized

skills per occupation are analyzed, are not presented in Figure 1 (details are given in

[11]). More specifically, the studies are analyzed in the sub-criteria of Basic,

Postgraduate and Doctoral studies, while the candidates‟ non-formal vocational

training is treated as two separate criteria. The first one concerns the training “with a

recognized title”, while the second “without a recognized title”. The basic target is the

assessment of a value to all criteria by examining each time their relevance to the

occupation for which the candidate asks accreditation.

Given that the candidates are sufficiently evaluated in phase A, they are

evaluated in phase B also for a set of other criteria that are connected to professional

experience and specialized skills per occupation, it is ensured that the evaluated

persons will fulfil to a certain degree the requirements for their accreditation in a

certain occupation. Therefore, the proposed evaluation procedure examines all the

possible ways to acquire knowledge and skills for a specific occupation and allows

the combination of the above, by choosing in each case those candidates who receive

high scores in all the criteria. Certainly, as the candidates differ from each other, the

aim is to attribute a unified kind of identity to each professional, that can have the

form of a formal document. For this purpose, it is proposed to classify the candidates

for accreditation in four homogenous groups of professionals and in particular in the

groups of beginners, competent, experienced and finally expert professionals [11].

MULTICRITERIA EVALUATION OF

INFORMATION TECHNOLOGY

COMPETENCES

SKILLS

GENERAL SKILLS

SPECIALIZED

SKILLS

BASIC STUDIES

INDIVIDUAL

QUALIFICATIONS

STUDIES TRAINING"with recognized title"

TRAINING"without recognized title"

Non-formal Training

POSTGRADUATE

STUDIES

DOCTORAL

STUDIES

Phase A'

Phase B'

Figure 1. The Multicriteria Evaluation Procedure for Inf. Technology Qualifications

It is indicatively mentioned that candidates who ask for accreditation as

“Computer Support Specialists”, can be classified in these four groups of

professionals. The candidates who fall into one of the four groups have the possibility

to be reaccredited later in order to improve their position and receive a documented

report that will mention the scores they received for each criterion. Finally, the

specific procedure is expected to contribute to the clarity and strengthening of the

employees‟ mobility within the countries of the European Union and to the

identification and attribution of value to knowledge and skills acquired through

professional experience and any other way of learning.

3. The system for the evaluation of accomplished studies

3.1 Basic evaluation model

The term “studies” refers to any type of attendance in a school/faculty of formal

education or training that belongs to the formal school system of each country, and

the studies that have been made after the completion of compulsory education. Taking

as a rule that the candidates for accreditation have completed certain studies, a system

is proposed which evaluates Basic, Postgraduate and Doctoral Studies. Therefore, the

target is first to assess a value to the three sub-criteria in which studies are analyzed

and then globally evaluate the studies of each candidate (Figure 2).

Doctoral Studies SubcriteriaPost Graduate Studies SubcriteriaBasic Studies Subcriteria

Multicriteria values

Basic Studies CriterionPostgraduate Studies

Criterion

Doctoral Studies

Criterion

GLOBAL EVALUATION

OF ACCOMPLISHED

STUDIES

60% 10% 30%

Figure 2. The Multicriteria Evaluation Model of Studies

More specifically, the global evaluation of studies (US) for each candidate for

accreditation, results from the aggregation of the three evaluations, as follows:

US= wBSU(es) + wMSCUMSC + wPHDUPHD (1)

where:

wBS =0.6 : Weight of Basic Studies criterion

wMSC=0.1 : Weight of Postgraduate Studies criterion

wPHD=0.3 : Weight of Doctoral Studies criterion

es : Alternative activity which corresponds to a specific combination of the type

of basic studies, marks and relevance of the basic studies to the occupation

for accreditation.

U(es): Value of es Basic Studies alternative in the interval [0,1]

UMSC : Value of Postgraduate Studies in the interval [0,1]

UPHD : Value of Doctoral Studies in the interval [0,1]

The determination of weights to the three sub-criteria of studies was based on

the years of attendance per case. Basic studies which include all types of schools or

faculties after the compulsory and up to tertiary education received a weight of 60%

(wBS = 0.6) expressing the longer period of time of certain studies. Postgraduate

studies which last a year received a weight of 10% (wMSC = 0.1), while Doctoral

studies which usually last three years received a weight of 30% (wPHD = 0.3).

3.2 Evaluation of Basic Studies : the Greek Paradigm

Basic Studies are considered all the possible cases of graduation after compulsory

education including the acquirement of a degree/diploma from Tertiary education.

Therefore, for the needs of the evaluation, all possible cases of acquiring relative

information technology knowledge through any type of school/faculty are examined.

Greece was selected as an example of implementation of the proposed methodology

and this led to the examination of all types of schools starting from the 70s and finally

to propose six different types of Basic Studies, as it is shown in Table 1.

Table 1. Types of Basic Studies in the Greek Educational System

Type

of

Studies

Description Explanation

A ΑΕΙ Highest Educational Establishment – Highest Tertiary Education

(University or Technical University).

B ΤΕΙ Technological Educational Establishment – Higher Tertiary Education.

C ΙΕΚ Institute of Vocational Training.

Post-Secondary (not Tertiary) Vocational Training.

D ΤΕΕ Cycle Β, Unified Lyceum, General Lyceum, 6-

class Gymnasium, ΤEL, ΕPL

Schools of (upper) Secondary Education-General and Secondary Vocational and Technical Education. Three-year duration.

E

ΤΕΕ Cycle Α, ΤΕΕ

Apprenticeship Cycle A-

OAED, TES

Schools of Secondary Vocational and Technical Education. Two-

year (or three-year) duration.

F ΙΕΚ 1 Vocational Training of 2 semesters after compulsory education.

In particular, the evaluation problem of Basic Studies is also treated as a

multicriteria problem, which is analyzed in three criteria:

Higher Title of Basic Studies possessed by the candidate.

Grade of the Title (degree, diploma, certificate, etc.).

Relevance of Basic Studies Title to a specific information technology

occupation.

For the modelling of the specific evaluation problem of Basic Studies, the

proposed qualitative evaluation scales are in detail the following: for the criterion

„title of basic studies‟ the six-degree scale has been adopted in correspondence with

the six types of studies (see Table 1), for the criterion „grade of title‟ a four-degree

evaluation scale is proposed (Excellent, Very Good, Good and Pass) and for the

„relevance‟ criterion a three-degree scale (High relevance of title with the occupation,

Medium relevance, and no-low relevance), as shown in Figure 3.

The proposed modelling of the criterion “grade of title” regards the Greek

educational system but can also be easily adjusted to any other. In particular, for the

evaluation of Basic Studies in the criteria: “type of studies”, “marks” and “relevance”,

a table of all possible combinations has been drawn up, based on the Greek example,

and the values of these combinations have been estimated (see Appendix) through a

multiplicative evaluation model estimated by a disaggregation model [6] presented

below.

Global Evaluation of

Basic Studies

Title of Basic Studies

Grade of Title

Relevance of Title

Α: AEI

B: TEI

C: IEK

D: TEE B, etc

Ε: TEE Α, etc

F: IEK 1

EXCELLENT

VERY GOOD

GOOD

PASS

HIGH

MEDIUM

NONE (or low)

T Y P E S

Figure 3. Evaluation Scales of Basic Studies

The set of alternative activities is defined the set of possible combinations of

the types of Basic Studies, Marks and Relevance, and is symbolized by A={ajkℓ,

j=1,…,J, k=1,…,K, ℓ=1,…,L} where j: type of Basic Studies, k: type of Marks and ℓ:

type of Relevance. The corresponding values for the case of Greece are: j=1,…,6

k=1,…,4 and ℓ=1,…,3. Considering the fact that the possible combinations in the

example of Greece are totally 48 (see [8]) a corresponding set of alternatives E is

defined, as follows: E={es, s=1,…,48}, which is a finite and predetermined set (see

Appendix).

The target now is to estimate a multiplicative value model that will allow the

ranking of all alternatives, by assessing the value of each activity. This model has the

following mathematical form:

U(es)= U(ajkℓ)= UsjUvkUrℓ (2)

where:

U(ajkℓ) : Global Value of candidate‟s Basic Studies:

j : type of Title of Basic Studies, j=1,…, 6 in correspondence to types Α-F.

k : type of Marks, k=1,…, 4 where 1 is given to Excellent and 4 to Pass.

ℓ : type of Relevance, ℓ=1,…,3 where 1 is given to High Relevance while 3 to

No (or low) Relevance.

Usj : value of j type of Basic Studies Title

Uvk : value of k type of Marks

Urℓ : value of ℓ type of Relevance

The equivalence of the alternative combinations: type of Basic Studies-j, type

of Marks-k and type of Relevance-ℓ with the 48 alternatives (es), is shown in the first

two columns of the Appendix.

For the use of the dissagregation approach (see [6]) a ranking of reference

activities (weak order) is presumed, which in this case is the ranking of the 48

activities of the appendix (third column), as it has been predetermined by certain

experts in the field of education.

According to the preference model (2), the following relations must be

effective:

U(es) > U(es+1) es P es+1 (strict preference of alternative es to es+1)

U(es) = U(es+1) es I es+1 (indifference between the two alternatives es and es+1)

For the estimation of the value function U(es), potential overestimation and

underestimation errors are introduced for all the activities and the following non

linear programme is solved which minimizes the sum of errors σ and -σ :

[min] F = 48

1

)}()({s

ss ee

subject to:

a. Preference constraints

)}()()()({)}()({ 111 ssssss eeeeeUeU , when es P es+1, for

s={1,…,32}

where δ=0.01

b. Indifference constraints

0)}()()()({)}()({ 111 ssssss eeeeeUeU , when es Ι es+1, for

s={33,…,47}

c. Normalization constraints at the value interval [0,1]

,11Us ,11Uv 11Ur

d. Additional constraints

The values of two successive types of Studies, Marks and Relevance are different by

at least 0.05:

05.01jj UsUs , 5,...,1j

05.01kk UvUv , 3,...,1k

05.01 UrUr , 2,...,1

and

σ+ (es), σ

-(es) 0 , s=1,…,48 ; 0jUs , 0kUv , 0Ur ,, kj

The optimal solution to the above non linear programme gives a zero total

error which shows the absolute compatibility of the value function with the initial

ranking of the alternatives. Moreover, the values Usj, Uvk and Urℓ of the relation (2)

are estimated and therefore the Global Value U(es) of each alternative is also

estimated. Analytically, the values for each type of Studies, Mark and Relevance are

shown in Table 2 while the Global Evaluation of the alternatives according to relation

(2) appears in the last column of the Appendix.

Table 2. Marginal Value Functions for Basic Studies

Variables

Value of

the types of Basic

Studies

Variables

Value of

the types

of Marks

Variables

Value of

the types of

Relevance

Us1 (Α:ΑΕΙ) 1 Uv1 (Excellent) 1 Ur1 (High) 1

Us2 (Β:TEI) 0.88 Uv2 (Very Good) 0.95 Ur2 (Medium) 0.64

Us3 (C:IEK) 0.50 Uv3 (Good) 0.89 Ur3 (None) 0

Us4 (D:TEE B, etc) 0.39 Uv4 (Pass) 0.84

Us5 (Ε:TEE A, etc) 0.31

Us6 (F:IEK1) 0.04

According to the estimated values, note that even in the worst case of studies

of type ΙΕΚ1 a 4% value is attributed. The same goes for the case of Pass grade in the

title of studies, while on the contrary if the title of studies has low or no relevance to

the occupation for accreditation it is valued by zero.

3.3 Evaluation of Postgraduate and Doctoral Studies

Postgraduate studies are considered in Greece the studies that lead to the obtaining of

the Postgraduate Specialization Diploma (PSD) and require at least the completion of

a certain programme of tertiary education. Similarly, the evaluation of doctoral

studies concerns the candidates who hold a title of doctoral studies.

If a candidate for accreditation has made postgraduate and/or doctoral studies, then

the relevance of his/her studies to the specific occupation for which he/she asks to be

accredited is examined. A value must be estimated for each case.

If the value of postgraduate and doctoral studies is symbolized with UMSC and

UPHD respectively, then:

UMSC = UΤ-MSCUR-MSC (3)

UPHD = UΤ-PHDUR-PHD (4)

where:

UT-MSC, UT-PHD: value of corresponding title (=1 if the candidate has the title; =0 if

not).

UR-MSC, UR-PHD: relevance value of corresponding title to the information technology

profession (=1 High Relevance; =0.60 Medium Relevance; =0 No-

low Relevance).

If the candidate has made postgraduate and/or doctoral studies with low or no

relevance to information technology, relevance is marked with zero as opposed to

cases when there is medium relevance to information technology and in the best case

of high relevance of the studies to the specialized information technology profession

when the score is 0.60 or 1 respectively. Therefore, in the worst case (anti-ideal),

where there is no relevance of the postgraduate and/or doctoral title to the information

technology profession for which the candidate requests accreditation or the candidate

has not made any postgraduate or doctoral studies, then UMSC=0 or/and UPHD=0, while

in the best case (ideal), of high relevance of the title to the information technology

profession, then UMSC=1 or/and UPHD=1. Therefore, the high relevance of

postgraduate or/and doctoral studies gives one point respectively to the candidate,

while the relative studies when they have no relevance to information technology or

are not judged sufficient for the specific occupation for accreditation, then they are

not taken into account during evaluation.

4. The evaluation system for non-formal training

The need to be updated in new technologies and the products that are constantly

introduced requires the continuous attendance of non-formal training programmes

that do not belong to the formal school system. Consequently, the non-formal training

programmes can be an evaluation element for the accreditation of the candidates. So,

training is divided in two criteria: the one that takes place by following certain

standards and usually a specific and similar in each country way of examination,

described as “Training with a recognized title” and the one that does not fulfil the

above prerequisites and is quite common, described as “Training without a

recognized title”. For example, in Greece the training programmes that take place

through Liberal Studies Laboratories, Vocational Training Centres-KEK, intra-

company programmes, etc., are cases that belong to the training criterion of type

“without a recognized title”. Similarly, in the case of non-formal training of type

“with a recognized title” are included for example the certificates of: ECDL,

Microsoft, Oracle, etc. Our aim is to evaluate the non-formal training programmes

with or without a “recognized” title, that the candidate is certified to have attended,

by giving a value to each criterion.

4.1 Evaluation of non-formal “training with a recognized title”

In this case two elements are examined: the existence of such training titles and the

relevance of each title to the occupation for which the candidate requests

accreditation. If, iVT

U expresses the value of i programme of non-formal vocational

training “with a recognized title”, then the global value of non-formal vocational

training “with a recognized title” (UVT+), of the candidate, results from the type:

UVT+

= VT VT1 1

if

1 otherwise

U Un n

i ii i

k k (5)

where:

n : number of non-formal vocational training programmes “with a recognized

title” attended by the candidate

k : higher number of evaluated non-formal vocational training programmes “with

a recognized title, where in our case k=3

UVT+ : value of vocational training “with a recognized title”, at the value interval

[0,1]

where:

iii VTRVTTVTUUU for each i Certificate (“recognized title”) (6)

when:

iVTTU : value of i title (=1 if available by the candidate; =0 if not available).

iVTRU : relevance value of i title to information technology profession (=1 High

relevance; =0.6 Medium relevance; =0 Low-no relevance).

If the candidate has attended a training programme with low or no relevance to

information technology then a zero value is attributed, while he/she is marked with

0.60 if the programme concerns a certain issue of information technology and has

offered him/her some relative knowledge but possibly not completely relevant to the

occupation for accreditation. In the best case of high relevance of training to the

occupation for accreditation is given the score one.

4.2 Evaluation of non-formal “training without a recognized title”

During the evaluation of non-formal training programmes “without a recognized title”

that the candidate is certified to have attended, the relevance of each programme to

the occupation for accreditation and the duration of the training are examined.

If, i

VTU expresses the value of i non-formal training programme “without a

recognized title”, then the global value of non-formal training of type “without a

recognized title” (UVT-) results from the type:

UVT- = VT VT

1 1

if

1 otherwise

U Un n

i ii i

m m (7)

where:

n : number of non-formal vocational training programmes “without a

recognized title” attended by the candidate

m : higher number of evaluated programmes of non-formal training “without a

recognized title”, where in our case m=6

UVT- : value of vocational training “without a recognized title”, at the value

interval [0,1]

where:

iiii VTRVTDVTTVTUUUU for each i Certificate (“non recognized title”) (8)

when:

: value i of title, receiving value 1 or 0 depending on whether the candidate

has or not a relative certificate.

iVTDU : value of duration of i training programme “without a recognized title”.

iVTRU : relevance value of i certificate to a specific information technology

profession, when relevance is evaluated as follows:

High relevance: 1, Medium relevance: 0.6, No (low) relevance: 0,

and the quantification of the qualitative scale follows a similar justification

as the respective case of training “with a recognized title”.

The period of time for the realization of a vocational training programme

usually varies from a few days to several months. Taking as a rule that the period of

time for the realization of a training programme can be counted in teaching hours, it is

proposed to take into account the duration in the evaluation methodology based on the

value scale shown in Table 3. This value scale results from a mid value point

technique [7], which suggests that a seminar of at least 200 hours is judged sufficient

for the education of the trainees, while a lower value is given to seminars of a shorter

duration. For example, a seminar of 150 hours covers requirements by 70%, while a

iVTTU

seminar of 100 hours by 40%. So, an intensive training programme is separated from

a longer duration programme, but even a seminar of few hours is counted in the

evaluation. For all the cases where the time duration of a training seminar varies

between the periods of time of Table 3, the linear interpolation type takes effect.

Table 3. Time Duration Evaluation Scale for Vocational Training Programmes “Without a

Recognized Title”

Duration in Hours 0 50 100 150 200 Over 200

Value UD-VTi- 0 0.1 0.4 0.7 1 1

5. Example of implementation

Let‟s suppose a candidate who has been sufficiently evaluated in phase A of the

evaluation and is evaluated in phase B on the studies also criterion and the two non-

formal training criteria from the whole set of qualitative criteria. Let‟s also suppose

that he/she asks for accreditation in the profession of “Computer Support Specialist”

and is a graduate of the Department of Electronic Engineers and Computer Engineers

of the Technical University of Crete, with grade “Very Good”, and has made

Postgraduate Studies at the same Department. He/she has attended two training

programmes by Microsoft, and in particular the MOUS (Microsoft Office User

Specialist) and MCSE (Microsoft Certified Professional-Systems Engineer) and a

100-hour programme, at a private school, with the title “Basic Principles of

Information Technology”. During his/her professional career he/she participated in

two intra-company seminars: “Introduction to network security issues” and “Unix

Operating System”, lasting 24 and 60 hours respectively.

Evaluation of Studies

Due to the high relevance of Basic and Postgraduate Studies to the specific

information technology profession and according to the values of Table 2 or the last

column of the Appendix and the relations (2), (3), (4) and (1) respectively, he/she will

receive the following scores:

Basic Studies: U(e2) = 1 x 0.95 x 1 = 0.95 (ΑΕΙ, Very Good, High Relevance)

Postgraduate Studies: UMSC = 1 x 1 = 1 (High Relevance)

Doctoral Studies: UPHD = 0 (Not available)

Global Evaluation of Studies: US = 0.95x0.6 + 1x0.1 + 0x0.3 = 0.67

Evaluation of Non-formal Vocational Training “with a recognized title”

If we consider that the two Microsoft programmes are classified in the

category: non-formal training “with a recognized title”, then according to the relations

(6) and (5) respectively, he/she receives:

MOUS Certificate: 6.06.011

xUVT

, (Medium relev. to the specific occupation)

MCSE Certificate: 1112

xUVT

, (High relevance to the specific occupation)

Global Evaluation of Non-formal Training “with a recognized title”:

UVT+ = (0.6 + 1) / 3 = 0.53

Evaluation of Non-formal Vocational Training “without a recognized title”

According to the non-formal vocational training programmes attended by the

candidate “without a recognized title” privately or inside the company where he/she

works and the relations (8), (7), the values of Table 3 and through the application of

the linear interpolation type where required, he/she receives:

Basic principles of information technology, 100 hours:

24.06.04.01 xx1VT

U (Medium relevance)

Introduction to security issues, 24 hours: 04.0104.01 xx2VT

U (High relevance)

Unix Operating System, 60 hours: 16.0116.01 xx3VT

U (High relevance)

Global Evaluation of Non-formal Training “without a recognized title”:

UVT- = (0.24 + 0.04 + 0.16) / 6 = 0.07

Therefore, the evaluation of the candidate in the three criteria goes as follows:

Studies: 0.67, Non-formal Training “with a recognized title”: 0.53 and Non-formal

Training “without a recognized title”: 0.07, which are the values of the three criteria

of phase B of the multicriteria methodology. In order to classify the candidate in one

of the four proposed categories of information technology professionals, he/she must

also be evaluated on a set of criteria referred to the specialized skills required by the

occupation or speciality according to the available professional experience (details are

given in [11]).

6. Concluding remarks

The accreditation problem of qualifications and skills has been treated realistically as

a multicriteria decision problem. The proposed methodology has been adjusted to the

needs of occupations/specializations of information technology and it can also be

applied with the proper modifications to other areas of expertise and occupations. The

developed methodology evaluates all the cases of acquiring knowledge and skills

through work, the formal school system, the non-formal vocational training system or

even personal activity, recognizing that the technical knowledge for the practice of a

profession can derive from a combination of the above.

The disaggregation approach applied in the case of attributing a value to the

candidates‟ basic studies includes all the possible cases of graduation from a formal

education school. The implementation made to the Greek educational system has

proven that a value can be attributed to the candidates‟ studies for the need of

accreditation. Moreover, an evaluation system is proposed for postgraduate and

doctoral studies and an evaluation system for any type of non-formal training

attended by the candidates.

The implementation made for a candidate has proven the effectiveness and

convenience in applying the methodology and the fact that the requested elements for

the evaluation can easily result from the study of the candidature personal folders or

the interviews with the candidates. It has also been proved that the multicriteria

approach can support the decision-maker and in this case an enacted committee that

can accredit the candidates for an occupation or speciality in a unified and clear way.

Certainly, the issue of accreditation has significant perspectives for further research

such as: (1) exploration of the evaluation system with the assistance of a competent

committee as the decision-maker; (2) exploration of the possibility to combine with

alternative approaches proposed in European countries at a pilot or applied level; and

(3) application of other multicriteria methods and comparison of the evaluation

results.

References

[1] Beuthe, M. and Scannella, G. (2001). Comparative analysis of UTA multi-

criteria methods. European Journal of Operational Research, 130, 2, 246-262.

[2] Bjørnåvold, J. (1997). Assessment of non-formal learning: the quality and

limitations of methodologies. The European Vocational Training Journal, (12),

52-67.

[3] Bjørnåvold, J., and Sellin, B. (1997). Recognition and transparency of

vocational qualifications; The way forward. Discussion paper. Cedefop

Panorama, Thessaloniki.

[4] Bjørnåvold, J. (2000). Making Learning Visible: Identification, assessment and

recognition of non-formal learning in Europe. Cedefop. Thessaloniki.

[5] DfEE (Department of Education and Employment), Skills Task Force (2000).

Second Report of the Skills Task Force. Towards a National Skills Agenda.

Chapt. 6.

[6] Jacquet-Lagrèze, E. and Siskos, Υ. (2001). Preference disaggregation: 20 years

of MCDA experience. European Journal of Operational Research, (130), 2,

233-245.

[7] Keeney, R. and Raiffa, H. (1976). Decisions with multiple objectives:

Preferences and value trade-offs. Wiley, New York.

[8] Krassadaki, Ε. (2001). Development of a multicriteria evaluation methodology

for information technology qualifications. Postgraduate Thesis, Technical

University of Crete, Department of Production Engineering & Management,

Chania.

[9] National Skill Standards Board, www.nsbb.org

[10] NWCET, Northwest Center for Emerging Technologies (1999). Building a

foundation for tomorrow. Skill standards for information technology, Bellevue

Community College.

[11] Siskos, Y., Grigoroudis, E., Krassadaki, E., Matsatsinis, N. (2001). A multi-

criteria accreditation system for information technology skills and

qualifications. Working Paper.

[12] Turner, C. (1999). Identification, assessment and recognition of non-formal

learning in Greece. Cedefop Panorama, Thessaloniki.

[13] UNESCO European Symposium (1999). Vocational Education and Training in

Europe, on the threshold of the 21st century. Conclusions. Conference Crete 23-

26 September 1998.

Appendix: Alternative activities of Basic Studies

Alternatives

es

(s=1,…,48)

Alternatives

ajkl, for each

combination j, k, ℓ

Ranking

by

increasing preference

order

Alternative Combinations of Basic Studies Titles,

Marks and Relevance

(according to the Greek example)

STUDIES MARKS RELEVANCE

Global

Eval.

of alterna-tives

e1 a111 1 ΑΕΙ EXCELLENT HIGH RELEVANCE 1

e2 a121 2 ΑΕΙ VERY GOOD HIGH RELEVANCE 0.95

e3 a131 3 ΑΕΙ GOOD HIGH RELEVANCE 0.89

e4 a211 4 ΤΕΙ EXCELLENT HIGH RELEVANCE 0.88

e5 a221 5 ΤΕΙ VERY GOOD HIGH RELEVANCE 0.83

e6 a231 6 ΤΕΙ GOOD HIGH RELEVANCE 0.79

e7 a112 7 ΑΕΙ EXCELLENT MEDIUM RELEV. 0.64

e8 a122 8 ΑΕΙ VERY GOOD MEDIUM RELEV. 0.61

e9 a132 9 ΑΕΙ GOOD MEDIUM RELEV. 0.58

e10 a212 10 ΤΕΙ EXCELLENT MEDIUM RELEV. 0.57

e11 a222 11 ΤΕΙ VERY GOOD MEDIUM RELEV. 0.54

e12 a232 12 ΤΕΙ GOOD MEDIUM RELEV. 0.51

e13 a311 13 ΙΕΚ (EXCELLENT) HIGH RELEVANCE 0.50

e14 a411 14 ΤΕΕ Β, etc EXCELLENT HIGH RELEVANCE 0.39

e15 a421 15 ΤΕΕ Β, etc VERY GOOD HIGH RELEVANCE 0.37

e16 a431 16 ΤΕΕ Β, etc GOOD HIGH RELEVANCE 0.35

e17 a441 17 ΤΕΕ Β, etc PASS HIGH RELEVANCE 0.33

e18 a312 18 ΙΕΚ (EXCELLENT) MEDIUM RELEV. 0.32

e19 a511 19 ΤΕΕ Α, etc EXCELLENT HIGH RELEVANCE 0.31

e20 a521 20 ΤΕΕ Α, etc VERY GOOD HIGH RELEVANCE 0.29

e21 a531 21 ΤΕΕ Α, etc GOOD HIGH RELEVANCE 0.28

e22 a541 22 ΤΕΕ Α, etc PASS HIGH RELEVANCE 0.26

e23 a412 23 ΤΕΕ Β, etc EXCELLENT MEDIUM RELEV. 0.25

e24 a422 24 ΤΕΕ Β, etc VERY GOOD MEDIUM RELEV. 0.24

e25 a432 25 ΤΕΕ Β, etc GOOD MEDIUM RELEV. 0.23

e26 a442 26 ΤΕΕ Β, etc PASS MEDIUM RELEV. 0.21

e27 a512 27 ΤΕΕ Α, etc EXCELLENT MEDIUM RELEV. 0.20

e28 a522 28 ΤΕΕ Α, etc VERY GOOD MEDIUM RELEV. 0.19

e29 a532 29 ΤΕΕ Α, etc GOOD MEDIUM RELEV. 0.18

e30 a542 30 ΤΕΕ Α, etc PASS MEDIUM RELEV. 0.17

e31 a611 31 ΙΕΚ1 (EXCELLENT) HIGH RELEVANCE 0.04

e32 a612 32 ΙΕΚ1 (EXCELLENT) MEDIUM RELEV. 0.02

e33 a113 33 ΑΕΙ EXCELLENT NO RELEVANCE 0

e34 a123 33 ΑΕΙ VERY GOOD NO RELEVANCE 0

e35 a133 33 ΑΕΙ GOOD NO RELEVANCE 0

e36 a213 33 ΤΕΙ EXCELLENT NO RELEVANCE 0

e37 a223 33 ΤΕΙ VERY GOOD NO RELEVANCE 0

e38 a233 33 ΤΕΙ GOOD NO RELEVANCE 0

e39 a313 33 ΙΕΚ (EXCELLENT) NO RELEVANCE 0

e40 a413 33 ΤΕΕ Β, etc EXCELLENT NO RELEVANCE 0

e41 a423 33 ΤΕΕ Β, etc VERY GOOD NO RELEVANCE 0

e42 a433 33 ΤΕΕ Β, etc GOOD NO RELEVANCE 0

e43 a443 33 ΤΕΕ Β, etc PASS NO RELEVANCE 0

e44 a513 33 ΤΕΕ Α, etc EXCELLENT NO RELEVANCE 0

e45 a523 33 ΤΕΕ Α, etc VERY GOOD NO RELEVANCE 0

e46 a533 33 ΤΕΕ Α, etc GOOD NO RELEVANCE 0

e47 a543 33 ΤΕΕ Α, etc PASS NO RELEVANCE 0

e48 a613 33 ΙΕΚ1 (EXCELLENT) NO RELEVANCE 0