Post on 27-Apr-2023
The Evaluation of Studies and Training in
Information Technology: A Multicriteria
Approach(*)
Evangelia Krassadaki1, Evangelos Grigoroudis
1, Nikolaos Matsatsinis
1,
Yannis Siskos2
1 Technical University of Crete, Decision Support Systems Laboratory,
University Campus, 73100, Chania, dsslab@ergasya.tuc.gr 2 University of Piraeus, Department of Informatics, 80, Karaoli & Dimitriou, 18534,
Piraeus, ysiskos@unipi.gr
Abstract
The accreditation of information technology executive officers is being approached in this
article as a problem of multicriteria decision analysis of professional skills, studies and
vocational training. The proposed approach includes the analysis and modelling of the
qualitative criteria that fall into the problem as well as the implementation of a set of
multicriteria aggregation-disaggregation methods. In particular, it includes the presentation of
the evaluation system for Basic, Postgraduate and Doctoral studies and Non-formal
Vocational Training “with recognized” and “without recognized” title. For the evaluation of
the Basic Studies a table has been drawn up with all the possible combinations based on the
Greek model, and then the evaluation takes place through a multiplicative value function that
has been assessed by a disaggregation method. The proposed evaluation approach is being
applied to a particular example of candidate for accreditation.
Keywords: Accreditation of skills, Information Technology, Multicriteria analysis
1. Introduction
On a global level, the accreditation of skills and competences has troubled many
researchers, mainly the intergovernmental organizations such as the European
Council, the Cedefop organization, Unesco and OECD [3, 12, 4, 13]. The speculation
is focused mainly on the development of commonly accepted methods through which
a value will be attributed to the competences and skills acquired during the
professional life. New skills and competences are acquired every day through non-
formal or informal ways of learning, and particularly the learning acquired through
professional experience or team work, self-education and self-training, work field
training, etc. Especially in the last 5 to 10 years many European countries have
introduced and enacted methodologies and systems for the identification,
(*)
This research has been financially supported by the European Union, Leonardo Da Vinci
programme, no. EL/97/1/28132/PI/I.1.1.b/FPC
accreditation, certification of qualifications, skills and competences acquired through
non-formal vocational training. In particular, such initiatives have been taken in
France with the implementation of the “Bilan de Compétence” and the initiative of
the Accreditation of Professional Experience with a diploma of formal vocational
education, in England and Ireland with the implementation of the APL (Accreditation
of Prior Learning) system, and in Germany with the “Criteria Referenced Testing”
system [2]. The methods and systems proposed on a European level seek to
strengthen the value and provide clarity to the general skills and competences of
people, beyond the specialized and standardized knowledge that is closely connected
to a specific occupation or job, and subsequently aim to attribute value and recognize
various forms of non-formal learning. This tendency has led the majority of the
countries-members of the European Union to seek commonly accepted
methodologies, that will ensure with reliable procedures of diagnosis and evaluation,
the identification and accreditation of knowledge and skills acquired outside of formal
education and training.
More specifically, the fast development of information technology combined
with the new globalized environment and the rapid changes in all the fields of
business activity make the human factor the prevailing element. The need to update or
acquire new information technology knowledge has led more and more people to look
for new ways of learning. The significant demand results in the development of
several non-formal learning initiatives and many times in the offering of several
forms of information technology knowledge accreditation. The assessment of
information technology knowledge, suggested mainly by major manufacturing
companies of the field, is usually materialized through examinations accompanied by
the issuing of a certificate. The specific effort is focused on the knowledge of
candidates on a sector or product without taking into account other features of the
assessed persons, such as their working experience, their prior studies, their
participation in training programmes, the requirements of the jobs, etc. Moreover, it
requires the participation of the assessed person in an examination, which as a process
does not necessarily correspond to the profile of all ages, and mainly to those who
have been withdrawn from the educational procedure many years ago but are
nevertheless successful professionals in the field of information technology.
The approach proposed in this article for the accreditation of knowledge,
skills and competences in information technology concerns an original multicriteria
evaluation methodology. The accreditation process, as it is proposed, evaluates a
candidate in relation to the practice of a specific occupation or specialization, which
in our case is the field of information technology. The proposed methodology collects
facts for each person through all the possible combinations of learning and
acquirement of skills for an occupation, such as: professional experience, studies,
non-formal vocational training and personal activity. These facts result from personal
interviews with the candidate and the study of the personal candidature folder. The
target is to attribute through a unified way of evaluation, a kind of “identity” to the
interested parties for the accreditation per occupation/specialization, by classifying
them in four groups of information technology professionals. This procedure can lead
to the issuing of a formal document. Before the classification of the candidates in
groups of professionals, the evaluation in a set of qualitative criteria takes place in
order to use these values as input data in the applied multicriteria classification
method.
The analysis of the qualitative criteria which compose the total evaluation
problem as well as the presentation of the general methodological framework are
described in section 2. The evaluation system for the candidates‟ studies, applied in
the Greek educational system, is presented in section 3, while the evaluation system
for non-formal training programmes attended by the candidates is presented in section
4. Section 5 includes an implementation example, and finally section 6 deals with the
evaluation of the results and the suggestion of new research avenues.
2. The evaluation methodology
According to approaches that have been made by European bodies and by
educational-research institutions the views converge that in order for an employee to
succeed in the actual working conditions he/she must have a number of basic skills or
key skills. According to the English version of Skills Task Force [5], the six key skills
that the employees must have are: communication skills, knowledge of arithmetic and
information technology, problem solving skills, ability to work in a team and finally
ability to upgrade knowledge and performance. There are similar suggestions by USA
bodies, such as the National Skills Standard Board-NSBB and the Northwest Center
for Emerging Technologies-NWCET [10], which focus on the necessary basic skills
that each employee must have. According to the NWCET, a person in order to work
in the field of information technology must have, other than the basic skills required
for other occupations, certain “core computer software and hardware skills”, which
refer to the minimum technical skills or the entry skills. According to the NWCET
such skills are the following: knowledge of Windows environment, word processing,
spreadsheet applications, presentation software, database applications, principles of
programming, network technologies, hardware and software installation, use of e-
mail and internet. Certainly, beside the core skills, as they are proposed by the
NWCET, the practice of any occupation as well as any information technology
profession requires the existence of a number of specialized professional skills per
case, as it is suggested by the analytical proposals of the above bodies.
The proposed multicriteria methodology adopts the general principle that the
evaluation of a candidate should refer to the degree of his/her adequacy to practise a
specific occupation or speciality. By adopting the proposal of the NWCET, the
procedure for the evaluation of candidates for an information technology profession
consists of two phases (Fig. 1): in phase A all the candidates are evaluated on a set of
criteria-general skills and those who are judged sufficiently are afterwards evaluated
in phase B on another set of qualitative criteria composed of a number of specialized
skills, each time adjusted per occupation, studies and training. The successful
evaluation of a candidate in phase A does not necessarily mean that he/she can
become a professional in information technology, for it is known that most employees
nowadays have some information technology knowledge. For this reason, on the one
hand the evaluation of specialized skills is connected to the candidate‟s professional
experience and on the other hand the relevance of the studies and training with a
specific occupation is controlled. A complete presentation of the multicriteria
evaluation methodology and the proposed criteria of general and specialized skills is
given in [11].
It is commonly accepted that a person during his/her life span is in a constant
process of learning and acquiring new skills. If this period of time is delimited in the
period of a person‟s professional career, then it can be considered that the starting-
point of professional specialization is the education or training that he/she acquires by
attending a certain special school of the formal school system and/or outside of it,
which is usually referred as studies inside the formal educational system or/and the
system of non-formal vocational training. This view connects professional career to
the initial attendance in a certain educational organization without excluding the
possibility for each interested person to re-enter the educational or training system,
whenever it is thought necessary. This possibility is feasible and often imperative
mainly due the demands of the labour market. The case of employees who have learnt
an occupation through experience or practice means that they have to a certain degree
the skills for its performance but do not have the corresponding formal education
and/or non-formal training. All the above cases are examined and analyzed in relation
to the profile of each candidate for accreditation.
In particular, the following sections contain the analysis and presentation of
the studies and training criteria, in relation to the practice of a specific occupation.
The evaluation of the candidates‟ studies and training is part of the evaluations of
phase B, as it appears in Figure 1. The criteria, in which the general and specialized
skills per occupation are analyzed, are not presented in Figure 1 (details are given in
[11]). More specifically, the studies are analyzed in the sub-criteria of Basic,
Postgraduate and Doctoral studies, while the candidates‟ non-formal vocational
training is treated as two separate criteria. The first one concerns the training “with a
recognized title”, while the second “without a recognized title”. The basic target is the
assessment of a value to all criteria by examining each time their relevance to the
occupation for which the candidate asks accreditation.
Given that the candidates are sufficiently evaluated in phase A, they are
evaluated in phase B also for a set of other criteria that are connected to professional
experience and specialized skills per occupation, it is ensured that the evaluated
persons will fulfil to a certain degree the requirements for their accreditation in a
certain occupation. Therefore, the proposed evaluation procedure examines all the
possible ways to acquire knowledge and skills for a specific occupation and allows
the combination of the above, by choosing in each case those candidates who receive
high scores in all the criteria. Certainly, as the candidates differ from each other, the
aim is to attribute a unified kind of identity to each professional, that can have the
form of a formal document. For this purpose, it is proposed to classify the candidates
for accreditation in four homogenous groups of professionals and in particular in the
groups of beginners, competent, experienced and finally expert professionals [11].
MULTICRITERIA EVALUATION OF
INFORMATION TECHNOLOGY
COMPETENCES
SKILLS
GENERAL SKILLS
SPECIALIZED
SKILLS
BASIC STUDIES
INDIVIDUAL
QUALIFICATIONS
STUDIES TRAINING"with recognized title"
TRAINING"without recognized title"
Non-formal Training
POSTGRADUATE
STUDIES
DOCTORAL
STUDIES
Phase A'
Phase B'
Figure 1. The Multicriteria Evaluation Procedure for Inf. Technology Qualifications
It is indicatively mentioned that candidates who ask for accreditation as
“Computer Support Specialists”, can be classified in these four groups of
professionals. The candidates who fall into one of the four groups have the possibility
to be reaccredited later in order to improve their position and receive a documented
report that will mention the scores they received for each criterion. Finally, the
specific procedure is expected to contribute to the clarity and strengthening of the
employees‟ mobility within the countries of the European Union and to the
identification and attribution of value to knowledge and skills acquired through
professional experience and any other way of learning.
3. The system for the evaluation of accomplished studies
3.1 Basic evaluation model
The term “studies” refers to any type of attendance in a school/faculty of formal
education or training that belongs to the formal school system of each country, and
the studies that have been made after the completion of compulsory education. Taking
as a rule that the candidates for accreditation have completed certain studies, a system
is proposed which evaluates Basic, Postgraduate and Doctoral Studies. Therefore, the
target is first to assess a value to the three sub-criteria in which studies are analyzed
and then globally evaluate the studies of each candidate (Figure 2).
Doctoral Studies SubcriteriaPost Graduate Studies SubcriteriaBasic Studies Subcriteria
Multicriteria values
Basic Studies CriterionPostgraduate Studies
Criterion
Doctoral Studies
Criterion
GLOBAL EVALUATION
OF ACCOMPLISHED
STUDIES
60% 10% 30%
Figure 2. The Multicriteria Evaluation Model of Studies
More specifically, the global evaluation of studies (US) for each candidate for
accreditation, results from the aggregation of the three evaluations, as follows:
US= wBSU(es) + wMSCUMSC + wPHDUPHD (1)
where:
wBS =0.6 : Weight of Basic Studies criterion
wMSC=0.1 : Weight of Postgraduate Studies criterion
wPHD=0.3 : Weight of Doctoral Studies criterion
es : Alternative activity which corresponds to a specific combination of the type
of basic studies, marks and relevance of the basic studies to the occupation
for accreditation.
U(es): Value of es Basic Studies alternative in the interval [0,1]
UMSC : Value of Postgraduate Studies in the interval [0,1]
UPHD : Value of Doctoral Studies in the interval [0,1]
The determination of weights to the three sub-criteria of studies was based on
the years of attendance per case. Basic studies which include all types of schools or
faculties after the compulsory and up to tertiary education received a weight of 60%
(wBS = 0.6) expressing the longer period of time of certain studies. Postgraduate
studies which last a year received a weight of 10% (wMSC = 0.1), while Doctoral
studies which usually last three years received a weight of 30% (wPHD = 0.3).
3.2 Evaluation of Basic Studies : the Greek Paradigm
Basic Studies are considered all the possible cases of graduation after compulsory
education including the acquirement of a degree/diploma from Tertiary education.
Therefore, for the needs of the evaluation, all possible cases of acquiring relative
information technology knowledge through any type of school/faculty are examined.
Greece was selected as an example of implementation of the proposed methodology
and this led to the examination of all types of schools starting from the 70s and finally
to propose six different types of Basic Studies, as it is shown in Table 1.
Table 1. Types of Basic Studies in the Greek Educational System
Type
of
Studies
Description Explanation
A ΑΕΙ Highest Educational Establishment – Highest Tertiary Education
(University or Technical University).
B ΤΕΙ Technological Educational Establishment – Higher Tertiary Education.
C ΙΕΚ Institute of Vocational Training.
Post-Secondary (not Tertiary) Vocational Training.
D ΤΕΕ Cycle Β, Unified Lyceum, General Lyceum, 6-
class Gymnasium, ΤEL, ΕPL
Schools of (upper) Secondary Education-General and Secondary Vocational and Technical Education. Three-year duration.
E
ΤΕΕ Cycle Α, ΤΕΕ
Apprenticeship Cycle A-
OAED, TES
Schools of Secondary Vocational and Technical Education. Two-
year (or three-year) duration.
F ΙΕΚ 1 Vocational Training of 2 semesters after compulsory education.
In particular, the evaluation problem of Basic Studies is also treated as a
multicriteria problem, which is analyzed in three criteria:
Higher Title of Basic Studies possessed by the candidate.
Grade of the Title (degree, diploma, certificate, etc.).
Relevance of Basic Studies Title to a specific information technology
occupation.
For the modelling of the specific evaluation problem of Basic Studies, the
proposed qualitative evaluation scales are in detail the following: for the criterion
„title of basic studies‟ the six-degree scale has been adopted in correspondence with
the six types of studies (see Table 1), for the criterion „grade of title‟ a four-degree
evaluation scale is proposed (Excellent, Very Good, Good and Pass) and for the
„relevance‟ criterion a three-degree scale (High relevance of title with the occupation,
Medium relevance, and no-low relevance), as shown in Figure 3.
The proposed modelling of the criterion “grade of title” regards the Greek
educational system but can also be easily adjusted to any other. In particular, for the
evaluation of Basic Studies in the criteria: “type of studies”, “marks” and “relevance”,
a table of all possible combinations has been drawn up, based on the Greek example,
and the values of these combinations have been estimated (see Appendix) through a
multiplicative evaluation model estimated by a disaggregation model [6] presented
below.
Global Evaluation of
Basic Studies
Title of Basic Studies
Grade of Title
Relevance of Title
Α: AEI
B: TEI
C: IEK
D: TEE B, etc
Ε: TEE Α, etc
F: IEK 1
EXCELLENT
VERY GOOD
GOOD
PASS
HIGH
MEDIUM
NONE (or low)
T Y P E S
Figure 3. Evaluation Scales of Basic Studies
The set of alternative activities is defined the set of possible combinations of
the types of Basic Studies, Marks and Relevance, and is symbolized by A={ajkℓ,
j=1,…,J, k=1,…,K, ℓ=1,…,L} where j: type of Basic Studies, k: type of Marks and ℓ:
type of Relevance. The corresponding values for the case of Greece are: j=1,…,6
k=1,…,4 and ℓ=1,…,3. Considering the fact that the possible combinations in the
example of Greece are totally 48 (see [8]) a corresponding set of alternatives E is
defined, as follows: E={es, s=1,…,48}, which is a finite and predetermined set (see
Appendix).
The target now is to estimate a multiplicative value model that will allow the
ranking of all alternatives, by assessing the value of each activity. This model has the
following mathematical form:
U(es)= U(ajkℓ)= UsjUvkUrℓ (2)
where:
U(ajkℓ) : Global Value of candidate‟s Basic Studies:
j : type of Title of Basic Studies, j=1,…, 6 in correspondence to types Α-F.
k : type of Marks, k=1,…, 4 where 1 is given to Excellent and 4 to Pass.
ℓ : type of Relevance, ℓ=1,…,3 where 1 is given to High Relevance while 3 to
No (or low) Relevance.
Usj : value of j type of Basic Studies Title
Uvk : value of k type of Marks
Urℓ : value of ℓ type of Relevance
The equivalence of the alternative combinations: type of Basic Studies-j, type
of Marks-k and type of Relevance-ℓ with the 48 alternatives (es), is shown in the first
two columns of the Appendix.
For the use of the dissagregation approach (see [6]) a ranking of reference
activities (weak order) is presumed, which in this case is the ranking of the 48
activities of the appendix (third column), as it has been predetermined by certain
experts in the field of education.
According to the preference model (2), the following relations must be
effective:
U(es) > U(es+1) es P es+1 (strict preference of alternative es to es+1)
U(es) = U(es+1) es I es+1 (indifference between the two alternatives es and es+1)
For the estimation of the value function U(es), potential overestimation and
underestimation errors are introduced for all the activities and the following non
linear programme is solved which minimizes the sum of errors σ and -σ :
[min] F = 48
1
)}()({s
ss ee
subject to:
a. Preference constraints
)}()()()({)}()({ 111 ssssss eeeeeUeU , when es P es+1, for
s={1,…,32}
where δ=0.01
b. Indifference constraints
0)}()()()({)}()({ 111 ssssss eeeeeUeU , when es Ι es+1, for
s={33,…,47}
c. Normalization constraints at the value interval [0,1]
,11Us ,11Uv 11Ur
d. Additional constraints
The values of two successive types of Studies, Marks and Relevance are different by
at least 0.05:
05.01jj UsUs , 5,...,1j
05.01kk UvUv , 3,...,1k
05.01 UrUr , 2,...,1
and
σ+ (es), σ
-(es) 0 , s=1,…,48 ; 0jUs , 0kUv , 0Ur ,, kj
The optimal solution to the above non linear programme gives a zero total
error which shows the absolute compatibility of the value function with the initial
ranking of the alternatives. Moreover, the values Usj, Uvk and Urℓ of the relation (2)
are estimated and therefore the Global Value U(es) of each alternative is also
estimated. Analytically, the values for each type of Studies, Mark and Relevance are
shown in Table 2 while the Global Evaluation of the alternatives according to relation
(2) appears in the last column of the Appendix.
Table 2. Marginal Value Functions for Basic Studies
Variables
Value of
the types of Basic
Studies
Variables
Value of
the types
of Marks
Variables
Value of
the types of
Relevance
Us1 (Α:ΑΕΙ) 1 Uv1 (Excellent) 1 Ur1 (High) 1
Us2 (Β:TEI) 0.88 Uv2 (Very Good) 0.95 Ur2 (Medium) 0.64
Us3 (C:IEK) 0.50 Uv3 (Good) 0.89 Ur3 (None) 0
Us4 (D:TEE B, etc) 0.39 Uv4 (Pass) 0.84
Us5 (Ε:TEE A, etc) 0.31
Us6 (F:IEK1) 0.04
According to the estimated values, note that even in the worst case of studies
of type ΙΕΚ1 a 4% value is attributed. The same goes for the case of Pass grade in the
title of studies, while on the contrary if the title of studies has low or no relevance to
the occupation for accreditation it is valued by zero.
3.3 Evaluation of Postgraduate and Doctoral Studies
Postgraduate studies are considered in Greece the studies that lead to the obtaining of
the Postgraduate Specialization Diploma (PSD) and require at least the completion of
a certain programme of tertiary education. Similarly, the evaluation of doctoral
studies concerns the candidates who hold a title of doctoral studies.
If a candidate for accreditation has made postgraduate and/or doctoral studies, then
the relevance of his/her studies to the specific occupation for which he/she asks to be
accredited is examined. A value must be estimated for each case.
If the value of postgraduate and doctoral studies is symbolized with UMSC and
UPHD respectively, then:
UMSC = UΤ-MSCUR-MSC (3)
UPHD = UΤ-PHDUR-PHD (4)
where:
UT-MSC, UT-PHD: value of corresponding title (=1 if the candidate has the title; =0 if
not).
UR-MSC, UR-PHD: relevance value of corresponding title to the information technology
profession (=1 High Relevance; =0.60 Medium Relevance; =0 No-
low Relevance).
If the candidate has made postgraduate and/or doctoral studies with low or no
relevance to information technology, relevance is marked with zero as opposed to
cases when there is medium relevance to information technology and in the best case
of high relevance of the studies to the specialized information technology profession
when the score is 0.60 or 1 respectively. Therefore, in the worst case (anti-ideal),
where there is no relevance of the postgraduate and/or doctoral title to the information
technology profession for which the candidate requests accreditation or the candidate
has not made any postgraduate or doctoral studies, then UMSC=0 or/and UPHD=0, while
in the best case (ideal), of high relevance of the title to the information technology
profession, then UMSC=1 or/and UPHD=1. Therefore, the high relevance of
postgraduate or/and doctoral studies gives one point respectively to the candidate,
while the relative studies when they have no relevance to information technology or
are not judged sufficient for the specific occupation for accreditation, then they are
not taken into account during evaluation.
4. The evaluation system for non-formal training
The need to be updated in new technologies and the products that are constantly
introduced requires the continuous attendance of non-formal training programmes
that do not belong to the formal school system. Consequently, the non-formal training
programmes can be an evaluation element for the accreditation of the candidates. So,
training is divided in two criteria: the one that takes place by following certain
standards and usually a specific and similar in each country way of examination,
described as “Training with a recognized title” and the one that does not fulfil the
above prerequisites and is quite common, described as “Training without a
recognized title”. For example, in Greece the training programmes that take place
through Liberal Studies Laboratories, Vocational Training Centres-KEK, intra-
company programmes, etc., are cases that belong to the training criterion of type
“without a recognized title”. Similarly, in the case of non-formal training of type
“with a recognized title” are included for example the certificates of: ECDL,
Microsoft, Oracle, etc. Our aim is to evaluate the non-formal training programmes
with or without a “recognized” title, that the candidate is certified to have attended,
by giving a value to each criterion.
4.1 Evaluation of non-formal “training with a recognized title”
In this case two elements are examined: the existence of such training titles and the
relevance of each title to the occupation for which the candidate requests
accreditation. If, iVT
U expresses the value of i programme of non-formal vocational
training “with a recognized title”, then the global value of non-formal vocational
training “with a recognized title” (UVT+), of the candidate, results from the type:
UVT+
= VT VT1 1
if
1 otherwise
U Un n
i ii i
k k (5)
where:
n : number of non-formal vocational training programmes “with a recognized
title” attended by the candidate
k : higher number of evaluated non-formal vocational training programmes “with
a recognized title, where in our case k=3
UVT+ : value of vocational training “with a recognized title”, at the value interval
[0,1]
where:
iii VTRVTTVTUUU for each i Certificate (“recognized title”) (6)
when:
iVTTU : value of i title (=1 if available by the candidate; =0 if not available).
iVTRU : relevance value of i title to information technology profession (=1 High
relevance; =0.6 Medium relevance; =0 Low-no relevance).
If the candidate has attended a training programme with low or no relevance to
information technology then a zero value is attributed, while he/she is marked with
0.60 if the programme concerns a certain issue of information technology and has
offered him/her some relative knowledge but possibly not completely relevant to the
occupation for accreditation. In the best case of high relevance of training to the
occupation for accreditation is given the score one.
4.2 Evaluation of non-formal “training without a recognized title”
During the evaluation of non-formal training programmes “without a recognized title”
that the candidate is certified to have attended, the relevance of each programme to
the occupation for accreditation and the duration of the training are examined.
If, i
VTU expresses the value of i non-formal training programme “without a
recognized title”, then the global value of non-formal training of type “without a
recognized title” (UVT-) results from the type:
UVT- = VT VT
1 1
if
1 otherwise
U Un n
i ii i
m m (7)
where:
n : number of non-formal vocational training programmes “without a
recognized title” attended by the candidate
m : higher number of evaluated programmes of non-formal training “without a
recognized title”, where in our case m=6
UVT- : value of vocational training “without a recognized title”, at the value
interval [0,1]
where:
iiii VTRVTDVTTVTUUUU for each i Certificate (“non recognized title”) (8)
when:
: value i of title, receiving value 1 or 0 depending on whether the candidate
has or not a relative certificate.
iVTDU : value of duration of i training programme “without a recognized title”.
iVTRU : relevance value of i certificate to a specific information technology
profession, when relevance is evaluated as follows:
High relevance: 1, Medium relevance: 0.6, No (low) relevance: 0,
and the quantification of the qualitative scale follows a similar justification
as the respective case of training “with a recognized title”.
The period of time for the realization of a vocational training programme
usually varies from a few days to several months. Taking as a rule that the period of
time for the realization of a training programme can be counted in teaching hours, it is
proposed to take into account the duration in the evaluation methodology based on the
value scale shown in Table 3. This value scale results from a mid value point
technique [7], which suggests that a seminar of at least 200 hours is judged sufficient
for the education of the trainees, while a lower value is given to seminars of a shorter
duration. For example, a seminar of 150 hours covers requirements by 70%, while a
iVTTU
seminar of 100 hours by 40%. So, an intensive training programme is separated from
a longer duration programme, but even a seminar of few hours is counted in the
evaluation. For all the cases where the time duration of a training seminar varies
between the periods of time of Table 3, the linear interpolation type takes effect.
Table 3. Time Duration Evaluation Scale for Vocational Training Programmes “Without a
Recognized Title”
Duration in Hours 0 50 100 150 200 Over 200
Value UD-VTi- 0 0.1 0.4 0.7 1 1
5. Example of implementation
Let‟s suppose a candidate who has been sufficiently evaluated in phase A of the
evaluation and is evaluated in phase B on the studies also criterion and the two non-
formal training criteria from the whole set of qualitative criteria. Let‟s also suppose
that he/she asks for accreditation in the profession of “Computer Support Specialist”
and is a graduate of the Department of Electronic Engineers and Computer Engineers
of the Technical University of Crete, with grade “Very Good”, and has made
Postgraduate Studies at the same Department. He/she has attended two training
programmes by Microsoft, and in particular the MOUS (Microsoft Office User
Specialist) and MCSE (Microsoft Certified Professional-Systems Engineer) and a
100-hour programme, at a private school, with the title “Basic Principles of
Information Technology”. During his/her professional career he/she participated in
two intra-company seminars: “Introduction to network security issues” and “Unix
Operating System”, lasting 24 and 60 hours respectively.
Evaluation of Studies
Due to the high relevance of Basic and Postgraduate Studies to the specific
information technology profession and according to the values of Table 2 or the last
column of the Appendix and the relations (2), (3), (4) and (1) respectively, he/she will
receive the following scores:
Basic Studies: U(e2) = 1 x 0.95 x 1 = 0.95 (ΑΕΙ, Very Good, High Relevance)
Postgraduate Studies: UMSC = 1 x 1 = 1 (High Relevance)
Doctoral Studies: UPHD = 0 (Not available)
Global Evaluation of Studies: US = 0.95x0.6 + 1x0.1 + 0x0.3 = 0.67
Evaluation of Non-formal Vocational Training “with a recognized title”
If we consider that the two Microsoft programmes are classified in the
category: non-formal training “with a recognized title”, then according to the relations
(6) and (5) respectively, he/she receives:
MOUS Certificate: 6.06.011
xUVT
, (Medium relev. to the specific occupation)
MCSE Certificate: 1112
xUVT
, (High relevance to the specific occupation)
Global Evaluation of Non-formal Training “with a recognized title”:
UVT+ = (0.6 + 1) / 3 = 0.53
Evaluation of Non-formal Vocational Training “without a recognized title”
According to the non-formal vocational training programmes attended by the
candidate “without a recognized title” privately or inside the company where he/she
works and the relations (8), (7), the values of Table 3 and through the application of
the linear interpolation type where required, he/she receives:
Basic principles of information technology, 100 hours:
24.06.04.01 xx1VT
U (Medium relevance)
Introduction to security issues, 24 hours: 04.0104.01 xx2VT
U (High relevance)
Unix Operating System, 60 hours: 16.0116.01 xx3VT
U (High relevance)
Global Evaluation of Non-formal Training “without a recognized title”:
UVT- = (0.24 + 0.04 + 0.16) / 6 = 0.07
Therefore, the evaluation of the candidate in the three criteria goes as follows:
Studies: 0.67, Non-formal Training “with a recognized title”: 0.53 and Non-formal
Training “without a recognized title”: 0.07, which are the values of the three criteria
of phase B of the multicriteria methodology. In order to classify the candidate in one
of the four proposed categories of information technology professionals, he/she must
also be evaluated on a set of criteria referred to the specialized skills required by the
occupation or speciality according to the available professional experience (details are
given in [11]).
6. Concluding remarks
The accreditation problem of qualifications and skills has been treated realistically as
a multicriteria decision problem. The proposed methodology has been adjusted to the
needs of occupations/specializations of information technology and it can also be
applied with the proper modifications to other areas of expertise and occupations. The
developed methodology evaluates all the cases of acquiring knowledge and skills
through work, the formal school system, the non-formal vocational training system or
even personal activity, recognizing that the technical knowledge for the practice of a
profession can derive from a combination of the above.
The disaggregation approach applied in the case of attributing a value to the
candidates‟ basic studies includes all the possible cases of graduation from a formal
education school. The implementation made to the Greek educational system has
proven that a value can be attributed to the candidates‟ studies for the need of
accreditation. Moreover, an evaluation system is proposed for postgraduate and
doctoral studies and an evaluation system for any type of non-formal training
attended by the candidates.
The implementation made for a candidate has proven the effectiveness and
convenience in applying the methodology and the fact that the requested elements for
the evaluation can easily result from the study of the candidature personal folders or
the interviews with the candidates. It has also been proved that the multicriteria
approach can support the decision-maker and in this case an enacted committee that
can accredit the candidates for an occupation or speciality in a unified and clear way.
Certainly, the issue of accreditation has significant perspectives for further research
such as: (1) exploration of the evaluation system with the assistance of a competent
committee as the decision-maker; (2) exploration of the possibility to combine with
alternative approaches proposed in European countries at a pilot or applied level; and
(3) application of other multicriteria methods and comparison of the evaluation
results.
References
[1] Beuthe, M. and Scannella, G. (2001). Comparative analysis of UTA multi-
criteria methods. European Journal of Operational Research, 130, 2, 246-262.
[2] Bjørnåvold, J. (1997). Assessment of non-formal learning: the quality and
limitations of methodologies. The European Vocational Training Journal, (12),
52-67.
[3] Bjørnåvold, J., and Sellin, B. (1997). Recognition and transparency of
vocational qualifications; The way forward. Discussion paper. Cedefop
Panorama, Thessaloniki.
[4] Bjørnåvold, J. (2000). Making Learning Visible: Identification, assessment and
recognition of non-formal learning in Europe. Cedefop. Thessaloniki.
[5] DfEE (Department of Education and Employment), Skills Task Force (2000).
Second Report of the Skills Task Force. Towards a National Skills Agenda.
Chapt. 6.
[6] Jacquet-Lagrèze, E. and Siskos, Υ. (2001). Preference disaggregation: 20 years
of MCDA experience. European Journal of Operational Research, (130), 2,
233-245.
[7] Keeney, R. and Raiffa, H. (1976). Decisions with multiple objectives:
Preferences and value trade-offs. Wiley, New York.
[8] Krassadaki, Ε. (2001). Development of a multicriteria evaluation methodology
for information technology qualifications. Postgraduate Thesis, Technical
University of Crete, Department of Production Engineering & Management,
Chania.
[9] National Skill Standards Board, www.nsbb.org
[10] NWCET, Northwest Center for Emerging Technologies (1999). Building a
foundation for tomorrow. Skill standards for information technology, Bellevue
Community College.
[11] Siskos, Y., Grigoroudis, E., Krassadaki, E., Matsatsinis, N. (2001). A multi-
criteria accreditation system for information technology skills and
qualifications. Working Paper.
[12] Turner, C. (1999). Identification, assessment and recognition of non-formal
learning in Greece. Cedefop Panorama, Thessaloniki.
[13] UNESCO European Symposium (1999). Vocational Education and Training in
Europe, on the threshold of the 21st century. Conclusions. Conference Crete 23-
26 September 1998.
Appendix: Alternative activities of Basic Studies
Alternatives
es
(s=1,…,48)
Alternatives
ajkl, for each
combination j, k, ℓ
Ranking
by
increasing preference
order
Alternative Combinations of Basic Studies Titles,
Marks and Relevance
(according to the Greek example)
STUDIES MARKS RELEVANCE
Global
Eval.
of alterna-tives
e1 a111 1 ΑΕΙ EXCELLENT HIGH RELEVANCE 1
e2 a121 2 ΑΕΙ VERY GOOD HIGH RELEVANCE 0.95
e3 a131 3 ΑΕΙ GOOD HIGH RELEVANCE 0.89
e4 a211 4 ΤΕΙ EXCELLENT HIGH RELEVANCE 0.88
e5 a221 5 ΤΕΙ VERY GOOD HIGH RELEVANCE 0.83
e6 a231 6 ΤΕΙ GOOD HIGH RELEVANCE 0.79
e7 a112 7 ΑΕΙ EXCELLENT MEDIUM RELEV. 0.64
e8 a122 8 ΑΕΙ VERY GOOD MEDIUM RELEV. 0.61
e9 a132 9 ΑΕΙ GOOD MEDIUM RELEV. 0.58
e10 a212 10 ΤΕΙ EXCELLENT MEDIUM RELEV. 0.57
e11 a222 11 ΤΕΙ VERY GOOD MEDIUM RELEV. 0.54
e12 a232 12 ΤΕΙ GOOD MEDIUM RELEV. 0.51
e13 a311 13 ΙΕΚ (EXCELLENT) HIGH RELEVANCE 0.50
e14 a411 14 ΤΕΕ Β, etc EXCELLENT HIGH RELEVANCE 0.39
e15 a421 15 ΤΕΕ Β, etc VERY GOOD HIGH RELEVANCE 0.37
e16 a431 16 ΤΕΕ Β, etc GOOD HIGH RELEVANCE 0.35
e17 a441 17 ΤΕΕ Β, etc PASS HIGH RELEVANCE 0.33
e18 a312 18 ΙΕΚ (EXCELLENT) MEDIUM RELEV. 0.32
e19 a511 19 ΤΕΕ Α, etc EXCELLENT HIGH RELEVANCE 0.31
e20 a521 20 ΤΕΕ Α, etc VERY GOOD HIGH RELEVANCE 0.29
e21 a531 21 ΤΕΕ Α, etc GOOD HIGH RELEVANCE 0.28
e22 a541 22 ΤΕΕ Α, etc PASS HIGH RELEVANCE 0.26
e23 a412 23 ΤΕΕ Β, etc EXCELLENT MEDIUM RELEV. 0.25
e24 a422 24 ΤΕΕ Β, etc VERY GOOD MEDIUM RELEV. 0.24
e25 a432 25 ΤΕΕ Β, etc GOOD MEDIUM RELEV. 0.23
e26 a442 26 ΤΕΕ Β, etc PASS MEDIUM RELEV. 0.21
e27 a512 27 ΤΕΕ Α, etc EXCELLENT MEDIUM RELEV. 0.20
e28 a522 28 ΤΕΕ Α, etc VERY GOOD MEDIUM RELEV. 0.19
e29 a532 29 ΤΕΕ Α, etc GOOD MEDIUM RELEV. 0.18
e30 a542 30 ΤΕΕ Α, etc PASS MEDIUM RELEV. 0.17
e31 a611 31 ΙΕΚ1 (EXCELLENT) HIGH RELEVANCE 0.04
e32 a612 32 ΙΕΚ1 (EXCELLENT) MEDIUM RELEV. 0.02
e33 a113 33 ΑΕΙ EXCELLENT NO RELEVANCE 0
e34 a123 33 ΑΕΙ VERY GOOD NO RELEVANCE 0
e35 a133 33 ΑΕΙ GOOD NO RELEVANCE 0
e36 a213 33 ΤΕΙ EXCELLENT NO RELEVANCE 0
e37 a223 33 ΤΕΙ VERY GOOD NO RELEVANCE 0
e38 a233 33 ΤΕΙ GOOD NO RELEVANCE 0
e39 a313 33 ΙΕΚ (EXCELLENT) NO RELEVANCE 0
e40 a413 33 ΤΕΕ Β, etc EXCELLENT NO RELEVANCE 0
e41 a423 33 ΤΕΕ Β, etc VERY GOOD NO RELEVANCE 0
e42 a433 33 ΤΕΕ Β, etc GOOD NO RELEVANCE 0
e43 a443 33 ΤΕΕ Β, etc PASS NO RELEVANCE 0
e44 a513 33 ΤΕΕ Α, etc EXCELLENT NO RELEVANCE 0
e45 a523 33 ΤΕΕ Α, etc VERY GOOD NO RELEVANCE 0
e46 a533 33 ΤΕΕ Α, etc GOOD NO RELEVANCE 0
e47 a543 33 ΤΕΕ Α, etc PASS NO RELEVANCE 0
e48 a613 33 ΙΕΚ1 (EXCELLENT) NO RELEVANCE 0