196

197

ON-LINE BENCHMARKING OF QUALITY ASSURANCE SYSTEMS IN VET ORGANISATIONS

Introduction

The on-line benchmarking tool for quality assurance in vocational education and training (BEQUAL tool, available at www.bequal.info) is a tool for VET organisations that want to make a self – assessment of their quality practices and processes and benchmark with peer organisations. BEQUAL tool has been developed based on the principles of EQAVET and EFQM. The users of BEQUAL tool can make a self-evaluation of their quality processes, following an extended questionnaire. As a result, the users get a benchmarking report of their achievements in quality assurance compared to their peer organisations. The benchmarking report that the tool generates can compare performance at European level, at national level and at the level of type of VET institute. The user can repeat the on-line self-assessment every year and keep a track of the progress and improvement.

The on-line benchmarking tool is based on

• the BEQUAL quality assurance model;

• the BEQUAL self-evaluation questionnaire;

• an on-line database of filled-in questionnaires;

• a benchmarking formula.

BEQUAL tool has been developed during the Leonardo da Vinci project BEQUAL (2008) and has been further expanded with the projects BEQUAL+ (2012) and EXPANDVET (2013). Currently the BEQUAL tool exists in 11 languages, with the goal to expand in all EU languages.

1 BEQUAL quality assurance for VET organisations model

Quality assurance in VET, both at system level and at VET organisations level has been stated as one of the key pillars for the modernisation of VET in Europe. Since the Lisbon strategy in early 2000s, the developments in the field of quality assurance are rapid. Quality assurance is the concept underpinning all European instruments of the Copenhagen process, the European Qualifications Framework (EQF), the European Credit System for VET (ECVET) and Europass. The central role of quality assurance is underlined in all policy documents at European level and it is incorporated as a requirement at national VET systems. The Recommendation on the establishment of a European Quality Assurance Reference

NATASSA KAZANTZIDOU - IDEC SA (EL)

198

Framework for Vocational Education and Training (EQARF Recommendation, 2009) provides a reference framework (EQAVET) for Member States and for VET organisations, to base on their quality assurance systems.

As a reference framework EQAVET is highly abstractive and can be implemented in many different ways, thus serving the needs and particularities of different countries and organisations. Vocational education and training has a high diversity of organisations and training courses at multiple levels. VET organisations vary extremely in their size from very big organisations to micro organisations, their scope including initial VET and continuing VET, their level of VET provision from lower secondary to higher VET programs, their source of funding including public organisations and private organisations. The VET programs vary as well in their duration from some hours to years, in their status from formal training courses to non�formal training course and in the source of funding from public funded to self�funded training courses.

Nevertheless, VET organisations share some common characteristics. The work of VET organisations is important for the individuals and for the society as a whole and it has long term effects. VET organisations need to raise their quality standards, to be excellent organisations and role models in their operating environment. The EFQM excellence model (EFQM, 2003) has been chosen as the most adequate model for VET organisations. EFQM is a well known and respected quality assurance framework. EFQM is based on eight fundamental concepts of excellence which are the underlying principles of the EFQM Excellence Model for achieving Sustainable Excellence for any organisation.

Based on the EQAVET reference framework and EFQM fundamental concepts, BEQUAL consist of a set of quality criteria, categorised in the four phases of quality assurance cycle: planning, implementation, evaluation and review. Table 1 presents the quality criteria of BEQUAL tool.

EQAVET Criteria BEQUAL Criteria

Planning 1 Definition of needs in human resources

2 Taking into account the context

3 Taking into account the results

4 Customer needs and general policies

5 Concrete training needs

6 Taking into account national and European policies

7 The training evaluation process

8 Social responsibility

9 Planning of material and technological resources

199

Implementation 1 Taking into account the social aspects of training

2 Taking into account the trainees’ needs

3 The key persons of the implementation

4 Definition and implementation of responsibilities

5 Follow up and supervising

6 Transparency of strategic and generic objectives

7 Communication of objectives to staff

8 Human resources development

9 Partnership with customers and providers

Evaluation and assessment 1 Evaluation by customers

2 Performance evaluation

3 Evaluation of trainers and other services

4 Good practices and comparison between objectives and results

5 Evaluation of social responsibility

Feedback and procedures for change 1 Enhancing reactivity

2 Capitalisation of experiences

3 Actions of improvement

4 Skills management

5 Customer feed-back

6 Risk management

Table 1 BEQUAL quality criteria for VET organisations

2 The BEQUAL questionnaire

The core of the BEQUAL tool is a self assessment questionnaire addressed to managers or quality managers of VET organisations. The self�assessment questionnaire presents the 29 quality criteria of the BEQUAL model (table

1), each one with a statement of the quality criterion and three to five practical examples on how the criterion can be applied in practice. In addition, the questionnaire gathers some profile data, including country and type of VET organisation, which are being used to filter the results of benchmarking.

200

The quality criteria are presented in the four steps of quality cycle: planning, implementation, evaluation and review in

the form of statements with examples. An extract of the first two questions (out of 29) is presented in table 2.

Questions on quality criteria

1. Definition of needs in human resourcesOnce we set our objectives, we draw up a human resources strategy or an equivalent plan to support their achievement.For example: • We determine systematically competencies which are necessary to achieve objectives • We evaluate annually competencies of our staff • We have an annual training plan in order to achieve our objectives • We evaluate the efficiency of human resources management policy

We continuously evaluate competencies of our trainers, including subcontractors1. Not started 2. Some progress 3. Considerable progress 4. Close to fully achieved/fully achieved

2. Taking into account the context We base our strategy and annual plans on information obtained through analysis of the operating environment.For example: • We have defined responsibilities to continuously keep a file of legal obligations • We have defined responsibilities for following competitors’ or similar organisations’

activities (financial, pedagogical, website etc.) • We have regular review meetings to evaluate the impact of the operating environment on

our processes and the content of our training1. Not started 2. Some progress 3. Considerable progress 4. Close to fully achieved/fully achieved

Table 1 BEQUAL quality criteria for VET organisations

201

The questionnaire uses an evaluation scale of four steps, as it is presented in table 3.

3 The on-line BEQUAL tool

Based on the BEQUAL questionnaire, the on-line tool has been developed as an extension for the open-source content management system (CMS) joomla. The on-line tool consists of a multiple questionnaire of 29 questions, a database of replies, a benchmarking formula and report templates.

The on-line BEQUAL tool requires user

registration. By creating a user account, the VET organisation can fill in the on-line questionnaire and store its results, browse the saved questionnaire and access an individual benchmarking report. Overall statistics as well as filtered statistics per country and per VET level provision are publicly accessible without registration. The BEQUAL tool is quite straight forward to fill in and it requires a time investment of about 15-20 minutes.

Table 3 BEQUAL evaluation scale

Evaluation scale

1. Not started • Not much happening in this area; perhaps some good ideas but they

have not progressed much beyond wishful thinking • No reference to any official document

2. Some progress

• Successful implementation or results for some key areas • Some actions are taking place on an ad�hoc basis, mostly reactive

and not pro�active • Reference to a policy statement or official documents • Random and occasional reviews resulting in improvement

3. Considerable progress

• The subject is well addressed for many key processes and areas • Regular and routine review and enhancement of the approach • Some concerns that this subject is not being addressed to its full

extent in all areas and processes • Linked with the overall policy and strategy of the organisation

4. Close to fully achieved / fully achieved

• Clear evidence that the subject is well achieved for most key processes and areas

• The subject is normal and fundamental part of the operational processes

• Approach and implementation are reviewed frequently and improved; positive trends have been recorded over a number of years

• A role model solution or achievement, which could be considered best practice

202

Figure 1: BEQUAL questionnaire

The benchmarking formula is based on a four level evaluation scale. The benchmarking formula calculates the position of the VET organisation among the peer organisations for each individual criterion. The result of the benchmarking tool is an individual report, where the user can see its performance against the performance of peer organisations for each criterion, the three strong points and the three weak points of own organisation. Strong and weak points are calculated in relation to peer organisations for each

question, using the distribution of replies of all users. The strong and weak points of a VET organisation are not stored in the database, instead they are calculated each time the user asks for a benchmarking report.

The on-line BEQUAL tool requires a minimum of 100 replies in overall and at least 20 replies from each country, in order to have credible results. When new country versions are added, we ensure to each first the threshold of 20 replies before launching the country version.

203

Figure 2: BEQUAL individual report

4 Reflections from the use of the tool

4.1 Reflections on Current Practice

BEQUAL tool has been launched in 2010, initially in English, Greek, German, French, Spanish and Romanian languages. In the first year of its operation it gathered a number of 145 filled�in questionnaires, the majority of them from Greece, Germany, France, Spain and Romania, while there was a small number of responses from other countries. In 2012, the tool expanded to new language versions,

including Bulgarian, Italian and Turkish that become fully operational in early 2013. In parallel, new language versions have been developed: Lithuanian and Polish that at the moment (May 2013) are in beta version and will become operational in few months. The total responses are above 300 and constantly rising. Tables 4 and 5 present the users of the benchmarking tool by country and by type of VET organisation (data 2/9/2013).

204

Users by country

France 27

Germany 32

Greece 28

Romania 33

Spain 27

Bulgaria 66

Italy 18

Turkey 41

Lithuania 0

Poland 0

Austria 0

UK 2

Other 51

TOTAL 325

Users by type of VET

Lower Secondary vocational education and training

19

Upper Secondary vocational education and training

65

Post Secondary non-tertiary vocational education and training

35

Higher vocational education 47

Further education and training 135

Apprenticeship 1

TOTAL 325

Table 4 Statistics extracted from BEQUAL tool – users by country

Table 5 Statistics extracted from BEQUAL tool – users by type of VET

The number of responses ensures the credibility of statistic results, nevertheless with further use of benchmarking profile and collection of a larger number of quality profiles, the benchmarking tool becomes a powerful and reliable tool for benchmarking and continuous quality improvement of VET organisations.

The first evaluation of the BEQUAL tool by end�users (training organisations) that was organised during 2010 in Greece, Germany, Spain, Romania and France revealed a positive feedback. The majority of the users (75�80%) rated high the quality of the benchmarking questionnaire, the clarity, simplicity and comprehensibility of the presented questions. A 52% of the responders rated positively the relevance of the tool, which leaves room for improvement.

The second evaluation performed during the first months of 2013 in Bulgaria, Italy and Turkey was also positive and the users made some valuable comments for the improvement of the benchmarking tool. The most important comment is the need to make the tool relevant to all types of organisations providing VET, i.e. VET schools, IVET centres, CVET centres etc.

A third evaluation is planned during second half of 2013, in Lithuania, Poland, United Kingdom and Austria. The Benchmarking tool will be revised in late 2013 � early 2014, to take into account the evaluation feedback from the first years of its operation, as well as recent developments in the field of quality assurance, especially the launch of a new revised EFQM excellence model (EFQM, 2013).

205

References

BEQUAL (2008). Retrieved from http://www.bequal.info/

BEQUAL + (2011). Retrieved from http://www.bequalplus.info/

EXPANDVET (2012). Retrieved from http://www.idec.gr/expandvet/

EFQM Excellence model 2003. Retrieved from http://ww1.efqm.org/en/PdfResources/PUB0723_InEx_en_v2.1.pdf

EFQM Excellence model 2013. Retrieved from http://www.efqm.org/en/tabid/132/default.aspx.

European Parliament and Council (2009). Recommendation of the European Parliament and of the Council of 18 June 2009 on the establishment of a European Quality Assurance Reference Framework for Vocational Education and Training. Retrieved from http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2009:155:0001:0010:EN:PDF