Geert Devos / Jef C. Verhoeven26

In search of 'best practice' in school self-evaluation: An exploratory study in secondary schools'

1 Statement of the problem and objective of the research Recent developments in secondary education are increasing school

autonomy. This is leading to greater responsibility being placed on the local administrative level. Starting from their own mission, schools have to ar-rive at an educational process whereby the perronnel, the financial, and the materiel policy are in harmony. In education, one can identify at least 3 evaluation levels: the central authorities (output control), the evaluation with external support, and the interral evaluation. School autonomy implies primarily the last, that is, being able to evaluate oneself. If schools are to receive greater responsibility, they must also be able to look critically at their own activities.

In this problem of interral evaluation, two approaches can be distin-guished: the school as a pedagogical-didactic process (with the clans as the microcosm) and the school as an organisational whole in which task-distribution and co-ordination processes take place. In this study, attention is given primarily to the evaluation of the school organisation. We consider school organisations as self-developing organisations. This point of depar-ture is important because evaluation cannot be seen apart from the organisa-tional dynamics that launch it.

According to experts, there are few secondary schools in Flanders in. which there is already a tradition of systematic self-evaluation. Moreover, self-evaluation is a concept that includes many things, and the concepts of self-evaluation diverge substantially. There is also little idea of the way in which secondary schools can deal with self-evaluation instruments. In order to obtain a more accurate picture of this problem area, an exploratory study was set up focussing on two research questions:

a) Hbw do educational experts perceive self-evaluation? b) How does the self-evaluation process proceed with the aid of external

self-evaluation instruments in secondary schools in general and in tech-

26 Geert Devos is associated with the Vierick Leuven Ghent Management School and Jef C. Verhoeven with the Centre for Educational Sociology of the Katholieke Universiteit Leuven.

27 This aaide is based on research conducted within the OBPWO 96.04 Project.

41

nical schools in particular? Do the schools achieve 'best practice' as re-gards self-evaluation?

2 What is self-evaluation? The literature offers a wide range of definitions of self-evaluation as

well as many concepts that refer to it (for example, school-based review, school self-evaluation, and school-level evaluation). We will use the notion of 'school self-evaluation'. In order to describe self-evaluation, we will use the definition offered by Van Petegem (1997: 127-132) and will examine the meaning of each component of the definition. We will also use these criteria to search for 'best practice' as regards self-evaluation.

In the first place, self-evaluation is seen as a process (van Aanholt et al., 1990: 20; Deckers & Jacobs, 1994: 22). Not the final result (product) but the process is central. Within the evaluation process, several phases can be distinguished: the preparatory phase, the execution phase, the scheduling, and the action phase. These phases succeed each other in the form of a regularly recurring cycle (Voogt, 1989: 28). Self-evaluation is in this form the keystone of a permanent and cyclical process of quality assurance (Boonen, 1990: 3-5). Previously conducted evaluations then constitute the basis on which the present functioning of the school is critically judged, al-though self-evaluation can also take the form of an occasional activity.

Second, self-evaluation is primarily initiated by the school `itself By `itself is intended the internal involvement of the school team and the exe-cution of all phases of the self-evaluation as much as possible on its own (Bunt & Van Hees, 1991: 15; Boonen, 1990: 3-4). The school (or parts thereof) is (are) the object of the evaluation /analysis, but it also has control over the analysis procedure (Pouwels & Jungbluth, 1991; Voogt, 1989). Schools thus, in a certain sense, play a "double role": they are themselves the object of the evaluation and at the same time its executors and/or con-trollers (Voogt, 1989: 25). Bunt & Van Hees (1991: 15) even consider it es-sential that the control of the information be retained by the school itself. However, the initiative can also come from outside the school. In this case, we speak of external evaluation, for example: the examination of the school by the Inspectorate. It is also possible for the self-evaluation to be initiated from the outside (for example, by pedagogical support services), but is con-ducted by the school itself.

Third, self-evaluation is done by well-chosen participants, that is, the people who are consulted for inventorying specific information (subjects of self-evaluation) are carefully selected. These are generally people who are closely involved with the school. With a view to good co-operation, it is

important that it be made clear why an evaluation is being made and what purpose the results will serve (Nevo, 1995: 123).

The fourth characteristic of self-evaluation is that it is done systemati-cally (Van den Bergh, 1990; Deckers & Jacobs, 1994: 15, 36; Nevo, 1995: 52) by making use of self-evaluation instruments and this in a reasonably formalised manner (Voogt, 1989: 26; Nevo, 1995: 52). Merely calling upon intuition is not sufficient.

Fifth, self-evaluation concerns the functioning of the school. Self-evaluation is intended to obtain information about the actual school praxis (Boonen, 1990; Cremers-van Wees, 1996: 2) and so can concern both parts of the school (specific analysis) and the entire functioning of the school (global analysis) (Voogt, 1989: 26; Boonen, 1990: 6). These aspects of the school can be organisational and/or pedagogical (Boonen, 1990: 6). The fact is that the actual functioning of the school is a very complex and dy-namic process (Voogt, 1989: 26; Miles, 1964, 1985; Marx, 1975; Fullan, 1982).

Sixth, self-evaluation comprises description and judgement. Evaluation not only refers to the systematic recording of school characteristics and facts but also includes a value judgement (Voogt, 1988, 1989: 25; Nevo, 1995: 11). The estimate of the value (i.e., judgement) of an action or prod-uct is also done in a more or less systematic manner (Van den Bergh, 1990). In this way, the strong and weak points of the functioning of the school are not only described but also judged.

The judgement supposes the presence of points of comparison (norms). For example, one can judge the present functioning by comparing it with certain stated or desired improvement objectives (Voogt, 1989: 25). Boonen (1990), too, would compare the factual situation with the desired situation. In addition, he would investigate the way that leads to the desired situation. Both the product and the process evaluation are important for this author. Other points of comparison can be obtained from results of a self-evaluation with different participants (for example, students, teachers, and heads), re-sults of a group of schools, or postulated indicators.

In the fact that evaluation also involves judgement lies precisely a dan-ger. Research shows that teachers feel threatened and uncomfortable with evaluation (House, 1973: 5). One wonders then if they even want to be evaluated. If not, those involved are apt to become suspicious and go on the defensive (Simons, 1985) or pretend to go along with it. This has to be taken into account in setting up self-evaluation.

42 43

Seventh, self-evaluation is done with a view to taking decisions or launching initiatives. After the inventorying and judging of the characteris-tics of the school and evaluation, recommendation for remedial action can follow (Voogt, 1989: 25). In other words, decisions are taken from which initiatives could flow. This also applies when it is established that the school scores highly. Indeed, the school then needs to be careful that it does not become complacent because of its good achievements. It is important to know that quality maintenance, too, constantly requires new actions.

Finally, self-evaluation takes place in the framework of the develop-ment of the school as a whole (Voogt, 1989: 25-26; Van Petegem, 1997: 126-127). Self-evaluation is not an objective of itself but a means (Ramsay & Clark, 1990; Holt, 1981) that contributes to aspects of the policy of the school so that the achievement of the students is influenced directly or indi-rectly by it.

This description constitutes the point of departure for the study of self-evaluation in the schools. The question, however, is whether all schools have already attained this level. The strict application of this definition could then lead to the observation that true self-evaluation is rare. There-fore, we opted to consult first with educational experts who have an over-view of school praxis. Before describing this, we will briefly present the method that is used in this study.

3 Method To respond to the first research question - how educational experts28

perceive self-evaluation - 6 experts were selected from the Inspectorate, the Services for Educational Development (Dienst voor Onderwijsontwik-keling: DVO), and the educational guidance services29 in the Catholic and the Community educational networks, and a staff member from an organi-sation that is intensely involved in the support of school.

The objective of the second research question was to examine the self-evaluation process in secondary schools. Eight schools were asked about their self-evaluation activities. In addition, a self-evaluation project was set up in the schools. The schools were asked to administer two questionnaires on the school policy and the school organisation to their staff30. The com-pleted questionnaires were retumed to the research team. The team proc-

28 No experts external to the education system were consulted. 29 "Pedagogische begeleidingsdiensten" (educational guidance services) are extemal

pedagogical support services to schools organised within the three main education networks in the Flemish Community.

30 Students were not questioned, because this could create more resistance in the schools to collaborate.

essed the data and compiled a report for each school with text and explana-tion of the results. The schools then had to interpret the results further. One basic condition for the selection of the eight secondary schools was that they were open to the idea of self-evaluation and that they were inter-ested in the topic. Pedagogical supervisors and members of the Inspectorate were asked to provide the names of the schools that met these conditions. Four Catholic schools, two Community schools, and two municipal schools participated in the study, i.e. a middle school (M8), a general secondary school(G4), three technical and vocational schools (V3, V6, V7), and three general, technical and vocational schools (GV1, GV2, GV5).3I

The initiation of a self-evaluation project by an external research team is, of course, an artificial process. In principle, the school itself must take the initiative to conduct self-evaluation and must also be able to determine itself what instrument it uses. These two conditions were not met in the study. At the time of the study, few schools in Flanders were intensively occupied with self-evaluation. Moreover, we wanted to guarantee a mini-mum degree of comparability of the self-evaluation process between the schools. Therefore, we decided to use external questionnaires whereby the entire functioning of the school was covered.32 Obviously, account had to be taken of this artificial situation when the re-sults were interpreted. In spite of the artificial character of the self-evaluation project, the researchers considered that the analysis of the self-evaluation process that resulted from it provided useful insights into the self-evaluation.

In order to restrict the artificial character of the self-evaluation as much as possible, the researchers remained apart from it as much as possible . They left the entire process as far as possible to the schools themselves.

In order to map the self-evaluation process, the schools were questioned at various times. First, the heads of the schools were interviewed as regards their previous experiences of the schools with self-evaluation and their view of self-evaluation. The purpose of the study of the self-evaluation project

31 Secondary schools in Flanders may offer education for the first two unstreamed years of secondary education (middle schools), and/or general education, technical, vocational or arts education in the 3rd —6th year. If a school is a middle school, we use the letter M and the identification number; if the school offers technical and vocational education we use the letter V and the identification number; and if the school offers general education we use the letter G. If a mixture is offered we use several letters.

32 The research also had the objective of comparing the results of the questionnaires in function of their content. This aspect is not considered further in the present paper. The questionnaires that were used were the following: Vokiso (De Cock et al. 1984), De lerende organisatie (Van Den Broeck, 1996), Meso (van Aanholt and Buys, 1990) and GIL (Guide to Institutional Learning) (Dalin, 1989)

44 45

was also explained. These open interviews proceeded on the basis of previ-ously established guidelines. These guidelines were drawn up on the basis of the literature and prior discussions with the experts. This was an unstruc-tured interview in which the guidelines served more as reminders for the re-searchers than for directing the conversation. We wanted to lead the re-spondents in their view of self-evaluation as little as possible in a specific direction.

In the second phase, the heads were asked to distribute two question-naires on the functioning of the school to all their staff. In this meeting, the second, additional information about the school was gathered. The results of these questionnaires were presented in a report to the heads and included an explanation of the structure and the major themes of the questionnaires, the numerical results of the inquiry, and a summary discussion of these re-sults. These reports were discussed with each of the heads individually.

Six months later, the schools were again contacted to determine what they had done with the results. For this purpose, the head and a teacher of each school were interviewed.

All interviews were taped and transcribed. The Folio Views program33 was used to analyse the interviews. The interviews were first coded. Miles and Huberman (1984: 56-60) stress the importance of providing a structural order to the codes. The codes have to be coherent and be part of a structure that is meaningful for the study and have to be derived from the research questions on which the research is based. The code structure that was used was derived from the initial guidelines that were used for the interviews.

Neither for the interviews nor for the coding was a prior conceptual ref-erential framework used. This choice was determined in part by the option of guiding the self-evaluation project as little as possible and in part to al-low the schools to remain as much as possible in their natural situation. The objective of the analysis was to seek important patterns in the data as much as possible on the basis of the interpretations of the respondents (Strauss & Corbin, 1994). In this way, we were able to obtain a smaller number of top-ics and to compare the schools in function of them.

4 Self-evaluation in the view of experts To obtain a picture of the way in which one perceives self-evaluation in

secondary education, a number of experts were asked about their practical experience and to compare it with what the self-evaluation of schools has to mean as far as they were concerned. Although these experts applied a con-

cept that fits closely the theoretical model given above, they all argued for partial self-evaluation steps also to be included in the analysis for they are the first steps toward a more mature self-evaluation policy.

4.1 What is self-evaluation of schools and what instruments are used?

On the basis of the perception of the experts on self-evaluation, three levels were distinguished. First of all, there is the systematic inventorying of the activities that take place in the schools (first level). Self-evaluation i s here primarily a description of the manner in which the school works in all of its aspects, for example, when it prepares an annual report. In fact, one cannot as yet speak here of a conscious process of self-evaluation because the school does not really judge its activities (cf. the definition of self-evaluation in Section 2). The second level concerns the evaluation because of specific problems in the school with the intention of finding a solution for them. The objective here is primarily to understand a particular problern in order to be able to approach it better and less to develop instruments or to collect data. Finally, schools can themselves evaluate the global school con-figuration or aspects of it without there being a question of a specific prob-lem (third level). This is what we call 'best practice'.

The second division concerns self-evaluation in the strict and in the broad sense. Self-evaluation in the strict sense is understood to be self-evaluation that is conducted completely by the school itself. Self-evaluation in the broad sense occurs when, to a certain extent, support is provided frorn outside the school (for example, by educational consultants). Thus, this cannot be considered "self'-evaluation in the strict sense. The experts were of the opinion that distinguishing between the two in practice is not mean-ingful. When self-evaluation in the broad sense is not included, a number of experts contended, there would then be little self-evaluation. In addition, by far not all schools are ready to apply self-evaluation in the strict sense. Ac-tually, according to the experts, it is not even so important to know whether support is involved or not. It is primarily the attitude and the readiness to question oneself that is important, irrespective of whether or not the evalua-tion is conducted completely by oneself. Still, schools have to have the op-portunity to conduct a self-evaluation from start to finish themselves. In-deed, there are schools that are capable of doing so.

At the time of the study, a considerable number of instruments were al-ready available to conduct the self-evaluation systematically oneself. The experts stressed, however, that instruments have to be adapted to the school.

33 A software programme for the analysis of qualitative data.

46 47

4.2 To what degree and why is self-evaluation done in Flanders?

In Flanders at the time of the study (1997-1998), a number of schools were intensively involved in self-evaluation. All of the respondents could name a number of schools that actively worked on self-evaluation on one or more of the three levels. What is noteworthy, however, is that it was a rather short list, particularly for self-evaluation on the second and third lev-els. From the various respondents, we always heard the same names. Self-evaluation appeared in these schools to be primarily a matter for the head. This is not to say that the teachers were not involved but that the impetus for working with self-evaluation always came from the head.

However, when schools developed instruments themselves, they mostly concerned small components of the life of the school. Further, according to the experts, they do this primarily with the intention of using the instru-ments only in the school itself and the instruments themselves are very practically oriented. Schools do not readily place their instruments at the disposal of other schools or explain them to other schools, even though this is requested. For the reasons for this, we refer to the interviews with the heads. There this comment was also made and the position justified.

The average school, however, was not constantly occupied with self-evaluation, and the use of self-evaluation instruments was in most schools quite limited. Where self-evaluation was conducted on this level, it was rather specific and often also quite amateurish. It often involved more the detection of, and the searching of solutions for, very specific problems than assessment of the school as an organisation in function of a broader vision (self-evaluation on the second level).

The impetus for working in function of self-evaluation, according to the experts, often comes from the examination by the Inspectorate, on the occa-sion of the development of a specific curriculum, or upon the introduction of Total Quality Management (TQM) programmes. In addition, a number of schools took the step of self-evaluation because of a specific problem they were being confronted with.

Starting from the concern for an impending examination by the Inspec-torate was, according to the experts, generally not a good stimulus. Schools should develop a kind of self-evaluation culture. Sometimes, the self-evaluation flowed from the results of the examination, here the work was, indeed, done on the basis of strong involvement but not in every area. Members of the Inspectorate had the impression that schools more readily devote attention to the pedagogical and didactic than to the administrative components of the school.

`TQM' would be an impetus more in technical schools. Other schools develop a kind of self-evaluation on the occasion of the formulation of a specific curriculum. Of course, this is only possible if the school sees the curriculum as something that must be realised and not purely as a document that has to be drawn up in compliance with the regulations.

In addition to these more systematic forms of self-evaluation, which were relatively scarce at the time of the study, there are many schools that set up limited self-evaluation because of a specific problem or because of a particular project. These are often one-time activities that could be an impe-tus to, but cannot yet be considered, self-evaluation on the third level.

4.3 What is the objective of good self-evaluation-instruments?

The most important message that emerged from the interviews with the experts is that self-evaluation has to become more an attitude than some-thing that one does at a particular moment. In everything the school does, a critical and self-evaluating attitude has to develop. Therefore, a self-evaluation instrument is only one small aspect of what self-evaluation should men for a school. An instrument must prompt a school to start re-flecting in order to see things from another perspective and to question the obvious. Another culture must develop from it. What is essential is that one does not limit oneself simply to the application of a self-evaluation instru-ment. A good self-evaluation instrument will lead the school to start think-ing about a number of aspects of what happens in the school.

Further, the experts stress that the instrument itself need not be a ready-made package but rather a kind of recipe book that one gives to the school and that still leaves the school a certain freedom and space for movement so that it can apply the instrument to its own situation. This can lead, of course, to the instrument becoming no longer scientifically justified. For some experts, this is a problem. For others it is a matter of lesser impor-tance. In their mind, a self-evaluation instrument is only one step in a proc-ess that can lead one to start thinking about the school and its functioning in various areas. They are of the opinion that a strictly scientific basis of an in-strument is not so important as long as it leads one to start thinking about the functioning of the school. The creation of a certain self-evaluation men-tality is more important than the making of a self-evaluation instrument.

Another criterion an instrument must satisfy is that it has to be practical and easy to apply in the school or it will miss its target. Ideally, this means a short instrument with an accompanying computer program for the proc-essing so that it requires only a minimum investment on the part of the school.

48 49

In addition, it is noted that it is important to develop an instrument that is not complete and that leads the schools in such a way that they will crea-tively fill it in function of their own individual context. This seems to con-tradict the previous comment. However, one must also take into account that each school is in a different phase as regards self-evaluation. A ready-made instrument may seem inappropriate from this point of view. For schools that are not so far along yet and perhaps are not ready for this crea-tive process, this can be an initial introduction or a first step in this process. Ideally, however, the school is here as autonomous as possible (in each phase of the self-evaluation process) and an instrument must be rather an impetus to thinking and continuous evolution.

Finally, the experts argue that the development of an instrument must start with the needs of a school. In order to be able to realise this, a total instrument is very important so that a school knows where it stands so that it can start from there in its search for where it wants to go.

5. The self-evaluation process in eight secondary schools 5.1 View of and experience with self-evaluation in the eight

schools In the second part of the study, the self-evaluation process was moni-

tored in eight secondary schools. Before the self-evaluation instrument was intro-duced, the heads of the schools were asked about their view of and their experience with self-evaluation. This section summarises the most im-portant elements derived from these interviews.

Experience with self-evaluation The interviews with the heads revealed the extent to which their schools

had actually engaged self-evaluation activities in the past. The following differences between the schools emerged:

• Some schools (schools GV1 and G4) considered the operation of the everyday policy organs also forms of self-evaluation. This intuitive form of self-evaluation, which is not based on systematic enquiry, does not correspond to our definition of self-evaluation. Although the other schools also work in this intuitive manner, they do not con-sider it explicitly as a form of self-evaluation.

• Two schools never had organised enquiries of specific target groups (teachers, parents, etc.) (Schools GV2 and G4). In the framework of their school development plan, a working group had made a strength/weakness analysis of the school. This report was given to all the teachers for comment. No responses occurred in School GV2,

while in School G4 the working group had organised a staff meeting to discuss the texts again.

• A few schools (Schools GV1 and GV5) had organised only re-stricted, very much pedagogy-oriented enquiries that were in direct line with the classroom activities of the teacher.

• Other schools had already set up both limited and global enquiries regarding the school organisation (Schools V3, V6, V7 and M8).

• Another important factor was the degree in which the schools set up their self-evaluation activities in function of a policy plan (school development plan). Half of the schools work with such a plan (Schools GV2, G4, GV5 and M8). Two of them (G4 and M8) started a policy plan in the past in the form of an annual report. The object of this annual report was initially to list the activities of a school year.

• Two schools (V6 and M8) explicitly indicated the inventorying of numerical data (absenteeism, intake, flow through, outflow of stu-dents, etc). In School V6, the data were not used as an instrument to adjust school policy. This is the case in School M8, where the evaluation data of the students per teacher were used in the annual performance discussions.

• Finally, in only one school was there a culture of systematically evaluating all the activities in the school and discusring these activi-ties with the various target groups (School M8). This was also the only school in which the priorities of the policy plan grew out of the self-evaluation activities. This middle school gives the best example of 'best practice in school self-evaluation'. This will become clearer below.

With account being taken of these observations, we could group the schools in categories in function of the following criteria:

• the organisation of enquiries in one or more school components; • exclusively pedagogical enquiries (teacher oriented) or also organ-

isational (school oriented); • working with an integrated plan in which the self-analyses proceed

in a co-ordinated manner.

50 51

Group 1: Little experience with enquiries

Schools GV1, GV2, G4 and GV5 had had almost no experience with self-evaluation. Some schools did work with a school development plan, but this plan was not used as a basis for systematic self-evaluation. The only form of self-evaluation that some used was associated strictly with the classroom activities of the teachers. One school had once set up an enquiry among the students. There was certainly no attitude of school involvement or of a school-oriented self-analysis.

Group 2: Much experience but no integrated approach

The second group of schools (V3, V6 and V7) did have considerable ex-perience with self-evaluation enquiries. These enquiries concemed both pedagogical and organisational matters. What was striking here was that these enquiries were isolated from each other. They were not set up in function of a coherent plan or a consistently developed policy. They were often executed when an occasion presented itself. Often, this was because the head had leamed of a new evaluation instrument or had compiled one himself. The self-evaluation proceeded on an ad hoc basis. The self-evaluation process was also to a large extent a matter of the school administration.

Group 3: Much experience and an integrated approach

Only in School M8 had self-evaluation become an integral part of the school culture. This school worked on the basis of an integrated policy plan and evaluated all of the activities in a more systematic way. Regu-larly, evaluation moments were scheduled for the various target groups by the co-ordinators responsible for them. These evaluations resulted in a number of priorities that were included in the policy plan for the next year. Nevertheless, the head of School M8 also stressed that it was not easy to involve teachers in the evaluation of the entire functioning of the school. Self-evaluation, he contended, always had to start from the im-mediate praxis of the teachers. The systematic working with this self-evaluation for several years as a process creates over the long term, ac-cording to the head, more school involvement on the part of the teachers. With it, they also begin to reflect on the functioning of the school and also want to have their say in it.

Still, almost all of the self-evaluation initiatives in School M8 have to be launched by the head, and in the follow-up the head continues to play a central role. Without follow-up from the head, the self-evaluation proc-ess would immediately disintegrate, as the head himself noted.

It is clear that the self-evaluation capacity in the eight schools differs considerably. While some schools had actually had virtually no experience with self-evaluation — and certainly the teachers were not yet familiar with an enquiry into aspects that were not immediately related to their classroont praxis — there is a school in which the work was being done systematically on the basis of a coherent policy plan and where self-evaluation was a basic element of the school culture. Still, in this school the head continued to play a key role in the initiation and follow-up of the self-evaluation.

Self-evaluation defined The discussions with the heads also covered their view of self-

evaluation. Some heads (Schools GV1, GV2, G4, GV5 and V6) stated that they did do self-evaluation but that they did not yet have a clearly defined view of it They referred to policy instruments such as the school develop-ment plan or the annual report, and they spoke about concrete self-evaluation activities in their school on specific occasions (for example, the tenth anniversary of a course of study, the division of tanks for relation al and sexual education, support of new teachers, enquiries among graduates). These heads had difficulty in situating these self-evaluation activities within their school policy. Actually, they had not yet thought about it.

Other heads (V3, V7 and M8) had a clear-cut view of self-evaluation. They considered self-evaluation to be the evaluation of the life of the school in all of its aspects. Self-evaluation was one of the priorities of the school policy, and they had formalised their view of it down on paper.

It is striking that the heads of Schools V3 and V7 did have clear-cut views on self-evaluation, while they still had not succeeded in putting this view into practice with an integrated approach. They initiated many self-evaluation activities, but these activities were all ad hoc and not part of a systematic policy. This was manifeste(' the most obviously in their use of various extemal instruments and in their failure to use them systematically. Things are different in School M8. The self-analysis was there developed primarily by the school itself and did not proceed on the basis of extemal self-evaluation instruments.

The heads of Schools GV2, G4 and GV5 were working on the basis of a school development plan but they translated it little or not at all in concrete self-evaluation enquiries. The school development plan was drawn up by a working group that plans the activities, occasionally on the basis of a strength/wealmess analysis. This analysis, however, was not based on a de-tailed enquiry or dialogue with some or all of the parties in the school.

53 52

Need for self-evaluation All of the heads were of the opinion that self-evaluation was necessary

for a school. It did emerge that not every instrument could be directly ap-plied in each school. Each school is different and, in this sense, it is impos-sible to devise a single instrument that can be taken over universally. More-over, it was noted that not each school was equally advanced in its self-evaluating capacity. Some schools had to limit themselves to smaller-scaled enquiries about classroom praxis because a global self-evaluation covering all of the life of the school (including school policy and school organisa-tion) would be too remote for many teachers. The teachers are too little in-volved in the general school events so the use of school-oriented self-evaluation instruments would not be meaningful.

The head of School M8 stressed that schools not only have to use self-evaluation instruments but also have to develop them themselves. Each in-strument starts from a specific framework of values while each school has its own values. In the ideal case, schools have to develop an instrument on the basis of their own framework of values. Primarily, schools have to be supported in the development of good instruments.

Skills one must master to engage in self-evaluation The heads stressed that self-evaluation can only succeed if the sphere in

the school is open and if it functions democratically. Further, it is important that one deals with self-evaluation professionally, not amateurishly. A col-lection of guidelines and particular points of attention in the self-evaluation process could make an important contribution here. It is also important for the objectives of the evaluation to be clearly established. Schools have to be able to handle the self-evaluation instruments creatively, for they also have to take the context of the school into account in the self-evaluation process.

In addition, some of the heads noted that global instruments that cover the entire operation of the school have to be offered from outside the school. The analysis must also be done externally, as the schools do not have the required expertise in house. The head of School M8 saw this mat-ter differently. A global school analysis can be developed gradually by the school itself. The self-evaluation is here not the same as using an external instrument. In the global self-analysis, ideas and specific techniques can be taken from external instruments but the analysis in particular must be de-veloped by the school itself.

Task of the authorities/external organisations in selfevalua-tion

Most of the heads consider that the authorities cannot impose interral self-evaluation on the schools. Schools have to decide themselves whether or not to engage in self-evaluation. Only when they are ready for it will self-evaluation lead to meaningful results. Only one head considered that the authorities may require schools to evaluate themselves. A condition for this is that the authorities provide training and support in order to aid the schools in this matter.

A number of heads considered that the authorities should take up this support task without any form of compulsion. At the time, the external sup-port was, in the rnind of some, deficient.

Some heads in the first group thought that the authorities should de-velop self-evaluation instruments and make them available to the schools. Others did not agree. This is primarily the case with the head of School M8 . If the authorities themselves provided the instruments, there would be a danger that the schools would simply take them over while it is essential that a school conduct the evaluation itself, perhaps by trial and error. The learning process is of itself a very important aspect of the self-evaluation according to the head of School M8.

Use of existing self-evaluation instruments As noted above, two schools (GV2 and G4) had never actually set up an

enquiry among their teachers, students, or parents. Two other schools (GV1 and GV5) rarely made use of external self-evaluation instruments. They prepared their instruments themselves. The heads of Schools V3 and V7 of-ten used external self-evaluation instruments. The head of School V3 used these instruments to develop new instruments himself. In School V6, in ad-dition to existing instruments, enquiries were set up that the school devel-oped itself. In School M8, examples from existing questionnaires were used to fill out the self-evaluation instrument used in the school. The head, how-ever, was strongly against using external questionnaires. These question-naires, the head contended, were usually based at least in part on values other than the values of the particular school. Only in School V7 was use primarily made of external, published self-evaluation instruments. The head of this school considered that the school did not have sufficient expertise to develop instruments. Also, there was no time to do so.

From the various conceptions of the heads, it is clear that self-evaluation is sometimes interpreted very differently. Thus, the opinion of the head of School V7 is almost diametrically opposed to the opinion of the head of School M8. Self-evaluation in School V7 was, first of all, the or-

55 54

ganisation of enquiries from which the head then drew conclusions for the school. In School M8, self-evaluation was oriented to the development of a critical attitude among the teachers, which cannot be realised by too much dependence on external instruments.

5.2 Distribution of the instruments In each school, two self-evaluation instruments were administered on

the school policy and the school organisation. The objective of the instru-ments was to sketch a general picture of the functioning of the school. The instruments were questionnaires that were presented to the staff.

The questionnaires were distributed in the school with an accompanying letter in which the objective of the study and the importance of the enquiry for the school were explained. The distribution of the two questionnaires within the individual school was left to the head. Entrusting the distribution to the head was opted for intentionally because this would most approach the natural situation. In a self-evaluation, the school also decides itself how it will proceed.

With this method, the distribution in the schools varied considerably so the response to the questionnaires also diverged considerably, as shown in Table 1. In some schools, the questionnaire was simply handed out to the staff with no explanation from the head (Schools GV1, GV2 and GV5). In three schools (G4, V6 and V7), the staff was asked at a full staff meeting to fill in the questionnaire. By way of introduction, the objective and the pur-pose of the enquiry was explained by the head. In Schools V3 and M8, the enquiry was also explained by the head to all of the staff. Then, the ques-tionnaires were distributed to the respondents with the request to return them within two weeks to the secretariat. In School M8, the head explicitly stressed that the questionnaires were external questionnaires that did not necessarily reflect the values of the school. The staff were urged to consider the enquiry as an experiment in which the school participated and not as a new part of the school policy.

Table 1. Response to the self-evaluation instruments per school School GV1 29.0% School GV2 29.9% School V3

61.4% School G4

93.9% School GV5

29.2% School V6

84.2% School V7

95.0% School M8

67.4%

The care with which the questionnaires were distributed by the heads al-ready gives an initial indication of the seriousness with which the school participated in the project. In School GV I, the head mentally dropped out in the course of the project. Although he initially had agreed enthusiastically to cooperate in the project and had been cooperative in the first interview, this positive attitude reversed as the project proceeded. His priorities were clearly situated elsewhere. He could or would only devote a minimum amount of time to the project.

In the two other schools with a low response, the communication about the enquiry of the school leadership to the staff was very summary. The schools with an explicit view of self-evaluation (Schools V3, V7 and V8) communicated clearly with the staff in advance. In Schools G4 and V6, the heads also explained the objective and purpose of the enquiry to all those involved.

5.3 Results of the self-evaluation The results of the enquiry were compiled in an easy-to-read report that

gave the most important results. The interpretation of the results (what the data meant for the school) and the suggestion of possible actions (what could be improved in the school on the basis of these conclusions) were left to the schools themselves. Self-evaluation implies that the extemal author-ity does not play a determinative role in the setting up of projects. The re-port was explained in the schools personally by the researchers.

The follow-up discussions with the head and the teacher who was part of a school policy organ (year coordinator, member of the pedagogical committee, etc.) took place six months after this explanation. In these dis-cussions, it became clear what the schools had done with the results of the enquiry.

No consultation In the first group, Schools GV1 and V6, no use was made of the results.

They were not discussed in any of the policy organs or work groups of the schools nor were they communicated to the teachers. In School GV1, the teacher who was interviewed was not even aware of the results. In School V6, the teacher had received the final report as preparation for the inter-view. The other members of the coodinator's group had not yet received them. While there was a clear lack of interest on the part of the head in the study in School GV1, in School V6, the lack of time was cited as the most important reason. The head of this school was also of the opinion that the teachers in general were not interested in it.

56 57

No communication to all parties In the second group, Schools G4 and GV5, the results were discussed in

a policy organ or a work group of the school. The results were not commu-nicated to the other teachers.

School G4, in the course of the project, had committed itself to another European self-evaluation project. The head intended to compare the results of this European project and the results of our enquiry with each other when the European project had been completed. For the discussion of the results of the two enquires, a study day would be organised for the teachers in the next academic year. Ultimately, the results of the enquiry were not dis-cussed.

In School GV5, the most important results of the school report were ex-plained by the head to the members of the school development plan work group. This was purely informative, for this work group did not really dis-cuss them. The reason the head cited for this was the low response (29.2%) so it was impossible to draw meaningful conclusions from them for the en-tire school population. We note here that the head himself made no effort to obtain a more substantial response to the enquiry.

Consultation and communication, no further action In the third group, Schools GV2, V3, V7 and M8, the results were dis-

cussed in a policy organ or in a work group and then explained at a general staff meeting. In Schools V3 and V7, no further action was taken. In School V3, the head said that this was due to the lack of interest on the part of the teachers. In School V7, it was due to lack of time.

In two schools, the respondents contended that actions had been launched as a result of the self-analysis. In School GV2, the head explained a summary of the results to a policy organ. In addition, the strong and weak points that emerged from the report were presented at a general staff meet-ing, and attention was given primarily to the weak points. An element of this was that the communication with the staff was sometimes deficient. This was also the case for the distribution of the questionnaires of the self-evaluation. In the future, such enquires would be filled in at a time when everyone was present at school. In addition, the respondents stated that the school had launched new initiatives as regards student support (for exam-ple, a work group on drug prevention). However, student support had not emerged as a weak point in the self-evaluation report. The enquiry did show that the respondents considered it an important priority in the school policy.

Action after the self-analysis In School M8, the report of the enquiry was discussed in the most im-

portant policy organ of the school. Attention was here given primarily to re-sults that were surprising for the members of the policy organ. The director set out to determine possible explanations by indirectly sounding out a number of teachers about whether the results were genuine and what the possible causes for them could be. One of the most important results of the enquiry was that the teachers thought that they were not sufficiently in-formed about decisions relating to school policy. This was surprising be-cause there were many information channels in the school. The further dis-cussions between the head and the teachers revealed that the teachers did not think so much that they were poorly informed but that they as a group had the feeling that they were insufficiently involved in the decisions. This subject was raised by the head at a staff meeting. It was then proposed that a number of discussion groups be set up in which all the participants in the school would be represented. By means of directing questions, the aspects of the enquiry that could be improved in the school were discussed in these groups. The results of these discussion groups were later taken up in the policy organ and converted to action. The head participated in these groups only as an observer.

6 Conclusions and discussion Different levels

The research shows that self-evaluation is a concept that is often inter-preted differently. Both the experts and some of the eight heads considered certain activities of the school to be a form of self-evaluation that does not fit the definition we used. The working of the policy organs in the school, the inventorying of the activities in a school year, or the compilation of stu-dent data are, of themselves, not self-evaluation. Self-evaluation includes a systematic (not intuitive) questioning and judgement of the activities of the school (not a compilation of data or inventorying of activities).

The three levels of self-evaluation the experts distinguished in the first part of the study we also found in the eight secondary schools. The first level, at which the schools simply compile data or inventory school activi-ties in an annual report, was cited by four schools as a step they had taken in the past toward self-evaluation (school GV2, G4, V6 and M8). In Schools GV2 and G4, this step provided the impetus for developing a school development plan on the basis of strength/weakness analyses in a work group. In School M8, the annual report was the first phase leading to a permanent, policy-oriented self-evaluation system of the school.

58 59

The second level, at which schools evaluate specific problem compo-nents, also was present in two schools (GV1 and GV5). These specific problem components were closely related to the classroom activities of teachers.

The third level, at which schools evaluate their global school policy, was observed in the vocational schools V3, V6, and V7 and the middle school M8.

Our study shows that another factor constitutes an important difference between the schools: the use of an integrated policy plan in order to situate the self-evaluation. In this respect, School M8 differs from the three other schools that have already evaluated their global school policy. In School M8, this global evaluation takes place each year. The self-evaluation en-quiries fit in with this policy. In the other schools, the enquiries are a more or less loose sequence of various actions. Therefore, a fourth level of self-evaluation can be distinguished at which the self-analyses proceed in the context of a policy plan. Here the global self-analysis of the school is a permanent process. In this case, we can speak of 'best practice'. On the third level, it is then a matter of a one-time process that is not part of a regu-larly recurring process.

The distinction between the four levels does not mean that schools can be situated on only one of them. Thus School V6 uses global school analy-ses without any problems (Level 3) while it also compiles data on students systematically (Level 1). Schools GV2 and G4 use a global policy plan (Level 4), but do not actually get around to conducting enquiries of the school population (Levels 2 and 3).

We would contend that only chose schools that reach the four levels do systematic self-evaluation.

External instruments versus self-constructed instruments The study shows two other important differences in the approach to

self-evaluation. The first approach is concerned with the use of extemal self-evaluation instruments or the development of the school' s own self-evaluation culture. School M8, which is the furthest along of all the schools in self-evaluation, works very much on the basis of development in the school itself. External instruments are looked on with mistrust. They can only be used insofar as they fit in with the existing school culture. The head of School V7 considers that his school has to use extemal instruments as much as possible for the school, in his opinion, does not possess the neces-sary expertise. The result of this position is that the self-evaluations are, in-deed, independent of each other. The question arises as to whether many of

the staff apart from the head are much involved in the self-evaluation proc-ess.

In School M8, the self-evaluation process was much more a part of the school culture. It is clear that the use of external self-evaluation instruments is no guarantee for the integration of the self-evaluation process in the over-all operation of the school. Still, this does not mean that the use of extemal self-evaluation instruments cannot be valuable for schools. This is also demonstrated in our study: School M8 used the external self-evaluation in-struments to adjust its own school operation.

The importance of pedagogical praxis The second approach to self-evaluation concerns the importance of

pedagogical praxis and of the global school policy. Some experts and some heads were of the opinion that self-evaluation primarily had to start from the praxis of the teachers. Self-evaluations of the organisational aspects of the school were, in their opinion, too far removed from the teachers. They did not consider themselves involved in it. Consequently, such self-evaluation makes little sense. Other heads stressed that self-evaluation must fit in with a global self-evaluation of the school. Specific elements that are related to, for example, the pedagogical praxis of the teachers, have to be able to be situated within it. It is precisely this approach, which is too strongly oriented to pedagogical praxis, that these heads rejected.

There may well be something to be said for both approaches. Self-evaluations that are not related to the praxis of the teachers will never mo ti-vate them strongly. Nevertheless, this motivation is essential for the success of the self-evaluation process. All of the panties involved in the study agreed that self-evaluation is related in the first place to an attitude of all of the members of the school, not only to that of the head. Schools that have no experience with self-evaluation would probably best start with the self-evaluation of educational activities that are of great importance for the teachers. This can be an important initial step for the evolution of the entire school to a system in which all of the school operations (also the school policy and school organisation) are evaluated. It is important that this ulti-mate goal be kept in mind. Finally, the school must strive for global self-evaluation that goes further than the individual functioning of the teachers. Without this goal, self-evaluation will never be part of the entire school cul-ture. It is clear that the degree in which the school leadership succeeds in involving the teachers in school policy will have an effect on this evolution.

60 61

Is school self-evaluation in vocational schools different from other schools?

With our data, it is not easy to answer this question indisputably. Of the eight schools of our sample, only three offered exclusively vocational edu-cation, and four others offered both general and vocational education. Al-though the data show differences between these categories, one cannot con-clude that these differences are the result of the typical characteristics of vocational schools since we find similar characteristics also in the schools with general and vocational tracks.

Nevertheless, some differences with other schools might be stressed. First, the three vocational schools had some experience in school self-evaluation and had conducted school surveys about both educational prac-tice and school management. The introduction of programmes of TQM in the school was often the reason for these surveys. Second, a large propor-tion of teachers of the vocational schools answered our questionnaires, which was very likely influenced by the better communication of the heads in comparison to which was the case in other schools. Third, in two of the vocational schools the head organised a discussion of the results of the self-evaluation instruments, but this was also the case in schools GV2 and M8. No action was taken in the vocational schools later on, however.

Our data do not show a clear difference between vocational schools and other schools as far as school self-evaluation is concerned. We have more reason to stress that best practice of school self-evaluation is related to the school culture, which certainly is influenced by the kind of education of-fered in a school but also by other factors.

Self-evaluation as the school culture The results of the self-evaluation project initiated in the study are quite

sobering. Of the eight schools, only four discussed the results in a work group and reported on this to all the staff. Only in two schools (GV2 and M8) was action actually taken that flowed from the self-evaluation. For School GV2, moreover, there is the question of whether the action really did flow from the self-evaluation. In any event, the actions taken did not concern a weakness that had been revealed by self-analysis. This was the case with M8, which at several points gave examples of 'best practice'.

One could say that the self-evaluation action was artificial. The schools had not spontaneously proceeded to self-evaluation. Moreover, they were assigned external self-evaluation instruments and could not themselves make a selection. However, it must be stressed that the eight schools in the study were included because they were open to the idea of self-evaluation. They agreed immediately at the first contact to participate in the study.

They knew clearly from the outset that the study would also check for pos-sible action. The only school that actually set up action as a result of the self-evaluation was M8, whose head was the most mistrustful of external self-evaluation instruments but whose school culture was appropriate. It was the only school about which we can state on the basis of the study that it had a high capacity for self-evaluation. It was the only school that achieved the four levels of self-evaluation (descriptive, problem solving, policy oriented, and permanent) and thus arrived at 'best practice'.

This study underlines the difficult route schools have to travel to arrive at a full-fledged, integrated self-evaluation policy. Schools that are not ready for this will perhaps make use of external instruments on the initiative of the head but the chance is great that this aspect will have little impact on school events. There is even the chance that such an enquiry will increase the distance between the head and the teachers and aggravate the cynicisin of the teachers. This has important implications for the development of self-evaluation procesces in schools. Although the development of good, vali-dated self-evaluation instruments can make an important contribution to the internal quality assurance in schools, it remains essenbal to get the schools to approach the self-evaluation process systematically. This means, first, taking initiatives in which the teachers and the other staff feel involved, and, second, developing a systematic approach to self-evaluation that inte-grate the various enquiries in a global policy plan. The best self-evaluation instrument will lead to nothing if the school does not engage in this process.

63 62

References

BOONEN, M. (1990) 'Zelfanalyse door scholen', Schoolleiding en -begeleiding, 1: 1-22. Zaventem: Kluwer.

BUNT, A. & TH. VAN HEES (1991) `Zelfdiagnose door scholen', Meso Magazine, 54: 13-17.

CREMERS-VAN WEES, J.M.C.M., I.J. REKVELD, H.P. BRANDSMA & R.J. BOSKER (1996) Instrumenten voor zelfevaluatie. Beschrijving van 31 instrumenten. Enschede: Universiteit Twente, Onderzoek Cen-trum Toegepaste Onderwijskunde.

DALIN, P. (1989) Organisatieontwikkeling in school en onderwijs. Alphen aan den Rijn: Uitgeverij Samson.

DECKERS, J. & J. JACOBS (1994) `Zelfevaluatie en visitatie. Instrumenten voor kwaliteitszorg in de autonome school', Mesofocus 17, Houten: Educatieve Partners Nederland.

DE COCK, G., R. BOUWEN, K. DE WITTE & J. DE VISCH (1984) Organisa-tieklimaat en —cultuur. Theorie en praktische toepassing van de or-ganisatieklimaat-index voor profit-oganisaties (OKIPO) en de ver-korte vorm (VOKIPO). Leuven: Acco.

FULLAN, M. (1982) The meaning of educational change. Columbia Univer-sity, New York: Teachers College Press.

HOLT, M. (1981) Evaluating the evaluators. London: Hodder & Stoughton. HOUSE, E.R. (1973) School evaluation, the politics and process. Berkeley:

McCutchan. MARX, E.C.H. (1975) De organisatie van scholengemeenschappen in on-

derwijskundige optiek. Groningen: Tjeenk Willink. MILES, M.B. (1964) Innovation in Education. New York. MILES, M.B.. & A.M. HUBERMAN (1984) Qualitative data analysis. Beverly

Hills, CA: Sage. NEVO (1995) School-based evaluation. A dialogue for school improvement.

Oxford: Elsevier Science. POUWELS, J. & R. JUNGBLUTH (RED.) (1991) Scholen toetsen op kwaliteit.

Studie ten behoeve van het WRR-advies inzake onderwijsverzorging. Nijmegen: Instituut voor Toegepaste Sociale Wetenschappen.

RAMSAY, W. & E.E. CLARK (1990) New ideas for effective school im-provement. Vision, Social Capital, Evaluation. London, New York & Philadelphia: The Falmer Press.

SIMONS, H. (1985) 'Against the rules: procedural problems in school self-evaluation', Curriculum perspectives, 5(2). 1-6.

STRAUSS, A. & J. CoRBIN (1994) Grounded theory methodology: An over-view. In N. K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 273-285). Thousand Oaks, CA: Sage.

VAN AANHOLT, TH. & TH. BUIS (1990) `De school onder de loep. Richtli-jnen en hulpmiddelen voor zelfevaluatie', Mesofocus 3, Culemborg: Educaboek.

VAN DEN BERGH, R. (1990) Bouwstenen voor evaluatie. Voor de praktijk van onderwijs en begeleiding. Houten/Antwerpen: Bohn Stafleu van Loghum.

VAN DEN BROECK, H. (1996) Lerend management. Verborgen krachten van managers en organisaties. Tielt: Lannoo.

VAN PETEGEM, P. (1997) Scholen op zoek naar hun kwaliteit. Effectieve scholenonderzoek als inspiratiebron voor de zelfevaluatie van scholen. Proefschrift voorgelegd tot het behalen van de graad van doctor in de pedagogische wetenschappen, Universiteit Gent.

VOOGT, J. (1989) Scholen doorgelicht. Een studie over schooldiagnose. De Lier: Academisch Boeken Centrum.

65 64

PETER LANG Europaischer Verlag der Wissenschaften

Band 8

PETER LANG Frankfurt am Main • Berlin • Bern • Bruxelles • New York • Oxford • Wien

Konzepte des Lehrens und Lernens

Klaus Breuer/Klaus Beck (eds.)

Herausgegeben von K. Breuer und G. Tulodziecki

Are European Vocational Systems

up to the Job? Evaluation in European

Vocational Systems

Die Deutsche Bibliothek - CIP-Einheitsaufnalune

Are European vocational systems up to the job? : evaluation in European vocational systems / Klaus Breuer / Klaus Beck (eds.). - Frankfurt am Main ; Berlin ; Bern ; Bruxelles ; New York ; Oxford ; Wien : Lang, 2002

(Konzepte des Lehrens und Lernens ; Vol. 8) ISBN 3-631-39550-7

Preface from the editors

The compilation of this volume is based in the COST All action program initiated by the European Commission. The program represents a pan-European collaboration focused upon research in vocational education and training. Working Group 5 has emphasized Evaluation and Assessment in the field of vocational education and training (VET). The papers published here address issues of evaluation. This is applied to the macro, the meso and the micro level. A common perspective is the attempt to give examples of best practice in evaluation of vocational education and training.

The papers are a selection from the contributions to presentations and dis-cussions within working group 5. Another volume on assessment is under preparation. All papers depict aspects of the collaboration within the initia-tive. In this respect we want to acknowledge the active participation of friends and colleagues from the European community in VET within work-ing group 5 without adding a full list of the names.

Mainz, May 2002 K. Breuer

K. Beck

ISSN 0724-6455 ISBN 3-631-39550-7

US-ISBN 0-8204-5984-4

C Peter Lang GmbH Europischer Verlag der Wissenschaften

Frankfurt am Main 2002 All rights reserved.

All parts of this publication are protected by copyright. Any utilisation outside the strict limits of the copyright law, without

the permission of the publisher, is forbidden and liable to prosecution. This applies in particular to reproductions, translations, microfilming, and storage and processing in

electronic retrieval systems.

Printed in Germany 12 3 4 6 7

www.peterlang.de

5

Contents

Klaus Breuer / Klaus Beck: Preface: Cultures of Vocational Education and Training 9

Jan Johansson / Torsten Bjorkman / Marita Olsson / Mats Lindeli: Qualified Vocational Education in Sweden - a New Form of Post-secondary Education 21

Geert Devos / Jef C. Verhoeven: In Search of `Good practice' in School Self-evaluation: An Explora- tory Study in Secondary Schools 41

Anu Raisanen: Education and Training of Vocational Sector - Object of External Evalution in Finland 67

Stan McHugh / Marie Ena Walsh Assessment in Vocational Education and Training in Ireland 91

Orsola Fornara, Stefano Volpi: The Integration of Educational, Vocational and Labour Market Systems in Italy: an Evaluation 109

Peter den Boer, Elly de Bruijn, and Truus Harms: Integrating a flexible, transfer oriented experiment into regular Dutch VET, has it worked? 133

Klaus Breuer: Evaluation of Authentic Assessment in Vocational Examinations: Approaches within the German Dual System 153

Contributors 167

7