5

Human–Machine Interaction Issues in QualityControl Based on On-Line Image ClassificationJ. E. Smith1, M. A. Tahir1, P. Caleb-Solly1, E. Lughofer2*, C. Eitzinger3, D. Sannen4 and M. Nuttin4

Abstract—This paper considers on a number of issues thatarise when a trainable machine vision system learns directlyfrom humans. We contrast this to the “normal” situation whereMachine Learning techniques are applied to a “cleaned” data setwhich is considered perfectly labelled with complete accuracy.This study is done within the context of a generic system forthe visual surface inspection of manufactured parts, however,the issues treated are relevant not only to wider computervision applications such as medical image screening, but alsoto classification more generally. Many of the issues we considerarise from the nature of humans themselves: they will be notonly internally inconsistent, but will often not be completelyconfident about their decisions, especially if they are makingdecisions rapidly. People will also often differ systematically fromeach other in the decisions they make. Other issues may arisefrom the nature of the process, which may require the machinelearning to have the capacity for real-time, on-line adaptationin response to users’ input. Because of this, it may be that theusers cannot always provide input to a consistent level of detail.We describe how all of these issues may be tackled within acoherent methodology. Using a range of classifiers trained ondata sets from a CD imprint production process, we presentresults which demonstrate that training methods designed totake proper consideration of these issues may actually lead toimproved performance.

Index Terms—Image classification, Human-Machine Interac-tion, on-line adaptation, resolving contradictory inputs, variableinput levels, partial confidence, insight into classifier structures

I. INTRODUCTION

In many machine vision applications, such as inspectiontasks for quality control, an automatic system tries to repro-duce human cognitive abilities. Even if the human expert istold to apply a set of well-defined rules, a lot of subjective, pastexperience will be incorporated in the decision. Any machinevision system that does not consider such experience will failto achieve a high classification accuracy. The most efficientand flexible way to transferring this experience into thesoftware of a machine vision system is to learn the task froma human expert [6]. Traditionally this is done either by theexpert providing pre-classified images for supervised learning,

J. E. Smith1, M. A. Tahir1, P. Caleb-Solly1 are with the Bristol In-stitute of Technology, University of the West of England, Bristol, UK,email:{muhammad.tahir,james.smith,praminda}@uwe.ac.uk

Corresponding author E. Lughofer2* is with the Department of Knowledge-based Mathematical Systems, Johannes Kepler University of Linz, Austria,email: edwin.lughofer@jku.at

C. Eitzinger3 is with the Profactor GmbH, A-4407 Steyr-Gleink, Austria,email: christian.eitzinger@profactor.at

D. Sannen4 and M. Nuttin4 are with Katholieke Universiteit Leuven, De-partment of Mechanical Engineering, Division PMA (Production engineering,Machine design and Automation), Celestijnenlaan 300B, B-3001 Heverlee(Leuven), Belgium, email: {davy.sannen,marnix.nuttin}@mech.kuleuven.be

or by knowledge acquisition from the human operators inform of rule bases. Typically Machine Learning (ML) systemsare trained in supervised batch mode from a set of exampledata items, each of which has a unique label. Although theremay be inconsistencies or noise within the data, these aregenerally considered to be random in nature, and each pointis considered to be labelled with complete accuracy.

However, as ML technology moves from research laborato-ries to practical applications such as Machine Vision, a rangeof issues arise concerning how humans relate to, and interactwith such systems [16] [8]. Not only does this question thefeasibility, or even relevance of considering “cleaned” datasets, there is an increasing demand for systems to operate insituations where off-line batch-mode processing is not appro-priate [13]. This can occur if data are hard, time-consuming orcostly to obtain, or if the underlying processes change fairlyrapidly, requiring reconfiguration. Both of these cases leadto the need for an element of incremental on-line training[18] [41]. Nevertheless, this prompts a renewed interest in thenature of the human interaction with adaptive ML systems [3][1].

In this paper, we focus on a number of issues relating toHuman-Machine Interaction (HMI) in the context of a genericsystem for the visual surface inspection of manufactured parts.Section II describes the basic architecture of the proposedsystem, and Figure 1 illustrates the impact of the issues therein.Following a description of the data sets (Section III), the HMIissues are considered as follows:

• Section IV deals with the issues arising when the natureof the application demands real-time on-line learningafter an initial batch-mode phase (HMI 1). We showhow classifiers which can be trained incrementally canoutperform static ones, which cannot be further refinedin response to incoming data.

• Section V considers the fact that different users will oftendiffer systematically from each other (HMI 2), and howbest to incorporate this diversity of information.

• Section VI considers how demand for rapid responsesmay reduce the level of detail in the feedback users canproduce (HMI 3), and suggests ways for dealing withthis.

• Section VII considers the fact that for a number ofreasons, the operator(s) may not be completely confidentin their decisions (HMI 4), and shows how an enhancedHMI interface for online labelling can be exploited tocapture this information and lead to performance im-provements.

• Section VIII issues related to the interpretability of the

6

classifiers are considered (HMI 5).

II. GENERIC ARCHITECTURE FOR IMAGE CLASSIFICATION

The whole framework is shown in Figure 1. Starting fromthe original image a “contrast image” is calculated. A pre-defined master image is used as ideal and fault-free refer-ence situation. Each red/green/blue pixel value of a newlyrecorded image is compared with the corresponding (samex-y coordinate position) red/green/blue pixel value of themaster image plus-minus a threshold, which may vary fordifferent regions in the image (e.g. in homogeneous regionstypically lower than in edgy regions). These thresholds arepre-estimated based on historic data and long-term experience.The absolute values of deviation in red, green and blue areaveraged to an overall deviation in gray value (ranging from 0to 255). In this sense, the gray value of each pixel representsthe degree of deviation from the normal appearance of thesurface. For all further discussions we disregard these low-level processing steps and assume that an appropriate andstable system of image acquisition, preprocessing and imagesegmentation is in place. In particular we assume that enoughrelevant information is captured to permit correct decisions tobe made purely based on each image.

From the contrast images Regions Of Interest (ROIs) areextracted, each of which contains a single object which mayor may not be a fault. Various ROI extraction methods forgrouping local and similar data clouds were applied andtheir accuracy tested, ranging from connected components,morphological approaches to clustering approaches for group-ing such as iterative k-means, hierarchical clustering [17],reduced Delaunay graph [32], DBSCAN algorithm [11] etc.For each ROI a total of 57 object features are calculatedsuch as area, brightness, homogeneity or roundness of objects,characterizing their shape, size etc. These are complementedby aggregate features which characterize images as a whole,such as the number of the objects, the maximal density ofobjects or the average brightness in an image. The full list ofaggregated features is shown in Table I. This list was derivedon the basis of discussions with experts on surface inspectionand wider machine vision applications. Some features appearas quite intuitive to have a high discriminative power, espe-cially the number and total area of all ROIs (typically, themore deviating pixels the contrast image show, the higher thelikelihood that the corresponding production item is bad) orthe maximal/average local density of ROIs (dense regions aremore likely to belong to bad parts). Depending on the natureof the operator’s feedback, the data describing individual ROImay be added to the aggregate image data in various ways.

The feature vectors are then processed by a trained classifiersystem that generates a final good/bad decision for the wholeimage. We have implemented and evaluated many differentclassifiers within this framework, and the “best” classifier isof course problem-dependent [48]. Here we focus on reportingthe results of three classifiers which performed well on thesedatasets. For off-line training these were two decision tree-based classifiers CART [2] and C4.5 [34], along with k-NearestNeighbors (kNN) [15]. These methods are widely known and

used, and were recently named among the top 10 data miningalgorithms [50]. We also applied two incremental learningalgorithms eVQ-Class [24] and FLEXFIS-Class [26]. Becausethe framework is generic, it produces many features describingeach ROI, not all of which may be relevant to any giventask. To reduce the “curse of dimensionality” effect [15] wetherefore applied Tabu Search to perform feature selection asdescribed in e.g. [27] [44].

The classifiers are trained using operator input. For a limitedperiod of time the operators judge the parts in parallel to themachine vision system. They usually inspect the real part (notthe image) and make a decision, which is then fed back into theclassifier. In the simplest case the operator just pushes a green(“good”) or red (“bad”) button. If the speed of the productionprocess allows, they may also provide more detailed traininginput (this will be treated in HMI issues #3 and #4, i.e. inSections VI and VII). For many reasons it may be desirableto aggregate input from different operators who are separatedin time (shift patterns...) or space (multiple lines or sites). Tocope with this, reflect different opinions and maintain userengagement, a separate classifier is trained using the inputfrom each operator. When these trained classifiers are appliedto new images, any contradicting decisions are resolved usingan ensemble (more precisely: classifier fusion) method [20],to provide the final decision of the system (see Section V,HMI issue #2). Additionally, there might be a quality controlsupervisor, which does not do the daily quality inspection him-/herself, but who wants to be in control as much as possible.If the supervisor also labels (part of) the data, the systemwill take into account this information as well by using itfor training the ensemble method in such a way that its outputmodels the supervisor’s decisions.

In general, classification performance will be measured interms of 10-fold CV error, except for the incremental on-linetraining issue, where the accuracy on a separate test set iscalculated (as CV is an off-line procedure). Sometimes, thiswhole training process may fail to produce satisfactory results,most often because there is important information missing inthe image. This may be due to inadequate lightning or failureof the image segmentation. To detect such situations, we havedeveloped an early warning system, that predicts the successor failure of the training process [42].

III. CHARACTERISTICS OF THE CD IMPRINT DATA SET

The whole recorded data set contains 1687 images, fromwhich 153 contrast images are fully black, hence containingno potential fault candidates on it and can be classified directlyas good. The images were labelled by four different operators,which assigned the labels not only to the whole images, butalso to each single ROI. A Graphical User Interface wasdesigned (see Figure 2) in consultation with the users to allowthem to rapidly assign both a class (0-12) (6 belonging topseudo-error = no real errors and 6 belonging to real errors)and a severity to each object segmented in the image. Inthese images, the class distributions were not well-balanced;in this sense, it was not possible to train a high-performancestand alone object classifier; however, when using their output

7

Fig. 1. Classification framework for classifying images into good and bad, the five major HMI issues marked with red ellipsoids

TABLE ILIST OF “AGGREGATED” IMAGE-LEVEL FEATURES USED WITHIN GENERIC FRAMEWORK.

No. Description No. Description1 Number of ROIs 10 Max. grey value in the image2 Average minimal distance between two ROIs 11 Average grey value of the image3 Minimal distance between any two ROIs 12 Total area of all ROIs4 Size of largest ROI 13 Sum of grey values of all faults5 Center Position of largest ROI, x-coord 14 Maximal local density of the ROIs6 Center Position of largest ROI, y-coord 15 Average local density of the ROIs7 Max. intensity of all ROIs 16 Average area of the ROIs8 Center Position of ROI with max. intensity, x-coord 17 Variance of the area of ROIs9 Center Position of ROI with max. intensity, y-coord

information as additional features (under the scope of HMI 3,according to Figure 1), we could improve the classification aswill be demonstrated in Section VI.

Examples of faults on the Compact Discs caused by theimprint system are a color drift during offset print, a pinholecaused by a dirty sieve (color cannot go through), occurrenceof colors on dirt, palettes running out of ink. These haveto be distinguished between so-called pseudo-faults, like forinstance shifts of CDs in the tray (disc not centered correctly),causing longer arc-type objects (e.g. see the upper left ROIin Figure 3) or masking problems at the edges of the imageor illumination problems (causing reflections). This leads toquite complex structures in the deviation images, ranging fromblob-type pixel clouds over arc-type objects (usually pseudo-errors) to large wide-spread pixel areas (usually stemmingfrom dirt, pallettes running out of ink). For an image ex-ample see Figure 3: different types of script nameplates (atdifferent positions) form separate regions of interest, alsothe different arc-type objects (two in the middle, one longone outside partially circumventing all other objects), (bothmarking pseudo-errors as due to shift of the CD in thetray) and the small but clearly visible blob in the middleof the image (a real error), all representing different ROIs;the different grey levels resp. colors represent a groupingachieved by the DBSCAN algorithm [11]. In Figure 4 anotherexample of a deviation image is presented from the CD imprintproduction process: different regions of interest found by theROI recognition method are highlighted in different colors(resp. grey levels); in fact, the longer arc-type object (in lightgrey) denotes a completely other type of error than the tworectangle and compact type of regions (in darker grey levels).

Fig. 2. GUI of labelling tool, here 6 regions of interest are found which canbe all labelled by a drop-down control element (lower left), image labels canbe assigned together with a confidence (lower right)

8

Fig. 3. A typical example of a contrast image from the CD imprintproduction process; different regions of interests found by DBSCAN markedwith different grey levels/colors

Fig. 4. Deviation image from the CD imprint process, different colors/grey-levels represent different ROIs; note that most of the image is white whichmeans that the deviation to the fault-free master is concentrated in smallregions; the faulty and non-faulty parts are exclusively marked as such.

An interesting point here is that on the right most part of thefaulty region (marked with blue pixels) would be not enough toclassify the whole image as ’bad’, whereas the whole region issignificant enough to classify the image as ’bad’. This meansthat aggregated features explaining the intensity and area ofthe pixels clouds may be necessary for discriminating between’good’ and ’bad’ images. However, this is not sufficient, as italso depends on the shape characteristics of the ROIs: imaginetwo arc type objects with same intensity and area as the faultyregion marked in Figure 4. This image would be classified as’bad’ although containing two pseudo-errors. In this sense,it is important to add object features characterizing shape,density etc. of single objects either supervised or unsupervisedto the aggregated features. The class distributions of thelabelled ROI were not well-balanced, which meant that it was

not possible to train a high-performance stand alone objectclassifier; however, when using their output information asadditional features (under the scope of HMI 3, accordingto Figure 1), we could improve the classification as will bedemonstrated in Section VI.

Applying these classification algorithms to the standardaggregated feature sets (containing 17 pre-defined features)of real-world data from an on-line CD imprint productionprocess, produced accuracies between 90% and 94% [39]within a 10-fold cross-validation procedure [43]. Even thoughthe accuracies lie in a reasonable range, they fall far belowthe standards needed for automated industrial quality control,or desirable for medical diagnostic support (often > 96%required). However, as the following sections detail, it ispossible to gain significant improvements by considering thehuman aspects of the sources of data from which the MLsystem learns. Another benefit is that the resulting systems arealso more transparent, user-friendly and generally applicable.

IV. INCREMENTAL CLASSIFIERS BASED ON OPERATORS’FEEDBACK

Incremental training involves the adaptation of parametersand evolving structures (e.g. evolving neurons, rules etc.) andis required whenever the operator gives a feedback upon theclassifier(s) decisions during on-line production mode. Thisis because a periodic rebuilding of the classifier using all thesamples seen so far is impractical as it slows down the trainingprocess too much. On the other hand, if the classifier(s) arenot updated at all (i.e. they are kept fixed after the initial off-line batch training phase) they cannot refine their parameters,react to changing operating conditions or system behaviors andhence may result in unsatisfactory performance. For dealingwith the on-line learning problem we exploited an evolvingclustering-based classifier (eVQ-Class [24]) and an evolvingfuzzy rule-based classifier (FLEXFIS-Class [26]). The first oneis a clustering-based classifier based on vector quantization[14] and exploits the idea of a vigilance parameter motivatedin [5] for forming an incremental and evolving variant. It isincremental in the sense that the cluster centers and surfacesare updated sample-wise based on new incoming samplesduring the on-line process. It is evolving in the sense that newclusters are evolved based on Mahalanobis distances from anewly loaded sample (feature vector extracted from a newlyrecorded image) to all the clusters: the minimal value ofthese Mahalanobis distances is compared with a threshold,also called vigilance parameter and if greater than this, a newcluster is born, otherwise the nearest cluster updated. In thissense, the latter controls the tradeoff between stability (i.e. thegeneration of new clusters) and plasticity (i.e. the updating ofexisting clusters). The cluster prototypes (centers) are updatedby shifting them towards new incoming samples, using alearning rate which starts with value 0.5 and decreases with thesize of the clusters, i.e. by 0.5

ki, with ki the number of samples

forming the cluster i, i.e. those samples for which this clusterwas the nearest one. The surfaces of the ellipsoidal clustersare estimated by the width of the cluster in each dimensionand synchronously updated by exploiting a recursive variance

9

formula [33]. Another modification of the conventional vectorquantization approach is that to find the winning cluster: itdoes not calculate the distance of a new incoming sampleto all the cluster centers, but to the ellipsoidal surfaces ofthe clusters. If a new sample falls inside some ellipsoids, thedistances to the center of these clusters are taken and thewinning cluster is elicited. This cluster is then updated, and inthis case a new cluster is never born. Each cluster is labelledwith the class most frequently represented among the samplesit contains. For further details, see [24].

The second incremental classifier evolves multiple Takagi-Sugeno fuzzy (regression) models [45] and a one-versus-restclassification scheme [9] in case of K > 2 classes. For eachclass a separate Takagi-Sugeno fuzzy model is trained basedon an indicator matrix, obtained by setting the label entries forthe current class to 1 and for all others to 0. The final multi-model fuzzy classifier is extremely flexible for integratingnew classes on demand as they arise, user-defined, duringthe on-line production and learning process. As fuzzy modelsare highly nonlinear and hence able to force the regressionsurface of a model to go towards 0 more rapidly comparing toinflexible linear hyper-planes, the masking effect (which is asevere problem in the case of linear regression by indicatormatrix [15]) is much weaker than in the pure linear case.The single Takagi-Sugeno fuzzy models are evolved by theFLEXFIS (=FLEXible Fuzzy Inference Systems) approach[25], which connects the eVQ [24] for evolving the fuzzyrules and for incremental training of the antecedent partswith the recursive least squares approach [23] for consequentadaptation in a way, such that near-optimality in the leastsquares sense is achieved. In these terms, we speak about a“robust” incremental learning approach coming close to thehypothetical solution if all on-line samples would have beensent into a batch learning process.

The methods are used to generate initial off-line batchtrained classifiers, which are further updated and evolved asnew samples arrive. However, it is necessary to alleviate theburden of input on the user, by reducing the number ofresponses they are required to give. Most so-called activelearning approaches (e.g. [46]) accomplish this by only query-ing the user if the system’s confidence in its own decision is abelow some threshold. However, this can be a risky strategy insituations where examples of previously unseen classes mayarise, or if the user wishes to over-ride previous decisions,which could happen for a range of human or production-related reasons. The approach that we have taken is as follows:

• The system predicts the class of each new sample.• This is shown visually to the user.• The default position is to assume the user’s tacit agree-

ment.• In this case the system is updated using its own decision

output, and taking into account the effect on class im-balance - for example a maximal imbalance of 1/3rd and2/3rd is allowed in 0/1 case.

• If the user disagrees with the prediction, they can providecontradictory input. In this case the system is updatedwith this input as a training signal.

TABLE IIPERFORMANCE OF INCREMENTAL ON-LINE VERSUS STATIC (AND

RETRAINED) BATCH CLASSIFIERS

Dataset Op01 Op02 Op03 Op04 Op05CART static 70.94 78.82 80.00 76.08 68.2CART retrained 82.74 90.59 90.39 88.24 81.7eVQ-Class static 68.82 76.67 76.27 74.71 70.0eVQ-Class inc. 83.33 88.82 88.43 88.43 82.0FLEXFIS-Class static 68.63 85.29 85.68 78.24 74.6FLEXFIS-Class inc. 84.12 88.06 89.92 84.90 84.2

From a HMI perspective, this changes the way that thequestion is posed to the user: the system is effectively askingthe user “Am I right that this a type X?” as opposed to thetypical “what type is this?”.

N-fold cross-validation assumes a fixed data set, and so isnot an appropriate measure here. Instead the CD imprint dataset was split into three. The first third of the images is used forinitial off-line training. The middle third is used to simulateincremental on-line training of the classifier, and is sent sampleper sample into eVQ-Class and FLEXFIS-Class. The final thirdis used as a test set for evaluating the trained classifiers. This isan appropriate way of estimating the true on-line accuracy asthe whole CD imprint data was recorded and stored on-line, sothe test samples do represent the most recent images. Table IIshows the performance of the incremental classifiers versustheir corresponding batch versions, i.e. trained in initial off-line mode with the first batch of data and not further updated(kept static): it can be seen that by doing an adaptation duringon-line mode the performance on new unseen samples signif-icantly increases by 10 to 15% for all operators. Furthermore,Table II also shows us that, when retraining CART in batchmode on all training samples, the accuracy on the new unseensamples is only marginally higher than for the incrementallytrained approaches; in this sense, the computationally intensiveretraining does not really pay off. Because we have used adifferent method for estimating the classifier accuracy, it s notpossible to directly compare exact values, but it is interestingto consider the effect of switching from shuffled data sets(used in N-fold CV), where all classes may be represented, totime-ordered data, where some operators perceive that certainclasses of defects only occur after a certain time. Closerinspection shows that for those operators such as Operator1 where there is a time-imbalance in the class distribution,the gap between the “shuffled” results and the “time-ordered”ones is less for the incrementally adapting eVQ-Class than it isfor even the retrained CART. From a HMI perspective, on-linelearning highlights some interesting points. In a batch-modemodel the user is asked to perform repeated interactions forimage labelling without feedback or reward. This can makethe process seem time-consuming and possibly pointless. Incontrast to this, on-line training means that the user can “see”the system learning from their input, building a progressivelymore accurate model, which can help to motivate them andincrease their focus and attention. Similar findings have beenfound in a range of other HMI studies: see e.g. [36].

10

V. HANDLING INPUT FROM MULTIPLE USERS

When learning systems are applied to industrial applicationsand the operators are actively involved in the training process,the fact that usually multiple operators will be working on thesystem needs to be taken into account. Similarly in the fields ofmedicine and science it may be desirable to incorporate inputfrom a wide range of people in order to broaden the range ofdata examples and human perspectives available. Inevitablyevery time a person makes a decision they implicitly considerand weight multiple competing criteria such as the desire toreduce the risk wrongly classifying “defective” samples as“OK”, or vice versa. One approach to this problem is tocombine the inputs and train multiple versions of classifierswith different parameters to produce ROC curves, and thenselect a global “winner”. However, this still assume consistentbehaviour between and within user’s inputs. It also assumesan fixed weighting of each user’s inputs. In the proposedframework we take a different approach based on the assump-tion that at some stage decisions will be taken to define e.g.company policy, or diagnostic standards, and that these canbe captured s a set of examples labelled by some “expert”.In the first stage of our system each operator trains their ownpersonal classifier as they think best. In the second stage theoutputs of these individual classifiers are then combined intothe final decision. Note that this approach could in principlework equally well whether the individual classifiers are trainedby the users labelling the same, or disjoint sets of examples.

The idea of classifier ensembles is to train not one, but aset (ensemble) of classifiers. Most research has considered thecase where there is only a single data set for which a diverseset of classifiers is trained. Here these ensemble methods areused in a different context: they are used to combine the out-puts of classifiers which are trained by different operators (i.e.using different training sets). This means the ensembles needto resolve the contradictions between the operators. Two levelsof contradiction among the operators can be distinguished:

• Inter-operator contradictions: systematic contradictionsbetween the decisions of different operators. They canbe caused e.g. by different levels of experience, training,skill, etc.

• Intra-operator contradictions: contradictions (often morerandom) between the decisions of a single operator. Theycan be caused by “personal” factors such as the level offatigue, attention, stress, boredom, etc. or “environmen-tal” factors, such as a changed company policy concern-ing the quality control, recent customer complaints, etc.

The intra-operator contradictions, which are mostly random,are dealt with by the operators’ own classifiers themselves.Several learning techniques can naturally handle noisy data(see e.g. [9]). To handle the systematic inter-operator contra-dictions, the operators train their own classifiers, the outputsof which are combined by an ensemble method. An advantageof having each operator train his “own” classifier is that theseclassifiers will be easier to train (provided sufficient data isavailable), since only the intra-operator contradictions needto be handled by these classifiers. Further, it is clear whathas been taught to the system by which operator, making it

possible to provide operator-specific feedback.Ensemble methods can be divided into two classes: gener-

ative and non-generative [47]. Generative ensemble methodsgenerate sets of classifiers by acting on the learning algorithmor on the structure of the data set, trying to actively improvethe diversity and accuracy of the classifiers. These includewell-known methods such as “Boosting” [12]. In generalthese algorithms build different classifiers iteratively, eachiteration focussing on the data points that were hard to classifyby the classifier in the previous iteration by weighing themappropriately. When a new data item is to be classified,the decisions of the classifiers from the different iterationsare combined using a weighted majority vote. This kind ofprocedure is not applicable here since there is a fixed set ofclassifiers (exactly one per operator). These classifiers cannotbe modified by the ensemble algorithm (as they represent thedecisions of an operator) and the ensemble method cannotcreate additional classifiers.

In contrast, non-generative ensemble methods do not ac-tively generate new classifiers but try to combine a set ofdifferent existing classifiers in a suitable way. Clearly the non-generative approach is necessary for the application in thispaper: the operators train their own classifiers in the waythey think is the best and there is no way for the systemto intervene in this process. Non-generative ensembles canbe generally divided into two groups: classifier selection andclassifier fusion [49]. The assumption of the former is thateach classifier is “an expert” in some local area of the featurespace. The latter assumes that all classifiers are trained overthe whole feature space. For the application in this paperclassifier fusion is more appropriate, since the operators trainthe system with the data which is provided by the visionsystem, which could be spread over the entire data space.The fusion of the outputs of the different classifiers (trainedby the different operators) can be done using fixed or, if a“supervisor” has labelled the data, trainable classifier fusionmethods. In the latter case, the ensembles are optimized tobest represent the decisions of the supervisor (i.e. this data isconsidered to be the ground truth), taking into account only thedecisions of the operators. The fixed classifier fusion methodsinclude Voting methods [22] and algebraic connectives suchas maximum, minimum, product, mean and median [19].Trainable classifier fusion methods include the Fuzzy Integral[7], Decision Templates [21], Dempster-Shafer combination[37] and Discounted Dempster-Shafer combination [38]. Alsothe Oracle, a hypothetical ensemble scheme that providesthe correct result if at least one of the classifiers in theensemble outputs the correct classification, was considered.The accuracy of the Oracle can be seen as a “soft bound”on the accuracy which can be achieved by the classifiersand classifier fusion methods. For a detailed survey of thesemethods, see e.g. [20]. We also considered Grading [40],a different approach to classifier fusion. Here a so-called“Grading” classifier is trained for each base-level classifierto predict whether the classifier will be correct for a givendata sample. The classifiers which are estimated to be correctare considered in a Majority Voting scheme to provide thefinal output. Thus Grading forms a non-linear weighting over

11

TABLE IIICLASSIFICATION ACCURACY (IN %) OF CART CLASSIFIERS FOR THE CD

DATA SETS. CLASSIFIERS ARE TRAINED USING DATA FROM ONEOPERATOR (ROW) AND EVALUATED FOR THE ABILITY TO PREDICT LABELS

PROVIDED BY ANOTHER(COLUMN).BOLD TYPE INDICATES CLASSIFIERSTESTED AND TRAINED ON SAME DATA SET.

Classifier trainedon data from

Evaluation on data from (supervisor)Op01 Op02 Op03 Op04 Op05

Op01 90.57 89.27 85.33 88.94 71.42Op02 88.83 95.38 91.88 93.10 69.42Op03 86.50 93.42 93.90 92.68 70.59Op04 88.82 93.67 92.08 94.38 71.91Op05 72.42 71.64 71.73 73.58 91.25

TABLE IVCLASSIFICATION ACCURACY (IN %) OF THE DIFFERENT ENSEMBLE

METHODS (ROWS) FOR THE CD DATA SETS. DIFFERENT OPERATORS ARECONSIDERED TO BE THE SUPERVISOR (COLUMNS); TRAINING INPUT FOR

THE ENSEMBLES IS PROVIDED BY THE OTHER OPERATORS.

Ensembles to combinethe operator classifiers

Evaluation on data from (supervisor)Op01 Op02 Op03 Op04 Op05

Voting 88.35 92.31 90.22 92.72 70.33Algebraic Mean 88.19 89.86 88.28 91.56 71.74Fuzzy Integral 88.50 94.42 91.61 93.69 71.25

Decision Templates 88.19 94.42 89.16 91.59 74.75D-S Combination 88.19 94.42 91.82 92.07 71.75

Disc. D-S Combination 88.31 94.42 92.24 93.54 71.75Grading (CART) 91.19 96.05 93.57 94.61 74.69

Oracle 95.83 98.89 97.07 98.60 78.90

the feature space for each classifier. In this sense it could beconsidered akin to the selection methods for classifier fusion.The difference is that in this case the weighting implicitlycaptures the preferences and opinions of the super-operatorin terms of conflicting risks etc. Thus in different regions ofthe feature space it will adapt to give more weight to thoseclassifiers representing operators whose decisions and weightsagree with the super-operator.

The CD data with 17 aggregate features described in Sec-tion II was labelled by four different operators. To investigatethe HMI issues arising from the inter-operator contradictions,a fifth operator (Operator05) was included in the tests, who hasa different position in the company than the other operatorsand thus has a different subjective view on the quality criteria,as will be demonstrated next. As Operator05 is not involved inthe day-to-day quality control, this operator is not consideredin the other sections of this paper (describing the other HMIissues – not considering the effect of training the system usingthe inputs of multiple operators). To investigate the effect ofthe inter-operator contradictions, each of these five operatorsis considered as the “supervisor” in turn. Classifiers are trainedfor the other operators and their outputs are then combined bythe ensembles in order to better model the decisions of thesupervisor. Note that this implies evaluating the classifiers ondata which they were not trained for.

In Table III the effect of training a classifier with the inputfrom one operator (row) and then evaluating this classifierusing the input from another operator (column) is shown. Itcan be seen that three operators make very similar decisions(Operators 02, 03 and 04), one operator differs slightly from

these three operators (Operator01), and one operator makesdecisions which are very different from all other operators(Operator05). Note that the results of about 90% to 95% onthe diagonal of this table (shown in bold) denote the evaluationof the classifiers on the same data they were trained on (i.e. anormal classification task) as seen in Section III.

In Table IV the effect of training a classifier for fouroperators and then combining them using different ensemblemethods to predict the labels provided by the fifth operator(considered the supervisor) is shown. From these results wecan see that the ensemble are able to represent the supervisorbetter than the classifiers trained by the different operatorsthemselves. The improvements range up to approximately 3%.In these experiments the Grading ensemble method performedthe best, outperforming the other classifier fusion methods(both fixed and trainable) in all experiments, except whenOperator05 is considered the supervisor (in this case thereis a marginal difference with the Decision Templates methodof 0.06%. Note that if all operators do not agree well withthe supervisor a drop in the accuracy is recorded – e.g.approximately 20% of the decisions of Operator05 (the op-erator who has a different position in the company than theother operators) do not agree with any of the other operators.In this case even the hypothetical Oracle does not performwell, bounding the achievable accuracy below 80%. Fromthese results we can conclude that the ensemble methodscan effectively be used to combine the decisions of differentoperators, in order to model the decisions of a supervisorbetter, with improvements of up to 3%.

Another interesting result is that by combining the outputsof the classifiers trained for the different operators and com-bining them by an appropriate ensemble method (in this casethe Grading ensemble), in several cases the performance iseven higher than the performance of a classifier specificallytrained on the labels provided by the supervisor (the resultson the diagonal of Table III). This is caused by the inter-operator contradictions among the operators, creating a diverseensemble of classifiers which is able to perform well, asshown in these results. The only experiment in which thiswas not the case is when Operator05 is considered to be thesupervisor (again due to the large deviations with all of theother operators).

VI. HANDLING VARIABLE LEVELS OF DETAIL IN USERINPUTS

A major problem of image classification is the fact that itis not known in advance how many regions of interest may besegmented from images occurring in the future, and yet mostclassification algorithms assume a fixed-size input data space.

As noted above, for each segmented ROI we calculate afixed number of features (57). The most straightforward way totackle this issue is to preprocess the ROI and characterize theirdistribution within the object-feature space by a fixed numberof descriptors. In practice, this requires first a preprocessingstage prior to training the image level classifier, and thenalso during testing/validation, an additional step to process theinformation about the ROI within each image. When operator-assigned labels are present supervised object level classifiers

12

can be constructed [4]. These created a labelled partition ofthe object feature space. When presented with the ROI froma new image, their outputs –i.e. the number of each type ofobject present on an image– are added to the aggregate imagedata features and then image-level classifiers trained and testedin a n-fold cross-validation regime. Such supervised learningmethods are highly useful, and may be easily interpreted,but obtaining this information requires significant operatorinput which may not be available off-line, or may simplybe infeasible on-line due to the speed of production. Whenobject-level labels are not available, it is necessary to usedifferent approaches to represent the distribution of objectfeatures. Such methods do not require operator input but arehighly reliant on the choice of input features. One approach isto apply first order statistics to this distribution to produceextra image-level descriptors. After some experimentation,it was found that using the maximum value of the object-level features was most useful - e.g. the size/intensity of thebiggest/brightest object in an image.

An alternative approach is to apply unsupervised object-level learning to produce a reduced set of extra features whichdescribe the distribution in a different way [3]. In this case, wehave proceeded as follows. During the pre-processing stage aclustering algorithm is applied to all the objects from all theimages in the training set. This provides us with C centroids inobject-feature space, and also, for each training object, a labelfrom the set {1, . . . , C}. Then, for each image in the trainingset we can obtain C extra features denoting the number ofobjects of each type it contains. After this pre-processing stagethe image classifier is then trained using the 17 + C featuresfor each labelled image. During testing/validation, each ofthe segmented ROI in a new image is assigned to the objectclass with the nearest centroid, and the appropriate hit-countincremented, to create the 17 + C dimensional vector that isinput to the classifier. In these experiments we used a relativelysimple clustering algorithm - k-means. As this is prone both tosettling in local optima, and to the effect of redundant features,we applied a Tabu search to perform “wrapper-style” featureselection in the object space. More details can be seen in [28].

The results of adding different levels of information fromthe operators are shown in Table V. Table VI shows theeffect of augmenting the 17 image level features with Ccluster-based object features, and how this changes with C. Ascan be seen, in each case the two-level supervised classifierapproach improves the results over our baseline. This is notsurprising given the extra levels of operator input available,although it is worth noting that training an object classifier is acomplex multi-classification approach, where just an accuracyof about 80% can be achieved at the object level. Whenthis level of information was not present, then simply usingfirst order statistics to describe the distribution of objects oneach image gave significant benefits. This is despite the factthat the result was a 74-dimensional space for the imageclassifier to learn. As Table VI shows, the results of using theclustering approach to partition the object-level feature spacedepend on an appropriate choice for C. More importantly, theresults demonstrate that with C = 12 the unsupervised cluster-based approach actually does as well, if not better, than the

TABLE VCLASSIFICATION ACCURACY USING DIFFERENT LEVELS OF USER INPUT.

Dataset CART C4.5 1NN 9NN eVQ-Class1:17 image-level aggregate features only

Op01 Acc. 91.6 91.8 92.2 90.2 90.5Op02 Acc. 95.5 95.5 95.1 95.1 95.2Op03 Acc. 94.2 94.2 93.0 94.3 92.9Op04 Acc. 94.9 94.1 93.8 92.6 92.9Op05 Acc. 91.9 90.5 90.4 88.1 85.2As 1 but with 13 extra features from supervised object classifierOp01 Acc. 93.9 93.1 93.4 92.3 92.3Op02 Acc. 96.7 96.7 96.3 96.2 95.2Op03 Acc. 96.1 96.9 95.2 94.8 92.6Op04 Acc. 96.4 95.6 95.4 94.8 92.7Op05 Acc. 94.2 94.8 93.9 91.8 90.0

As 1 but with 57 extra features-the max. values of the object-level features for each image

Op01 Acc. 93.2 91.5 92.7 92.4 92.8Op02 Acc. 95.6 95.1 95.8 95.4 96.1Op03 Acc. 94.5 94.0 93.4 94.3 93.4Op04 Acc. 94.9 93.2 94.8 93.5 94.6Op05 Acc. 92.3 92.1 92.8 91.7 87.1

TABLE VICLASSIFICATION ACCURACY (%) WITH 17 IMAGE FEATUES PLUS C

CLUSTER-BASED OBJECT FEATURES

Data Set KM/C4.5/FS KM/C4.5/FS KM/C4.5/FS(C=4) (C=8) (C=12)

Op01 Acc. 93.7 94.1 94.5Op02 Acc. 97.1 97.4 97.4Op03 Acc. 95.7 96.2 96.2Op04 Acc. 96.2 96.8 96.8Op05 Acc. 94.0 93.5 94.2

supervised approaches. The fact that 12 clusters were found inthe data suggests that the problem is one of correctly assigningan object to a class, rather than that those different classes donot actually exist in the data.

VII. ACCOMMODATING PARTIAL CONFIDENCE OFOPERATORS

During the setup phase of an image classification frame-work, the labelling of several images can be a difficult taskfor the operators, especially in cases where real faults arehard to distinguish among themselves or between so-calledpseudo-errors. This problem can become even worse when theoperators are not working in the relative calm of an off-linesetting, but are providing real-time decisions at a speed drivenby other factors. In this sense, it is promising, sometimeseven necessary for the operators to provide information abouthow confident they are when assigning the labels to certainimages or objects. Here, only the confidences in the wholeimage labels are taken into account. The simplest way isto represent the users’s confidence as a value in range 0.0(very unconfident) to 1.0 (very confident). This raises twoissues: with what precision should the confidence be used,and how should this information be obtained from the users?In keeping with research from other fields about how manycategories people can typically discriminate, rather than askingusers to spend time thinking of an exact value to assign,we ask them for one of five values. This means that wecan avoid the need to enter that data either via typing, or

13

TABLE VIICLASSIFICATION ACCURACIES, AND IMPROVEMENTS WHEN APPLYINGTHE REGRESSION APPROACH TO THE 74 AGGREGATED + SUPERVISED

FEATURES

Dataset CART C4.5 1NN 9NN eVQ-ClassOp01 Acc. 94.2 - 93.4 92.3 91.9

(+0.3) - (0.0) (0.0) (-0.3)Op02 Acc. 97.2 - 96.3 96.2 96.6

(+0.5) - (0.0) (0.0) (+1.4)Op03 Acc. 97.6 - 95.2 94.8 93.1 )

(+0.7) - (0.0) (0.0) (+0.5)Op04 Acc. 96.5 - 95.4 94.8 95.2

(+0.1) - (0.0) (0.0) (+2.5)Op05 Acc. 94.5 - 93.9 91.8 90.6

(+0.3) - (0.0) (0.0) (+0.6)

by a mouse click and drag on a slider, both of which aretime-consuming operations. The choice of five distinct values,i.e. {20%, 40%, 60%, 80%, 100%} confidence, is also partiallydriven by the needs of the Graphical User interface (see Figure2), resulting from an intensive round of discussion and designiteration with industrial users from a range of fields. Based onthis we worked out two principle approaches for incorporatingthe confidence values for training the classifiers.

The first approach is to apply regression approaches toconfidence values transformed by:

conf transformed =(1 + η · conf)

2(1)

where η takes the value +1/-1 for “good”/”bad” samples.Because this model treats the problem as a regression problemrather than a classification one it is only directly applicable in atwo-classification scenario. For multi-class problems, indicatormatrices have to be applied, see [15]. Table VII shows theresults obtained. We applied the regression variants of CART[2]. This type of regression is not possible with C4.5. ForkNN the confidence values of the nearest neighbors of a newincoming instance are averaged and if being greater than0.5, label “good” is assigned. For eVQ-Class the hit-rate,i.e. the relative proportion of samples is calculated for eachclass in the nearest cluster of the new incoming sample bysimply summing up all confidence values for each class. Theclass with the highest sum is assigned to the new incominginstance. The improvement over the results without includingthe confidence values is clearly visible using CART and eVQ-Class. For 1NN and 9NN there are no improvements. As thismethod does not provide benefits for several classifiers, anddoes not tackle the multi-class problems, other approacheswere investigated.

The second approach tried is based on duplicating the (ex-tended) aggregated feature vectors according to the assignedconfidence values. Thus a feature vector from an image whichis labelled with 1.0 confidence is duplicated five times, anotherone labelled with 0.8 confidence is duplicated four times etc.This means that feature vectors which are labelled with ahigher confidence are higher weighted in the training processthan those labelled with a lower confidence. In principle, thismay not affect necessarily a training process of a classifier,however the approaches used in this paper (CART, C4.5, k-

TABLE VIIICLASSIFICATION ACCURACY USING TWO-LEVEL APPROACH AND

DUPLICATING FEATURE VECTORS ACCORDING TO CONFIDENCE LEVELSOF OPERATORS. RESULTS IN BRACKETS SHOW THE IMPROVEMENT IN

PERCENTAGE BY EACH CLASSIFIER OVER TWO-LEVEL APPROACHWITHOUT DUPLICATION.

Dataset CART C4.5 1NN 9NN eVQ-ClassOp01 Acc. 94.4 94.0 93.4 94.2 93.4

(+0.5) (+0.3) (+0.0) (+1.9) (+1.2)Op02 Acc. 97.7 96.9 96.3 ) 96.2 96.7

(+1.0) (+0.8) (+0.0) (+0.0) (+1.5)Op03 Acc. 98.0 97.3 95.2 95.2 96.5

(+1.1) (+0.1) (+0.0) (+0.4) (+3.9)Op04 Acc. 97.0 96.7 95.4 95.9 95.6

(+0.6) (+1.1) (+0.0) (+1.1) (+2.9)Op05 Acc. 95.2 95.3 93.9 93.0 91.7

(+1.0) (+0.5) (+0.0) (+1.2) (+1.7)

NN and eVQ-Class) are definitely affected by the density resp.relative proportions of the classes. It should be noted that theidea of creating training sets by sampling from a given dataset is not new, e.g. the well known algorithm bootstrapping[10] does this. However, in one case the sampling is weighteduniformly, and in the other the sampling weights are deter-mined iteratively according to the performance of classifiersbuilt from the initial training sets. What we are proposing issomewhat different, in that (i) the weights governing samplingprobabilities are a function of the operators confidence – thusincorporating information which would be lost or ignored bytraditional approaches, and (ii) the sampling is deterministic,rather than stochastic.

Applying the duplicating feature vectors approach on thetwo-level CD feature data set gives the results shown inTable VIII. As can be seen, the results improve for allclassifiers except 1NN and can outperform the regressionapproach significantly (see Table VII). It is to be expected thatno difference is observed with 1NN, as of course duplicationhas no effect when only one instance is considered to makeeach decision. In contrast, when a larger groups of neighborsare used (9NN – column 5) the increase can be dramaticas “confident” images outvote others. Not only does thistechnique give improvements for all the different types ofclassifiers, it does so for all operators: towards 98% foroperator #2 and operator #3 and to 97% for operator #4.

VIII. PROVIDING INSIGHT INTO CLASSIFIER STRUCTURES

The operator may wish to have more insight into theclassifier structures in order to obtain a better understanding ofthe classifier decisions respectively the characteristics of thefaulty situations at the systems. Sometimes, he may even wishto interact with the classifier’s training process, i.e. to modifysome structural components of the classifier (e.g. rules, leafs,neurons etc.). This is because he has some intrinsic knowledgeabout the fault and non-fault characteristics in the recordedimages. Note that this will lead finally to a kind of grey-boxmodelling behavior [31] in an alternating scheme, which isto train an initial classifier based on some training data, thento modify the trained classifier by expert knowledge and thento adapt further the classifier with new on-line data. In order

14

to gain more insight into the classifier structure, a first issueis to consider which model architecture is a reliable one todo so, i.e. yielding transparent classifiers. From the repertoireof methods that were used in this paper, the decision-treeapproaches seem to be most convenient, as they produce aclassifier with a tree structure, where each path from the root tothe terminal node can be transferred to a linguistically readableclassification rule [29] [34].

A second important issue is to consider complexityreduction processes for making trained complex modelsslimmer and hence more transparent and interpretable.For decision trees this is achieved by so-called pruningtechniques, which perform complexity reduction based onnested tree sequences (from simple to more complex trees)[30]. For an example of a simple (pruned) tree structure seethe right image Figure 5, whereas the left image representsthe unpruned (quite unclear) tree. For the pruned tree theclassification rules can be read as follows (xi denotes the ithfeature):

Rule 1: IF x7 ≥ 4.72 THEN Image is “BAD”Rule 2: IF x7 ≤ 4.72 AND x12 < 3384.5 THEN Image is“GOOD”Rule 3: IF x7 ≤ 4.72 AND x12 ≥ 3384.5 AND x2 ≥ 3.38THEN Image is “GOOD”and so on

In fact, this means that if feature x7 is greater or equalthan 4.72, the image is already classified as “BAD”. In orderto give these rules a better linguistic representation, it can becross-checked what 4.72 means with respect to the range ofthe feature x7, which is in this case the aggregated feature“maximal intensity of an object”, whereas intensity is definedby multiplying the size with the average (normalized) greyvalues of the object: 4.72 is a quite low value when takinginto account that this feature ranges from 0.078 to 98.14, thismeans the rule “IF x7 ≥ 4.72 THEN Image is BAD” can betransferred to the linguistic rule “IF the maximal intensityof an object is NOT LOW THEN Image is BAD” resp. “IFthere are any ROIs in the (contrast) image which are clearlyvisible THEN image is BAD”. In this sense, the user candecide whether it is a correct or wrong rule.

Please note that pruning is not only for obtaining a bettertransparency of the classification rules and more compactrepresentation of the intrinsic knowledge in the training data,but also for improving the accuracies of the classifiers, ascomplex trees tend to overfit on new unseen samples. Inthis sense, pruning is definitively always a successful stepwhen designing a decision-tree based classifier. The resultsin Table VIII with respect to the CART algorithm were alsoachieved with an implicit pruning step, the pruning level wasset to 1 in this case, leading to a still quite complex treewith depth of 11 and number of non-terminal nodes of 37.It was also examined how a stronger pruning improves thetransparency (measured in terms of two numbers: tree depthand number of nodes) while weakening the predictive powers.This is underlined in Figure 6, where for the training datalabelled by Operator 6 the number of non-terminal nodes

Fig. 6. Complexity versus accuracy of the pruned decision tree

versus the elicited accuracy are visualized. Here, it can be seen,that indeed by using a pruning level of 1 the best accuracy canbe obtained, however when increasing the pruning level to avalue of 6 and decreasing the number of non-terminal nodes(i.e. the number of rules) to just 9, only a fraction (i.e. 0.8%)of the accuracy is lost. In this sense, this pruned tree representsa good tradeoff between performance in terms of accuracy andtransparency of the classifier.

IX. CONCLUSIONS AND OUTLOOK

As machine learning systems move out of the laboratoryand into real-world applications such as vision and image pro-cessing, it is valuable to reconsider some of the assumptionsthat have been made about how such systems can best learnfrom users. In this paper we have discussed some importantissues, and suggested how they can be handled. Experimentsconducted with “real” data within the context of a genericimage processing system show that when properly handled, thehuman factors can represent an additional form of informationto these systems for improving performance and may widenthe applicability and usability, rather than to be a disagreeablesource of noise. Key issues of these factors include on-lineguidance and feedback, a diversity of user skills, uncertaintiesas well as different levels of knowhow and detail in users’input. Crucial to this change is paying proper considerationto the way that the users actually interact with the system.They can be distilled to two questions. The first of theseis: what should the users be shown and how? Many yearsof experience of the project partners have taught us thatmaintaining users’ involvement in the value of the system iscrucial, so especially during on-line training it is vital for theinterface to show that the system is actually learning fromtheir input. The second question is: what information shouldthe users provide, and how? We have shown how asking fortoo much information –e.g. object level labels– can sometimescause complications, and may not always be possible. Incontrast, other information such as the human’s uncertainty,

15

Fig. 5. Left: complex decision tree not being pruned; right: by applying a pruning step, the tree gets much easier to interpret by loosing only a fraction ofits predictive power

can be easily obtained and lead to significant performanceimprovements. The improvements are made possible by recentadvances in the speed with which graphical user interfaces canoperate. The next generation of user interaction devices offersthe potential to build on this research, creating much richerhuman-machine learning interaction.

ACKNOWLEDGEMENTS

This work was supported by the European Commission(project Contract No. STRP016429, acronym DynaVis). Thispublication reflects only the authors’ views.

REFERENCES

[1] H. Bekel, I. Bax, G. Heidemann, and H. Ritter. Adaptive computervision: online learning for object recognition. In Proceedings of the 26thDAGM Symposium on Pattern Recognition, pages 447–454. Springer-Verlag Berlin, 2004.

[2] L. Breiman, J. Friedman, C.J. Stone, and R.A. Olshen. Classificationand Regression Trees. Chapman and Hall, Boca Raton, 1993.

[3] P. Caleb-Solly and J.E. Smith. Adaptive surface inspection via interactiveevolution. Image and Vision Computing, 25(7):1058–1072, 2007.

[4] P. Caleb-Solly and M. Steuer. Classification of surface defects on hotrolled steel using adaptive learning methods. In Proc. of the IEEE FourthInternational Conference on Knowledge-Based Intelligent EngineeringSystems and Allied Technologies,vol. 1, pages 103–108.

[5] G. A. Carpenter and S. Grossberg. Adaptive resonance theory (art).In M. A. Arbib, editor, The Handbook of Brain Theory and NeuralNetworks, pages 79–82. MIT Press, Cambridge, MA, 1995.

[6] E. Castillo and E. Alvarez. Expert Systems: Uncertainty and Learning.Springer Verlag New York Inc., New York, USA, 2007.

[7] S. Cho and J. Kim. Combining multiple neural networks by fuzzyintegral for robust classification. IEEE Transactions on Systems, Man,and Cybernetics, 25(2):380–384, 1995.

[8] R. Cipolla and A. Pentland (editors). Computer Vision for Human-Machine Interaction. Cambridge University Press, Cambridge, UK,1998.

[9] R.O. Duda, P.E. Hart, and D.G. Stork. Pattern Classification - SecondEdition. Wiley-Interscience, Southern Gate, Chichester, West Sussex PO19 8SQ, England, 2000.

[10] B. Efron and R. Tibshirani. An Introduction to the Bootstrap. Chapmanand Hall/CRC, 1993.

[11] M. Ester, H.P. Kriegel, J. Sander, and X. Xu. A density-based algorithmfor discovering clusters in large spatial databases with noise. InProceedings of 2nd International Conference on Knowledge Discoveryand Data Mining (KDD-96), 1996.

[12] Y. Freund and R.E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer andSystem Sciences, Vol. 55, No. 1, pp. 119–139, 1997.

[13] F. Gayubo, J.L. Gonzalez, E. De La Fuente, F. Miguel, and J.R. Peran.On-line machine vision system for detect split defects in sheet-metalforming processes. In Proc. of the 18th International Conference onPattern Recognition. IEEE Comput. Soc., Los Alamitos, CA, USA,2006.

[14] R.M. Gray. Vector quantization. IEEE ASSP Magazine, pages 4–29,1984.

[15] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of StatisticalLearning: Data Mining, Inference and Prediction. Springer Verlag, NewYork, Berlin, Heidelberg, Germany, 2001.

[16] A. Jaimes and N. Sebe. Multimodal human computer interaction: Asurvey. In Proceedings of the International Workshop on Human-Computer Interaction, HCI/ICCV 2005, pages 1–15. Springer Verlag,2005.

[17] A.K. Jain and R.C. Dubes. Algorithms for Clustering Data. PrenticeHall, 1988.

[18] N. Kasabov. Evolving Connectionist Systems - Methods and Applicationsin Bioinformatics, Brain Study and Intelligent Machines. SpringerVerlag, London, 2002.

[19] J. Kittler, M. Hatef, R. Duin, and J. Matas. On combining classi-fiers. IEEE Transactions on Pattern Analysis and Machine Intelligence,20:226–239, 1998.

[20] L. I. Kuncheva. Combining pattern classifiers: Methods and algorithms.Wiley, 2004.

[21] L. I. Kuncheva, J.C. Bezdek, and R. P.W. Duin. Decision templatesfor multiple classifier fusion: an experimental comparison. PatternRecognition, 34(2):299–314, 2001.

[22] L. I. Kuncheva, C.J. Whitaker, C.A. Shipp, and R. P.W. Duin. Limitson the majority vote accuracy in classifier fusion. Pattern Analysis &Applications, 6(1):22–31, 2003.

[23] L. Ljung. System Identification: Theory for the User. Prentice Hall PTR,Prentic Hall Inc., Upper Saddle River, New Jersey 07458, 1999.

[24] E. Lughofer. Extensions of vector quantization for incremental cluster-ing. Pattern Recognition, 41(3):995–1011, 2008.

[25] E. Lughofer. FLEXFIS: A robust incremental learning approach forevolving TS fuzzy models. IEEE Trans. on Fuzzy Systems (special issueon evolving fuzzy systems), to appear, doi 10.1109/TFUZZ.2008.925908,2008.

[26] E. Lughofer, P. Angelov, and X. Zhou. Evolving single- and multi-modelfuzzy classifiers with FLEXFIS-Class. In Proceedings of FUZZ-IEEE2007, pages 363–368, London, UK, 2007.

[27] A. Bouridane M. A. Tahir and F. Kurugollu. Simultaneous featureselection and feature weighting using hybrid tabu search/k-nearestneighbor classifier. Pattern Recognition Letters, 28, 2007.

[28] J. Smith M. A. Tahir and P. Caleb-Solly. A novel feature selectionbased semi-supervised method for image classification. In To appearin the Proceedings of 6th International Conference on Computer VisionSystems, Santorini Island, Greece, 2008.

[29] S. Mitra, K.M. Konwar, and S.K. Pal. Fuzzy decision tree, linguisticrules and fuzzy knowledge-based network: generation and evaluation.IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applica-tions and Reviews, 32(4):328–339, 2002.

16

[30] K. Muata and O. Bryson. Post-pruning in decision tree induction usingmultiple performance measures. Computers and Operations Research,34(11):3331–3345, 2007.

[31] O. Nelles. Nonlinear System Identification. Springer Verlag Berlin,Germany, 2001.

[32] G. Papari and N. Petkov. Algorithm that mimics human perceptualgrouping of dot patterns. In Brain, Vision, and Artificial Intelligence,pages 497–506. Springer, Berlin, 2005.

[33] S.J. Qin, W. Li, and H.H. Yue. Recursive PCA for adaptive processmonitoring. Journal of Process Control, 10:471–486, 2000.

[34] J. R. Quinlan. C4.5: Programs for Machine Learning. MorganKaufmann Publishers Inc, U.S.A., 1993.

[35] S. Raiser, E. Lughofer, and C. Eitzinger. Impact of object extractionmethods on classification performance in surface inspection systems.Machine Vision and Applications (special issue surface inspection),submitted, 2008.

[36] J. Rasmussen. Information Processing and Human-Machine Interaction:An Approach to Cognitive Engineering. Elsevier Science Ltd, 1986.

[37] G. Rogova. Combining the results of several neural network classifiers.Neural Networks, 7:777–781, 1994.

[38] D. Sannen, H. Van Brussel, and M. Nuttin. Classifier fusion usingdiscounted dempster-shafer combination. In Proc. of InternationalConferernce on Machine Learning and Data Mining (MLDM 2007),pages 216–230.

[39] D. Sannen, M. Nuttin, J.E. Smith, M.A. Tahir, E. Lughofer, andC. Eitzinger. An interactive self-adaptive on-line image classificationframework. In A. Gasteratos, M. Vincze, and J.K. Tsotsos, editors,Proceedings of ICVS 2008, volume 5008 of LNCS, pages 173–180.Springer, Santorini Island, Greece, 2008.

[40] J. Furnkranz A.K. Seewald. An evaluation of grading classifiers.In Proceedings of the 4th International Conference on Advances inIntelligent Data Analysis, Springer.

[41] A. Shilton, M. Palaniswami, D. Ralph-D, and A.-C.-Tsoi. Incrementaltraining of support vector machines. IEEE Transactions on NeuralNetworks, 16(1):114–131, 2005.

[42] J.E. Smith and M.A. Tahir. Stop wasting time: On predicting the successor failure of learning for industrial applications. In Proccedings ofthe 8th International Conference on Intelligent Data Engineering andAutomated Learning (IDEAL’08), Lecture Notes in Computer Science,LNCS 4881, Springer Verlag, pages 673–683, Birmingham, UK, 2007.

[43] M. Stone. Cross-validatory choice and assessment of statistical predic-tions. Journal of the Royal Statistical Society, 36:111–147, 1974.

[44] M. A. Tahir and J. Smith. Improving nearest neighbor classifier usingtabu search and ensemble distance metrics. In Proceedings of the IEEEInternational Conference on Data Mining (ICDM), Hong Kong, 2006.

[45] T. Takagi and M. Sugeno. Fuzzy identification of systems and itsapplications to modeling and control. IEEE Trans. on Systems, Manand Cybernetics, 15(1):116–132, 1985.

[46] S. Tong and D. Keller. Support Vector Machine Active Learningwith Applications to Text Classification. Journal of Machine Learningresearch, 2:45-66, 2001

[47] G. Valentini and F. Masulli. Ensembles of learning machines. InM. Marinaro Series Lecture Notes in Computer Sciences and R. Taglia-ferri, editors, Neural Nets WIRN Vietri-02. Springer-Verlag, Heidelberg,Germany, 2002.

[48] D.H. Wolpert. The lack of a priori distinctions between learningalgorithms. Neural Computation, 8(7), 1341-1390, 1996

[49] K. Woods, W. P. Kegelmeyer, and K. Bowyer. Combination of multipleclassifiers using local accuracy estimates. IEEE Transactions on PatternAnalysis and Machine Intelligence, 19:405–410, 1997.

[50] X. Wu, V. Kumar, J.R. Quinlan, J. Gosh, Q. Yang, H. Motoda, G.J.MacLachlan, A. Ng B. Liu, P.S. Yu, Z.-H. Zhou, M. Steinbach, D.J.Hand, and D. Steinberg. Top 10 algorithms in data mining. Knowledgeand Information Systems, 14(1):1–37, 2006.