Catchline (head of first page only) International Journal of Remote Sensing, Vol. X, No. X, Month 2008, xxx–xxx

Running heads (verso) Luisa Gonçalves, Cidália Fonte, Eduardo Júlio and Mario Caetano (recto) Evaluation of possibilistic classifications

Article Type (Research Paper)

Evaluation of soft possibilistic classifications with non-specificity uncertainty measures

LUISA M S GONÇALVES*†, CIDÁLIA C FONTE §, EDUARDO N B S JÚLIO&, MARIO CAETANO‡

†Civil Engineering Department, Polytechnic Institute of Leiria, Portugal

§Institute for Systems and Computers Engineering at Coimbra, Portugal

§Department of Mathematics, University of Coimbra, Portugal

&ISISE, Civil Engineering Department, University of Coimbra, Portugal

‡Portuguese Geographic Institute (IGP), Remote Sensing Unit (RSU), Lisboa, Portugal

‡ Institute for Statistics and Information Management (ISEGI), Universidade Nova de Lisboa, Lisboa, Portugal

*†Email: luisa@estg.ipleiria.pt

Abstract

The aim of this paper is to investigate the usefulness of the non-specificity uncertainty measures to evaluate soft classifications of remote sensing images. In particular, it is analysed if these measures can be used to identify the difficulties found by the classifier and to estimate the classification accuracy. Since two non-specificity uncertainty measures can be considered, namely the non-specificity measure (NSp) and the U-uncertainty measure, the behaviour of both measures is analysed to evaluate if one is more appropriate for this application than the other. To overcome the fact that these two measures have different ranges, a normalized version (Un) of the U-uncertainty measure is used. Both measures are applied to evaluate the uncertainty of a soft classification of a very high spatial resolution multispectral satellite image, performed with an object

oriented image analysis based on a fuzzy classification. The classification accuracy is evaluated using an error matrix and the user’s and producer’s accuracy are computed. Two uncertainty indexes are proposed for each measure, and the correlation between the information given by them and the user’s and producer’s accuracy is determined, to assess the relation and compatibility of both sources of information. The results highlight that there is a positive correlation between the information given by the uncertainty and accuracy indexes, but mainly between the uncertainty indexes and the user’s accuracy, where the correlation achieved 77%. This study shows that uncertainty indices may be used, along with the possibility distributions, as indicators of the classification performance, and may therefore be very useful tools.

Keywords: Soft classification; Possibility distributions; Uncertainty measures; Non-specificity measures; Accuracy assessment AMS Subject Classification:

1. Introduction

During the last decade considerable research has been done in the development of soft classifiers to extract information from remote sensing images (e.g. Maselli et al., 1995; Brown et al., 2000; Zhang and Foody, 2001;Tso and Mather, 2001; Ibrahim, et al., 2005; Doan and Foody, 2007). These soft classifiers overcome some of the limitations of the traditional hard methods, where information is represented in a one pixel - one class base and where all data regarding the other possible classes is discarded (e.g. Wang 1990; Foody 1996; Bastin, 1997; Ibrahim, et al., 2005; Lu and Weng, 2007). Soft classification methods assign to each spatial unit (pixel or object) several classes with different degrees of probability, possibility or membership, depending on the mathematical theory applicable in each case. The additional data provided by the soft classification methods for each spatial unit, besides providing more information about land cover, can also be used to assess the classification uncertainty. Within the context of remote sensing, several semantics may be associated to the degrees of probability, possibility or membership obtained with soft classifiers. These may represent partial membership of the classes to the spatial units, as in the case of mixed pixels or objects; a degree of similarity between what exists in the ground and the pure classes; or the uncertainty associated to the classification. However, in practice, it is not generally possible to know if the first or second interpretations are the correct ones, since the spectral responses are used to determine what exists in reality and therefore the real conditions are not known in advance. For this reason, in most cases, the degrees of membership actually reveal the degree of uncertainty associated to the classification, even though this information is frequently used to estimate the mixture between classes (e.g. Zhang and Foody, 2001; Maselli et al., 1996; Bastin, 1997). Several types of uncertainty have been identified, such as nonspecificity (or imprecision), fuzziness (or vagueness) and discord (e.g. Klir and Yuan, 1995; Pal and Bezdek, 2000; Klir, 2004). The different types of uncertainty are evaluated using different mathematical frameworks and different measures may be applied within very different contexts and areas. Nonspecificity, in particular, considers that the fewer elements a set contains the more specific it is. Then, specificity measures the extent to which a set restricts a variable to a small number of values and, therefore, when applied to possibility distributions, it measures the degree to which a possibility distribution allows one and only one element as possible (Yager, 1992; Pal and Bezdek, 2000). This concept may be applied to image classification to evaluate the difficulty found by the classifier to attribute only one class to each spatial unit. The fewer possibilities found the more specific the set is and less uncertainty is associated with the classification. Ricotta (2005) suggested, from a theoretical point of view, the use of the non-specificity measures developed within the fuzzy sets and possibility theory community, namely

the specificity measure developed by Yager (1982) and its non-specificity counterpart, and the U-uncertainty measure, proposed by Higashi and Klir (1983). Nevertheless, its potential within the context of remote sensing has not been investigated, no further explanation is available on the information given by these measures and no application to real data was made. The evaluation of the classification accuracy is usually done building an error matrix based on the information collected for a sample of points in the classified image and a reference source of information (e.g. Foody, 2002). The matrix may then be used to compute accuracy indexes, such as the global accuracy or the user’s and producer’s accuracy. A number of approaches to assess the accuracy of soft classifications have also been proposed (e.g. Foody, 1996; Arora and Foody, 1997; Binaghi et al., 1999; Woodcock and Gopal, 2000; Oki et al., 2004; Pontius and Cheuk, 2006), however, several of these approaches are based on generalized cross tabulation matrices from which a range of accuracy measures, with similarities to those widely used with hard classifications, may be derived (Doan and Foody, 2007). The construction of accuracy matrixes is an expensive and time consuming process, since it requires the identification of the class at the reference source of information for each sample location. Furthermore, an important limitation of the approaches with error matrixes is that they provide only an estimate of the overall accuracy of the entire classification and do not provide information about the spatial distribution of error. In addition, the assignment of the ground truth classes to the samples is frequently subjective and may change the accuracy information of the map without meaning that it is more or less accurate, mainly when the classification of the reference is made by a photointerpreter. The development of new methods or a combination of methods to assess the classification accuracy continues to be an area of research. The work herein presented is dedicated to: (1) analyze the information provided to the user by the non-specificity uncertainty measures in the context of remote sensing; (2) evaluate if they have similar behaviours when applied to the same possibility distributions; (3) determine if one of them proves to have advantages over the other within the context of image classification; and (4) assess the relation and compatibility between the information provided by the uncertainty measures and the accuracy indexes. Even though uncertainty measures can be computed to the whole image, to perform the last analysis, only the sample objects were used, so that the results were not influenced by the representativeness of the sample used in the accuracy assessment.

2. Non-specificity measures: theoretical background

Two measures of specificity/non-specificity have been developed within the context of possibility theory and fuzzy sets theory: the specificity measure (Sp), proposed by Yager (1982); and the U-uncertainty measure, proposed by Higashi and Klir (1983). The measures of non-specificity are adequate to assess the possibilistic soft classification uncertainty, since they quantify the ambiguity in specifying an exact solution (e.g. Pal and Bezdek, 2000; Yager, 1992). Both these measures can be applied to fuzzy sets and to ordered possibility distributions. For an ordered possibility distribution Π defined over a universal set X, that is, a possibility distribution such that

( ) ( ) ( )Π ≥ Π ≥ ≥ ΠK1 2 nx x x , the specificity measure proposed by Yager is given by

( )max

0

1Sp d

α

α

αΠ =Π

⌠⌡

(1)

where Πα is the α-possibility level of Π , which corresponds to the subset of elements having possibility at

least α , that, in mathematical terms, can be expressed as ( ){ }x xαΠ = Π ≥ α ; where Πα is the cardinality of

Πα , that is, the number of elements of Πα ; and maxα is the largest value of ( )xΠ for every element of the

universal set X, which can be expressed as ( )= Πmax maxx

xα , ∀ ∈x X .

The specificity measure Sp can be used to define a non-specificity measure NSp. This measure is given by equation (2).

( ) ( )NSp 1 SpΠ = − Π (2)

Then,

( )max

0

1NSp 1 d

α

α

αΠ = −Π

⌠⌡

(3)

Integrating equation (3), it is obtained

( ) ( )11

1NSp( ) 1

n

i ii

x xi+

=

Π = − Π − Π ∑ (4)

where n is the number of elements in the universal set and ( )+Π 1nx is assumed to be zero.

Higashi and Klir (1983) proposed a measure of imprecision called U-uncertainty, which can be interpreted as a measure of non-specificity (Pal and Bezdek, 2000; Klir, 2000). For an ordered possibility distribution Π

( ) ( )max

2 max 2

0

U log 1 logd nα

α α αΠ = Π + −∫ (5)

where Πα , Πα , maxα and n have the same meaning as above.

An approximate solution of equation (5) is (Mackay et al., 2003)

( ) ( ) ( ) ( )1 2 1 22

U 1 log logn

i ii

x n x x i+=

Π = − Π + Π − Π ∑ (6)

where ( )+Π 1nx is assumed to be zero.

Properties of both measures can be found in Pal and Bezdek (2000) or Klir (2000). From these properties we stress the following: • For a possibility distribution Π , 0 NSp( ) 1≤ Π ≤ , where the minimum is obtained for

( ) ( )1 2, ,..., 1,0,...,0nx x xΠ = and the maximum for ( ) ( )1 2, ,..., 0,0,...,0nx x xΠ = ;

• For a universal set X and a possibility distributionΠ , ( )U 20 log≤ Π ≤ n , where the minimum is obtained

for ( ) ( )1 2, ,..., 1,0,...,0nx x xΠ = and the maximum value for ( ) ( )1 2, ,..., , ,...,nx x x α α αΠ = , [ ]0,1α∀ ∈ .

In the context of remote sensing, a possibilistic soft classification generates a possibility distribution associated to each spatial unit. Consider, for example, the ordered possibility distribution

( ) ( )Π =1 2 3, 4, , 1;0.8;0.6;0.4c c c c associated to a spatial unit (pixel or object). The degrees of possibility 1, 0.8,

0.6 and 0.4 express the possibility that the spatial unit is occupied by, respectively, classes c1, c2, c3 and c4. The universal set is the set of all classes, that is, the n classes under consideration, which are four in this illustrative example. For this possibility distribution ( )NSp Π and ( )U Π are obtained considering equations

(4) and (6), and are computed as shown in equations (7) and (8).

( ) ( ) ( ) ( ) ( )1 1 1NSp 1 1 0.8 0.8 0.6 0.6 0.4 0.4 0 0.53

2 3 4 Π = − − + − + − + − =

(7)

( )U 1 0.8 log 4 (0.8 0.6) log 2 (0.6 0.4) log 3 0.4log 4Π = − + − + − + =( ) ( ) ( )2 2 2 21 1 log 4 0.8 0.6 log 2 0.6 0.4 log 3 0.4 log 4 1.32− + − + − + = (8)

The analysis developed within this paper requires the comparison of the results obtained with the two measures. This comparison is made considering the normalized U-uncertainty measure nU , given by

( ) ( )n

2

UU

log

ΠΠ =

n (9)

This new metric is used because, on one hand, NSp and U have different ranges and therefore an immediate comparison between the results given by them is not possible. On the other hand, the information given by normalized measures is more easily understandable to the user and therefore their use is more convenient than non-normalized ones. In this way, a value of zero means that there is no ambiguity in assigning only one class to the spatial unit, and therefore there is no uncertainty, and a value of one means that the uncertainty is maximum. For example, for the ordered possibility distribution ( ) ( )Π =1 2 3, 4, , 1;0;0;0c c c c associated to a

spatial unit, where the uncertainty is minimum, ( ) ( )nNSp U 0Π = Π = and for ( ) ( )Π =1 2 3, 4, , 0;0;0;0c c c c , where

the uncertainty is maximum, ( ) ( )nNSp U 1Π = Π = , since no class is adequately assigned to the spatial unit.

The differences in the meaning of the values given by both measures are subsequently analysed in this paper.

3. Data set and case-study area

For the purposes of this research the image data used was a CARTERRA-Geo image (Jacobsen, 2002) obtained by the IKONOS-2 sensor, with a spatial resolution of 1m in the panchromatic mode and a spatial resolution of 4m in the multispectral mode (XS) (Figure 1). Additional acquisition details of the image can be seen in Table 1. The study was performed using the 4 multispectral bands and their geometric correction consisted of its orthorectification. The average quadratic error obtained for the geometric correction was 1.95 m, lower than half the pixel size, which guarantees an accurate geo-referencing. The image covers an area of 81.5 km2 located near the Portuguese coast, and includes regions with different characteristics, such as: built up areas; agricultural fields and forest. The dominant forestry species in the region are eucalyptus and coniferous trees.

Figure 1. Case-study area (RGB 321)

Table 1. Acquisition details of IKONOS image

Date 02 September 2009

Time 11h14m

Sun elevation angle (°) 53.37588

Nominal sun elevation (°) 81.06776

Sun azimuth angle (°) 145.0290

Nominal sun azimuth (°) 50.9100

Radiometric Resolution 8 bits

Dimension (m x m) 10928x7460

Table 2. Map nomenclature.

Continuous residential areas Discontinuous residential areas Public services Industrial activities Road transport Extractive industries Waste dump deposit Building sites Green or leisure areas

Artificial Surfaces

Sport facilities Annual crops Agricultural

Areas Complex cultivation patterns Coniferous forest

Forests Broad-leaved forest Herbaceous vegetation Semi-Natural

Areas Forest clear-cuts

4. Methodology

Very High Spatial Resolution (VHSR) images allow the identification of smaller objects and landscape units, and therefore more fine maps can be produced. With the object based approach the goal is to capture the spatial patterns of the land unit classes and to produce a map, on a 1:10 000 scale. The soft classifier, used to obtain degrees of possibility, comprised an object oriented image analysis based on a fuzzy classification available in the commercial software eCognition. The used methodology includes the following steps: (1) classification of the satellite image using an object oriented fuzzy classification, in order to obtain degrees of possibility that translate the compatibility between the objects characteristics and the considered classes; (2) application of the NSp and Un non-specificity measures to the possibility distributions obtained with the classification; (3) analysis and comparative evaluation of the behaviour of both uncertainty measures; (4) evaluation of the classification accuracy using an error matrix; and (5) comparison between the results obtained with the uncertainty measures and the error matrix. Since the aim of this study is to evaluate the information given by non-specificity measure when applied to soft classifications of remote sensing images and investigate if the uncertainty indexes proposed can be used as indicators of the classification accuracy, the paper focuses mainly on this analysis and only a briefly explanation about the classification methodology is presented.

4.1 Classification

The methodology used to extract the thematic information from the multi-spectral image was: (1) definition of a map nomenclature; (2) image segmentation to create objects; (3) image classification based on a hierarchical model. A classification scheme with sixteen classes was established (see Table 2), based on the CORINE Land Cover nomenclature (Bossard et al., 2000). The rationale behind the development of the nomenclature was two-fold: (1) a nomenclature that is compatible with an established one (in this case the CORINE Land cover) and (2) a nomenclature that matches the spatial resolution of the used satellite imagery. Despite some authors consider that mixing land use and land cover classes should be avoided (e.g. Ottavianelli, 2007) the option in the study was to use a nomenclature that mixes them. In fact, two different areas may have similar land cover but different land use, and by using only the spectral information is not possible to discriminate the land use classes. In addition, mixing them may increase the spectral confusion which will affect the classification accuracy. However, the approach used allows the identification of objects using characteristics like shape information, texture, neighborhood relationships and the operation within a network of a whole bundle of relations to connect image objects, allowing contextual information. The possibility of incorporating shape and context, which are some of the main clues used by a human interpreter, allow the discrimination of some classes that have the same land cover but different human uses. For example, it is possible to construct a hierarchy of classes to identify, the land cover class Artificial Areas in a hierarchical level and, by using the shape attributes, extract the football stadiums in other hierarchical level, where the football stadiums are included (son in the hierarchy) of the Artificial Areas class. Another example is the classification of land cover class ‘Forest’ and land use class ‘Green or leisure areas’. In this situation it is possible to discriminate the objects which are situated within urban fabric and has the same (bio)physical cover on the Earth’s surface as forest, by using the attributes neighborhood relationships and area, and classify them correctly as ‘Green or leisure areas’. The objects extraction was made using the Fractal Net Evolution Approach (FNEA), which can be described as a region merging technique (Baatz and Schape, 2000). The implementation of this algorithm in software eCognition allows a multiresolution segmentation of images. In the present study, four segmentations of the image were done and registered in different levels. The developed classification process includes four hierarchic levels, each one with different scales and different purposes: (1) the first level aims to allow a classification at the pixel level, which will be used as a texture level in the other supra levels, and to classify surface elements like roof materials; (2) the second level aims to allow the classification of objects whose size is related to the scale of interest, which in this case is 1:10 000; (3) the third and fourth levels allow the classification of land units (Gonçalves and Caetano, 2004). The classification process is similar to a decision tree. A decision tree classifier for geographical objects is a hierarchical structure with several levels. At each level a test is applied to one or more attribute values. The developed class hierarchy relies on a hierarchy of objects in which each object knows its own context and its horizontal and vertical neighbours. Each geographical object is completely described by a set of attributes and a class label. The objective of using a hierarchical structure for classifying objects is to gain more comprehensive understanding of relationships between objects at different scales of observation, or levels of detail. Furthermore, the information extracted at each level may be used as context information in the other levels, to improve the classification process in an iterative way. The developed classification rules describe the properties of the classes and were established using fuzzy membership functions defined for the several image attributes such as spectral bands intensities shape, texture, neighbourhood relationships and image object hierarchy. Figure 2 shows a scheme of the classification workflow.

Figure 2. Land cover map workflow.

4.2 Comparative evaluation of non-specificity measures

The classification results are degrees of possibility that translate the compatibility between the characteristics of each object and the characteristics of the classes considered. The NSp and the Un non-specificity measures were applied to the possibility distributions obtained for each object with the classification and a comparison of the results was made. The behaviour of both measures was analyzed considering several aspects, including the sensitiveness of the measures to the variation of the compatibility between the objects characteristics and the classes, and their sensitiveness to the dispersion of possibilities over several classes. The notion of compatibility is expressed by the magnitude of the degrees of possibility that, for each object, are associated to each class. High degrees of possibility correspond to large compatibility, while small degrees of possibility correspond to small compatibility. The term “dispersion” refers to the number of positive possibilities assigned to each object; the larger the number of positive possibilities, the larger the number of possible classes. To further illustrate the behaviour of both uncertainty measures, some additional experiments were made using synthetic values. Given that uncertainty measures are computed for all image objects, the spatial distribution of uncertainty may be obtained and used to visualize the differences between the results obtained with both uncertainty measures.

4.3 Comparison between uncertainty and accuracy indexes

One important aspect in the production of thematic maps from remote sensing data is the evaluation of their accuracy. To determine if the information given by the uncertainty measures may be used as an indicator of the classifier performance and if it is correlated with the results obtained with the classification quality assessment, an error matrix was made based on the information obtained from a sample of 800 points, where

the (i,j) entry is the number of the objects that are class i in the map and class j in the reference. The reference dataset consisted of a stratified random sample of 50 objects per class. Two accuracy indexes were considered, namely the normalized user’s and producer’s accuracy; and two indexes for each uncertainty measure: the mean based indexes, corresponding to the complement of the mean uncertainty per class (

1 NSpI −

and n1 U

I − ) and the low uncertainty indexes, which correspond to the normalized percentage of objects per

class with uncertainty lower than 0.5 (NSp 0.5I < and nU 0.5I < ), corresponding to relatively small uncertainty. Since

in the mean based indexes high uncertainty values are expected to correspond to low accuracy values, and vice-versa, the complement of the uncertainty measures is considered to allow an easy comparison with the accuracy indexes. The low uncertainty indexes were proposed because they are computed using an approach similar to the one used to obtain the accuracy indexes, where the values are obtained counting the number of sample sites corresponding to the several classes on the map and on the ground. The correlation coefficient between the several accuracy and uncertainty indexes was also computed. The considered uncertainty indexes were computed using just the values obtained for the sample objects. This approach was used because error matrixes are generated using only the information obtained for the considered sample and the assignment of the results obtained with the matrix to the whole image classification requires the assumption that a representative sample was used. Even though uncertainty measures can be computed to the whole image, since the objective was to compare the results given by the uncertainty measures with the ones given by the error matrix, only the sample objects were used, in order that the results were not influenced by the sample representativeness.

5. Results and discussion

5.1 Comparative evaluation of non-specificity measures

NSp and Un non-specificity measures were applied to the results of the image classification. Error! Reference source not found. shows the results obtained for some classified objects randomly selected and ordered with increasing NSp values. A similar general tendency can be observed for both measures, since uncertainty increases for distributions

( )Π x with smaller degrees of possibility, although some considerable differences can be found. In general, Un

gives larger uncertainty values than NSp. Equal values are only obtained when there is just one degree of possibility different from zero, that is, only one class is assigned to the object, with any degree of possibility. For some objects, the uncertainty values given by NSp and Un are considerably different. For example, for object 26, for which the three degrees of possibility are 1, 0.97 and 0.92, NSp only gives an uncertainty value of 0.64, while Un gives an uncertainty value of 0.95. Since the software used (eCognition) only provides the three largest degrees of possibility for each object, to analyze how both measures behave when the number of theoretically possible classes increases, even though the non zero values of the possibility distribution are the same, nine classes were considered as theoretically possible with all additional degrees of possibility equal to zero. This means that the dispersion of the classification over all theoretically possible classes’ decreases, since the same number of positive possibilities is obtained over a larger number of theoretically possible classes. The results are shown in Figure

3.

0.00.10.20.30.40.5

0.60.70.80.91.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

NSp Un

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

PossibilityDistributions

UnNSp

Figure 3. In this case, the values of NSp remain unaltered while the values of Un decrease considerably, becoming, in general, smaller than the values obtained for NSp. This shows that Un is sensitive to the classification dispersion over all the classes considered as theoretically possible, while the NSp measure is insensitive to it. That is, Un depends upon the dispersion of possibility over all classes considering that less dispersion means less uncertainty, while NSp only evaluates the uncertainty having in consideration the classes with positive possibility values.

0.00.10.20.30.40.5

0.60.70.80.91.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

NSp Un

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

PossibilityDistributions

UnNSp

Figure 3. Values of NSp and Un determined for objects 1 to 37, considering nine classes as theoretically possible and considering null possibilities from the fourth to the ninth class.

To further illustrate the behaviour of both measures with the classification dispersion, some additional experiments were made using synthetic values. Error! Reference source not found. shows the behaviour of NSp and Un uncertainty measures with possibility distributions translating different degrees of dispersion over all classes but with the same first three degrees of possibility. The obtained results show that the variation of Un with the dispersion is larger than the variation of NSp. For NSp, the difference in the uncertainty values obtained for the possibility distributions two and ten is only 0.17, while for Un it is 0.40. Another aspect of interest is the behaviour of both measures with the compatibility of the classification results to the classes’ ideal characteristics. Error! Reference source not found. shows the behaviour of NSp and Un uncertainty measures for possibility distributions with varying degrees of compatibility to the classes but with the same variation between consecutive possibilities. Notice that the Un uncertainty measure remains unaltered, meaning that it is not sensitive to the compatibility between the objects characteristics and the classes but to the variation of the possibilities over the possibility distribution, while NSp increases with the decrease of the degrees of possibility. Figure 4 shows the behaviour of NSp and Un uncertainty measures with different possibility distributions when all values in the possibility distributions are equal, that is, equal possibilities are assigned to all classes and therefore the confusion between them is maximum. In this situation, the Un uncertainty measure takes the unit value for all possibility distributions, that is, according to this uncertainty measure, the uncertainty is maximum. In this figure, it is also visible that Un uncertainty measure is not sensitive to the possibility values but to the variation of possibilities within the possibility distribution. On the other hand, NSp varies between

0.67 and 1, demonstrating that it is sensitive to the variation of compatibility between the objects characteristics and the classes. These results mean that uncertainty measures NSp and Un provide different information to the user. NSp is more sensitive to the compatibility between the classes and the object characteristics, while Un uncertainty is sensitive to the difference between the consecutive degrees of possibility and not to their absolute value and, therefore, it is also sensitive to the dispersion of the classification over the total number of classes considered theoretically possible.

0.00.20.40.60.81.0

1 2 3 4 5 6 7 8 9 10 11

NSp Un

0.97

1

0.93

1

10.900.870.830.800.770.730.700.67NSp

Un 111111111

0.97

1

0.93

1

10.900.870.830.800.770.730.700.67NSp

Un 111111111

0.1

0.1

0.1

10

0.2

0.2

0.2

9 1187654321

Third ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

00.30.40.50.60.70.80.91

00.30.40.50.60.70.80.91

00.30.40.50.60.70.80.91

0.1

0.1

0.1

10

0.2

0.2

0.2

9 1187654321

Third ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

00.30.40.50.60.70.80.91

00.30.40.50.60.70.80.91

00.30.40.50.60.70.80.91

PossibilityDistributions

UnNSp

Figure 4. Behaviour of NSp and Un uncertainty measures with possibility distributions where all possibilities are equal.

Figure 5 and Figure 6 show the spatial distribution of uncertainty considering four intervals with equal ranges, namely [ [0,0.25 , [ [0.25,0.5, [ [0.5,0.75 and [ ]0.75,1 designated, respectively, by low, low to medium, medium

to high and high level of uncertainty. It should be noticed that, when a maximum of three classes is considered for each object, the spatial distribution of the classification uncertainty given by both measures is considerably different. It was observed that the main differences occur in the interval [ [0.25,0.75. For example, according to

the Un uncertainty measure, 70% of the classified area has uncertainty levels in the interval [ [0.5,0.75 and 6%

inside the interval [ [0.25,0.5, while according to NSp only 18% of the objects are inside the interval [ [0.5,0.75

and 59% inside [ [0.25,0.5.

Figure 5. Spatial distribution of the values obtained for the Un uncertainty measure, considering intervals [ [0,0.25 , [ [0.25,0.5 ,

[ [0.5,0.75 and [ ]0.75,1 .

Figure 6. Spatial distribution of the values obtained for the NSp uncertainty measure, considering intervals[ [0,0.25 , [ [0.25,0.5 ,

[ [0.5,0.75 and [ ]0.75,1 .

The spatial difference of the maps representing the uncertainty obtained with the two measures (Figure 7) shows that the regions with uncertainty smaller than 0.25, according to NSp, are completely included in the regions with uncertainty smaller than 0.25 according to Un. Moreover, these are almost completely coincident, since both occupy an area of approximately 21% of the image. The regions with uncertainty larger than 0.75 according to NSp correspond to 0.4% of the image and are included in the regions with uncertainty larger than 0.75 according to Un, which occupy 2.3% of the image.

Figure 7. Spatial distribution of the difference between values obtained for the NSp an Un uncertainty measure.

A more detailed analysis of the objects with uncertainty values larger than 0.75 for NSp and Un was performed. For NSp, it was observed that the maximum degree of possibility associated to the objects with this level of uncertainty was always smaller than 0.69 and 55% of these objects were only allocated to one class. For Un, it was observed that the maximum degree of possibility associated to the objects was 1.0 and that 78% of the objects were allocated to the three classes. Since dispersion has almost no influence over NSp, the objects with large degrees of possibility to several classes have lower levels of uncertainty according to NSp then according to Un. These results confirm that NSp is highly sensitive to compatibility (meaning more compatibility less uncertainty), while Un is strongly sensitive to dispersion. From the above analysis, it can be concluded that the sensitivity of the Un uncertainty measure to the dispersion of possibilities through all theoretically possible classes means that, according to this measure, the smaller the number of classes with positive possibilities (up to one, since the uncertainty is maximum if all possibilities are zero), the smaller the uncertainty. The sensitivity of the NSp uncertainty measure to the degrees of possibility means that this measure mainly evaluates the compatibility of the objects or pixels characteristics to the classes and not the dispersion. Furthermore, the Un uncertainty measure takes larger values when: (1) all values in the possibility distributions are equal; (2) differences between all consecutive possibilities are small; (3) high possibilities are assigned to all classes considered theoretically possible and (4) all values in the possibility distribution are small. NSp uncertainty measure takes larger values when (1) all values in the possibility distribution are small and (2) when large degrees of possibility are assigned to a large number of classes. Both measures assign low uncertainty to objects when the highest possibility is close to one and the remaining values of the possibility distribution are either zero or close to zero. A considerable difference between the results given by the two measures occurs when very high degrees of membership are assigned to some classes (for instance to three or four), since in these cases Un takes large uncertainty values while NSp takes only moderate values.

5.2 Comparison between uncertainty and accuracy indexes

To evaluate the soft classification accuracy, a ground truth data set containing 800 samples was selected. A stratified random sampling of 50 objects per class was drawn considering the entire image scene. The error matrix of the classification was computed (see Figure 11) and the land cover map produced presented an overall accuracy of 78%. Figure 12 shows the uncertainty indexes

1 NSpI − and

n1 UI − along with the user’s and

producer’s accuracy, ordered with increasing values of 1 NSp

I − and Figure 13 shows the uncertainty indexes

NSp 0.5I < and nU 0.5I < along with the user’s and producer’s accuracy, ordered with increasing values of NSp 0.5I < .

The information provided by the uncertainty measures clearly presents some correlation with the accuracy indexes, despite the fact that some classes present considerable differences between the user’s and producer’s accuracy.

Figure 8. Error matrix of the IKONOS image classification.

Figure 9.

1 NSp−I and 1 nU

I − uncertainty indexes along with the user’s and producer’s accuracy.

Figure 10. NSp 0.5I < and

nU 0.5I < uncertainty indexes along with the user’s and producer’s accuracy.

Artificial classes presented in general better results. Some classes like ‘Extractive Industries’, ‘Industrial Activities’, ‘Public Services’, ‘Sports Facilities’ and ‘Waste Dump Deposit’, were very well identified according to both accuracy indexes and also presented considerably low levels of uncertainty, corresponding to large values of

1 NSpI − ,

n1 UI − , NSp 0.5I < and

nU 0.5I < . It can also be observed that, as expected from the behaviour

of the Un measure, the values of 1 NSp

I − are slightly larger than those obtained for n1 U

I − , however, the classes

presenting higher and lower values for both uncertainty indexes are the same. This tendency is kept for the

NSp 0.5I < and nU 0.5I < indexes, but for some classes the differences are much larger. The information provided by

the mean based uncertainty and accuracy indexes is consistent for almost all classes. The main difference occurs for the class ‘Coniferous Forest’, which showed good results for both the user’s and producer’s accuracy, respectively 94% and 63%, but presented higher levels of uncertainty, corresponding to lower values of the

1 NSpI − and

n1 UI − indexes. A more detailed analysis of the possibility distributions showed that

there is considerable confusion between ‘Broad-leaved Forest’ and ‘Coniferous Forest’. Table 4 shows the sum of the degrees of possibility associated to the several classes with positive possibilities for respectively the ‘Coniferous Forest’ and the ‘Broad-leaved Forest’. It can be seen that for both classes only positive degrees of possibility are obtained for these two classes, and they are very high, meaning that the classifier has some difficulty in differentiating one from the other. The error matrix confirmed that there is significant confusion between ‘Broad-leaved Forest’ and ‘Coniferous Forest’, probably due to the proximity of their spectral signatures. Even though, medium levels of uncertainty were obtained for these classes, and despite the fact that the error matrix also shows that there is some confusion between these two classes, the accuracy indexes show better results, which means that, even though there is some confusion between the classes characteristics, the classifier managed to make a good classification and the uncertainty indexes have reflected mainly the difficulties of the classifier. The results given by the NSp 0.5I < index are very similar to the information given by the two mean based

uncertainty indexes. It should be noticed that this index gives a value for the ‘Coniferous Forest’ considerably close to the results given by the user’s and producer’s accuracy, and presents, in general, results also significantly close to the accuracy indexes. The difference for the ‘Coniferous Forest’ between this index and the previous two occurs because the values of NSp for the objects classified as Coniferous are mainly between 0.4 and 0.5, which gives a value of

1 NSpI − between 0.5 and 0.6. Nevertheless, the great majority of values are

really below 0.5 and therefore the NSp 0.5I < index has a much larger value. Contrarily, the nU 0.5I < index shows

clearly worse results for some classes, such as ‘Broad-leaved Forest’ and ‘Coniferous Forest’. This occurs because Un gave large uncertainty values for these two classes since only three theoretically possible classes were considered and therefore only a few sample objects presented uncertainty values lower than 0.5. To quantify the correlation between the several uncertainty and accuracy indexes, the correlation coefficients between them were computed. The results can be seen in Table 3 and Table 4 show that there is a positive correlation between the uncertainty and accuracy indexes. Higher correlation coefficients are observed between the user's accuracy and all uncertainty indexes.

Table 3. Sum of the degrees of possibility associated to the several classes with positive possibilities.

Classes in the classified map Classes with positive degrees of

possibility Broad-leaved Forest

(sample size: 50 points) Coniferous Forest

(sample size: 50 points) Broad-leaved Forest 45 39

Coniferous Forest 38 49

Table 4. Correlation coefficients between accuracy and uncertainty indexes.

Mean based indexes Low uncertainty based indexes

1 NSpI −

n1 UI − NSp 0.5I <

nU 0.5I <

User’s accuracy 0.73 0.71 0.77 0.53

Producer’s accuracy 0.53 0.50 0.52 0.33

This can be explained by the fact that, with the soft classification, the sample sites are assigned to the classes based on the highest possibility value and the uncertainty value associated to that site is used as an indicator of the likeliness that the assignment is correct. This approach is very similar to the computation of the user's accuracy, which is an indicator of the commission errors, where the sample sites are considered to belong to the class determined by the classification procedure, and it is analyzed if that assignment is true or not. The main difference between these two approaches is that, when the error matrixes are used, the commission errors are evaluated with ground truth, while with the uncertainty indexes they are estimated using the uncertainty measures. The relatively low correlation between the accuracy indexes and the

nU 0.5I < index may be explained by the fact

that only three theoretical classes were considered possible and therefore Un values were relatively large,

which results in the identification of less objects inside the interval [0,0.5]. As illustrated in

0.00.10.20.30.40.5

0.60.70.80.91.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

NSp Un

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.000.110.380.000.000.710.670.170.520.630.850.920.230.000.230.790.000.550.450.000.000.000.000.000.350.000.120.340.000.000.350.000.000.000.000.000.00Third ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fourth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Fifth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Sixth ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Seventh ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Eight ΠΠΠΠ(x)

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.00Ninth ΠΠΠΠ(x)

Second ΠΠΠΠ(x)

Best ΠΠΠΠ(x)

0.240.350.470.200.000.730.740.390.730.660.950.970.260.000.730.970.290.550.970.620.590.980.960.000.810.430.280.640.700.140.360.510.340.300.260.000.00

0.270.370.530.370.270.770.770.530.770.770.981.000.530.370.771.000.530.771.000.770.770.981.000.531.000.770.771.001.000.770.981.000.980.981.000.981.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

0.800.760.690.700.730.590.580.620.550.550.480.480.590.630.500.450.560.500.390.420.410.330.300.470.320.360.340.260.220.270.200.160.130.110.080.020.00Un

NSp 0.850.820.770.730.730.710.710.690.680.660.640.640.640.630.630.620.610.590.560.540.520.510.480.470.460.440.390.380.350.290.260.260.190.170.130.020.00

PossibilityDistributions

UnNSp

Figure 3, if more classes would have been considered theoretically possible, a larger correlation might have been found, since the Un values would decrease. Figure 14 enables the visualization of the correlation between the user’s accuracy and the NSp 0.5I < index.

Perfect correlation would correspond to values in the main diagonal. Considering that several factors may influence the results of the accuracy and uncertainty assessment, an area corresponding to a variation of 20% was considered as the region showing good correlation, and only two classes are not within this area, namely ‘Herbaceous Vegetation’ and ‘Continuous Residential Areas’, both presenting less accuracy than expected from the uncertainty indexes. This may be explained by the fact that, in this case-study area, it was very difficult to decide weather an object should be included in the classes ‘Herbaceous Vegetation’, ‘Annual Crops’ or ‘Complex Cultivation Areas’, since most of these land covers are in backyards and these difficulties may influence the results obtained with the accuracy indexes. The same happens with the class ‘Continuous Residential Areas’, which, in this region, was hard to differentiate from ‘Discontinuous Residential Areas’, since all urban area is quite scattered.

Figure 11. Visualization of the correlation between the user’s accuracy and the NSp 0.5I < uncertainty index. The central diagonal

corresponds to perfect correlation and the extreme diagonal lines delimit the region considered as having good correlation (corresponding to a 20% variation to the sides of the central diagonal).

The results showed that the information of the uncertainty indexes, along with the analysis of the degrees of possibility, allowed to detect the main problems of the classifier performance that were responsible for the final classification results and can be caused by several factors such as: data; classification approach; nomenclature definition; sample protocols for training the classifier and/or testing the classification. A positive correlation between the uncertainty indexes and the user’s accuracy index was obtained, which supports the fact that the uncertainty indexes may be used as indicators of the final classification results. Better results were obtained for artificial classes, whereas for natural classes, such as forest, the results were relatively worse. One possible explanation may be because the spectral resolution of the IKONOS images is lower than that of sensors with smaller spatial resolution (e.g. Landsat-TM), resulting in some limitations in the characterization of this kind of classes which difficults the separability of the different species (Goetz et al., 2003). The results also showed that the uncertainty indexes can not replace the accuracy indexes, but enable the identification of the classification major problems and a previous estimation of the classification accuracy. Therefore, they can be very useful in the classification decision process and may be used as an intermediate step between the classification and the generation of confusion matrixes, enabling the improvement of the classification, for example, reformulation the nomenclature, before evaluating the classification accuracy with confusion matrixes, which are expensive and time consuming.

6. Conclusions

Measures of non-specificity of fuzzy sets and possibility distributions were proposed by Yager (1982) (the specificity measure Sp and its counterpart, the non-specificity measure NSp) and Higashi and Klir (1983) (the U-uncertainty measure). Ricotta (2005) presented these measures as possible ways to analyze the classification uncertainty of remote sensing images, from a theoretical point of view. The more detailed analysis of both measures carried out in this study, and their application to an object soft possibilistic classification, led to the conclusion that these actually have different behaviours. Furthermore, it was concluded that the use of one or the other, together with the definition of the assumptions to apply the U-uncertainty measure, provides different information to the user. A new metric Un, corresponding to the

normalized version of the U-uncertainty measure, was used to enable the comparison of both measures and to establish a variation interval understandable to the user. The presented study showed that: NSp uncertainty measure is sensitive to the absolute values of the possibility distributions, while the Un is only sensitive to the differences between the consecutive degrees of possibility; NSp uncertainty measure evaluates uncertainty considering only the classes to which positive possibilities were assigned, while Un uncertainty measure is sensitive to the dispersion between all classes theoretically considered as possible, even if degrees of possibility equal to zero have been assigned to some of them. For this reason, to assess the dispersion or the variation between the consecutive degrees of possibility, the Un uncertainty measure should be used and to assess the compatibility between the objects characteristics and the classes, NSp should be adopted. To determine if the uncertainty measures may be considered as indicators of the classifications accuracy, a comparison was made between the results given by the uncertainty measures and the information obtained from an error matrix. This analysis showed that there is positive correlation between the information given by the uncertainty measures and the user’s and producer’s accuracy but mainly with the user’s accuracy, showing that they may be used as indicators of the classifier performance and consequently as indicators of the final classification results, but they can not replace the accuracy indexes. This study showed that uncertainty measures are an easy, non time consuming, and non-expensive approach to perform a wide range of analysis operations, giving a preliminary evaluation of the classification difficulties. Also, they are also not dependent from the subjectiveness of the reference classification, which may influence the accuracy results. These can therefore be used as an intermediate step between the classification and the generation of error matrixes, contributing to an iterative improvement of the classification process, identifying, for example, regions and classes with higher levels of uncertainty and enabling the implementation of measures to reduce the classification uncertainty, which will likely result in accuracy improvement. This study is a first approach to the application of uncertainty measures to evaluate the classification performance. Even though additional studies are necessary to identify the sensitiveness of these measures to the classification techniques, variations in nomenclature or images with different characteristics, the study reported here shows that the application of these measures to estimate classification accuracy seems to be promising, providing valuable information to the user, and deserves therefore further attention.

References

ARORA, M. K., FOODY, G. M., 1997, Log-linear modelling for the evaluation of the variables affecting the accuracy of probabilistic, fuzzy and neural network classifications. International Journal of Remote Sensing, 18, 785–798.

BAATZ, M., SCHAPE, A., 2000, Multiresolution segmentation – an optimization approach for high quality multi-scale image segmentation. In: Strobl, J. et al. (Eds.): Angewandte Geographische Informationsverarbeitung XII. Beitrage Zum AGIT – Symposium Salzburg 2000, Karlsruhe, Herbert Wichmann Verlag, pp. 12-23.

BASTIN, L., 1997, Comparison of fuzzy c-means classification, linear mixture modeling and MLC probabilities as tools for unmixing coarse pixels. International Journal of Remote Sensing, 18, 3629-3648.

BINAGHI, E., BRIVIO, P.A., GHEZZI, P., RAMPINI, A., 1999, A fuzzy set-based accuracy assessment of soft classification. Pattern Recognition Letters, 20, 935-948.

BOSSARD, M., FERANEC, J., OTAHEL, J., 2000, CORINE Land Cover Technical Guide – Addendum 2000. Technical report No 40, Copenhagen (EEA).

BROWN, M., LEWIS, H., GUNN, S., 2000, Linear spectral mixing models and support vector machines for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 38, 2346–2360.

DOAN, H. T. X., FOODY, G. M., 2007, Increasing soft classification accuracy through the use of an ensemble of classifiers. International Journal of Remote Sensing, 28, 4609-4623.

FOODY, G. M., 1996, Approaches for the production and evaluation of fuzzy land cover classifications from remotely-sensed data. International Journal of Remote Sensing, 17, 1317-1340.

FOODY, G. M., 2001, Estimation of sub-pixel land cover composition in the presence of untrained classes. Computers & Geosciences, 26, 469-478.

FOODY, G. M., 2002, Status of land cover classification accuracy assessment. Remote Sensing of Environment, 80, 185-201.

GONÇALVES, L., CAETANO, M., 2004, Classificação das imagens do satélite IKONOS utilizando uma abordagem orientada por objectos, Actas da Conferência de Cartografia e Geodesia, (ed. L. Bastos e J. Matos) Lidel, Lisboa, pp. 287-298.

HIGASHI, M., KLIR, G., 1983, On measure of fuzziness and fuzzy complements. International Journal of General Systems, 8, 169-180.

IBRAHIM, M. A., ARORA, M. K., GHOSH, S. K., 2005, Estimating and accommodating uncertainty through the soft classification of remote sensing data. International Journal of Remote Sensing, 26, 2995-3007.

JACOBSEN, K., 2002, Mapping with IKONOS images. 22 EARSeL symposium, Prague, Czech Republic, 4-6 June 2002.

KLIR, G., 2000, Measures of uncertainty and information. In: Fundamentals of Fuzzy Sets, Dubois, D., Prade, H. (eds.), pp. 439-457. (The Handbook of Fuzzy Sets Series, Kluwer Acad. Publ).

KLIR, G., 2004, Generalized information theory: aims, results and open problems. Reliability Engineering and Systems Safety, 85, 21-38.

KLIR, G., YUAN, B., 1995, Fuzzy sets and Fuzzy Logic-Theory and Applications. (Prentice Hall PTR, New Jersey).

LU, D. AND WENG, Q., 2007, A survey of image classification methods and techniques for improving classification performance. International Journal of Remote Sensing, 28, 823-870.

MACKAY, D., SAMANTA, S., AHL, D., EWERS, B., GOWER, S., BURROWS, S., 2003, Automated Parameterization of Land Surface Process Models Using Fuzzy Logic. Transaction in GIS, 7, 139-153.

MASELLI, F., RODOLFI, A., CONESE, C., 1996, Fuzzy classification of spatially degraded Thematic Mapper data for the estimation of sub-pixel components. International Journal of Remote Sensing, 17, 537-551.

MASELLI, F., CONESE, C., FILIPPIS, T.D., NORCINI, S., 1995, Estimation of forest parameters through fuzzy classification of TM data. IEEE Transactions on Geoscience and Remote Sensing, 33, 77–84.

OKI, K., UENISHI, T.M., OMASA, K., TAMURA, M., 2004, Accuracy of land cover area estimated from coarse spatial resolution images using an unmixing method. International Journal of Remote Sensing, 25, 1673–1683.

OTTAVIANELLI, G., 2007, Synthetic aperture radar remote sensing for landfill monitoring.PhD thesis, University of Cranfield.

PONTIUS, R.G., CHEUK, M.L., 2006, A generalized cross-tabulation matrix to compare soft-classified maps at multiple resolutions. International Journal of Geographical Information Science, 20, 1–30.

PAL, N., BEZDEK, J., 2000, Quantifying different facets of fuzzy uncertainty. In Fundamentals of Fuzzy Sets, Dubois, D., Prade, H. (eds.), pp. 459-480. (The Handbook of Fuzzy Sets Series, Kluwer Acad. Publ).

RICOTTA, C., 2005, On possible measures for evaluating the degree of uncertainty of fuzzy thematic maps. International Journal of Remote Sensing, 26, 5573-5583.

TSO, B., MATHER, P.M., 2001, Classification Methods for Remotely Sensed Data. (Taylor and Francis, London).

WANG, F., 1990, Improving remote sensing image analysis though fuzzy information representation. Photogrammetric Engineering and Remote Sensing, 56, 1163-1169.

WOODCOCK, C.E., GOPAL, S., 2000, Fuzzy set theory and thematic maps: accuracy assessment and area estimation. International Journal of Geographical Information Science, 14, 153–172.

YAGER, R., 1982, Measuring tranquility and anxiety in decision making: an application of fuzzy sets. International Journal of General Systems, 8, 139-146.

YAGER, R., 1992, On the specificity of a possibility distribution. Fuzzy Sets and Systems, 50, 279-292.

ZHANG, J., FOODY, G. M., 2001, Fully-fuzzy supervised classification of sub-urban land cover from remotely sensed imagery: statistical and artificial neural network approaches. International Journal of Remote Sensing, 22, 615-628.