Interest Point Detection for Hyperspectral Imagery

Leidy P. Dorado-Munoza, Miguel Velez-Reyesa,Badrinath Roysamb, and Amit Mukherjeeb

Center for Subsurface Sensing and Imaging SystemsaLaboratory for Applied Remote Sensing and Image Processing, University of Puerto Rico at

Mayaguez, Mayaguez PR.00681-9048, USA.bDepartment of Electrical, Computer and Systems Engineering, Rensseleaer Polytechnic

Institute, Troy NY. 12180-3590, USA.

ABSTRACT

This paper presents an algorithm for automated extraction of interest points (IPs)in multispectral and hyper-spectral images. Interest points are features of the image that capture information from its neighbours and theyare distinctive and stable under transformations such as translation and rotation. Interest-point operators formonochromatic images were proposed more than a decade ago and have since been studied extensively. IPs havebeen applied to diverse problems in computer vision, including image matching, recognition, registration, 3Dreconstruction, change detection, and content-based image retrieval. Interest points are helpful in data reduction,and reduce the computational burden of various algorithms (like registration, object detection, 3D reconstruc-tion etc) by replacing an exhaustive search over the entire image domain by a probe into a concise set of highlyinformative points. An interest operator seeks out points in an image that are structurally distinct, invariant toimaging conditions, stable under geometric transformation, and interpretable which are good candidates for in-terest points. Our approach extends ideas from Lowe’s keypoint operator that uses local extrema of Difference ofGaussian (DoG) operator at multiple scales to detect interest point in gray level images. The proposed approachextends Lowe’s method by direct conversion of scalar operations such as scale-space generation, and extremepoint detection into operations that take the vector nature of the image into consideration. Experimental resultswith RGB and hyperspectral images which demonstrate the potential of the method for this application and thepotential improvements of a fully vectorial approach over band-by-band approaches described in the literature.

Keywords: Interest Points, Local Descriptors, SIFT Descriptors

1. INTRODUCTION

Interest point detection is usually the first step in image processing tasks such as object recognition, image regis-tration, image matching, and change detection since these methods need the extraction of salient and distinctivestructures or points in an image. Interest points should be invariant and stable under geometric transformations,have a well defined position in image space, and the neighborhood around the interest point is highly informativeabout the local structure reason for which interest points are helpful in dimensionality reduction.

Interest points are detected by operators which detect intensity changes and use the autocorrelation functionof the signal within a neighborhood to find regions of maximum change. Other detectors add scale invariance likethe methods reported by Lindberg,1 Mikolajczyk,2 and Lowe.3 The Lowe’s operator searches local extrema inthe scale-space of an image generated by Gaussian Smoothing. Lowe’s detector, besides being invariant to scaleand affine transformations, performs feature extraction more efficient than others and identify a large numberof features allowing reliable feature matching through a considerable range of affine distortions. The operatorsmentioned before have shown great performance on monochromatic images but their extension to hyperspectralor multispectral images is limited because structural information of these images is not limited to a single band.

Contact information:leidy.dorado@ece.uprm.edu,m.velez@ieee.org, Telephone: 787-832-2825

Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV,edited by Sylvia S. Shen, Paul E. Lewis, Proc. of SPIE Vol. 7334, 73340O · © 2009 SPIE

CCC code: 0277-786X/09/$18 · doi: 10.1117/12.819487

Proc. of SPIE Vol. 7334 73340O-1

Rather it is distributed over several bands and it is often difficult to map a higher dimensional image accuratelyonto a plane by preserving all the necessary structural information. In addition, the computational cost andtime invested in the processing of these images is high, therefore, the research community of remote sensingand computer vision are investigating methods that permit them to perform a good detection of features withlow computational cost. An extension of Lowe’s Detector for Vector images was proposed by Mukherjee et.al.4

In this approach, the multi-scale representation is carried out for each band of hyperspectral image, then themulti-band information is combined by means of a function that maps the information of each band of the imageinto a scalar image, which is used to detect extremes and in turn to detect interest points.

In this paper a new extension of Lowe’s detector for vector images is proposed. Our approach is inspiredtotally in Lowe’s approach but fully takes into consideration the vector nature of HSI. The first modificationis performed in the representation of scale-space, which is generated by anisotropic diffusion for hyperspectralimages.5 Other modification is related with the location of the local extreme, which is performed by means of acomparison of vector-pixels instead of comparison between scalar-pixels (original approach), using lexicographi-cal vector ordering,6,7 and finally second fundamental form concept8,9 is used as an extension of Hessian Matrixfor computing principal curvatures, which are used for rejecting poorly defined peaks in the scale space function.

The rest of this paper is organized as follow: Section 2 introduces relevant fundamentals of Lowe’s detector forgrayscale images as well as the approach for hyperspectral images proposed by Mukherjee.4 Section 3 presentsthe proposed approach to extend Lowe’s detector to vector images. Section 4 presents some experiments andresults obtained with approach for vectorial images, such as RGB and hyperspectral images. Section 5 presentsconclusions.

2. PREVIOUS WORK

In every process of interest point detection, the first step is to identify locations and scales of distinctive andstable features. The searching of these points is performed across a scale space representation of the image.3

A scale-space representation is a multi-scale representation of image data, obtained by embedding any givensignal into a one-parameter family of derived signals.1 The image family is parameterized by a scale parameterand contains at coarse levels, simplified structures corresponding to more complex structures located at fine scales.

Witkin10introduced a methodology to construct a linear scale-space representation by convolution of a Gaus-sian kernel with the image:

L(x, y) = G(x, y, σ) ∗ u(x, y) (1)

where u(x, y) is original image and G(x, y, σ) is the Gaussian kernel with σ as parameter given by:

G(x, y, σ) =1

2πσ2e−(x2+y2)/2σ2

(2)

In general, this is the methodology used by most operators that find interest points in images and it is usedby Lowe3 for identifying locations and scales that can be assigned to the features, under differing views of thesame object.

Proc. of SPIE Vol. 7334 73340O-2

2.1 Lowe’s Approach for Grayscale Image

The methodology of Lowe proposes to perform detection of interest points and to compute a descriptor oflocal region around of each interest point. Initially, Lowe’s Operator detects interest points using a cascadefiltering approach and the scale-space function is used to ensure that interest point locations are invariant toscale changes. The scale-space representation is performed by means of Difference of Gaussian function (DoG),which is computed from the difference of two nearby scales separated by a constant multiplicative factor k:

DoG(x, y, σ) = L(x, y, kσ) − L(x, y, σ) (3)

where L(x, y, σ) and L(x, y, kσ) are smoothed images that are computed according to equation (9). The differ-ence of Gaussian function provides a close approximation to the scale-normalized Laplacian of Gaussian studiedby Lindeberg,11 which is required for true scale invariance.

Once DoG(x, y, σ) is computed, the extreme detection is performed by means of comparison among currentsample point and its 26 neighbors located in the current scale (8 neighbors) and scales of above and below(18 neighbors). If the intensity value of the sample pixel is a minimum or maximum, the pixel is consideredinterest point candidate. For each extrema point, the location is interpolated by 3D quadratic function, specifi-cally a Taylor expansion of the scale space function DoG. More detail explanation can be found in Lowe’s paper.3

The next step, is to eliminate unsuitable extrema. So a threshold for difference function DoG is introducedto eliminate points with low contrast, and a criterion for edge or non-edge points is used for eliminating pointswith a strong response along edges but with a location poorly defined. Thus, the principal curvature of the DoGis used to detect low contrast points. The principal curvatures are proportional to the eigenvalues of Hessianmatrix computed at the location and scale of the extrema points

H(DoG(x, y, σ)) =

[∂

∂x2 DoG ∂∂xy DoG

∂∂yxDoG ∂

∂y2 DoG

](4)

The sum and product of the Hessian eigenvalues can be calculated from the trace and determinant of HessianMatrix H:

Tr(H) =∂

∂x2DoG +

∂

∂y2DoG = α + β,

Det(H) =∂

∂x2DoG

∂

∂y2DoG − (

∂

∂xy)2DoG = αβ

(5)

where α and β are the maximum and minimum eigenvalues of H. Let r = α/β the ratio between the eigenvaluesα and β of H, then

Tr(H)2

Det(H)=

(α + β)2

αβ=

(rβ + β)2

rβ2=

(r + 1)2

r(6)

The following test is performed to check that the ratio of principal curvatures is below some threshold r:

ifTr(H)2

Det(H)<

(r + 1)2

rthen extrema is a Interest Point (7)

This methodology is summarized in Figure 1.

Proc. of SPIE Vol. 7334 73340O-3

22

2

2

2

),(DoGDoG

yxDoG

xDoG

yxDoGH

-Increasing

ofScales

2yDoG

xyDoG

rr

HDetHTr 22 )1(

)()(

-Increasing

ofScalesOriginal Image

Local Maxima Pixel

Threshold around of each LocalMaxima Pixel

Generation of Scale SpaceGaussian Smoothing

Difference of GaussiansDoG

g g

Gaussian Smoothing

Interest Points

Figure 1. General Scheme for Interest Point detection proposed by David Lowe for grayscale images

2.2 Mukherjee et.al extension for Hyperspectral Images

An approach to extend Lowe’s operator to hyperspectral images was proposed by Mukherjee et.al4 and sum-marized in Figure 2. Initially, PCA (principal component analysis) is used for reducing unstable interest pointsarising from noise or spatial irregularities and to perform a fast implementation of Difference-of Gaussians func-tion on hyperspectral data. In addition, for the selection of a subset of projections from all principal components,the authors propose a methodology where components whose eigenvalues span more than 5% of the data varianceare retained, and all the remaining components that are acceptably smooth in a sense defined by (8) are retainedtoo.4 The metric to quantify the smoothness of ith component is:

Snxn =std(Ai)

std(Ai − Ai)(8)

where Ai is the minimum mean absolute error estimate in the local nxn neighborhood of each pixel, and std(·)is the standard deviation. More detailed information can be found in paper.4 Once the hyperspetral data havebeen reduced to some PCA projections, the generation of scale space representation is performed on each PCindividually, and is discretized as follow:

σs ∈ {ks}; k = 21/3, s ={−1, 0, 1, 2, 3...,

log(smax)1/3log2

}(9)

This discretization is motivated by Lowe3 and the factor of 3 in (9) implies that each octave of scale is sampledat 3 sample points, smax is specified according to the desired scale interest point. The next step is the choice ofa combining function fl: RL → R+ that maps the Difference-of-Gaussians values for each PC component into ascalar image, which is used to locate the extrema points of similar manner that Lowe’s approach. Four combiningfunctions were evaluated: L1-norm, L1-top50, earth movers distance12 and diffusion distance13 by Mukherjee.In our implementation we used euclidean distance. Flowchart of the methodology proposed by Mukherjee et.al4

is showed in Figure 2:

Proc. of SPIE Vol. 7334 73340O-4

Increasingof

Increasingof

Scales -Comp. 1 Comp. 1 Comp. 1

Comp. 2

C

Scales

Original Image

Comp. 2 Comp. 2

Comp. M

Increasingof

Generation of Scale SpaceDifference of Gaussians

-Comp. M

Comp. M

Increasingof

Scales

PCA projectionScales

Function for combiningDoG responses along

l di iGeneration of Scale SpaceGaussian Smoothing

DoG

2

2

2 DoGDoG

spectral dimension

2

22

2

),(

yDoG

xyDoG

yxxyxDoGH

rHTr 22 )1()(

Differentscales

Local Maxima Pixel

Interest Points

rHDet )(

Threshold around of each local Maxima Pixel

Figure 2. General Scheme for Interest Point detection proposed by Mukherjee et.al. for hyperspectral images

3. A VECTORIAL APPROACH FOR DETECTION OF INTEREST POINTS

The proposed IP detection methodology is described in Figure 3. The approach extends Lowe’s interest pointdetection to vector images (i.e. multispectral and hyperspectral images). Initially, a pre-processing step fordimensionality reduction is performed using SVD Band Subset Selection.14–16 Next we describe how scale-spacerepresentation, local extreme detection and Hessian Matrix are extended to vector images.

Increasingof

Scales

Increasingof

Scales

Diff f dj t lOriginal Image

-

Scale Space Representationby Anisotropic Diffusion

Difference of adjacent scalesDoSBand Subset

Selection

MM 22

Mi

Mi

M

i

iM

i

i

DoSDoSyx

DoSx

DoS

II 221

2

12

2

Scale t+1Increasing

ii yxy 12

1

rIID

IITr 22 )1(

Scale t

Scale t-1

ofScales

Local Maxima Pixel vectorVector Ordering

rIIDetInterest Points

Threshold around of eachlocal Maxima Pixel vector

Figure 3. General Scheme of our approach for detecting interest points in vector images

Proc. of SPIE Vol. 7334 73340O-5

3.1 Generation of Scale Space with Anisotropic Diffusion

Lowe’s methodology establishes that the first stage in the interest point detection is generation of scale spacerepresentation by means of Gaussian smoothed filtering and difference-of-gaussians. In this research, the scale-space representation is carried out by means of Anisotropic Diffusion to improve the detection of interest points,taking advantage of the fact that anisotropic diffusion smoothes uniform areas and enhances images edges. Thelinear diffusion equation is given by:

∂

∂tu(x, y) = ∇ · (c∇u(x, y)) (10)

with the original image used as initial condition u(t = 0) = u0 and, c is the constant diffusion coefficient. Inlinear scale-space, the locations of boundaries at coarse scale are shifted from their true locations, this is dueto gaussian blurring, which does a homogeneous smoothing across all the image without considering the naturalboundaries, hindering edge detection and distinctive features detection such as interest points. This problemcan be addressed by using anisotropic diffusion, which sharpens natural edges of the image and smooths theirinternal regions. The use of Anisotropic Diffusion on image processing was proposed by Perona-Malik,17 whereby using a nonlinear diffusion coefficient function c it is possible to achieve a blurring in separated regionswithout interaction among them and in addition, it can satisfy homogeneity and isotropy criterions. c becomesa nonlinear function of the image gradient | ∇u | and its values ranges between 0 ≤ c(x, y, t) ≤ 1.The extension of Anisotropic Diffusion for vector valued images was proposed by Weickert18 and it is given by(11):

∂

∂tui(x, y) = ∇ ·

⎡⎣c

⎛⎝ M∑

j=1

|∇uj(x, y, t)|2⎞⎠∇ui

⎤⎦ , i = 1, ...,M (11)

The implementation of anisotropic diffusion for vector images such as hyperspectral images is computationallyexpensive. A way to solve this is to use semi-implicit discretization of equation (11) as was proposed by Duarte.5

In this approach the computational cost added by the solution of a system of linear equations in each iterationis compensate by fast linear solvers and approximation schemes such as Additive Operator Splitting and others.We have used the Duarte’s approach for generating a scale-space representation and once the smoothed imagesare generated, the difference between adjacent scales (DoS) is performed and this new function is used to findextrema points.

3.2 Local Extrema Pixel-Vector

Next stage is the determination of extrema points in the DoS function. Since in HSI images a pixel is avector, we use methods of vector ordering for sorting vectors in a neighborhood of 3x3x3 to find extremas.Our methodology is based on lexicographical ordering,7 where a maximum vector pixel in a neighborhood isdetermined by comparison of each pixel-vector with its 26 neighbors in the same scale and adjacent scales(above and below scales). Equation (12) corresponds on the rule that is followed for determining the maximapixel-vector.

a < b

⎧⎨⎩

a1 < b1 ora1 = b1 and a2 < b2 ora1 = b1 and a2 < b2 ... aM < bM

(12)

where a and b are vector pixels with M components and the determination of maxima vector depend on itscomponents.

Proc. of SPIE Vol. 7334 73340O-6

3.3 Accurate Extrema LocalizationOnce the maxima pixel-vector are determined, the accurate location is performed by use of thresholds andprincipal curvature ratio. In this case, the vector image of difference of scales is a regular surface in RM , whereM is the number of spectral bands of the image:

DoSi(x, y) = (DoS1(x, y), DoS2(x, y), ...DoSM (x, y)) (13)

Thus, we should establish the directions of curvature of this surface for determining a threshold that permitus to reject pixel-vector with a strong response along of the edges but poorly located, as Lowe proposed. So,with Riemanian geometry, the extension of the Hessian Matrix is the Second Fundamental Form8 which encodesthe information about how the normal vector to the surface varies as one moves in different directions on thesurface and is given by:

II =

[ ∑Mi=1

∂2

∂x2 DoSi

∑Mi=1

∂2

∂x∂y DoSi∑Mi=1

∂2

∂x∂y DoSi

∑Mi=1

∂2

∂y2 DoSi

](14)

The principal curvatures of the surface, can be derived from eigenvalues of II(DoS) and to perform a similaranalysis as described in Section 2.1 for carrying out the accurate location of interest points, by using:

ifTr(II)2

Det(II)<

(r + 1)2

rthen extrema is a Interest Point (15)

where the value of r is equal to value used in Mukherjee’s approach.

4. EXPERIMENTS AND RESULTSThe experiments are focused on the evaluation of our approach for detection of interest points. We have workedwith vector images specifically RGB images and hyperspectral images. Evaluation of the interest points foundby our approach was performed by means of visual comparison with interest point found using Mukherjee’sapproach.

4.1 Experiments with RGB ImagesThe first image is a 256x320 RGB image from the MATLAB image database. Figure 4 shows the original imageand the Interest Points (IPs) found by Mukherjee, and the vector approach. The number of IPs found by thevector approach is larger and the selected points are better located since they are closer to edges and othersalient spatial features.

4.2 Experiments with Hyperspectral ImagesThe first hyperspectral image studied is from the Enrique reef area in southwestern of Puerto Rico. This imagehas a size of 167x372 pixels and 128 bands. Fourteen bands were selected using SVDSS.14–16 Figure 5 presentsIPs found by the vector approach and Mukherjee’s approach. Again the vector approach resulted in more interestpoints and IPs closer to the image edges.

The next hyperspectral image studied was taken by the AISA Eagle sensor over another area in southwesternof Puerto Rico. This is an image of 400x380 pixels and 128 bands. SVD band subset selection was used to select14 bands. Figure 6 shows the IPs found by the two approaches. Once again, the evaluation of interest pointsfound by vector approach is performed from a visual view point and by means of comparison with interest pointsfound by Mukherjee’s approach. We can see that our approach selected more interest points and that thesepoints were primarily over the coast and reef mangroves. In our opinion those are better points for use in imageregistration.

Proc. of SPIE Vol. 7334 73340O-7

C.

a) c)

Figure 4. Interest Points found by Our vectorial approach and Mukherjee’s Approach. a) Original Image. b) ProposedApproach, 3405 IPs were found. c) Mukherjee’s Approach, 1197 IPs were found.

Figure 5. Interest Points found by Our vectorial approach and Mukherjee’s Approach. a) Original Image. b) ProposedApproach, 5102 IPs were found. c) Mukherjee’s Approach, 971 IPs were found.

5. CONCLUSIONS

In this paper, we presented initial results of a new approach for interest point detection in vector images such asmultispectral and hyperspectral. The proposed method is a direct extension of the SIFT operator proposed byLowe to vector images with no simplification. The proposed method is visually compared with another extensionto Lowe’s approach proposed by Mukherjee. Work to develop a more quantitative approach is under way. Thevector approach selects more interest points and the selected points are closer to edges and other features thatare more likely to be used as interest points in image processing tasks such as image registration.

ACKNOWLEDGMENTS

Dr. Velez-Reyes and Ms. Dorado were supported primarily by the Department of Defense. All participantsreceived partial support from the Bernard M. Gordon Center for Subsurface Sensing and Imaging Systems, underthe Engineering Research Centers Program of the National Sciences Foundation (Award Number EEC-9986821)

REFERENCES[1] Lindeberg, T., “Feature detection with automatic scale selection,” Int.J.Comput.Vision 30(2), 79–116

(1998).[2] Mikolajczyk, K. and Schmid, C., “Scale and invariant interest point detectors,” Int.J.Comput.Vision 60(1),

6386 (2004).[3] Lowe, D., “Distinctive image features from scale-invariant keypoints,” Int.J.Comput.Vision 60(2), 91–110

(2004).[4] Mukherjee, A., Velez-Reys, M., and Roysam, B., “Interest points for hyperspectral image data,” IEEE

Trans. on Geosciences and Remote Sensing 47(3), 748–760 (2009).

Proc. of SPIE Vol. 7334 73340O-8

a C)

Figure 6. Interest Points found by Our vectorial approach and Mukherjee’s Approach. a) Original Image. b) ProposedApproach, 4380 IPs were found. c) Mukherjee’s Approach, 2506 IPs were found.

[5] Duarte-Carvajalino, J., Castillo, P., and Velez-Reyes, M., “Comparative study of semi-implicit schemes fornon-linear diffusion in hyperspectral imagery,” IEEE Trans. on Image Processing 16(5), 1303–1314 (2007).

[6] Apotula, E. and Lefevre, S., “On lexicographical ordering in multivariate mathematical morphology,” Pat-tern Recogn. Lett. 29(2), 109–118 (2008).

[7] Apotula, E. and Lefevre, S., “A comparative study on multivariate mathematical morphology,” PatternRecogn. Lett. 40(11), 2914–2929 (2007).

[8] Hameiri, E. and Shimshoni, I., “Estimating the principal curvature and the darboux frame from real 3drange data,” Proc.3 Data Processing Visualization and Transmission, 258–267 (2002).

[9] Taubin, G., “Estimating the tensor of curvature of a surface from a polyhedral approximation,”Proc.Comput.Vision,, 902–907 (1994).

[10] Witkin, A., “Scale-space filtering,” Proc.8th.Int. Conference on Artificial Intelligence,, 1019–1022 (1983).[11] Lindeberg, T., “Scale-space theory: A basic tool for analysing structures at different scales,” J. Applied

Statistics 21(2), 224–270 (1994).[12] Rubner, Y., Tomasi, C., and Guibas, L., “The earth mover’s distance as a metric for image retrieval,” Int.

J.Comput. Vision 40(2), 99–121 (2000).[13] Ling, H. and Okada, K., “Diffusion distance for histogram comparison,” IEEE Compu.Society

Conf.Comp.Vison Pattern Recognition 1, 246–253 (2006).[14] Velez-Reyes, M. and Jimenez, L., “Subset selection analysis for the reduction of hyperspectral imagery,”

Proc. IEEE Int.Geosciences and Remote Sensing Symposium, 1577–1581 (1998).[15] Velez-Reyes, M., Linares, D., and Velazquez, H., “Comparison of matrix factorization algorithms for band

selection in hyperspectral imagery,” Proc. SPIE 4049, 288–297 (2000).[16] Velez-Reyes, M., Linares, D., and Velazquez, H., “Comparison of principal-component-based band selection

methods for hyperspectral imagery,” Proc. SPIE 4541, 361–369 (2001).[17] Perona, P. and Malik, J., “Scale-space and edge detection using anisotropic diffusion,” IEEE Trans. on

Pattern Analysis Machine Intelligence 12(7), 629–639 (1990).[18] Weickert, J. and Brox, T., “Diffusion and regularization of vector- and matrix-valued images,” Contemporary

Mathematics 313, 251–268 (2002).

Proc. of SPIE Vol. 7334 73340O-9