Twenty Second National Radio Science Conference (NRSC 2005) -1 7 ,

March 15-17,2005 Cairo - Egypt

A New Super-Resolution Image Reconstruction Algorithm Based On Wavelet Fusion

S. E. ECKhamy*, Fellow IEEE , M. M. Hadhoud*, Member IEEE, M. I. Dessoukf, l3. M. Salam', and E'. E. Abd El-Samie'

*Department of Electrical Eng., Faculty of Engineering, Alexandria University, Alexandria 21544, EGYPT. ,

Dept. of Inform. Tech., Faculty of Computers m d Information, Menoufm Unfv., 32511, Shebin Ekom , EGYPT.

# Dept of Electronics and Elect. Communications, Fac. of Electronic Eng., Menoufia Univ., 32952, Menouf, EGYPT.

E-mails: tlkhamv@icee.orp, mmhadhoud@mufic.com , fathi sayed@whoo.com

Abstract d

In this paper, a wavelet based image super-resolution reconstruction algorithm is presented. The super-resolution image reconstruction problem is known to be an ill-posed inverse problem. The objective of the suggested algorithm is to remove this ill posedness. The suggested algorithm is based on breaking the problem into four consecutive steps, a registration step, a multi-channel maximum entropy restoration step, a wavelet based image hsion step and finally a maximum entropy hage interpolation step. The objective of the wavelet firsion step is'to integrate the data obtained from each output h m the multi-channel restoration step into a single image, which is then interpolated using the maximum entropy interpolation scheme. The paper explains the implementation of each step of the suggested algorithm. The simulation results indicate that the proposed algorithm has succeeded to obtain a high-resolution (HR) image from multiple degraded observations with a high PSNR. The computation time of the proposed algorithm is small.

.

Keywords:

Maximum Entropy Restoration, Image Fusion, Multi-Channel Restoration, Maximum Entropy Interpolation, Toplitz to circulant approximation.

I. Introduction

Super-resolution reconstruction of images is a very important field of image processing. The development of this field has been motivated by the requirement to have high-resolution images from degraded observations obtained by multiple sensors. By super-resolution reconstruction algorithms we can obtain a single high-resolution image either from several degraded still images or from several degraded multifiames. There are numerous applications of super-resolution image reconstruction such as remote sensing, medical imaging, satellite imaging and High Definition Television (HDTV).

The super resolution reconstruction problem is 8n ill-posed inverse problem having matrices of very large dimensions. This special nature of the problem forces most image super- resolution algorithms to have 811 iterative nature [1-11]. These algorithms aim at reducing the computational complexity of the matrix inversion processes involved in the sokution by using successive approximation methods for the estimation of the high- resolution image. Although these algorithms avoid the matrix inversion process, they are still time consuming [12]. The first treatment to the supper-resolution reconstruction problem was an iterative frequency domain

.

Twenty Second National Radio Science Conference (NRSC 2005) -1 March 15-17,2005 Cairo - Egypt

tffatment since khe Fourier transform has superior properties for transnational shifts between observations [l-31. The maximum a posteriori (MAP) estimation algorithm has also bepn implemented in the field of image super- resolution [4-6]. Some other set theoretic approaches have been also presented [3,7].

In this paper, we present wavelet based maximum entropy approach for the super-resolution image reconstruction problem. This approach is based on breaking the problem into three consecutive steps to avoid the large dimension complexity. The image fusion technique is used in our suggested appmach as a tool to integrate the i n f o d o n obtained from all the outputs of the multi-channel restoration step into a single image. This image is then interpolated to give a high-resolution image. The concept of maximum entropy is exploited in both the multi-channel restoration step and the interpolation step to obtain a high-resolution image with the maximum amount.of information

The paper is organized as follows: in section I1 the multi Sensor image degradation model is presented. Section Ill summarizes the steps of the suggested approach. Section IV explains the implementation of the multi-channel maximum entropy restoration algorithm. Section V discusses the wavelet based image fusion pmcess, In section VI, the implementation of the maximum encrppy image interpolation algorithm is presented. The experimental results are given in section VU. Finally the concluding remarks are given in section V111.

II. Multi-Channel Image Degradation Model

For a multi channel imaging system with P channels each of size NXN, the imaging model hecomes [13]:

where g,, fh and are the observed image, the ideal image and the noise of the k* channel, respectively.

H, i s the degradation matrix of the lc* channel.

g k = Hkfk + ' k far k = 1,2, .... P (1)

Equation(1) can be written in the following formE131: g = H f + n

.

where

where g,f and Pare Pm2 in length. The degradation H of the multi channel-imaging model is of

dimension (pm2)x(pm2). In this model, only, inter-channel degtadations exist.

IU. Proposed Approach

In this section, we suggest a computationally efficient approach of super-resolution image reconstruction, which is composed of three consecutive steps. In our suggested approach, we assume that the available degraded observations are correctly registered, If this assumption is not achieved a registration step will be needed prior to the implementation of the suggested approach [IS]. Shple and efficient registration can be achieved using any motion estimation algorithm such as the three-step method [15]. The consecutive steps are shown in Fig.( 1) and can be summarized as follows:

Multi-channel maximum entmpy image restordon of the re@stered depaded observations. Image fusion of the multiple images obtained from step 2 to form a single image (Wavelet based h i o n is used). Maximum Entropy interpolation of the resulting image to obtain a high-resolution image.

1-

2-

3-

Twenty Second.National Radio Science Conference (NRSC 2005) 1-1 M a d 15-17,2005 Cain, - Egypt

IV: Multi-Channel Maximum Entropy Restoration Model:

Image restoration on a multi channel basis aims at iMlorporating all the information existing in dl the channels into the restoration process insteal of restoring each channel separately. In this section a mathematical model for image restoration is derived based on the maximization of the entropy of the output images a priori. If aH the samples of the vector fare ~ssumed to have unit energy, they can be treated as if they ate probabilities, possibty of so many photons, which are present at the i* sample (pixel value) of the vector f [16). The required sample values of the vector f are assumed to be treated as light quanta associated with each pixel ,value, Thus, the entropy of the required vector f is then defined as follows:

D"l

where E is the entropy and fi is the i' pixel value of the required vector f. This quation can be written in the vector form as follows: .

E = -if log2 (f) ( 5 )

2 2 For image restoration, to maximize the entropy subject to the constraint that llg - Hfll =Jn11 , the following cost function must be minimized:

where 2 is a Lagrangian multiplier. Differentiating both sides of the above equation with respect to f and equating the result to zero:

Solving for the estimated vector f:

tn(i) = -1 - R 1n(2)[2~' (g - ~ i ) ] Thus:

i = exp[- 1 - A ln(2)kH' 6 i ~ - M)]l Expanding the above equation using Taylor expansion and neglecting all but the fust two terms, since g - Hf must be a small quantity leads to the following form:

f = - A ln(Z)[2H' (g -Hi)]

i (HrH + VI)-' Hlg

(10)

hlvi,ng for i leads to:

(11) . .

where 7 = -1 (212 ln(2)). This equation can be solved using the Toplitz to circulat approximation [16 1.

IV. Wavelet Based Image Fusion

This section explains the image fusion process. Image fusion is a process by which information from dinerent observation images am inwrpomted inta a single image. There. are different techniques for image fusion. The most famous fusion technique is the wavelet-based technique. The importance of image fusion lies in the fact that each observation image contains complementary information. When this complementary information is integrated with that of another observation, an image with the maximum amount of informdon is obtained [17,18].

m Twenty Second National Radio Science Conference (NRSC 2005) March 1517,2005 Cairo - Egypt

In this paper, we adopt the wavelel transform image fusion approach to integrate the data from the multiple outputs of the maximum entropy image restoration step. This is due to the fact that the multiple outputs of the maximum entropy restoration step are correctly registered and aligned. The registrash of the multiple inputs to the wavelet fusion step is a very important prerequisite to the success of the h i o n step [ 17, i 81.

In the application of the simple wavelet image fusion scheme, the wavelet packet decomposition is calculated for each observation to obtain the multi resolution levels of the images to be fused. In the transform domain the coefficients in all resolution levels whose absolute values are larger are chosen between the available observations. This rule is known as the maximum frequency rule [ 17,181. Generally, the process of wavelet packet image fusion shown in Fig.(2) can be summarized in the following steps [17,18]: .

1- The available images are first registered (This is achieved for the outputs of the maximum entropy restoration step).

2- The wavelet packet decomposition of the observations is calculated using a suitable basis function and decomposition level.

3- The maximum frequency fusion rule is used to select the wavelet coefficients from the source observations

4- A wavelet packet reconstruction is performed on the combinational created coefficients.

V. Maximum Entropy Image Interpolation

This step aims at interpolating the resulting image from the fusion step. The relationship between the LR fused image and the required high resolution image can be represented by[ 19-24]:

where x , y and v are lexicographically ordered vectors of the &own high resolution image, the measured f b e d image and additive noise values respectively. The vector X is of size 12N* XI and the vectors y and V are of size N2 XI. The matrix D represents a filtering and down-sampiing process of dimensions N2 X 12N2 where I is the resolution enhankanent factor in both dimtiow.

Under separability assumption, the matrix D which transforms the IN X 1N high resolution image to the NXN low resolution image is given by 119-241:

D=D1@D1 (13)

where 8 represents the Kronecker product, and the matrix D1 represents the one dimensional (I-D) low pass

filtering and downsampling. When 1=2 the matrix D1 will be given by f19-241:

y = D x + v (12)

p I o o ... o 01

Lo 0 0 0 a - - 1 I ] From the above model, it is clear that the process of obtaining 8 high-resolution image from a low resolution image is an inverse problem which requires inverting the operator D. This inversion process is impassible because D is not a square matrix The objective of image interpolation is to estimate h e vector X given the samples of the recorded image y . The maximum entropy solution to Eq.( 12 ) will be given by [21,24]:

. . - . f G (p'D + Df y

t This solution is based on the direct inversion of the termD D -t qI. This matrix inversion can be performed easily depending of the special nature of this matrix which is a sparse diagonal matrix [21,241.

~ - . - l _ ” . . . . . - _ .._... ......I .... .-- . . ._. .. . . . . _ - ...-

Twenty Second National Radio Science Conference (NRSC 2005) ml March 15-17,2005 Cairo - Egypt

VI. Experimental Results

In this section, the suggested maximum entropy super-resolution image reconstruction algorithm is tested. An experiment has been conducted to test the suggested implementation. In this experiment, three degraded observations of an MRI image blurred with different blurring operatoCS are used to test the suggested algorithm. Each observation is of size (128x128) pixels and the signal to noise ratio in each observation is 40 dB. The original image and its maximum entropy interpolated version are given in Fig.(3). The degraded observations are given in Fig.(4). The image obtained from the fusion of the maximm entropy restoration results is given in Fig.(5-a). The rule used in image fusion is the maximum frequency rule and the fusion process is performed in a one decampsition level. Applying the maximum entropy image interpolation algorithm on that image in Fig.(S-a) gives the high resolution image in Fig@&).

The computation time of this experiment is IO sec on a lGWz processor. It is clear that the suggested algorithm has succeeded in obtaining a high-resolution image with a good visual quality as compared to the available observations and a high PSNR. The computation time of the suggested algorithm is also small as compared to other previously published work, which require several minutes on very high-speed computers applying iterative algorithms to get the high-resolution image [1-12].

VaI, Conclusion 1

In this paper, a wavelet based maximum entropy approach has been developed for the reconstruction of a high-resolution image from multiple degraded observations. This approach is based on breaking the high- resolution reconstruction probiem into three consecutive steps. First the available observations are subjected to a multi-cfiannel maximum‘ entropy image restomtion algorithm. The resulting images f” this multi-channel restoration step are fused on a wavelet basis to form a single image. This image is then interpolated using the maximum entropy algorithm to give a high-resolution image.. Experimentel results show that the proposed approach gives good high-resolution images as compared to the available observations. It is a h clear that the suggested approach is computationally efficient when compared to the traditional iterative high-resolution image reconstruction aigorithms.

References

[I] S. P. Kim, N. K. Bose and H. M. Valemela, Recursive Reconstruction of High Resolution Image From Noisy Undersampled Multiframes” EEE Trans. Acoustics, Speech and Signal Processing, Vol. 38, No. 6, pp. 1013-1027, June. 1990.

[2] S. P. Kim and W. Y. Su, ’‘ Recursive High Resolution Reconstruction of Blurred Multiframe Images” IEEE Trans. lmage Processing, Vol. 2, No. 4 ,pp. 534-539, October 1993,

[3] S. C. Park, M. K. Park and M. G. Kang, “ Super-Resolution Image Reconstruction: A Technical Overview” IEEE Signal Processing Magazine, Vol. 20, No. 3, pp. 21-36, May 2003.

[4] M. Elad and A. Feua, “Restoration of a Single Superresolution Image from Several Blurred, Noisy, and Undersampled Measured Images” IEEE Trans. Image Processing, Vol. 6, No. 12 ,pp. 1646-1658, December 1997.

[5 ] P. E, Eren, M. I, Sezan and M. Tekalp, “ Robust, Object Based High Resolution Image Reconstruction from Low Resolution Vedio” IEEE Trans. Image Processing, Vol. 6, No. 10 ,pp. 146-1451, October 1997.

[6] D. Capel and A. Zissennan, “ Computer Vision Applied to Super-Resolution” IEEE Signal Processing Magazine, Vol. 20, No. 3, pp. 75-86, May 2003.

[7] N. Nguyen, P. Milanfar and G. Golub, A computationally Efficient Superresolution Image Reconstruction algorithm” IEEE Trans. Image Processing, Vol. 10, No. 4 ,pp. 573-583, April 2001.

[SI M. Elad and A. Feuer, ‘‘ Super-resolution Restoration of an Image Sequence: Adaptive Filtering Approach” IEEE Trans. Image Processing, Vol. 8, No. 3 ,pp. 387-395, March 1999,

Twenty Second National Radio Science Conference (NRSC 2005) 1-1 March 15-17,2005 Cairo - Egypt

(91 C. A. Segall, R. Molina and A. K. Katsaggelas “ High-Resolution Images from Low- Resolution Compressed Video” IEEE Signal Processing Magazine, Vol, 20, No. 3, pp. 37-48, May 2003.

Multi-Objective Super-Resolution: Concepts and Examples” IEEE Signal Processing Magazine, Vol. 20, No. 3, pp. 4%1, May 2003.

M. K. Ng and N. K. Bose, Mathematical Analysis of Super-Resalution Methodology” IEEE Signal Processing Mapine, Vol. 20, No. 3, pp. 62-74, May 2003.

M. Vega, J. Mteos, R. M o l i and A. K. Katsagegelos, Bayesian Parameter Estimation In Image Reconstruction From Subsampled Blurred Observations” pmceedings of IEEE iClp 2003.

N. P. Galatsanos. and R. T. Chin, Digital Restomtion Of Multichannel Images,“ IEEE Trans. Acoustics, Speech and Signal Processing, 37(3), 415421 (1989).

S. Baker and T. Kanade, “ Super-Resolution: Reconstruction or Recognition” Proceedings of the IEEE-EURASIP Workshop On Nonlinear Signal and Image Processing 2001.

[lo1 D. Rajan, S. Chmdhuri and M. V. Joshi,

c111

[I21

1131

[I43

[ 151 5. S. Lim, “Two-Dimensional Si@ and Image Processing”, hnt ice Hall, inc, 1990. [16]H.C. Anderws and E.R. Hunt, Digital Image Restoration. Englewood Cliffs, N2 Prentice-

Hall, (1977). [I71

[IS1

[I93

I201

H. Li, B. S, Manjunath and S. IC. Mitra, Multi-Sensor Image Fusion Using The

G. Piella and H. Heijmans, “ Multiresolution Image Fusion Guided By A Multimodal

W. Y. V Leung, P. J. Bones “ Statistical Interpolation of Sampled Images” Opt. Eng.

S. E: El-Khamy , M. M. Hadhoud, M. I. Dessouky, B. M. Salam, and F. E. A M El-Samie Optimization of lmage Interpolation as an Inverse Problem Using The LMMSE Algorithm” Proceedings of IEEE MELECON , (2004).

S. E. El-Khamy , M. M. Hadhoud, M, I. Dessouky, 9, M. Salam, and F. E. A M El-Samie. Efficient Implementation of Image Interpolation As An lnverse Problem” Accepted for publication in the Elsevier Journal of Digital Signal EVocessing.

El-Samie Sectioned Implementation of Regularized Image Interpolationn 4 8 lEEE MWSCAS , (2003).

J. H. Shin, J. H. Jung, J. K. Paik, “Regularized Iterative Image lnterpolation And I t s Application To Spatially Scalable Coding” lEEE T m . Consumer Electronicsi vol. 44, no.3,pp. 1042-1047, August 1998.

F. E. A M El-Samie, M. M, Hadhoud, M. 1. Dessouky, B. M. Salam, and S. E. El-Khamy, “A computationally EMicient Image Interpolation Approach ” Accepted for publication in the International Journal of Signal Processing.

Wavelet Transform” Proceedings of IEEE ICIP 1994.

Segmentation” Proceedings of ACIVS 2002.

40(4) 547-553 (April 2001).

E213

P21 S. E. El-Khmy, M. M. Hadhod, M. I. Dessouky, 8. M. Salam, and F. E. AM

1231

1241

Twenty Second National Rndio Science Conference (NRSC 2005) 1-1 March 15-17,2005 Cairo - Egypt

e c , -+ Multi- * Wavelet Max.

? Registration 7 Image hterpobtion Entropy ? Based + EnmPY Channel Max Image

Fusion .--, Restoration

--+

Fig.( I ) Schematic diagram of the proposed Super-resolution algorithm

-1-1 Twenty Second National Radio Science Conference ( W C ZOOS) March 15-17,2005 Cairo - Egypt

Registered DWT of observations Images

Fused DWT

Fig.(2) Schematic diagram of the wavelet image fusion process

, ',

Fused Image

(a) Original image (b) Max Entropy interpolation of the original image

Fig.(3) Original undegraded image

m Twenty Second National Radio Science Conference (NRSC 2005) March 15-17,2005 Cairo - Egypt

(a) observation (1) (b) observation (2) (c) observation (3) 5x5 Blur operator 7x7 Blur operator 9x9 Blur operator

Fig,(4) Available observations SNR=40 dB.

(a) Fused (b) Obtained HR Image image PSNR=29.27 dR PSNR=29.3 dB

CPU=lO sec on lGHz nmwsnnr

Fig.(5) Results of the suggested Maximum Entropy Super-resolution reconstruction Algorithm.