218 IEEE Transactions on Consumer Electronics, Vol. 57, No. 1, February 2011

Original manuscript received 01/07/11 Revised manuscript received 01/12/12 Current version published 02/21/11 Electronic version published 02/21/11. 0098 3063/11/$20.00 © 2011 IEEE

Vaguelette-Wavelet Decomposition for Frequency Adaptive Image Restoration Using Directional Wavelet Bases

Sangjin Kim, Wonseok Kang, Eunsung Lee, and Joonki Paik, Member, IEEE

Abstract — Image restoration plays a major role in

improving image quality as a preprocessing step in various imaging systems. While conventional image restoration techniques, such as the constrained least-squares (CLS) and the Wiener filters, often exhibit either noise amplification or over-smoothing problem. On the other hand, advanced image restoration techniques, such as the iterative regularization and the combined Fourier and wavelet domain thresholding filters, are not suitable for real-time applications because of high computational overhead. For overcoming these problems, we present a novel vaguelette-wavelet decomposition (VWD) approach to real-time, space-frequency adaptive image restoration based on frequency-adaptive shrinkage and directional wavelet bases. The proposed algorithm effectively reduces noise by adaptively shrinking wavelet coefficients based on alpha map and entropy, and restores a degraded image using the vaguelette-wavelet transform. In generating vaguelettes, we use the estimated point-spread function (PSF) and new directional wavelet bases. The extended set of experimental results confirm that the proposed algorithm can restore sharp details without amplifying noise, and is suitable for commercial low-cost, high-quality imaging devices, such as digital and computational cameras, visual surveillance systems, and consumer's camcorders 1.

Index Terms — Real-time image restoration, vaguelette-wavelet decomposition (VWD), frequency-adaptive shrinkage, directional wavelet bases.

I. INTRODUCTION

As the demand for digital image acquisition devices increases, various digital techniques are required for higher-resolution, better color reproduction, and higher signal-to-noise ratio (SNR), to name a few [1][2]. Among various

1 This research was supported by Basic Science Research Program through

National Research Foundation (NRF) of Korea funded by the Ministry of Education, Science and Technology (2009-0081059), and supported by the MKE (The Ministry of Knowledge Economy), Korea, under the HNRC (Home Network Research Center)–ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2010-C1090-1011-0010).

Sangjin Kim is with the Department of Image Engineering, Chung-Ang University, Seoul, Korea (e-mail: layered372@wm.cau.ac.kr).

Wonseok Kang is with the Department of Image Engineering, Chung-Ang University, Seoul, Korea (e-mail: kandws12@wm.cau.ac.kr).

Eunsung Lee is with the Department of Image Engineering, Chung-Ang University, Seoul, Korea (e-mail: lessel7@wm.cau.ac.kr).

Joonki Paik is with the Department of Image Engineering, Chung-Ang University, Seoul, Korea (corresponding author, e-mail: paikj@cau.ac.kr).

image processing techniques, image restoration is of great importance for analyzing and removing image degradation. For this reason, there have been various, in-depth researches on image restoration in the literature. Since deconvolution of a degraded image is an ill-posed problem, conventional frequency-domain restoration filters, such as the constrained least-squares (CLS) and the Wiener filters, either amplify undesired noise or result in over-smoothing artifacts. Some edge-preserving methods [3][4][5] can provide better results at the cost of increased computational overhead and processing time.

A hybrid Fourier-wavelet regularized deconvolution (ForWaRD) method has been proposed in [6] which can effectively combines and balances scalar Fourier and wavelet shrinkage without using iteration steps. On the other hand, the ForWaRD method cannot completely remove ringing artifacts at the restored image’s boundary because of the circular symmetry of the Fourier transform. The shape-adaptive DCT (SADCT) [7] method uses an intersection of confidence intervals technique to define the shape of the transforms support in a point-wise adaptive manner, and the two-step iterative shrinkage/thresholding (TwIST) [8] method handles a class of convex unconstrained optimization problems arising in image restoration and other linear inverse problems. Although both the SADCT and the TwIST methods can produce near optimal solution, the increased computational overhead is another burden.

In this context, under the real-time framework, we present a novel vaguelette-wavelet decomposition (VWD) approach to real-time, spatially adaptive image restoration using frequency-adaptive shrinkage and directional wavelet bases. The proposed algorithm consists of three steps; (i) wavelet transform of an input degraded image, (ii) adaptively shrinking a selective part of wavelet coefficients, and (iii) inverse vaguelette transform using vaguelettes. More specifically, we first reduce noise by adaptively shrinking wavelet coefficients using alpha map and entropy, and then compose a new directional transform using combined wavelet functions and an adaptive Gaussian low-pass filter. Finally, we generate vaguelettes by deconvolving the estimated PSF from directional wavelet bases.

II. VWD-BASED IMAGE RESTORATION: AN OVERVIEW

A typical image formation system of a digital camera consists of; (i) a set of optical lenses, (ii) an image sensor covered with a color filter array (CFA), (iii) an analog front

S. Kim et al.: Vaguelette-Wavelet Decomposition for Frequency Adaptive Image Restoration Using Directional Wavelet Bases 219

end (AFE) module, and (iv) a digital back end (DBE) module including various digital signal processing blocks, and (iv) the storage and display devices, as shown in Fig. 1 [1].

Fig. 1. The image formation system of a digital camera.

The proposed VWD-based image restoration framework

belongs to the DBE module in the image formation system, and is shown as a diagram in Fig. 2. The PSF is first estimated from an input image based on the method proposed in [9], which assumes that out-of-focus blur results in isotropic (or circularly symmetric) and an input image has sufficiently strong isolated linear edge. The adaptive classification is preformed based on the method proposed in [1] for an extended depth of field (EDoF) camera which is adaptively weighted mixing method between high-frequency regions and flat regions using alpha map based on local variance and mean of an input image. We reduce noise by adaptively shrinking wavelet coefficients based on alpha map in flat regions and based on entropy in high-frequency regions. We then compose a new directional transform using combined wavelet functions and an adaptive Gaussian low-pass filter. Finally, we generate vaguelettes by deconvolving the estimated PSF from directional wavelet bases, and the restored image is obtained by inverse vaguelette transform.

Fig. 2. The proposed VWD-based image restoration framework.

III. WAVELET SHRINKAGE FOR NOISE REDUCTION

Primary purpose of using wavelet decomposition is to compact significant information into the reduced number of basis functions, and to truncate the trivial components which contain amplified noise (observation error) or undesired restoration artifacts. Thresholding methods of wavelet coefficients have been proposed in [10][11]. Let

w be an individual wavelet coefficient, and the corresponding threshold value. Then the soft thresholding

function is given as

,w sign w w (1)

and the hand thresholding function is

, if .

0, otherwise

w ww

(2)

One of soft and hard thresholding can be applied to wavelet coefficients with possibly different threshold values for each level. Furthermore, various thresholding approaches can be used with consideration of inter-scale and spatial correlations [12]. However, these methods cannot fully utilize the frequency localization property of wavelets.

In this context, we propose adaptive wavelet shrinkage method by using spatial information. More specifically, wavelet coefficients in the flat regions are inner-producted by alpha map in the detail sub-band. As a result, noise can be reduced in flat regions.

After filtering noise in flat regions, we need to remove noise in the neighborhood of edge. To this end, we compute entropy as a measure of both complexity and energy of an image in the following manner.

ln ,

{ , , }, 1, , ,

i k kWk B

E p p

for i H V D k MN

(3)

where B represents an M N local block in the wavelet

sub-band, iWE

the entropy of the local block, k a number of

wavelet coefficients in the corresponding block, and kp a

probability of wavelet coefficients. Combination of the entropy

and wavelet’ s directional frequency property can completely

localize features in the spatial-frequency domain [13]. More specifically, local entropy of each wavelet coefficients in high-frequency regions contains information of local complexity in the regions. Because a noisy high-frequency region has higher complexity than purely noisy regions, a sub-band with higher entropy can be considered to contain more noise than other sub-bands. Based on this analysis, we can reduce noise by shrinking wavelet coefficients with high entropy. The proposed noise reduction algorithm is shown in Fig. 3.

Fig. 3. The proposed entropy-adaptive noise reduction in the high-frequency regions.

220 IEEE Transactions on Consumer Electronics, Vol. 57, No. 1, February 2011

IV. GENERATION OF DIRECTIONAL WAVELET BASES AND

VAGUELETTES

After selecting one-dimensional (1D) scaling function and wavelet function which satisfy the following equation [14]

2 2

2 2 ,

x h n x nn

x h n x nn

(4)

where h n and h n respectively represents the scaling

and the wavelet function coefficients. The corresponding two-dimensional (2D) scaling and three 2D wavelets are

,

,

,

, .

H

V

D

x y x y

x y x y

x y x y

x y x y

(5)

We can generate a 2D scaling function and three 2D directional wavelet functions using (5). We employ a symlet filters that satisfy the condition of 1D scaling and wavelet functions. The directional wavelet bases are given by convolving with the Gaussian low-pass filter as

, ,

( , ) ( ){ ( )* ( )},

, { ( )* ( )} ,

, { ( )* ( )}{ ( )* ( )},

V

HV G

VV G

DV G G

x y x y

x y x h y y

x y h x x y

x y h x x h y y

(6)

where Gh represents the Gaussian low-pass filter whose

bandwidth is smaller than that of wavelets, and indicates * convolution. The proposed directional wavelet bases have properties to reduce the remaining noise in each block. The resulting directional wavelet bases are shown in Fig. 4.

,V x y

( , )HV x y

(a) (b)

,VV x y

,DV x y

(c) (d)

Fig. 4. The directional wavelet bases using symlet filters.

We generate vaguelettes deconvolving the estimated PSF from directional wavelet bases [12][15] as

1

1

1

1

( , ) ( , ) , ,

( , ) ( , ) , ,

( , ) ( , ) , ,

( , ) ( , ) , ,

AV

H HV

V VV

D DV

v x y h x y x y

v x y h x y x y

v x y h x y x y

v x y h x y x y

(7)

where ( , )h x y represents the estimated PSF.

At each decomposition level, the next higher level approximation is obtained using wavelet coefficients by

inverse wavelet transform until level 1J as

,

,

,

,

( , , ) ( , ) 1, ,

( , ) 1, ,

( , ) 1, ,

( , ) 1, , ,

( , 2, 1),

A A A UP

H H UP

V V UP

D D UP

W j m n m n W j m n

m n W j m n

m n W j m n

m n W j m n

for j J J

(8)

where ,i UPW represents up-sampling,

AW inverse coefficients using wavelets.

The inverse vaguelette transform reconstructs the deconvolved image by adding four results as

,

,

,

,

( , ) ( , , )

( , ) 1, ,

( , ) 1, ,

( , ) 1, ,

( , ) 1, , ,

A

A A UP

H H UP

V V UP

D D UP

f m n W J m n

v m n W J m n

v m n W J m n

v m n W J m n

v m n W J m n

(9)

where ( , )f x y

represents final restored image through the

inverse vaguelette transform.

V. EXPERIMENTAL RESULTS

The performance of the proposed real-time vaguelette-wavelet image restoration algorithm is evaluated in the sense of peak-to-peak signal-to-noise ratios (PSNRs), mean structural similarity (MSSIM) [16], and the CPU computation time in seconds on a PC equipped with 2.66 GHz CPU and 4GB RAM, and a set of standard images and in-house outdoor images acquired by a digital single lens reflected (DSLR) camera are used. We first evaluated the proposed algorithm using the Lena image, as shown in Fig. 5(a). Fig. 5(c) is a degraded image with 7x7 Gaussian PSF and 45 dB additive white Gaussian noise (AWGN). Fig. 5(e) is the restored image by the proposed algorithm using the estimated PSF shown in Fig. 5(g). Fig. 5(b), (d), and (f) are cropped images of red dotted rectangle in Fig. 5(a), (c), and (d), respectively. As

S. Kim et al.: Vaguelette-Wavelet Decomposition for Frequency Adaptive Image Restoration Using Directional Wavelet Bases 221

shown Fig. 5, the proposed algorithm successfully restored a degraded image without amplifying noise and losing sharp details.

(a) (b)

(c) (d)

(e) (f)

(g) Fig. 5. The experimental results using the Lena image: (a) the original image, (b) the cropped image of the red dot rectangle in (a), (c) the degraded image, (d) the cropped image of the red dot rectangle in (c), (e) the restored image by the proposed algorithm using (g), and (f) the cropped image of the red dot rectangle in (e), and (g) the estimate PSF.

The proposed algorithm is also compared with the well-

known algorithms, such as the Wiener filter, the ForWaRD [6], the SADCT [7], and the TwIST [8] methods. For comparison with other existing algorithms, we first use the Cameramen

image blurred with 7x7 Gaussian PSF and 35 dB AWGN. As shown in Fig. 6, the Wiener filter tends to amply noise and lose details in the restored image. On the other hand, the proposed algorithm gives subjectively acceptable image quality in the sense of both suppressing noise and preserving the sharp details compared with the existing state-of-the-art image restoration methods.

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Fig. 6. Comparison of different restoration algorithms using the Cameraman image: (a) the original image, (b) the degraded image, (c) the restored image by the Wiener filter, (d) the restored image by the ForWaRD, (e) the restored image by the SACDT, (f) the restored image by the TwIST, and (g) the restored image by the proposed algorithm using (h), and (h) the estimated PSF.

222 IEEE Transactions on Consumer Electronics, Vol. 57, No. 1, February 2011

Similar results for a real defocused image are shown in Fig. 7. Furthermore, Table I and II summarize PSNR/MSSIM values and the CPU computation time in seconds of the Wiener filter, the ForWaRD, the SADCT, the TwIST, and the proposed methods with noisy (AWGN), out-of-focus blurred input images of size 1504x1000. Although the proposed method provides lower PSNR and MSSIM values than the SADCT and the TwIST methods in high SNR images, it gives acceptable performance and produces better result for CPU computation time than these methods.

(a) (b)

(c) (d)

(e) (f)

(g) Fig. 7. Comparison of different restoration algorithms using a real defocused image: (a) the defocused image, (b) the restored image by the Wiener filter, (c) the restored image by the ForWaRD, (d) the restored image by the SADCT, (e) the restored image by the TwIST, (f) the restored image by the proposed algorithm using (g), and (g) the estimated PSFs.

TABLE I PSNR / MSSIM VALUES COMPARISON OF FIVE DIFFERENT ALGORITHMS

Blur Size Restoration

Type PSNR/

MSSIM SNR (dB)

20 dB 30 dB 40 dB 50 dB

5x5

Wiener PSNR 9.434 16.523 22.923 26.4321

MSSIM 0.339 0.510 0.582 0.701

ForWaRD PSNR 20.496 22.346 25.324 31.342

MSSIM 0.551 0.638 0.700 0.891

SADCT PSNR 29.350 32.321 33.433 34.679

MSSIM 0.898 0.930 0.932 0.950

TwIST PSNR 30.7707 32.182 34.496 35.7 12

MSSIM 0.902 0.929 0.968 0.9502

Proposed Algorithm

PSNR 20.123 22.391 25.958 31.210

MSSIM 0.549 0.639 0.711 0.889

7x7

Wiener PSNR 9.023 15.332 21.232 23.172

MSSIM 0.319 0.500 0.540 0.612

ForWaRD PSNR 19.532 21.423 25.031 30.132

MSSIM 0.505 0.620 0.684 0.882

SADCT PSNR 28.172 30.423 31.531 33.002

MSSIM 0.871 0.888 0.890 0.931

TwIST PSNR 29.123 31.182 32.183 34.218

MSSIM 0.881 0.915 0.923 0.963

Proposed Algorithm

PSNR 20.321 21.512 24.213 30.134

MSSIM 0.521 0.621 0.661 0.883

9x9

Wiener PSNR 7.833 14.021 19.732 20.002

MSSIM 0.300 0.421 0.492 0.500

ForWaRD PSNR 19.511 20.823 24.531 26.132

MSSIM 0.502 0.610 0.651 0.699

SADCT PSNR 28.001 29.533 31.032 31.872

MSSIM 0.855 0.863 0.885 0.902

TwIST PSNR 28.999 30.235 31.192 32.283

MSSIM 0.877 0.897 0.901 0.930

Proposed Algorithm

PSNR 19.324 21.008 22.013 25.002

MSSIM 0.501 0.611 0.630 0.683

TABLE II

PERFORMANCE CPU COMPUTATION TIME IN SECONDS FOR THE

PROPOSED AND EXISTING ALGORITHMS

Blur Size Restoration

Type SNR (dB) Average

(Sec.) 20 dB 30 dB 40 dB

5x5 Wiener 7.931 8.031 8.231 8.064ForWaRD 133.123 134.392 133.322 133.612SADCT 110.313 109.321 109.213 109.616TwIST 1833.512 1835.245 1838.502 1835.753

Proposed Algorithm

15.002 14.864 14.154 14.673

7x7 Wiener 7.545 7.953 8.048 7.849ForWaRD 132.485 135.001 135.099 134.195SADCT 112.562 112.215 112.441 112.406TwIST 1852.552 1853.259 1855.548 1853.786

Proposed Algorithm

16.008 16.823 17.921 16.917

9x9 Wiener 8.564 8.128 8.426 8.373ForWaRD 134.864 136.129 135.741 135.578SADCT 112.569 112.458 113.851 112.959TwIST 1828.865 1828.185 1829.555 1828.868

Proposed Algorithm

16.548 15.196 16.845 16.196

V. CONCLUSION

In this paper, we presented a novel VWD approach to real-time, space-frequency adaptive image restoration. We presented frequency-adaptive shrinkage using alpha map in flat regions and entropy in high-frequency regions. We also proposed a new

S. Kim et al.: Vaguelette-Wavelet Decomposition for Frequency Adaptive Image Restoration Using Directional Wavelet Bases 223

directional transform using combined wavelet functions and adaptive Gaussian low-pass filters, and generated the vaguelettes by deconvolving the estimated PSF from directional wavelet bases. The proposed VWD-based image restoration algorithm can remove image degradation caused by defocusing, and reduce noise amplifying noise and losing sharp details. Experimental results demonstrate the proposed algorithm can overcome limitations in conventional image restoration filters, and is suitable for commercial low-cost, high-quality imaging devices, such as digital and computational cameras, surveillance systems, and consumer's camcorders.

REFERENCES [1] S. Kim, S. Jun, E. Lee, J. Shin, J. Paik, "Real-Time Bayer-Domain Image

Restoration for an Extended Depth of Field (EDoF) Camera," IEEE Transactions on Consumer Electronics, vol. 55, no. 4, pp. 1756-1764, November 2009.

[2] Y. Kim, B. Choi, and J. Paik, "High Fidelity RGB Video Coding Using Adaptive Inter-Plane Weighted Prediction," IEEE Transactions on Circuits and Systems for Video Technology, vol. 19, no. 7, pp. 1051-1056, July 2009.

[3] A. Katsaggelos, J. Biemond, R. Schafer, and R. Mersereau, "A Regularized Iterative Image Restoration Algorithm," IEEE Transactions on Signal Processing, vol. 35, no. 4, pp. 914-929, April 1991.

[4] M. Belge, M. Kilmer, E. Miller, "Wavelet Domain Image Restoration with Adaptive Edge-Preserving Regularization," IEEE Transactions on Image Processing, vol. 9, no. 4, pp. 597-608, April 2000.

[5] P. Charbonnier, L. Blanc-Feraud, G. Aubert and M. Barlaud, "Deterministic Edge-Preserving Regularization in Computed Imaging," IEEE Transactions on Image Processing, vol. 6, no. 2, pp. 298-311, February 1997.

[6] R. Neelamani, H. Choi, and R. Baraniuk, "ForWaRD: Fourier-Wavelet Regularized Deconvolution for Ill-Conditioned Systems," IEEE Transactions on Signal Processing, vol. 52, no. 2, pp. 418-433, February 2004.

[7] A. Foi, V. Katkovnik, and K. Egiazarian, "Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images," IEEE Transactions on Image Processing, vol. 16, no. 5, pp. 1395-1411, May 2007.

[8] M. Bioucas, and A. Figueirdo, "A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration," IEEE Transactions on Image Processing, vol. 16, no. 12, pp. 349-366, December 2007.

[9] J. Jeon, I. Yoon, D. Kim, J. Lee, and J. Paik, "Fully Digital Auto-Focusing System with Automatic Focusing Region Selection and Point Spread Function Estimation," IEEE Transactions on Consumer Electronics, vol. 56, no. 3, pp. 1204-1210, August 2010.

[10] D. Donoho, “Nonlinear Solution of Linear Inverse Problems by Wavelet-Vaguelette Decomposition,” Applied Computational and Harmonic Analysis, vol. 2, pp. 101-26, 1995.

[11] F. Abramovich and B. Silverman, “Wavelet Decomposition Approaches to Statistical Inverse Problems,” Biometrika, vol. 85, no. 1, pp. 115-129, 1998.

[12] E. Balster, Y. Zheng, and R. Ewing, “Feature-based Wavelet Shrinkage Algorithm for Image Denoising,” IEEE Transactions on Image Processing, vol. 14, no. 12, pp. 2024-2039, December 2005.

[13] H. Zheng, C. Xiaoqing, and L. Guoming, "Wavelet Entropy Measure Definition and Its Application for Transmission Line Fault Detection and Identification," Int. Conf. Power System Technology 2006, pp. 1-6, October 2006.

[14] R. Gonzalez and R. Woods, Digital Image Processing, 3rd ed., PrenticeHall, 2007.

[15] S. R. Nowak and M. Thul, "Wavelet-Vaguelette Restoration in Photon-Limited Imaging," Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing, May 1998.

[16] Z. Wang, A. Bovik, H. Sheikh and E. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 2004.

BIOGRAPHIES

Sangjin Kim was born in Seoul, Korea in 1978. He received the B.S. degree in electronic engineering from Kangnam University, Korea, in 2003. He received the M.S and Ph.D. degrees in image engineering from Chung-Ang University, Korea, in 2005 and 2009 respectively. Dr. Kim is currently Post-Doctorial researcher in Image Processing and Intelligent Systems Laboratory at Chung-Ang University. His research interests include image

restoration, computational camera, and real-time object tracking.

Wonseok Kang was born in Jeju, Korea in 1983. He received the B.S. degree in electronic engineering from Korea Aerospace University, Korea, in 2010. He is currently pursuing M.S. degree in image processing at Chung-Ang University. His research interests include image restoration and computational camera.

Eunsung Lee was born in Seoul, Korea in 1982. He received the B.S. degree in electronic engineering from Chung-Ang University, Korea, in 2009. He is currently pursuing M.S. degree in image processing at Chung-Ang University. His research interests include image restoration, computational camera, and image enhancement.

Joonki Paik was born in Seoul, Korea in 1960. He received the B.S. degree in control and instrumentation engineering from Seoul National University in 1984. He received the M.S. and the Ph.D. degrees in electrical engineering and computer science from Northwestern University in 1987 and 1990, respectively. After getting the Ph.D. degree, he joined Samsung Electronics, where he designed the image stabilization chip sets for consumer’s camcorders. Since 1993, he has

worked for Chung-Ang University, Seoul, Korea, where he is currently a Professor in the Graduate school of Advanced Imaging Science, Multimedia and Film. From 1999 to 2002, he was a visiting Professor at the Department of Electrical and Computer Engineering at the University of Tennessee, Knoxville. From 2005 to 2007 he served as Dean of the Graduate School of Advanced Imaging Science, Multimedia, and Film. Dr. Paik is currently the head of the Image Processing and Intelligent Systems Laboratory, which is supported by the Brain Korea 21 Project, by the Korean Ministry of Information and Communication under ITRC-HNRC, and by Seoul Future Contents Convergence Cluster established by Seoul Research and Business Development (R&BD) Program.