Image fusion-based adaptive regularization for image expansion

Jeong Ho Shin, Joon Shik Yoon, and Joon Ki Paik

Department of Image Engineering,

Graduate School of Imaging Science, Multimedia, and Film, Chung-Ang University

221 Huksuk-Dong, Tongjak-Ku, Seoul 156-756, Korea

ABSTRACT

This paper presents a regularized image sequence interpolation algorithm, which can restore high frequency details byfusing low-resolution frames. Image fusion algorithm gives the feasibility of using di�erent data sets, which correspondto the same scene to get a better resolution and information of the scene than the one obtained using only one dataset. Based on the mathematical model of image degradation, we can have an interpolated image which minimizesboth residual between the high resolution and the interpolated images with a prior constraint. In addition, by usingspatially adaptive regularization parameters, directional high frequency components are preserved with e�cientlysuppressed noise. The proposed algorithm provides a better-interpolated image by fusing low-resolution frames. Weprovide experimental results which are classi�ed into non-fusion and fusion algorithms. Based on the experimentalresults, the proposed algorithm provides a better interpolated image than the conventional interpolation algorithmsin the sense of both subjective and objective criteria. More speci�cally, the proposed algorithm has the advantageof preserving high frequency components and suppressing undesirable artifacts such as noise.

Keywords: image interpolation, image expansion, regularization, image fusion, adaptive regularization parameter.

1. INTRODUCTION

Image interpolation is used to obtain a higher resolution image from a low resolution image, and therefore it is veryimportant in multi-resolution or high-resolution image processing. For example, the spatial scalability function inMPEG-2 and multiscale decomposition-based image processing techniques require image interpolation techniques.On the other hand, high-resolution image processing applications such as digital HDTV, aerial photo, medicalimaging, and military purpose images, need high-resolution image interpolation algorithms. Recently, it can alsobe used in changing the format of various types of images and videos, and in increasing the resolution of images.Moreover, there has been increasing interests in integrating information of a scene, obtained from di�erent data sets.The terminology \fusion" means the integration of information from multiple sources to produce more speci�c andcomprehensive uni�ed data about an entity. In this paper we mainly deal with image interpolation techniques toenhance the image quality in the sense of resolution.

Conventional interpolation algorithms, such as zero-order or nearest neighbor, bilinear, spline-based, and theDFT-based interpolation, can be classi�ed as signal synthesis or reconstruction from basis functions, and theymainly focus on just increasing the size of images.1{3 Those algorithms have been developed under assumption thatthere are no mixture among adjacent pixels in the imaging sensor, no motion blur due to �nite shutter speed ofthe camera, no isotropic blur due to out-of-focus, and no aliasing in the process of sub-sampling. Since the abovementioned assumptions are not satis�ed in general low-resolution imaging systems, it is not easy to restore theoriginal high-resolution image by using the conventional interpolation algorithms.

In order to improve the performance of the above mentioned algorithms, a spatially adaptive cubic interpolationmethod has been proposed.4 Although it can preserve a number of directional edges in the interpolation process, itis not easy to restore original high frequency components which are lost in the sub-sampling process.

As an alternative, multi-frame interpolation techniques which use sub-pixel motion information have been pro-posed in the literature.5,6 It is well-known that image interpolation is an ill-posed problem. More speci�cally, weregard a sub-sampling process as a general image degradation process. Then the regularized image interpolation is

Further author information-J. H. Shin : E-mail: shinj@ms.cau.ac.kr; WWW:http://ms.cau.ac.kr/�shinjJ. S. Yoon : E-mail: redeagle@ms.cau.ac.kr; WWW:http://ms.cau.ac.kr/�redeagleJ. K. Paik : E-mail: paikj@chungang.edu; WWW:http://cau.ac.kr/�paikj

(a) (b) (c)

Figure 1. Relationships between two images with di�erent resolutions: (a) Sampling grids for two di�erent images;\ " for y 1

4

, and \�" for x, (b) an array structure of photo-detectors for x(n1; n2), and (c) an array structure of

photo-detectors for y 1

4

(m1;m2).

to �nd the inverse solution de�ned by the image degradation model subject to a priori constraints.7,8 Since theconventional regularized interpolation methods used isotropic smoothness as a priori constraints, their interpolationperformance for images with various edges is limited. To solve that problem, Schultz and Stevenson have addresseda method for nonlinear image interpolation algorithm which preserves the discontinuities of the original image basedon the maximum a posteriori (MAP) estimation technique using Huber-Markov random �eld (HMRF).9

A spatial image interpolation algorithm using projections onto convex sets (POCS) theory, which obtains highresolution images from low resolution image sequences, has been studied in.10 In,11 modi�ed edge-based lineaverage (ELA) techniques for the scanning rate conversion is proposed. To obtain a high resolution image frame,Stochastic estimation such as Maximum likelihood (ML) and MAP, and POCS are combined in a hybrid method.12

In this paper we present theoretic fundamentals, which are possible to communicate e�ciently in the format ofhigh resolution images, applying spatially adaptive regularization parameters �tted to each image properties between at and complex region of images. By applying image fusion algorithms, the pixel values of the fused low-resolutionimages with dominant feature of local area make us possible to obtain better low resolution images. As a result ofboth algorithms above, edge-preserving image interpolation algorithms can be employed.

This paper is organized as following. Section 2 de�nes mathematical models of low resolution image formationsystem. In Section 3, we summarize the iterative algorithm with real-time structure brie y. We propose a spatiallyadaptive image interpolation algorithm based on image fusion in Section 4. Some experimental results for highresolution image interpolation are provided in Section 5. Finally, Section 6 concludes the paper.

2. DEGRADATION MODEL FOR A LOW RESOLUTION IMAGING SYSTEM

Let xc(p; q) represent a two-dimensional continuous image, and x(n1; n2) the corresponding digital image obtainedby sampling xc(p; q), with size N �N , such as

x(m1;m2) = xc(m1Tv;m2Th); for m1;m2 = 0; 1; � � � ; N � 1; (1)

where Tv and Th respectively represent the vertical and the horizontal sampling intervals. We consider the N �Ndigital image, x(m1;m2), as a high resolution original image, and the N

q� N

qimage, y(n1; n2), with q times lower

resolution in both horizontal and vertical directions can be represented as

y(n1; n2) =1

q2

q�1Xi=0

q�1Xj=0

x(qn1 + i; qn2 + j); for n1; n2 = 0; 1; � � � ;N

q� 1: (2)

Two di�erent sampling grids to obtain the image with 4 times lower resolution in both horizontal and verticaldirections, which respectively correspond to images in (1) and (2), are shown in Fig. 1a. And the correspondingarray structures of photo-detectors are shown in Figs. 1b and 1c, respectively.

N NN/2

2Horizontal LPF Vertical LPF 2

N

N/2 N/2

x(m1,m2) y(n1,n2)

Figure 2. Sub-sampling process from the N �N high resolution image to the N=2�N=2 low resolution image.

As shown in (1) and (2), a sub-sampling process from the high resolution image to the low resolution image canbe given as

y(n1; n2) =Xm1

Xm2

x(m1;m2)h(n1; n2;m1;m2) + �(n1; n2); (3)

where y(n1; n2), x(m1;m2), and �(n1; n2) respectively represent two-dimensional arrays of the low resolution image,the high resolution image, and additive noise. And h(n1; n2;m1;m2) represents the space-variant point spreadfunction (PSF), which determines the relationship between the high resolution image and the low resolution image.In this paper we assume that the sub-sampling process is separable and space-invariant.

Under the separability assumption, the sub-sampling process from the N � N high resolution image to theN=2�N=2 low resolution image is shown in Fig. 2. The corresponding discrete linear space-invariant degradationmodel can be expressed in matrix-vector form as

y = Hx+ �; (4)

where the�N2

�2� 1 vectors, y and � respectively represent the lexicographically ordered low resolution image and

noise, and the N2 � 1 vector, x, represents the original high resolution image. And H represents a�N2

�2�N2 block

Toeplitz matrix can be written asH = H1 H1; (5)

where represents the Kronecker product, and the N2�N matrix, H1, represents the one dimensional (1D) low-pass

�ltering and sub-sampling by a factor of 2, for example, such as

H1 =1

2

26664

1 1 0 0 � � � 0 00 0 1 1 � � � 0 0...

......

.... . .

......

0 0 0 0 � � � 1 1

37775 : (6)

In the right hand side of (5), the �rst and the second H 0

1s respectively represent the low-pass �ltering and sub-sampling process in the horizontal and the vertical directions. In Eq. (5) and (6), H represents a matrix which iscomposed of asymmetric structure.

As shown in Fig. 1, one way to increase the spatial resolution is to increase the density of photo-detectors byreducing their sizes. However, the smaller the size of each photo-detector is, the less the amount of incoming lightenergy becomes. The decrease in incoming light energy degrades the picture quality since shot-noise is unavoidable,in principle. Therefore, in current sensor technology, there is limitation in reducing the size of the photo-detectorin order to keep shot noise invisible, and its size limitation is known to be approximately 50�m2.19 In20 it hasbeen noted that current CCD technology has almost reached the bound. In other words, higher resolution over thatbound should be obtained by using digital image interpolation techniques.

3. ITERATIVE ALGORITHM WITH REAL-TIME STRUCTURE

In order to solve (4), the regularized iterative image interpolation algorithm tries to �nd the estimate, x̂, whichsatis�es the following optimization problem.

x̂ =arg minxi f(xi); (7)

where,f(xi) = jjyi �Hxijj

2 + �jjCxijj2: (8)

In Eq. (8), C represents a high-pass �lter, and jjCxijj represents a stabilizing functional whose minimization sup-presses high frequency components due to noise ampli�cation. And � represents the regularization parameter whichcontrols the �delity to the original image and smoothness of the restored image.

The cost function given in (8) can be rewritten as

f(xi) =1

2xTi Txi � bTi xi; (9)

whereT = HTH + �CTC and bi = HT yi: (10)

The best estimate which minimizes (9) is equal to the solution of following linear equation,

Txi = bi: (11)

In order to solve the above equation given in (11), the successive approximation equation describing the interpolatedimage, x, at the k + 1st iteration step, is given by

xk+1 = xk + �fHT yk � (HTH + �CTC)xkg; (12)

where yk represents the motion compensated image frame of the k-th frame yk, and � the function that controls theconvergence rate.16

In dealing with iterative algorithms, the convergence and the rate of convergence are very important. Unlessconvergence is guaranteed, the iterative algorithm cannot be used. Even if convergence is guaranteed, su�cientlyhigh rate of the convergence is needed for practical applications. In-depth convergence analysis has been proposedin.17

4. SPATIALLY ADAPTIVE IMAGE EXPANSION ALGORITHMS

The regularized iterative image restoration algorithms presented in the previous subsection does not satisfy thehuman visual system. However, we introduce adaptive image interpolation algorithm which is adequate to humanvisual system and present its e�cient implementation in this section. In addition, the pixel values of the fused lowresolution images have dominant feature of local area, so it makes us possible to obtain better low resolution images,which can be able to make a high resolution image by using image fusion algorithms.

4.1. Preprocessing using image fusion algorithm

The purpose of image interpolation is to recover or enhance the resolution of original image or scene. Fusion may bean alternative approach for image interpolation. The reason why we use fusion concept in this paper is that it canprovide a better resolution and information of the scene than the one obtained using only one data set. Data fusioncan be implemented at either the signal, the pixel, the feature or the symbolic level of representation. We use pixellevel fusion which is to merge multiple images on pixel-by-pixel basis to improve the performance of many imageprocessing tasks. From this idea, we can apply it to our new interpolation algorithm, which uses the feature of localarea as a fusion criterion.

In order to apply the fusion algorithm to (12), the corresponding iteration process can be rewritten as

xk+1 = xk + �(bkfused �HTH � �CTCxk); (13)

where bkfused represents the fused image obtained from low-resolution frames.

Computinggradients oflocal area

Acitivity levelmeasurement

ConsistancyVerification

Fusion Aspect

RegistrationMaximumselection

rule

Figure 3. The structure of the proposed image fusion method.

To obtain the k-th fused frame, bkfused in (13), we should de�ne an appropriate image fusion scheme, which isdescribed in Fig. 3. First, low-resolution frames have to be registered so that the corresponding pixels are aligned.Second, the gradient of the low-resolution image frames is computed to obtain the feature of local area. In general,the gradient has magnitude and direction in continuous images as

jgradf(x; y)j =

s�@f

@x

�2

+

�@f

@y

�2

(14)

� = arg

�@f

@x;@f

@y

�: (15)

Among various types of gradient operators, we choose the Laplacian and LoG operator to calculate the gradient oflocal regions in this paper. These operators have properties, such as: (i) They have isotropic zero-crossings of thesecond derivative, (ii) They have a certain degree of accuracy in �nding edge positions, (iii) They approximate thesecond derivative which gives the gradient magnitude only, (iv) LoG operator is insensitivity to noise. We measurethe activity level measurement as preliminary to choice of the gradient magnitude. In our implementation we usethe maximum absolute value within the window as an activity measure associated with the center pixel. As a result,we can determine the maximum value among the gradient magnitude by using the maximum selection rule. Finally,median �ltering of binary map, which represents the origin of coe�cient values to provide a constraint in the processof consistency veri�cation.

The proposed image fusion algorithm is summarized as follows:

Algorithm 1. Image fusion

1. Registration of images.

2. Computation of gradient magnitude using Laplacian and LoG.

3. M(i; j) = maxfjgradf(m;n)jg; for m� P � m � m+ P; n�Q � n � m+Q

4. gradient map =

�1; if Mk(i; j) > Mk�1(i; j)0; if Mk(i; j) �Mk�1(i; j);

5. gradient map = median(gradient map)

By using the proposed fusion algorithm, we can de�ne the fused frame as

bkfused(i; j) =

�HT yk(i; j); if gradient map = 1HT yk�1(i; j); if gradient map = 0;

(16)

According to (16), we can choose HT yk(i; j) as a proper pixel value, on condition that k-th gradient is bigger thank � 1-th one. After the image fusion process, we perform the interpolation process by using iterative regularizedinterpolation algorithm summarized in Sec. 3. In general, low resolution images have more lowpass components

than original high resolution image frames. Based on this observation, high resolution images, interpolated byusing these low resolution images, may be sub-optimal in the sense of resolution. In our proposed algorithm,however, low resolution images have maximum gradient at each pixel, so it enables us to obtain better low resolutionobservations for making higher resolution image. The proposed interpolation algorithm uses the current and previouslow-resolution frames in order to obtain a high-resolution frame. It means that we can reuse the information of low-resolution frames without information loss.

4.2. Interpolation with spatially adaptive regularization parameters

The human visual characteristics are partly revealed by psychophysical experiments. According to those experiments,human visual system is sensitive to noise in at regions, while it becomes less sensitive to noise at sharp transitionsin image intensity. In conclusion, human visual system is less sensitive to noise in edges than in at regions .15

Based on the results of the experiments, various ways to subjectively improve the quality of the restored image havebeen proposed in.13,21 According to the characteristics of human noise sensitivity, the regularization parametercan be adaptively be chosen depending on spatial activity of the corresponding region. As a result, high frequencycomponents in complex regions are ampli�ed by adaptively varying the regularization parameters, � in (10), andhigh frequency components in at regions are suppressed.21

In order to adopt the spatial adaptivity that re ects human visual system, we de�ne the visibility function ofimages. Based on the gradient magnitude of images, the local visibility function can be de�ned as

v(i; j) =1

jgradf(i; j)j+ 1; (17)

where represents a parameter which make visibility function become close to unity or zero, and jgradf(i; j)jrepresents the gradient magnitude in (14).

It is clear from (17) that in regions with high spatial activity the visibility function becomes smaller (noise is notvisible), while in at regions the visibility function becomes larger (noise is visible).

An image interpolation algorithm using locally adaptive regularization parameters is derived as the following.We �rst de�ne two constraint sets which include the feasible solution in their intersection, then the functional, f(x),which satis�es above constraints sets, is de�ned as

f(x) = f(�i; x) = jjy �Hxjj2 + �jjV Cxjj2; (18)

where V TV represents N2 �N2 diagonal matrix whose diagonal elements takes the value of v2(i; j).

The minimization of (18) with respect to x is straightforward and leads to the spatially adaptive regularizedsolution, as

(HTH + �CT V TV C)x = HT y: (19)

The local visibility function of each pixel, which is determined by (17), is applied to the interpolation process asfollows. A regularization parameter, �, in (10) is replaced by v2(i; j) � �. Here, we note that � represents themaximum value of v2�� because v has the value between 0 and 1. On the other hand, distribution of v depends on .

In order to solve (19), the following iteration is used.

x0 = �HT y; xk+1 = xk + �fHT y � (HTH + �CTV TV C)xkg: (20)

5. EXPERIMENTAL RESULTS

In order to demonstrate the performance of the proposed algorithm, we make an image sequence with twenty130 � 90 image frames. Fig. 4 shows the 520 � 360 high resolution �ghter image, Fig. 5 shows the 130 � 90subsampled image of Fig. 4 by using the degradation model given in (3). Resulting interpolated images using zero-order interpolation, bilinear interpolation, and cubic B-spline interpolation are shown in Figs. 6, 7, and 8 respectively.Cubic B-spline based-interpolated image of Fig. 8 is shown using the `congrid(,/cubic)' function of IDL. Figs. 9 and 10respectively show interpolated images by using the conventional non-fusion based algorithm and the proposed fusion-based adaptive regularized interpolation algorithms. For both interpolated images, only one iteration is performed per

each image frame. Therefore, almost real-time interpolation can be implemented based on the proposed algorithm.To implement the fusion-based adaptive interpolation algorithm, we use the gradient magnitude in (14) and (17) asLoG. Because of LoG's shape, the gradient operator is commonly called a Mexican hat. LoG operator is give by2

666640 0 �1 0 00 �1 �2 �1 0

�1 �2 16 �2 �10 �1 �2 �1 00 0 �1 0 0

377775 : (21)

Under assumption that the regularization parameter's range is [0:001; 1], the parameter, in (17) can be calculatedas

=999

Pth; (22)

where the threshold of the gradient magnitude, Pth is given by

Pth =arg

Pth

hmaxfhistogram(jgradj)g

�

100

i; (23)

and � represents the cuto� degree for histogram of gradient magnitude.

As we know from the experimental results, the proposed fusion-based adaptive algorithm gives better interpolationresults in the sense of both subjective and objective measures.

6. CONCLUSIONS

In this paper we proposed a spatially adaptive image interpolation algorithm using data fusion. To obtain a higherresolution image, the fused image from low resolution image frames are used as input of iterative regularizationalgorithm. Based on the regularized iteration, moreover, the proposed algorithm adaptively changes the regularizationparameter � depending on the spatial activity.

From experimental results, the proposed adaptive algorithm has shown to be able to restore high frequencydetails, and at the same time to give higher PSNR value than the algorithms without image fusion.

The proposed adaptive algorithm can easily be extended to motion compensated image sequence interpolationunder assumption that each frame has a space-invariant translational motion.16 This means that the resolutionof image communication systems, such as video phone and video conferencing system, etc. can be improved with-out additional transmission overhead, and that higher quality decoded image or higher compression ratio may beachievable by applying the proposed interpolation algorithm into spatial scalability of MPEG-2 and digital HDTV.

Acknowledgments

*This work was supported in part by the Ministry of Science and Technology of Korea as a part of the NationalResearch Lab Project.*This work was supported in part by DOE's University Research Program in Robotics (URPR), grant numberDOE-DE-FG02-86NE37968.

REFERENCES

1. A. K. Jain, Fundamentals of Digital Image Processing, Prentice- Hall, 1989.

2. M. Unser, A. Aldroubi, and M. Eden, \Fast B-spline transforms for continuous image representation and inter-polation," IEEE Trans. Pattern Analysis, Machine Intelligence, vol. 13, no. 3, pp. 277-285, March 1991.

3. J. A. Parker, R. V. Kenyon, and D. E. Troxel, \Comparison of interpolating methods for image resampling,"IEEE Trans. Med. Imaging, vol. 2, no. 1, pp. 31-39, March 1983.

4. K. P. Hong, J. K. Paik, H. J. Kim, and C. H. Lee, \An edge-preserving image interpolation system for a digitalcamcoder," IEEE Trans. Consumer Electronics, vol. 42, no. 3, pp. 279-284, August 1996.

5. S. P. Kim, H. K. Bose, and H. M. Valenzuela, \Recursive reconstruction of high-resolution image from noisyundersampled frames," IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, pp. 1013-1027, June 1990.

6. A. Patti, M. I. Sezan, and A. M. Tekalp, \High-resolution image reconstruction from a low-resolution imagesequence in the presence of time varying motion blur," Proc. 1994 Int. Conf. Image Processing, November 1994.

7. M. C. Hong, M. G. Kang, and A. K. Katsaggelos, \An iterative weighted regularized algorithm for improving theresolution of video sequences," Proc. 1997 Int. Conf. Image Processing, vol. 2, pp. 474-477, October 1997.

8. B. C. Tom and A. K. Katsaggelos, \An iterative algorithm for improving the resolution of video sequences," Proc,SPIE Visual Comm., Image Proc., pp. 1430-1438, March 1996.

9. R. R. Schulz and R. L. Stevenson, \Extraction of high-resolution frames from video sequences," IEEE Trans.

Image Processing, vol. 5, no. 6, pp. 996-1011, June 1996.

10. A. J. Patti, M. I. Sezan, and A. M. Tekalp, \High-resolution standards conversion of low resolution video," Proc.1995 Int. Conf. Acoust., Speech, Signal Processing, pp. 2197-2200, 1995.

11. C. J. Kuo, C. Liao, and C. C. Lin, \Adaptive interpolation technique for scanning rate conversion," IEEE Trans.

Circuit, System, Video Technology, vol. 6, No. 3, pp. 317-321, June, 1996.

12. M. Elad and A. Feuer, \Restoration of a single superresolution image from several blurred, noisy, and undersam-pled measured images," IEEE Trans. Image Processing, vol. 6, no. 12, pp. 1646-1658, December 1997.

13. A. K. Katsaggelos, \Iterative image restoration algorithms," Optical Engineering, vol. 28, pp. 735-748, 1989.

14. R. W. Schafer, R. M. Mersereau, and M. A. Richards, \Constrained iterative restoration algorithms," Proc.

IEEE, vol. 69, no. 4, pp. 432-450, April 1981.

15. G. L. Anderson and A. N. Netravali, \Image restoration based on a subjective criterion," IEEE Trans. Sys.,

Man, Cybern., vol. SMC-6, no. 12, pp. 845-853, December 1976.

16. J. H. Shin, J. H. Jung, and J. K. Paik, \Regularized iterative image interpolation and its application to spatiallyscalable coding," IEEE Trans. Consumer Electronics, vol. 44, no. 3, pp. 1042-1047, August 1998.

17. J. H. Shin, J. S. Yoon, J. K. Paik, and M. A. Abidi, \Fast superresolution for image sequence using motionadaptive relaxtion parameters," Proc. 1999 Int. Conf. Image Processing, vol. 3, pp. 676-680, October 1999.

18. J. S. Lim, Two-Dimensional Signal and Image Processing, Prentice-Hall, pp. 495-497, 1990.

19. K. Aizawa, T. Komatsu, and T. Saito,\A scheme for acquiring very high resolution images using multiplecameras,", Proc. 1992 Int. Conf. Acoust., Speech, Signal Processing, vol. 3, pp. 289-292, 1992.

20. T. Ando, \Trend of high-resolution and high-perfomance solid state imaging technology," J. ITE Japan, vol. 44,no. 2, pp.105-109, Feburuay 1992.

21. A. K. Katsaggelos, J. Biemond, R. W. Schafer, R. M. Mersereau, \A regularized Iterative image restorationalgorithms," IEEE Trans. Signal Processing, vol. 39, no. 4, pp. 914-929, 1991.

22. H. J. Trussell and M. R. Civanlar, \The feasible solution in signal restoration," IEEE Trans. Acoust. Speech

Signal Processing, vol. ASSP-32, no. 2, pp. 201-212, April 1984.

23. D. C. Youla and H. Webb, \Image restoration by the method of convex projections: Part 1-theory," IEEE Trans.

Medical Imaging, vol. MI-1, no. 2, pp. 81-94, October 1982.

Figure 4. Original image

Figure 5. The �rst image frame of 20 synthetically subsampled images by factor of 14

Figure 6. Zero-order interpolated image (PSNR=27.60[dB])

Figure 7. Bilinear interpolated image (PSNR=26.40[dB])

Figure 8. Cubic B-spline interpolated image (PSNR=26.37[dB])

Figure 9. Regularized interpolated image without fusion algorithm (PSNR=29.17[dB])

Figure 10. Adaptively regularized interpolated image with fusion algorithm (PSNR=29.69[dB])