A Proposed System for Colorization of a Gray Scale Facial Image using Patch Matching Technique

Santanu Halder, Abul Hasnat Department of Computer Science and Engineering Government College of Engineering and Textile

Technology, Berhampore, India

sant.halder@gmail.com, abul_05@sify.com

D. Bhattacharjee, M. Nasipuri Department of Computer Science and Engineering

Jadavpur University Kolkata-700032, India

debotosh@ieee.org, mitanasipuri@gmail.com

Abstract— This paper proposes a novel approach to colorize a gray scale facial image with the selected reference face using patch matching technique. The colorization methodology has been applied on facial images due to their extensive use in various important fields like archaeology, entertainment, law enforcement etc. The experiment has been conducted with 150 male and female facial images collected from different face databases and the result has been found satisfactory. The proposed methodology has been implemented using Matlab 7.

Index Terms—Patch matching, Exhaustive Method, Colorization.

I. INTRODUCTION OLOR information plays an important role as a key

attribute of an image due to its extensive use in various important fields like medical image processing, facial

image processing, satellite image processing, video processing etc. [1][2][3][4]. Colorization is a process that converts a gray scale image to a color one. Though it has a huge application in the area of archaeology, medical application, entertainment, law enforcement etc., but a very little research work has been observed in literature survey on this topic. For example, in medical application, sometimes it is not possible to diagnose the fracture in an X ray image which is in gray scale format. In 2010, Dr. M. S. Sathik and N. R. S. Parveen proposed a method to convert a gray scale X-ray image into a RGB space to diagnose the fracture properly [5]. In archaeology, the gray scale archive documents could be preserved by enhancing it into color image. In entertainment, many old movies and video clips, which are in gray scale mode, have been colorized manually which is normally a laborious process. In Law enforcement, gray level face image on Voter ID card or face sketch demands the color conversion to make it more prominent and to apply the color based facial image processing algorithm on them. Also during data transmission, the channel bandwidth utilization will be more effective when the gray scale images are transferred over the channels compared to transfer the color images over the same channel as a gray scale image takes only 8 bit per pixel to store the pixel intensity whereas a color image takes 24 bits in RGB mode [2][4]. Again, as the gray scale face images do not contain any color information, the color based

facial image processing algorithms [6][8] cannot be applied on them. Govind Haldankar et al. [9] proposed a system which modifies a gray scale image into a color one by the luminance effect of the source image. But the time needed for colorization is huge for a large size image and hence the method is less effective in real time system. For these large applications of facial images, this paper proposes novel system which aims to convert a gray scale face image into color one in a simple method.

This work is a part of our main research work which aims to colorize a gray scale facial image based on the textual color description given by the user. The system works through five phases: (i) Extraction of facial components from a given facial image (ii) Analysis of the extracted components and storage of them along with their corresponding textual color labels into the database for later use (iii) Handling of queries for reference face image using textual color description (iv) Searching for the reference face into the database and finding the most suitable one from the selected faces (v) Colorizations of the given gray level face with the selected reference face using patch matching technique. This paper only concentrates on the last phase i.e. the colorization phase. This phase works in four steps:

I. Finding the gray level value of the reference face.

II. Searching for the best patch on gray level reference face for a particular patch of the problem face using Exhaustive patch matching technique.

II. Colorization of the gray level problem patch with the color values of selected reference patch.

III. Tuning the colorized face image to remove the noise generated by some erroneous patch selection.

A variety of face databases have already been described in literature. Here we have worked with FRAV2D color face database [10] and ORL face database [11] from AT & T Laboratories Cambridge.

This paper is organized as follows: Section II presents the preprocessing required for the images in the present work. Section III briefly discusses the proposed method. Section IV discusses the patch matching technique. Section IV describes

C

2013 IEEE Recent Advances in Intelligent Computational Systems (RAICS) | 19 - 21 December 2013

the method of colorization. Section V discusses how the colorized image is tuned to remove noise generated by some erroneous patch selection. Section VI shows the experimental results and finally section VII concludes and remarks about some of the aspects analyzed in this paper.

II. PREPROCESSING As the present work concentrates on the colorization of the

face images only, only preprocessing carried out on the face database is to crop the face images from hair to chin (top to bottom) and from right ear to left ear (left to right).This is done to focus on the face regions not excluding the background subjects. Fig. 1 shows some face images and their cropped images.

(a) (b)

Fig. 1. Example of preprocessing. (a)Original images (b)Cropped images.

III. PROPOSED METHOD This paper aims to generate a color image of a gray scale

face using a reference color face image IRC(a,b,c). Let the gray level face image IRG(a,b) be the converted grayscale image of the RGB color face image IRC(a,b,c).Assume the sizes of ISG(a,b) and IRG(a,b) are S1×S2 and R1×R2 respectively. Now the input images including the reference image IRG(a,b) are divided into patches. A patch is considered to be a p×p pixel grid which is used to subdivide the images into smaller problem space. Before dividing an image into patches, the size of the image needs to be normalized to a multiple of p. The new height (H�) and width (W�) of a size normalized image are calculated from its original height (H) and width (W) using Eq. 1 and Eq. 2.

ppHH ×⎥⎦

⎥⎢⎣

⎢=' (1)

pp

WW ×⎥⎦

⎥⎢⎣

⎢=' (2)

Fig. 3 shows the patches of different sizes on a color face image after the normalization of size. The patches are numbered from 1 to P, where P is total number of patches for an image as shown in Fig. 4.

Patch Size: 4×4 10×10 20×20

Fig. 2. Example of a face image showing patches of different sizes

Fig. 3. An example face with patch numbers ( P=80)

Suppose the total number of patches on ISG(a,b) and IRG(a,b)

are PS and PR respectively. Then the value of PS and PR are obtained from Eq. 3 and Eq. 4.

pS

pSPS

21×= (3)

where 21 SS × is size of source grayscale image ISG(a,b)

pR

pRPR

21 ×= (4)

where 21 RR × is size of reference RGB image ISG(a,b)

Next, for each patch yj of image ISG, the system finds the best matching patch jx̂ from the patches xi of the gray scale

image IRG. Actually, the time complexity of colorization process is directly proportional to the time needed for patch matching. A patch xi on IRG contains the gray level value of pixel intensities and the same patch on IRC contains the color information in RGB color space. Now the patch yj is colorized using the color information stored in patch jx̂ on reference

color image IRC. Finally the system removes the noise generated by some erroneous patch selection during patch matching from the constructed color face image ISC and outputs the image. The steps for colorization are shown in Fig. 4.

Fig. 4. Steps for Colorization

IV. PATCH MATCHING This section describes the methodologies to find the best

matching patch jx̂ from the patches ix s of reference grayscale

converted image for the patch yj on problem image. The distance between patch yj on ISG and patch xi on IRG is calculated using Eq. 5.

[ ]2

1 1),(),(∑ ∑

= =−=

p

k

p

l RGSG lkIlkIdist (5)

Next the present system aims to find the patch jx̂ on IRG for

which the distance between patch yj and patch xi is minimum and the color information for patch yj is obtained from the patch corresponding to jx̂ on IRC. Minimum distance signifies

that patches yj and jx̂ is most similar in terms of their color

content. Now all the patches xi on IRG is explored against each patch yj of ISG. Thus a set of distance values {di} is produced for patch yj and patch jx̂ is calculated corresponding to

minimum {di} value. Let the total number of patches of ISG and

IRG are PS and PR respectively. The steps are described as follows:

Step 1: Repeat Step 2 to Step 7 for j=1 to PS times

Step 2: {d} = �

Step 3: Repeat Step 4 to Step 5 for i=1 to PR times

Step 4: Calculate distance disti (1≤ i ≤ PR) between patch j and patch i.

Step 5: idistdd ∪= }{}{

Step 6: }{minˆ1 i

RPij dd≤≤

=

Step 7: Mark xi corresponding to jd̂ as jx̂ .

Fig. 5 shows an example image colorized by the method using the Exhaustive search based patch matching technique.

(a) (b) (c)

Fig. 5. Example of colorization by patch matching using exhaustive method (a) Problem image (b) Reference image (c) Generated image

Thus the most suitable patch in the reference image is found

and the next section describes the colorization process.

V. COLORIZATION In the previous sections, the methodologies for the selection

of a patch jx̂ on ),( baI RG similar to a patch jy on

),( baI SG have been discussed. But in the position of patch

jx̂ on reference color image ),,( cbaI RC , the RGB color

information is stored as shown in Fig. 6. This color information is used to colorize the patch jy on gray scale source image.

The process of colorization is described in this section.

Fig. 6. Patch selection on reference grayscale image

Suppose, patch yj of size p×p is represented as ),(ˆ baISG

and jx̂ of size p×p×c on ),,( cbaI RC is represented

as ),,(ˆ cbaIRC . The sum of pixel intensities on ),,(ˆ cbaIRC is calculated as:

( )∑ ∑ ∑

= = ==Φ

p

a

p

b c RCjx cbaI

1 1

3

1

ˆ ,,ˆ (6)

Now the mean values jxkˆψ ( { }BGRk ,,∈ ) of Red (R),

Green (G) and Blue (B) components of the sub-window

),,(ˆ cbaIRC are computed as:

∑ ∑= =

×Φ

=p

a

p

b RCjxjx

R baI1 1

ˆ

ˆ )1,,(1ψ (7)

∑ ∑= =

×Φ

=p

a

p

b RCjxjx

G baI1 1

ˆ

ˆ )2,,(1ψ (8)

∑ ∑= =

×Φ

=p

a

p

b RCjxjx

B baI1 1

ˆ

ˆ )3,,(1ψ (9)

Next the R, G and B values for the sub-window ),(ˆ baI SG (or

problem patch yj) are calculated using Eq. 10 to Eq. 12 and stored in ),,(ˆ cbaI SR .

),(ˆ)1,,(ˆ ˆ baIbaI SGjx

RSC ×=ψ (10)

),(ˆ)2,,(ˆ ˆ baIbaI SGjx

GSC ×=ψ (11)

),(ˆ)3,,(ˆ ˆ baIbaI SBjx

BSC ×=ψ (12)

In this way every patch of the grayscale problem image is colorized using the selected RGB patch in the reference RGB color face image. The next section describes how the system removes the noise in the colorized face image due to some erronous patch selection.

VI. TUNING THE COLORIZED FACE IMAGE Now some noises may be generated after colorization of the

gray scale problem image due to erroneous patch selection. To remove this unwanted noise, a median filtering technique is used. The filtering is applied on the patch matrix. A patch

matrix )( kPM is defined as a p

Sp

S 21 × matrix for a S1 × S2

image with p × p patch size which contains the jxk

ˆψ value for each patch of ),( baISG . Thus three patch matrices PMR, PMG and PMB are generated and the filtering technique is applied on each of them separately. These modified jx

k

ˆψ values are used to find the RGB intensities for the problem image using Eq. 10 to Eq. 12. Fig. 7(b) shows the effect of filtering technique on the noisy face.

(a) (b)

Fig. 7. Effect of filteing technique (a) Colorization of a gray scale face without filtering (b) Colorization of same face after filtering

The next section shows the experiemntal results of the

proposed system.

VII. EXPERIMENTAL RESULT The proposed methodology has been tested on 110 male

and 40 female face images of FRAV2D database [11] and ORL face database [12].

Fig. 8 shows the effect of colorization of same grayscale face image using different reference color face images.

Fig. 8. Colorized face images using different Reference face images

Fig. 9 shows the colorized face images using the proposed method.

(a) (b) (c)

Fig. 9. Examples of Colorized Images (a) Gray level problem Images (b) Color Reference Images (c) Colorized Images

The proposed methodology has been applied also on the

gray scale images of ORL face database [11] to color the images. Some output images are shown in Fig. 10.

(a) (b)

Fig. 10. Some example images of ORL database after applying the colorization method (a)Grayscale images (b)Colorized images.

VIII. CONCLUSION The proposed system colors a gray level face image from a

reference color face using patch matching technique based on exhaustive search. The method selects a suitable patch on the reference image and colorizes the problem image by finding the color information from it. The proposed method has a huge application in the area of archaeology, entertainment, law enforcement etc. This paper concentrates on the colorization of gray level facial images only which may be extended for any kind of images and this may open new era in colorization of grayscale images in future.

ACKNOWLEDGMENT Authors are thankful to the "Center for Microprocessor

Application for Training Education and Research", "Project on Storage Retrieval and Understanding of Video for Multimedia" at Computer Science & Engineering Department, Jadavpur University, for providing infrastructural facilities during progress of the work. Mr. Santanu Halder is thankful to Government College of Engineering and Leather Technology, Kolkata and Mr. Abul Hasnat, is thankful to Government College of Engineering and Textile Technology, Berhampore, for kindly permitting them to carry on the research work.

REFERENCES [1] R. C. Gonzalez, R. E. Woods, S. L. Eddins, “Digital Image processing

using MATLB”, Mc-Graw Hill, 2011. [2] W. K. Pratt, “Digital image processing”, A Wiley Interscience

Publication, 1991. [3] Scott E Umbaugh, “Computer vision and image Processing”, Prentice

Hall, 1999. [4] R. C. Gonzalez, R. E. Woods, “Digital Image Processing”, Addison

Wesley, 2002. [5] M. M. Sathik, N. R. S. Parveen, “Feature Extraction on Colored X-Ray

images by Bit-Plane Slicing Technique”, International Journal of Engineering Science and Technology, Vol 2(7), pp. 2820-2824, 2010.

[6] A. Hadid, M. Pietikinen and B. Martinkauppi, “Color-Based Face Detection using Skin Locus Model and Hierarchical Filtering”, Proceedings. 16th International Conference on Pattern Recognition, pp. 196 – 200, 2002.

[7] D. Bhattacharjee, S. Halder , M. Nasipuri, D.K. Basu, M. Kundu, “Construction of Human Faces from Textual Descriptions”, Soft Computing - A Fusion of Foundations, Methodologies and Applications, Vol. -15, No. – 3, pp. 429-447, November 2009.

[8] D. Chai, and K.N. Nghan, “Face Segmentation using Skin Color Map in Videophone Applications”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 9, No. 4, 1999.

[9] Govind Haldankar, Atul Tikare and Jayprabha Patil, “Converting Gray Scale Image to Color Image”, Proceedings of SPIT-IEEE Colloquium and International Conference, Mumbai, India, Vol. 1, 189.

[10] FRAV2D Database (2004), Freely available from: http://www.frav.es/databases/frav2d/

[11] ORL Database, http:// www.cl.cam.ac.UK/ Research/ DTG/ attarchive:pub/data/att_faces.zip