1

ABSTRACT

In this paper we describe ongoing work in an advanced

disparity algorithm developed for interactive systems. First we have developed an adaptive interpolation algorithm based on a bilinear interpolation for image resizing; and second we have integrated this algorithm to create a multi-screen view suitable for the interactive television. We propose a new hardware architecture that enables implementation of custom video enhancement algorithms.

Keywords: Image processing, scaling algorithm, multi-resolution analysis, multi-tuner, Set Top Box, video.

I. INTRODUCTION The interactive television is undoubtedly taking place. It has

created a new way to watch television and covered a wide range of systems and services inextricably linked to the multimedia concept. Literally, the terms refers to any presentation, any program composed of images, audio, graphics, text or other representations viewable. In practice, these elements are interconnected by a degree of interactivity in the sense that the user can control the assembly of items or move from one to another.

The new model of digital broadcasting, including digital television, which is in the form of a container of digital data rather than a transport mechanism for a specific programs channel, pushes the interactive mode and supports the development of techniques giving it life. Indeed, the rapid growth of multimedia documents requires the development of descriptors for content analysis and cooperative techniques providing. The content description is one of the main vectors of interactivity and has several application fields such as: creation of video summaries, search for a category of channels (News, newspaper, cartoon, sports ...), and event detection (registration of a goal in a football game, survey of the end of the advertisement ...). The results of these applications should be returned to viewers as multiple display modes such as PiP (Picture in Picture), POP (Picture out Picture), PBP (Picture by Picture).

The viewer can therefore have a local interactivity in the form of a mosaic style display which displays multiple videos in a screen simultaneously. In order to make these display modes suitable for viewers, we propose to size and resize images and video with a specific algorithm. The task at hand will be to evaluate and compare different interpolations methods. We motivate this study by using one of these methods to implement suitable modes of multi-screen display such as picture in picture (PiP), picture out picture (PoP) and mosaic style screen.

II. IMAGE SCALING METHODS There is a variety of image scaling methods that may be

used for such functions as enlarging or reducing the size of an image. Three extended methods are used for image scaling and resizing:

The first method performs resizing without becoming resembling. It keeps the same image resolution. However, the second method changes the image resolution. Indeed, the resembling is a change in size or resolution of an image that will cause a change in the number of pixels. Coordinates and color values of resembled image pixels are calculated by interpolation techniques. The third method, we will call smart resizing. It performs removing or adding connected path of low energy pixels called seams.

In this paper, we will focus on interpolation techniques used for image scaling and resizing.

Image interpolation is widely used in digital image processing tasks such as image scaling, image warping and image restoration. Interpolation is a method of constructing new data points within the range of discrete set of known data points. In communication, interpolation is most important for signal transition. These methods could be processing signal including video and audio. Interpolation methods can be divided into two categories: the non-adaptive interpolation methods and the adaptive interpolation methods. [1, 2]

Adaptive methods change depending on what they are interpolating, but non-adaptive methods treat all pixels equally. A comparison between different techniques will be made. In this paper, we are mainly concerned about image interpolation. However, interpolation in two dimensions for the general case is sometimes difficult to describe. For gridded

Software Application for Interactive Video Multi-Screen

Mhamed Frad EμE laboratory, CSR group, university of

Monastir Email:

mhamed.frad@gmail.com

Asma Chaouch EμE laboratory, CSR group, university of

Monastir Email:

chaouch.asma@gmail.com

Lamjed Touil EμE laboratory, CSR group, university of

Monastir Email:

lamjedtl@yahoo.fr

Abdellatif Mtibaa EμE laboratory, CSR group, university of

Monastir Email:

abdellatif.mtibaa@enim.rnu.tn

978-1-4577-2209-7/11/$26.00 ©2011 IEEE

2

data, the n-dimensional interpolation function can be described as the product of n one-dimensional interpolation functions. Therefore it is permitted to look at one dimensional interpolation functions to discuss the behavior of the n-dimensional interpolation functions.

In the present paper we made a comparison between the following algorithms: Box Algorithm (Nearest Neighbour- NN) [3,4] ; Bilinear Interpolation (BI) [5,4]; Bicubic Interpolation (BCI) [6,7] to chose the preferment method for our algorithm.

A. Quantitative Comparison Estimating the quality of the scaling process of the image is

accomplished by following a strict succession of actions. These three steps are: scaling the image up or down using some of the tested algorithms, recovering the original resolution using the same methods and measuring MSE, PSNR and d (Kullback distance) between the original and the final image. There is no limitation for the value of the scaling coefficient, such as being integer [11]. It can be even a floating point number. When processing the selected image, for obtaining the final resolution is formed a so called “Frame”, comprising of four images: the original one, the intermediate one – obtained at the first step of the processing (reduction or magnification); the final one – obtained at the second step of the processing (for one-step image processing the intermediate and the final images are identical) and another one – represents the visual (RGB) difference between the original and the final images.

For objective estimate of the scaling algorithm’s quality is used the main approach in digital image processing, based on computing MSE, PSNR and KLD. MSE (1) represents the average square error (difference) in the intensities [I (i; j) and I’(i; j)] of a given color of the corresponding pixels from the both images and it is dimensionless quantity. Because of its simplicity, it is the most common method of objectively measuring image quality given a reference image.

1 1

0 0

1( , ) ( ( , ) ( , ))M N

o r o ry x

MSE I I I x y I x yMN

− −

= =

= −∑∑

(1)

PSNR represents the square of the ratio between the maximum of the signal (the maximal possible value of color’s intensity (Imax)) and the mean square error (difference) in the intensities [I(i; j) and I’(i; j)] of a given color of the corresponding pixels from the both images and is measured in decibels (dB)

2

10255( , ) 10log ( )

( , )o ro r

PSNR I IMSE I I

=

(2)

KLD Kullback-Leibler Distance (KLD) is a measure of distance between two probability density functions (pdfs) KLD is minimal when the pdfs are equal.

[ ]0,255

( )( ) log( )( )x

p xKLD p xq x∈

=∑

(3)

p(x) , q(x) : probability in two point for two different image.

The estimate can be done by computing MSE, PSNR and KLD for every single color (Red, Green, Blue), as well as totally for the three of them. Experimental Results

TABLE 1

PSNR (DB) IN DIFFERENT SCALING FACTOR (SF) AND FIGURES SF Bilinear Nearest

neighbor Bi-cubic

2.5 27.5404 25.1534 28.2121 1.66 30.6506 28.2728 32.0667 1.43 31.7364 29.4947 33.4378 1.33 32.4638 29.6443 34.3431 1.25 32.7368 30.6012 34.6872 KLD 0,242 0,342 0,158 As shown in Table 1, the nearest neighbor interpolation

function, as the simplest one, is the method which has the smallest PSNR and the biggest error of KLD while the bicubic method is the method which has the largest PSNR and the smallest KLD.

So, the table 1 shows that the PSNR and KLD values of bicubic interpolation is greater than those of others interpolation method.

B. Qualitative Comparison In this part, we resize the image by nearest neighbor,

bilinear and cubic interpolation algorithms. The experimental results show that cubic interpolation algorithm is better than other scaling-up algorithms in terms of speed and quality. [8, 9]

To follow as next two figures, we first make original image scale up by these three methods then scale down by these three methods respectively. The scaling up factor is 2.7 and the scaling down is 0.77.

As is shown in (Fig.1.a) and (Fig.1.b) Bicubic interpolation have better sharpness than other methods; nearest neighbor interpolation creating obvious pixilation (Fig.5.b) or edges that break up curves into steps or jagged edges(in scaling up) , also called "jaggies”, Although in bilinear interpolation edge contrast is reduced by averaging neighboring values together (Fig.1.c). So, the better quality results obtained with bicubic interpolation algorithm but it is the slowest one compared by the other methods, because Bicubic interpolation analyzes the 16 pixels around each pixel and uses that information for resizing and for this reason that we work with bilinear interpolation algorithm in the rest of our project.

3

Fig.1. Detail of image magnified by 2 using:

III. PROPOPSED MULTI-TUNER ARCHITECTURE FOR SET-TOP BOX

In the conventional method, mosaic screen might be available in two different cases. The first case is that service provider prepares an extra mosaic screen by organizing individual different contents providers and broadcast it through a separate channel. Users can watch mosaic screen by decoding only this channel at the receiver. The other case is to receive all the streams of mosaic screen, then decode them simultaneously and displays them at a time.

The first case is only possible when service provider has a control of all the contents of different content providers such as satellite and cable TV broadcasters. And they should prepare a separate channel for mosaic screen. However, in terrestrial broadcasting, each channel is owned by different stations. In this case there is no way to create mosaic screen of contents of different broadcasters.

Although there were separate mosaic channels, it is also impossible for users to compose their own mosaic screen of desired channels. In this paper we only consider the case that there is no separate mosaic channel.

Fig3. Proposed multi-tuner Architecture for Set –Top Box

Bilinear, bicubic and nearest neighbor interpolation

The second case is to display multiple channels by receiving and decoding streams for mosaic screen simultaneously in the receiver part. P. Poterbic et al. proposed a method to display mosaic of multiple still images on the screen by regularly updating a snapshot of the channels using a second tuner. However this method only shows a still mosaic image on the screen, furthermore when a user chooses to view the channel that snapshot represents.

In this paper, we propose a new architecture of Set-Top Box that enables viewers to watch multiple channels simultaneously [12,13].

The hardware architecture includes a limited number of tuners, a public processor with the on-chip programmable logic portion. The on-chip programmable logic portion can be configured or programmed to implement enhancement video algorithms. There are numerous such video enhancement algorithms. One particular example is an up or downscaling algorithm. Furthermore, the proposed architecture needs as many decoders as to decode received streams simultaneously.

Currently, researchers have several options. They can develop code from scratch, but the complex nature of common image processing makes this approach impractical.

4

A more commonly used option is to use standard libraries to build codes with the desired functionality. Allegro, is a library of plain C functions and procedures that are used specifically for image processing. The whole library is in plain C except for one class (fix) which has been made for those who want to code in C++. We therefore designed and implemented our algorithm using allegro primitives.

IV. IMAGE SCALING ALGORITHM SPECIFICATION

In the present paragraph, we describe the proposed algorithm. This latter supports various functions related with image scaling. This section is intended to give the reader a feel for algorithm functionality. We outline the major functions defined by this algorithm that illustrate its power and use[12,14].

The first function is used to perform image scaling. We have chosen a bilinear interpolation technique for the function of resize because it is the convenient tool that offers simple implementation and efficiency in the execution of the resize.

To understand this function, it is helpful to understand the data types provides. The basic abstractions in this function are:

Bitmap: A Bitmap Object is declared as a BITMAP Structure

Screen: This is a bitmap that point directly to the screen located in the hardware of the video card. Drawing on a screen bitmap draws directly at the screen.

Buffer: is a memory bitmap, this kind of bitmap is used to store images or create video buffers.

Thus, this function takes three arguments: Src: this is the original image having size of 800 x 600. wd: is the width of the resized image. hd: is the height of the resized image.

This function gives a resized image with the wanted width and height. First, it scales up the original image then it compares the obtained size with the desired size. Then, it scales down the original image if it is bigger than the obtained size else it scales up.

A. Picture in Picture (PIP) implementation

The present function relates to a Picture in Picture television which can reproduce two different pictures simultaneously on a common screen display.

Fig.3. Picture In Picture mode

So, the first steps to create the PIP effect can be briefly

stated as follows: given two input image in the buffer, resize them according to the the wanted size, paste the smaller image in the other image, and display the buffer in the full screen.

This function resizes the two pictures loaded from the HDD and places them into the adequate location in the screen. To give viewer better control, this function gives user the ability to change by enlarging one of them and reduce the other using the mouse.

void PIP(BITMAP *image1,BITMAP *image2) {

BITMAP *dest1,*dest2,*img; int x1,x2,y1,y2;

clear_bitmap(buffer); dest1=ajuster_image(image1,800,600); dest2=ajuster_image(image2,200,200);

blit(dest1,buffer,0,0,600,0,200,200); blit(dest2,buffer,0,0,600,200,200,200);

blit(buffer,screen,0,0,0,0,800,600);

While(mouse_b & 1) { remplacer(dest1,dest2) ; Afficher(dest1,dest2,x1,x2) ; } }

B. Picture out Picture implementation The present function performs generating a plurality of

picture out picture screens and allows dividing a display screen into a plurality of sub-screens. The size of written small pictures is specified into the resizing function

Fig.4. Picture Out Picture mode

As shown in the example, the screen is divided in four sub-screens: three favorite programs configured by the user and a main program. Therefore, the viewer can easily select his preferred program. A personalized mosaic can be then realized according to his interests.

Figures 4 and 5 show the sample screen shot for picture in picture and picture out picture style screen which is playing many videos simultaneously.

5

Fig 5. Different Model of picture Out Picture

void POP(BITMAP *image1,BITMAP *image2,BITMAP *image3,BITMAP *image4) { BITMAP *dest1,*dest2,*dest3,*dest4,*img; int x1,x2,x3,x4,y1,y2,y3,y4,t1; clear_bitmap(buffer); t1=1; dest1=ajuster_image(image1,200,200); dest2=ajuster_image(image2,200,200); blit(dest1,buffer,0,0,600,0,200,200); blit(dest2,buffer,0,0,600,200,200,200); blit(dest3,buffer,0,0,600,400,200,200); blit(dest4,buffer,0,0,0,0,600,600); blit(buffer,screen,0,0,0,0,800,600); While(mouse_b & 1) { img=Chercher(mouse_x,mouse_y) ; remplacer(img,image1) ; Afficher(dest1,dest2,dest3,dest4,x1,x2,x3,x4) ; } }

C. Zooming function implementation

This function performs the zoom operation. It allows to the viewer to scale up and down the image by the mouse buttons.

1 void resize(BITMAP *image1) 2 { 3 BITMAP *dest1,*img; 4 clear_bitmap(buffer); 5 dest1=ajuster_image(image1,200,200); 6 blit(dest1,buffer,0,0,600,0,200,200); 7 blit(buffer,screen,0,0,0,0,800,600); 8 if(mouse_b & 1) 9 { 10 img=zoom(dest1) ; 11 Afficher(dest1) ; 12 } 13 Else if (mouse_b & 2) 14 { 15 Img=reduce(dest1) ; 16 Afficher(dest1); 17 } 18 }

Fig 6. Scaling Mode

V. APPLICATION EXAMPLE FOR VIDEO-SURVEILLANCE

To confirm the feasibility of our proposed algorithm, we

have considered a system with multiple IP cameras which was elaborated in our laboratory [17].

Figure 7 shows the different algorithm integrated in the video style screen which is playing many videos simultaneously.

6

Fig.7. Video result of the different display mode

VI. CONCLUSION Interactive video has been used in several areas as television

interactive (Set Top Box), mobile, games, video-surveillance. In this paper, we have presented a software application for

interactive video that could be implemented in hardware/software architecture of multi-screen video.

Our application is based on interpolation algorithms. We have realized a comparative study between different coexisting interpolation algorithms based on comparison between the two parameters: PSNR coefficient and KLD distance. We have chosen at last the bilinear interpolation algorithm.

We have used this algorithm to create four types of interactive displaying video modes which are Picture out Picture, Picture in Picture, scaling mode and mosaic mode. These four displaying modes give to the user the possibility to interact with TV programs by using a Set Top Box. To implement this algorithm, we have suggested a new Architecture of Set Top Box based on FPGA.

ACKNOWLEDGMENT This work was supported by the CSR group under contract

A.M. (Abdellatif Mtibaa), we thank the members of the group for their support and encouragement, and we also thank Nidhal Krifa to his contribution in the implementation part of this work.

REFERENCES [1] El Hassani M, "Segmentation basée-région pour l’amélioration de

séquences vidéo : Algorithme-Architecture dans un contexte temps réel", Thèse de doctorat de l’université de Caen /Basse Normandie, 07 août 2006.

[2] B B Stival F, Delubac, "Algorithme d'interpolation d'images utilisant la variation totale", Projet de méthode de traitement des images", Master in Televison Systems and image, 2000-2001. http://www.tsi.enst.fr/tsi/enseignement/ressources/mti/interpol_tv/index.html

[3] Hurley T.R., “Evolution of the Digital Set-top Box,” International Broadcasting Convention, (CONF. Publ N. 428) pages: 277-282, 1996.

[4] Robert G. Keys. "Cubic Convolution Interpolation for Digital Image Processing[J]". IEEE, Speech and Signal Processing, Vol.Assp-29, No.6, December 1981.

[5] DE Lescure B, Roudier F, "Method for processing video data to be displayed on a screen and implementing device", United States Patent, 13 January 2000

[6] P. Potrebic, J. D. Yaksick, G. R. Smith, “Systems and methods Using Multiple Tuners,” United States Patent, US6804824, October 2004.

[7] A. Boucherat, J C Aribaud, "Rééchantillonnage", Projet de méthode de traitement des images, ", Master in Televison Systems and image, 1998 http://www.tsi.enst.fr/tsi/enseignement/ressources/mti/reechantillonnage/plan.html

[8] Y Wintz, "Rapport d'étude et de recherche redimensionnement d'image intelligent", 2007-2008

[9] Y Kyeong Seong and Y-Hee Choi, "A Method for Watching Multiple Channels Simultaneously in a Digital Television", Consumer Communications and Networking Conference, 2008. CCNC 2008. 5th IEEE Volume, Issue, 10-12 .pp.:841 – 842 . Jan. 2008.

[10] Christophe Boyanique, " Rééchantillonnage d'images ", http://www.raceme.org /imac/redim/

[11] N Mueller," Image Interpolation Using Multiscale Geometric Representations", IEEE, University of Illinois at Urbana-Champaign, November 1995.

[12] M.Frad, A.Mtibaa, "Contribution à l’étude et l’implantation d’un système de réception numérique multi-tuners sur des systèmes embarqués, Master in Electrical Engineering from National Engineering School of Monastir, 13 February 2010.

[13] M.Frad, N.Elgbasi, A.Mtibaa, "Contribution à l’étude et à l’intégration sur SOPC des récepteurs numériques pour les TV numériques (décodage canal/décodage source)", Interne memo in CSR-ENIM-Groupe. Ref. : ENIM, 620.07.072, 09 juin 2008.

[14] A. Chaouch, A.Mtibaa, Contribution a l’étude et à la génération des IPs pour le multi affichage multi-screeen", Interne memo in CSR-ENIM-Groupe. Ref. : ENIM, 620.07.182, 27 juin 2009

[15] M.Frad, N.Elgabsi, L.Touil, A.Mtibaa "FPGA based satellite Set-Top Box prototype design", International Conference on sciences and techniques of Automatic control & computing engineering, 20 – 23 December 2008.

[16] D Hubaux1, L Guedria1, L Vandendorpe, M Verleysen et J-D Legat, "Nouvelles méthodes de conception de systèmes électroniques intégrés", Hammamet, Tunisie, 10 - 13 avril 2002.

[17] A.Hichri, " Etude et développement d’architectures dédiées à la navigation robotique", thesis in preparation.