OPTIMIZED RESIDUAL IMAGE FOR STEREO IMAGE CODING

Walid Hachicha 1 , Mounir Kaaniche 1 , Azeddine Beghdadi 1 , Faouzi Alaya Cheikh 2

1L2TI, Institut Galilee, Universite Paris 13, Sorbonne Paris Cite, France2The Norwegian Colour and Visual Computing Lab, Gjøvik University College, Norway

walid.hachicha@univ-paris13.fr, mounir.kaaniche@univ-paris13.fr, azeddine.beghdadi@univ-paris13.fr, faouzi.cheikh@hig.no

ABSTRACTMany research works have been developed for stereo imagecompression purpose by focusing on the disparity compensa-tion technique. For this reason, a great attention should bepaid to the generation of the disparity-compensated residualimage. Generally, the residual image is computed througha simple substruction of the disparity-compensated referenceimage from the target one. In this paper, we investigate twotechniques for optimizing the computation of the residual im-age. The obtained results confirm the benefits of these opti-mization approaches in the context of stereo image coding.

Index Terms— Stereo image, compression, disparitycompensation, optimization, `2 and `1 minimization.

1. INTRODUCTION

Stereoscopic imaging has gained a growing interest in manyemerging applications such as 3D TV, 3D Digital cinema, im-mersive games and videoconferencing. The main advantageof stereoscopic technology is that it enhances the depth per-ception and makes the 3D experience more vivid. This tech-nology requires the acquisition of two images, called left andright images, by recording two slightly different views an-gles of the same scene. Thus, adding the third dimension tothe viewer/player leads to the doubling of the image data sizecompared to the monoscopic case, and consequently, involvesa large amounts of stereo data. Therefore, it becomes nec-essary to design efficient stereo compression techniques forstoring and transmitting purpose.To this end, the simplest way for compressing stereo image(SI) is the independent coding scheme where the left and rightimages are encoded separately by using existing still imagecoders. However, since these images result from the pro-jection of the same 3D scene and so present similar visualcontents, it has been shown that more efficient joint codingschemes can be developed by taking into account the inter-image redundancies. More precisely, the common idea be-hind most of the existing methods is based on the followingsteps [1, 2]:

À First, one image is selected as a reference image (forexample the left one).

Á Then, the disparity map, representing the displacementfield between the pixels of the right and left images, isoften estimated using a block-based approach.

 After that, the right view, referred to as target image,is predicted from the reference one using the estimateddisparity map, and the difference between the originaltarget image and the predicted one, called residual im-age, is generated.

à Finally, the reference and residual images as well as thedisparity map are encoded.

While the disparity map is often losslessly encoded using aDPCM technique, the reference and residual images can beencoded in different transform domains [3–6].Therefore, the third step related to the generation of the resid-ual image plays a key issue in the design of an efficient jointstereo image compression scheme. Most of the reportedmethods apply a simple subtraction between each pixel of thetarget image and its homologous one in the reference image.In order to better exploit the inter-view dependencies, we pro-pose in this paper to use the neighborhood of the homologouspixel to predict the pixel of the target image. To this end,we investigate two minimization techniques, based on theuse of `2 and `1 criteria, for optimizing the generation of theresidual image. Note that the benefits of such optimizationcriteria have been recently shown in the context of the designof lifting operators in a wavelet-based coding scheme [7].

The rest of this paper is organized as follows. In Section2, the disparity compensation technique for stereo image cod-ing is presented. The developed optimization techniques forgenerating the residual image are described in Section 3. InSection 4, experimental results are given and conclusions aredrawn in Section 5.

2. DISPARITY COMPENSATED RESIDUAL CODING

Stereo matching algorithms consist of assigning to each pixelof the target image I(r) a displacement value that allows tofind its homologous point in the reference image I(l). Suchproblem has been extensively studied in computer vision, anda review of disparity estimation techniques can be found in

[8]. In the context of stereo image coding, block-matchingalgorithms are often used because all pixels belonging tothe same block can be represented by only one value, andso requires a few bits to encode the resulting disparity map.More precisely, after partitioning the target image into non-overlapping blocks, the disparity value associated to eachblock is obtained by minimizing a similarity criterion D asfollows:

d = arg mind0∈{1,...,dmax}

D(I(r)(i, j), I(l)(i+ d0, j)

), (1)

where (i, j) are the spatial coordinates of the top left pixel inthe block, and dmax represents the potential maximum dis-parity value. In general, the similarity criterion is the sumof square difference (SSD) or the sum of absolute difference(SAD).Once the disparity map is estimated, the target image I(r) canbe predicted by shifting the reference image I(l) along thehorizontal direction thanks to the disparity information. Then,the residual image I(e) is generated as follows:

I(e)(i, j) = I(r)(i, j)− I(l)(i+ d(i, j), j), (2)

Thus, the reference image, the residual one and the disparitymap are encoded and transmitted to the decoder side. Notethat the reconstruction of the stereo images from these datais straightforward. Indeed, the reference and residual imagesas well as the disparity map are firstly decoded. Then, thetarget image is predicted using the decoded reference imageand the disparity information. Finally, the target image is re-constructed by adding the decoded residual image to the pre-dicted target one.

3. OPTIMIZED RESIDUAL IMAGE

3.1. Motivation and computation strategy

Although most of the reported stereo image coding methodsgenerate the residual image according to Eq. (2), it is worthpointing out that this computation strategy may be subopti-mal. Indeed, some authors have proposed to weight the com-pensation term to take into account the illumination variationbetween the two views [9]:

I(e)(i, j) = I(r)(i, j)− αI(l)(i+ d(i, j), j), (3)

where α ∈ R∗+.Most importantly, the homologous pixel I(l)(i + d(i, j), j)may not appear exactly on the epipolar line due to the pres-ence of noise, the numerical rectification error, and the devi-ation from the pinhole camera model. Therefore, during thesubtraction (i.e prediction) operation, instead of only takingthe homologous pixel I(l)(i+ d(i, j), j), it becomes more in-teresting to use also its neighboring samples. More precisely,

we will consider the following equation for computing theresidual image:

I(e)(i, j) = I(r)(i, j)− p>I(c)(i, j), (4)

where

I(c)(i, j) =(I(l)(i+d(i, j)+m, j+n)

)−M/2≤m≤M/2−N/2≤n≤N/2

(5)

is the vector, of dimension (M + 1) × (N + 1), containingthe pixels of the left image used in the prediction step, and pis the prediction vector.Once the computation strategy of the residual image is de-scribed, we focus now on the optimal design of the vector p.The objective is to produce a compact representation of theresidual image by minimizing a given cost functional J :

popt = arg minp∈RL

J (p), (6)

where L = (M + 1)× (N + 1).In what follows, two minimization criteria will be investi-gated.

3.2. `2 optimization technique

Since the residual image corresponds to a prediction error, thevector of the prediction weights can be optimized by minimiz-ing the variance of the residual coefficients (i.e their `2-norm).Thus, the corresponding criterion is expressed as follows:

J (p) =

W∑i=1

H∑j=1

(I(r)(i, j)− p>I(c)(i, j)

)2, (7)

where W and H are the width and the height of the stereoimages.By minimizing this criterion, it can be checked that the opti-mal prediction vector popt must satisfy the well-known Yule-Walker equations:

E[I(c)(i, j)I(c)(i, j)>]popt = E[I(r)(i, j)I(c)(i, j)], (8)

where E[·] denotes the mathematical expectation.

3.3. `1 optimization technique

Sparse representations have attracted a great deal of atten-tion in many application fields during the last years such ascompressive sensing, image debluring and image compres-sion [7, 10]. Therefore, with the ultimate aim of increasingthe sparsity of the residual coefficients, we propose to studyalso an `1 optimization technique for designing the optimalprediction vector. More precisely, the objective consists inminimizing the following `1 criterion:

J (p) =

W∑i=1

H∑j=1

|I(r)(i, j)− p>I(c)(i, j)|. (9)

It should be noted that the sparsest residual coefficients couldbe obtained by minimizing an `0 criterion. However, sucha problem is inherently non convex and NP-hard. For thisreason, we have focused on the use of `1 criterion which isconvex and can be efficiently solved thanks to a class of prox-imal optimization algorithms [11]. Among them, we adoptthe Douglas-Rachford algorithm which was found to be sim-ple and effective for this problem [12].Before describing the algorithm, let us first address the nec-essary backgrounds on convex analysis and proximity opera-tors [11].

3.3.1. Background on convex optimization tools

We denote by RK the K-dimensional Euclidean space withnorm || · ||. The main definitions which will be used in thiswork are the followings:

• The indicator function of a convex setC ⊂ RK is givenby :

∀x ∈ RK , ıC(x) =

{0 if x ∈ C,+∞ otherwise.

(10)

• The distance function to a nonempty set C ⊂ RK isdefined by:

∀x ∈ RK , dC(x) = infy∈C||x− y||. (11)

• The projection of x ∈ RK onto a nonempty closed con-vex set C ⊂ RK is the unique point PC(x) ∈ C suchthat dC(x) = ||x− PC(x)||.

• Γ0(RK) is the class of lower semicontinuous convexfunction from RK to ] − ∞,+∞] and not identicallyequal to +∞.

• The proximity operator of a function f ∈ Γ0(RK), de-noted by proxf , is defined as follows:

proxf : RK → RK

x 7→ arg miny∈RK

f(y) +1

2||x− y||2. (12)

3.3.2. Douglas-Rachford algorithm

In the following, the algorithm adopted to generate the resid-ual image is described. We recall that the aim consists ofminimizing the `1-norm of the difference between the currentpixel of the right image I(r)(i, j) and its predicted value. Inthis context, the vector I(r) =

(I(r)(i, j)

)1≤i≤W1≤j≤H

can be seen

as an element of the Euclidean space RW×H . The minimiza-tion problem (9) can be rewritten as follows:

minX∈V

W∑i=1

H∑j=1

|I(r)(i, j)−X(i, j)|, (13)

where V = {X =(X(i, j)

)1≤i≤W1≤j≤H

| ∃ p ∈ RL, ∀(i, j) ∈

{1, . . . ,W} × {1, . . . ,H}, X(i, j) = p>I(c)(i, j)}.

According to the definition of the indicator function (10),the minimization problem (13) is equivalent to the followingone:

minX∈RW×H

f1(X) + f2(X), (14)

where

f1(X) =||I(r) −X||`1

=

W∑i=1

H∑j=1

|I(r)(i, j)−X(i, j)|, (15)

andf2(X) = ıC(X). (16)

By writing it under this form, the minimization problem (14)can be solved by using the Douglas-Rachford algorithm. Theobtained numerical solution can be obtained based on the fol-lowing iterative algorithm :

Fix S0 ∈ RW×H , γ > 0, ε ∈]0, 1[ andλ ∈ [ε, 2− ε]for k = 0, 1, . . . do

Xk = proxγf2Sk,Sk+1 = Sk + λ

(proxγf1(2Xk − Sk)−Xk

).

endAlgorithm 1: Douglas-Rachford algorithm.

As it can be seen in this algorithm, each iteration requires thecomputation of two proximity operators for the functions f1and f2. Note that closed-form expressions of the proximityoperator of some functions belonging to Γ0(RK) are devel-oped in [11].In our case, the proximity operator of the function γf1 isgiven by :

∀Sk ∈ RW×H , proxγf1(Sk) =(πk(i, j)

)1≤i≤W1≤j≤H

, (17)

where ∀(i, j) ∈ {1, . . . ,W} × {1, . . . ,H},

πk(i, j) = soft[−λ,λ](Sk(i, j)− I(r)(i, j)

)+ I(r)(i, j) (18)

and ∀t ∈ R,

soft[−λ,λ](t) =

{sign(t)(|t| − γ) if |t| > γ

0 otherwise.(19)

Concerning γf2, the proximity operator is given by:

∀Sk ∈ RW×H , proxγf2(Sk) = PV (Sk) (20)

=(p>I(c)(i, j)

)1≤i≤W1≤j≤H

,

wherep =

(∑i,j I

(c)(i, j)I(c)(i, j)>)−1∑

i,j Sk(i, j)I(c)(i, j).Finally, we should note that it has been shown in [13] thatevery sequence Xk generated by the Douglas-Rachford algo-rithm converges to a solution to problem (14) if the selectedparameters λ and γ satisfy the conditions given in Algorithm1.

4. EXPERIMENTAL RESULTS

The simulations are performed on different standard stereoimages downloaded from some public stereovision datasets 1,

2 and 3. A block-matching technique with a 8 × 8 block sizeis employed to estimate firstly the disparity map. The latteris then losslessly encoded by using a DPCM technique fol-lowed by an arithmetic coder. Finally, to encode the referenceimage as well the residual one, the 9/7 (resp. 5/3) wavelettransform, retained for the lossy (resp. lossless) compressionmode of JPEG2000 [14], is applied over three resolution lev-els. The resulting wavelet coefficients are then encoded byusing the entropy coder EBCOT.In order to study the proposed optimization strategies, we fo-cus on the generation of the residual image by considering thethree following experiments:

À The first one uses the standard computation proceduredefined by Eq. (2). We refer to this method by “Stan-dard”.

Á The second one consists of applying a weighting termas given by Eq. (3). The weight is computed by min-imizing the variance of the residual coefficients. Thismethod will be denoted by “Weighted-version”.

 The third and the fourth ones correspond to the pro-posed optimization strategies. More precisely, accord-ing to the notations used in Eq. (5), the pixels takenduring the disparity compensation process are obtainedby setting m ∈ {−1, 0, 1} and n ∈ {−1, 0, 1}. The op-timization based on the `2 and `1 criteria will be desig-nated by “Proposed-`2-OPT” and “Proposed-`1-OPT”,respectively.

In addition to these methods, we have also considered the in-dependent coding scheme, denoted by “Independent”, wherethe retained wavelet transform is applied separately to the left

1http://vision.middlebury.edu/stereo/2http://vasc.ri.cmu.edu/idb/html/stereo/index.html3http://vasc.ri.cmu.edu/idb/html/jisct/

and right images.First, the performance of these different methods will be eval-uated in the context of lossless coding mode. Since the ref-erence image is encoded in the same way for all the testedmethods, we give in Table 1 the entropy of the multiresolu-tion representations of the target image. The advantages ofsuch measure are that it is easily computed and it is indepen-dent of the performance of any embedded coder. Comparedto the standard approach, the proposed optimized computa-tion strategy of the residual image results in a gain which canreach 0.25 bits per pixel (bpp). It can be also observed thatthe `1 optimization technique achieves a further improvementcompared to the `2 optimization approach.

Table 1: Results in terms of entropy (in bpp) of the targetimages for the considered methods.

SI Shrub Art Cones DrumsticksIndependent 4.95 4.38 5.39 4.49

Standard 3.73 3.51 4.27 3.88Weighted-version 3.70 9 3.51 4.26 3.87Proposed-`2-OPT 3.39 3.51 4.22 3.88Proposed-`1-OPT 3.37 3.46 4.19 3.83

Moreover, these methods have also been evaluated in the con-text of lossy coding mode. Note that a closed loop-based cod-ing structure has been employed [5]. The reported results aregiven in terms of the average bitrateRav and its correspondingPSNR measure:

Rav =R(l) +R(e) +R(d)

2, (21)

PSNR = 10 log10

2552

(MSE(l) + MSE(r))/2, (22)

where R(l), R(e) and R(d) denote respectively the bitrate ofthe left, target, and disparity images. MSE(l) and MSE(r) cor-respond respectively to the mean squared error of the recon-structed left and right images. Figure 1 illustrates the perfor-mance of the different considered stereo images compressionmethods for the stereo image “Shrub” and “Houseof”. It canbe observed that the proposed strategy outperforms the stan-dard computation procedure by about 0.3-1 dB. Furthermore,we have also compared the `1-based optimization techniqueto the standard one in terms of Bjontegaard metric, oftenused to measure the distance between two R-D curves [15].The PSNR differences and the bitrate saving between thesetwo approaches are provided in Table 2 for low, middle andhigh bitrates corresponding to the four target bitrate points{0.15, 0.2, 0.3, 0.4}, {0.5, 0.6, 0.7, 0.8} and {0.9, 1, 1.1, 1.2}bpp, respectively. This table shows that the average of thePSNR differences between the R-D results varies between0.14-1.2 dB.Finally, we illustrate in Fig. 2 the reconstructed “Pentagon”stereo image. The reconstruction quality is evaluated in

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

32

34

36

38

40

42

bitrate (bpp)

PS

NR

(dB

)

Shrub

IndependentStandardWeighted-versionProposed-ℓ1-OPT

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

27

28

29

30

31

32

33

34

bitrate (bpp)

PS

NR

(dB

)

Houseof

IndependentStandardWeighted-versionProposed-ℓ1-OPT

Fig. 1: PSNR (in dB) versus the bitrate (bpp) after JPEG 2000encoding for the stereo images “Shrub” and “Houseof”.

terms of PSNR. As it can be seen in Fig. 2, the proposed`1 optimization strategy leads to better reconstruction qualitycompared to the standard approach.All these results confirm the benefits which can be drawnfrom the proposed techniques for optimizing the residualimage.

5. CONCLUSION

In this work, we have focused on the optimization of the resid-ual image to improve the stereo coding performance. Moreprecisely, for each pixel in the target image, we have proposedto use the homologous pixel in the reference image as wellits neighboring to generate the residual image. To this end,two optimization techniques including the `2 and `1 criteriahave been investigated. Experimental results show the inter-

Table 2: The average PSNR differences and the bitrate savingat low, middle and high bitrates for different Stereo images.The gain of the Proposed `1-OPT method w.r.t to the standardone.

SIPSNR gain (dB) Bitrate saving (%)

low middle high low middle highShrub 0.60 0.91 1.07 -11.97 -14.04 -13.81

Houseof 0.34 0.59 0.72 -10.19 -11.31 -10.80Pentagon 0.52 0.91 1.20 -14.89 -16.92 -17.36

Art 0.22 0.24 0.25 -4.14 -3.42 -3.01Cones 0.14 0.27 0.35 -3.42 -4.85 -4.49

average 0.37 0.58 0.72 -8.92 -10.11 -9.89

est of this procedure in the design of a residual-based stereocoding scheme. While the developed optimization strategyis performed by using a criterion defined in the spatial do-main, it would be interesting to investigate in future a moresophisticated criterion defined on the whole multiresolutionrepresentation of the residual image.

6. REFERENCES

[1] M. E. Lukacs, “Predictive coding of multi-viewpointimage sets,” in IEEE International Conference onAcoustics Speech and Signal Processing, Tokyo, Japan,1986, pp. 521–524.

[2] M. G. Perkins, “Data compression of stereopairs,” IEEETransactions on Communications, vol. 40, no. 4, pp.684–696, 1992.

[3] W. Hachicha, A. Beghdadi, and F. A Cheikh, “1D direc-tional DCT-based stereo residual compression,” in Eu-ropean signal Processing Conference, Marrakech, Mo-rocco, 2013.

[4] M. S. Moellenhoff and M. W. Maier, “Transform codingof stereo image residuals,” IEEE Transactions on ImageProcessing, vol. 7, no. 6, pp. 804–812, June 1998.

[5] N. V. Boulgouris and M. G. Strintzis, “A family ofwavelet-based stereo image coders,” IEEE Transactionson Circuits and Systems for Video Technology, vol. 12,no. 10, pp. 898–903, 2002.

[6] M. Kaaniche, A. Benazza-Benyahia, B. Pesquet-Popescu, and J-C. Pesquet, “Vector lifting schemes forstereo image coding,” IEEE Transactions on Image Pro-cessing, vol. 18, no. 11, pp. 2463–2475, 2009.

[7] M. Kaaniche, B. Pesquet-Popescu, A. Benazza-Benyahia, and J.-C. Pesquet, “Adaptive lifting schemewith sparse criteria for image coding,” EURASIP Jour-nal on Advances in Signal Processing: Special Issue on

(a) PSNR = 33.53 dB (b) PSNR = 34.04 dB

Fig. 2: Reconstructed “Pentagon” stereo image coded at Rav=0.23 bpp, (a) Standard and (b) Proposed-`1-OPT.

New Image and Video Armentieroises Based on Sparsity,vol. 2012, 22 pages, January 2012.

[8] D. Scharstein and R. Szeliski, “A taxonomy and eval-uation of dense two-frame stereo correspondence algo-rithms,” International journal of computer vision, vol.47, no. 1-3, pp. 7–42, 2002.

[9] W. Miled, J.-C. Pesquet, and M. Parent, “A convex opti-mization approach for depth estimation under illumina-tion variation,” IEEE Transactions on Image Process-ing, vol. 18, no. 4, pp. 813–830, 2009.

[10] D.L. Donoho, “Compressed sensing,” IEEE Transac-tions on Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.

[11] P.L. Combettes and J-C. Pesquet, “Proximal splittingmethods in signal processing,” Fixed-Point Algorithmsfor Inverse Problems in Science and Engineering, p.185, 2010.

[12] J. Eckstein and D.P. Bertsekas, “On the douglas-rachford splitting method and the proximal point algo-rithm for maximal monotone operators,” MathematicalProgramming, vol. 55, no. 1-3, pp. 293–318, 1992.

[13] P.L. Combettes and J-C. Pesquet, “A Douglas-Rachfordsplitting approach to nonsmooth convex variational sig-nal recovery,” IEEE Journal of Selected Topics in SignalProcessing, vol. 1, no. 4, pp. 564–574, 2007.

[14] D. Taubman and M. Marcellin, JPEG2000: Im-age Compression Fundamentals, Standards and Prac-tice, Kluwer Academic Publishers, Norwell, MA, USA,2001.

[15] G. Bjontegaard, “Calculation of average PSNR dif-ferences between RD curves,” Tech. Rep., ITU SG16VCEG-M33, Austin, TX, USA, April 2001.