2

State-of-the-art and trends in scalable videocompression with wavelet based approaches

Nicola Adami, Member, IEEE, Alberto Signoroni* Member, IEEE and Riccardo Leonardi, Member, IEEE

Abstract—Scalable video coding (SVC) differs form traditionalsingle point approaches mainly because it allows to encode ina unique bit-stream several working points corresponding todifferent quality, picture size and frame rate. This work describesthe current state-of-the-art in SVC, focusing on wavelet basedmotion-compensated approaches (WSVC). It reviews individualcomponents that have been designed to address the problem overthe years and how such components are typically combined toachieve meaningful WSVC architectures. Coding schemes whichmainly differ from the space-time order in which the wavelettransforms operate are here compared, discussing strengths andweaknesses of the resulting implementations. An evaluation ofthe achievable coding performances is provided considering thereference architectures studied and developed by ISO/MPEG inits exploration on WSVC. The paper also attempts to draw alist of major differences between wavelet based solutions andthe SVC standard jointly targeted by ITU and ISO/MPEG. Amajor emphasis is devoted to a promising WSVC solution, namedSTP-tool, which presents architectural similarities with respectto the SVC standard. The paper ends drawing some evolutiontrends for WSVC systems and giving insights on video codingapplications which could benefit by a wavelet based approach.

Index Terms—Scalable video coding, wavelets, spatiotemporalmultiresolution representations, motion compensated temporalfiltering, entropy coding, video coding architectures, MPEG,video quality assessment

I. INTRODUCTION

Traditional single operating point video coding systemscan be surpassed by using scalable video coding (SVC)architectures where, from a single bit-stream, a number ofdecodable streams can be extracted corresponding to variousoperating points in terms of spatial resolution, temporal framerate or reconstruction accuracy. Different scalablity featurescan coexist in a single video coded bit-stream with codingperformance approaching the state-of-art single point codingtechniques. This has been more and more a trend in the lastyears [1] and it has become a reality thanks to the developmentof SVC systems derived either from hybrid schemes [2] (usedin all MPEG-x or H.26x video coding standards) or fromspatiotemporal wavelet technologies [3]. A great part of thisdevelopment and relative debates regarding SVC requirements[4], applications and solutions has been carried out by peopleparticipating in the ISO/MPEG standardization.This paper offers an overview of existing SVC architectures

All authors are with the Department of Electronics for Automation, Facultyof Engineering, University of Brescia, via Branze n.38, I-25123, Brescia, Italy.Email: firstname.lastname@ing.unibs.it

* Corresponding author, Tel. +39 030 3715432, Fax. +39 030 380014.Copyright c©2007 IEEE. Personal use of this material is permitted. How-

ever, permission to use this material for any other purposes must be obtainedfrom the IEEE by sending an email to pubs-permissions@ieee.org.

which are based on multi-resolution spatiotemporal represen-tation of video sequences. In particular, wavelet-based SVC(WSVC) tools and systems will be classified and analyzedwith the following objectives: a) to give a quite comprehen-sive tutorial reference to those interested in the field, b) toanalyze strong and weak points of the main WSVC tools andarchitectures, c) to synthesize and account for the explorationactivities made by the MPEG video group on WSVC and todescribe the issued reference platform, d) to compare suchplatform with the ongoing SVC standard (jointly targeted byITU and ISO/MPEG) in qualitative and quantitative terms, ande) to discuss and propose promising evolution paths and targetapplications for WSVC. For this purpose the presentation hasbeen structured as follows. In Sec. II WSVC fundamentalsas well as basic and advanced tools which enable temporal,spatial and quality salability are presented and discussed.Sec. III shows how such components are typically combined toachieve meaningful WSVC architectures, which typically dif-fer from the space-time order in which the wavelet transformoperates, discussing strengths and weaknesses of the resultingimplementations. In the same section, an emphasis is placedon a promising architecture which presents some similarities tothe SVC standard. Subsequently, Sec. IV explains the WaveletVideo Reference architecture(s) studied by ISO/MPEG inits exploration on Wavelet Video Compression. The paperattempts as well (Sec. V) to draw a list of major differencesbetween such architecture(s) and tools with respect to theSVC reference model, providing performance comparison interms of coding efficiency and giving a critical analysis of thepresented results. In Sec. VI further investigations on WSVCcritical aspects are presented. A discussion on perspectivesand applications that appear better aimed to wavelet basedapproach is made in Sec. VII. Finally, conclusions are derived.Note that in order to be consistent with the other papers ofthis special issue, the acronym SVC will be used withoutdistinction to indicate generic scalable video coding systemsand the SVC standard. The appropriate meaning will emergefrom the context.

II. BASIC PRINCIPLES OF WAVELET-BASED APPROACHESTO SVC

A. WSVC fundamentals

Fig.1, shows a typical SVC system which refers to thecoding of a video signal at an original CIF resolution anda framerate of 30 fps. In the example, the highest operatingpoint and decoding quality level correspond to a bit rate of2Mbps associated with the data at the original spatiotemporal

3

SV extractorSV coder SV decoder

orig CIF

30fps

QCIF

15fps

150kbps

texture mvCIF

30fps

2Mbps decodable bit-stream

at the target operating point

Figure 1. Scalable Video Coding rationale. A unique bit-stream is producedby the Coder. Decodable sub-streams can be obtained with simple extractions.

7.5 fps 15 fps 30 fps 60 fps

GOP 1 GOP 2 GOP 3 GOP 4 GOP N

MV4

MV3 MV1

QCIF CIF 4CIF

MV2

GOP header

Spatial levels

header

Temporal levels header plus

entropy coder header

(containing truncation info)

Figure 2. Example of a scalable bit-stream structure.

resolution. For a scaled decoding, in terms of spatial, temporaland/or quality resolution, the decoder only works on a portionof the originally coded bit stream according to the specificationof a desired working point. Such a stream portion is extractedfrom the originally coded stream by a functional block calledextractor. As shown in Fig.1 the extractor is arranged betweenthe coder and the decoder. In any practical application, it canbe implemented as an independent block within any node ofthe distribution chain or it can be included in the decoder.The extractor receives the information related to the desiredworking point – in the example of Fig.1, a lower spatialresolution QCIF, a lower frame rate (15 fps), and a lower bit-rate (quality) (150 kbps) – and extracts a decodable bit streammatching or almost matching the working point specification.One of the main differences between an SVC system and atranscoding solution is the low complexity of the extractorwhich does not require coding/decoding operations and typi-cally consists of simple parsing operations on the coded bit-stream. For practical purposes, let us consider the followingexample where a 4CIF video is coded allowing to extract threedifferent spatial scalability layers and four embedded temporalresolutions. In this case, a possible structure of the bit-streamgenerated by an SVC system is shown in Fig.2. As indicated,the usual partition in independently coded groups of pictures(GOP) can be adopted for separating the various video frames.The GOP header contains the fundamental characteristics of

15 fps

QCIF CIF

15 fps

CIF

QCIF

Figure 3. Example of a sub-stream extraction: CIF at 15fps is extracted froma 4CIF sequence at 60fps.

the video and the pointers to the beginning of each independentGOP bit-stream. In this example, each GOP stream containsthree bit-stream segments, one for each spatial level. Notethat in an ideal scalable bit-stream, except for the lowestworking point, all information is incremental. This means, forexample, that in order to properly decode higher resolution(e.g. 4CIF), one needs to use also the portions of the bit-stream which represent the lower spatial resolutions (e.g. QCIFand CIF). Given this bit-stream organization, for each spatialresolution level a temporal segmentation of the bit-stream canbe further performed. Accordingly, the 7.5 fps portion containsdata that allow to reconstruct the lowest temporal resolutionof the considered spatial resolution reference video sequence.Similarly the 15fps, 30fps, 60fps parts refer to informationused to refine the frame-rate until the allowed maximum. Sincemost successful video compression schemes require the use ofmotion compensation (MC), each temporal resolution level bit-stream portion should include the relative motion information(usually consisting of coded motion vector fields), as shownin Fig.2. Further quality layers can be obtained, for example,by arranging the signal representation, for every temporalsegment, in a real or virtual bit-plane progressive fashion (seeSec.II-B3). With such a bit-stream, every allowed combinationin terms of spatial, temporal and quality scalability can begenerally decided and achieved according to the decodingcapabilities. Fig.3 provides a graphical representation of atarget bit-rate extraction of a CIF 15fps coded sequence, fromthe 4CIF 60fps bit-stream of the original video. Note thatcorresponding parts of QCIF and CIF bit-streams have beenextracted to reconstruct the sequence at the desired workingpoint.A suitable property for concrete applications (e.g. for contentdistribution on heterogeneous networks) is the possibilityto adapt the bit-stream according to a multiple (chained)extraction path, from the highest towards lower operatingpoints; this has been called bit-stream multiple adaptation[5]. Multiple adaptations should be allowed, of course, alongevery advisable path, and in general without any a-prioriencoding settings. Multiple adaptation extraction differs from

4

Mathematics

Multiresolution

spaces

Signal processing

Filter banks

Wavelets

Polyphase

systems

DWT

Lifting

Motion E+C

MCTF

Mult

ires

olu

tion

Sca

lable

Vid

eo C

od

ing

Bit-plane quantization

Information theory

Video coding

Wavelet codingContext based

arithmetic coding

S

T

Q

Figure 4. WSVC theoretical context.

a single extraction only in one aspect; for multiple adaptationsthe bit-stream extraction information must be conserved (i.e.processed and re-inserted throwing out what become useless)from the input bit-stream. Clearly there is no need to re-insert(and spend bits for) the extraction information if no furtherextraction is expected.

B. Tools enabling scalability

In the past two decades there has been a proliferation ofactivities focalized on theory and solutions for efficient (min-imally redundant) multiresolution representation of signals. Ingeneral, a multiresolution signal decomposition (transform)inherently enables a corresponding low to high resolutionscalability by the representation of the signal in the trans-formed domain. The tools that enable scalable multidimen-sional (space-time-quality) signal representation and codingare multidisciplinary. This fact is clearly depicted in Fig.4 thatgives a quick overview of these tools and how they can becombined in order to achieve the desired scalability feature:Spatial, Temporal and Quality (S,T and Q). In the following,each scalability dimension, S,T and Q, will be consideredand the most popular and effective solutions will be brieflydescribed. In this context, Fig.4 can be helpful in keeping thebig picture when analyzing a specific tool. Due to lack ofspace to describe each tool in detail, support references willbe provided for the interested reader.

1) Spatial scalability: Tools producing a multiresolutionrepresentation of n-dimensional signals can be of differenttypes. They can be linear or non linear, separable or not,redundant or critically sampled. Linear decompositions havebeen studied for a long time. The simplest mechanism toperform a two resolution representation of a signal is based onwhat here we call inter-scale prediction (ISP): an original (fullresolution) signal x can be seen as a coarse resolution signalc, obtained by a decimation of x (e.g., by filtering with a low-pass filter h and downsampling of factor 2), added to a detailsignal d calculated as the difference of the original signal andan interpolated version of c, d = x − I(c) (throughout thisdocument a ∼ symbol over a d signal means that the detailis not critically sampled). The Laplacian pyramid introducedby Burt and Adelson [6] is an iterated version of such an

h

h~

2

2+

h

h~

2

2+

d2

d1

~

~

~

c1x

h

h~

2

2+

dn

~

cn-1 c

n

c2

Figure 5. A n-level Laplacian pyramid analysis filter bank.

h

x

2

cn

h2

g 2 g2

dn

+

x

h 2

g 2

h2

g2

+

d1

~

~

~

~

Figure 6. A n-level (bi)orthogonal DWT analysis/synthesis implementedwith a dyadic tree filter bank.

ISP mechanism resulting in a coarsest resolution signal c anda set of details d(l) at decreasing levels of spatial resolutionl = 1 . . . n. Fig.5 shows a Laplacian decomposition of the one-dimensional signal x in n-stages (from full resolution 0 to thecoarsest one n) where I(cl) is implemented by upsamplingby a factor of 2 and interpolating with h. The discretewavelet transform (DWT) is a linear operator which projectsthe original signal in a set of multiresolution subspaces [7]allowing a critically sampled (orthogonal or biorthogonal)representation of the signal in the transformed domain andguaranteeing perfect reconstruction synthesis. Similarly to theLaplacian pyramid, the DWT decomposition of x generates acoarse signal c and a series of detail signals d(l) at variouslevels of spatial resolutions. By using the terminology ofmultirate filter banks [8], the transformed domain signals arealso called subbands. In fact, the DWT can be implementedby means of a two-channel filter bank, iterated on a dyadictree path, as shown in Fig.6 [9] [10]. Symbols h and grepresent respectively the low-pass and high-pass analysisfilters, whereas h and g are the synthesis for an orthogonalor biorthogonal wavelet decomposition. Symmetric filter re-sponses are preferred for visual data coding applications dueto their linear phase characteristics and can be obtained onlywith biorthogonal decompositions (except for length 2 filters,i.e. Haar basis). A popular filter pair for image coding purposesis the 9/7 one [10, pag. 279], which gives an analysis filter hwith 9 taps and a synthesis filter h with 7 taps.For multidimensional signals, e.g. images, separate filteringon rows and columns are usually adopted to implement the socalled separable pyramidal and DWT decompositions. In thecase of separable 2D-DWT on images this generates, at eachlevel of decomposition, one coarse subband c and three detailsubband that we continue to indicate altogether as d.Sweldens [11] introduced the lifting scheme as an alternativespatial domain processing to construct multiresolution signalrepresentations. The lifting structure is depicted in Fig.7 and

5

-P Usplitx

x2i+1

x2i

di

ci

Figure 7. Dyadic decomposition by lifting steps: split, prediction P andupdate U .

reveals a quite intuitive spatial domain processing of theoriginal signal capable to generate a critically sampled (c, d)representation of signal x. According to this scheme, signal xis split in two polyphase components: the even and the oddsamples x2i and x2i+1. As the two components (each one ishalf the original resolution) are correlated, a prediction P canbe performed between the two; odd samples can be predictedwith an operator which uses a neighborhood of even samplesand that produces the residue

di = x2i+1 − P{(x2i)i∈N}. (1)

The transform x → (x2i, di) is not satisfactory from amultiresolution representation point of view as the subsampledsignal x2i could contain a lot of aliased components. As such,it should be updated to a filtered version c. The update step

ci = x2i + U{(di)i∈N} (2)

can be constrained to do this job and it is structurally invertibleas well as the preceding stages. Perfect reconstruction issimply guaranteed by

x2i = ci − U{(di)i∈N} (3)x2i+1 = di + P{(x2i)i∈N} (4)

Daubechies and Sweldens [12] proved that every DWT can befactorized in a chain of lifting steps.The lifting structure is not limited to be an alternative and com-putationally convenient form to implement a DWT transform.Actually, it also allows non-linear multiresolution decompo-sitions. For example, lifting steps implementing a DWT canbe slightly modified by means of rounding operations in orderto perform a so called integer to integer (non-linear) wavelettransform [13]. Such a transform is useful as a componentof wavelet-based lossless coding systems. The kind and thedegree of introduced non-linearity can depend on the structuralor morphological characteristics of the signal that one maywant to capture [14], keeping in mind that non linear systemscan have unexpected behaviors in terms of error propagation(which may turn out to be a burden in presence of quantizationof the transformed signal). Moreover, the lifting structure has afundamental role for Motion Compensated Temporal Filtering(MCTF) as we will see in the next subsection.

-MCP MCUsplitx

x2i+1

x2i

di

ci

ME

mv

Figure 8. The prediction and update lifting steps with explicit motioninformation conveyed to the P and U operators.

2) Temporal scalability: A key tool which enables temporalscalability while exploiting temporal correlation is the MCTF.An introduction to the subject can be found in [15]. Afterthe first work of Ohm proposing a motion compensatedversion of the Haar transform [16], other studies [17] [18]began to show that video coding systems exploiting this toolcould be competitive with respect to the classical and mostsuccessful hybrid ones (based on a combination of block-based spatial DCT transform coding and block-based temporalmotion estimation and compensation). Other works (e.g. [19][20]) focus on the study of advanced MC lifting structuresdue to their versatility and strength in increasing the codingperformance. Basically, an MCTF implementation by liftingsteps can be represented by the already seen eqs.(1)-(4) whereindex i has now a temporal meaning and where the predictionand update operators P and U can be guided by motioninformation generated and coded after a motion estimation(ME) stage becoming MCP and MPU operators respectively.Fig.8 shows a conceptual block diagram of this general idea.ME/MC are implemented according to a certain motion modeland in the case of spatiotemporal multi-resolution SVC andWSVC systems they usually generate a set of motion descrip-tions consisting of motion vector fields mv(l, k). These areestimations of the trajectory of groups (usually rectangularblocks) of pixels between the temporal frames, at spatial levell, involved in the k-th MCTF temporal decomposition level.Haar filters based [16] [17] [21] and longer filters MCTF

solutions have been proposed [22] [23] [24]. For exampleMCTF implementations of the 5/3 transform exist [20]. 5/3and in general longer filter solutions usually achieve betterperformance in terms of prediction efficiency and visual qual-ity at lower temporal resolution frames, but make use of abidirectional prediction and thus motion information is morebit-demanding than Haar based solutions. The two solutionsrecall unidirectional and bidirectional block mode predictionused in hybrid schemes. Similarly to advanced hybrid schemes,an adaptive selection among the two modes is also possiblein the case of MCTF, leading to block-based adaptive MCTFsolutions [25] [26]. Alternatively, deformable motion modelshave also been explored with promising results [27].By exploiting the local adaptability of the MCP and MCUoperators and thanks to the transmitted (usually lossless coded)mv(l, k) information, the MCTF can also be aware of andhandle a series of issues affecting motion compensation:a) occlusion and uncovered area problems can be handled

6

recognizing unconnected and multiple connected pixels andadapting the MCP and MCU structure accordingly [22][17] [21]; b) in the case of block based ME/MC, blockingeffects can be reduced in various ways by considering ad-jacent blocks (spatiotemporal lifting) realizing for exampleoverlapped block-based motion compensation (OBMC) [21][26]; c) when MV’s are provided with fractional pel precisionthe lifting structure can be modified to implement the neces-sary pixel interpolations [22] [28] [20] while preserving thetemporal invertibility of the transform. It must also be notedthat MCU makes use of a reversed motion field with respectto the one used by MCP . This may constitute a problemespecially in those areas where inversion cannot be performedexactly, e.g. on occluded/uncovered areas, or does not fallon integer pixel positions because fractional pel accuracy isinvolved. In order not to duplicate the motion informationand the associated bit-budget, mechanisms to derive or predictMCU fields from the MCP ones have been proposed [27][29] [30] as well as methods that identify inversion problemsand adopt interpolation based solutions (e.g. the Barbell liftingscheme [31]).With lifting structures non dyadic temporal decompositionare also possible [32], leading to temporal scalability factorsdifferent from a power of two.As already noticed, the update step could be omitted, withoutimpairing perfect reconstruction, generating a so called uncon-strained MCTF (UMCTF) [33]. This guarantees the direct useof the original frames for reduced frame-rates, then reduces thecomplexity of the transform stage and presents the benefit ofeliminating ghosting artifacts. Ghosting artifacts can appearin some regions of the low pass temporal frames as a conse-quence of unavoidable local MC failures (associated to scenechanges, unhandled object deformations, uncovered/occludedregions). In such erroneously motion-compensated regions,the low pass temporal filtering updates the correspondingtemporal subbands with “ghosts” (which “should not exist”in the current frame) coming from past or future frames.Depending on the entity of the ghosts, this can locally impairthe visual appearance. On the other hand, for correctly motioncompensated regions, the update step has a beneficial temporalsmoothing effect that can improve visual perception at reducedframe-rates and also coding performance. Reduction of ghost-ing artifacts is then an important task and it has been obtainedby the implementation of MCTF adaptive update steps [34][35] or by clipping the update signal [36].Either MCTF or UMCTF generate a temporal subband hi-erarchy starting from higher temporal resolution to lowerones. Alternatively, the hierarchical B-frames decomposition(HBFD) [2] can be built enabling a similar temporal detailhierarchical structure but starting from the lower resolutions(predictions between more distant frames). This allows closed-loop implementations of the temporal prediction [37].Efficient MCTF implementations have also been studied toreduce system complexity [38], memory usage [39] [23] andcoding delay [40].

3) Quality scalability: Among the best image compres-sion schemes, wavelet-based ones currently provide for highrate-distortion (R-D) performance while preserving a limited

computational complexity. They usually do not interfere withspatial scalability requirements and allow a high degree ofquality scalability which, in many cases, consists in the possi-bility of optimally truncating the coded bit-stream at arbitrarypoints (bit-stream embedding). Most techniques that guaranteean optimal bit-stream embedding and consequently a bit-wisequality scalability are inspired from the zerotree idea firstintroduced by Shapiro’s embedded zerotree wavelet (EZW)technique [41] and then reformulated with the set partitioningin hiearchical trees (SPIHT) algorithm by Said and Pearlman[42]. A higher performance zerotree inspired technique is theembedded zero-block coding (EZBC) algorithm [43], wherequad-tree partitioning and context modeling of wavelet coeffi-cients are well combined. The zerotree idea can also be refor-mulated in a dual way, allowing to directly build significantcoefficients maps. To implement this idea, a morphologicalprocessing based on connectivity analysis has been used andalso justified by the statistical evidence of the energy clusteringin the wavelet subbands [44]. The EMDC technique [45] [46]exploits principles of morphological coding and guarantees,for 2D and 3D data [47], coding performance comparableor superior to state-of-the-art codecs, progressive decodingand fractional bit-plane embedding for a highly optimized bit-stream.Another popular technique, which does not use the zerotreehypothesis, is the embedded block coding with optimizedtruncation (EBCOT) algorithm [48], adopted in the JPEG2000standard [49], which combines layered block coding, fractionalbit-planes [50], block based R-D optimizations, and context-based arithmetic coding to obtain good (adjustable) scalabilityproperties and high coding efficiency.Proposed techniques for coding coefficients of spatiotemporalsubbands generated by WSVC systems usually are extensionsof image coding techniques. Obviously such 3D extensionsshall not interfere with temporal scalability requirements aswell, and can be achieved by considering a) single temporalframe coding with multiple frame statistical contexts updating,b) extended spatiotemporal non significance (or significance)prediction structures, c) spatiotemporal statistical contexts, ora combination of them. Temporal extensions of SPIHT [18],EZBC [51] [52], EBCOT [36] [24] and EMDC [53] have beenproposed and used in WSVC video coding systems.

III. CODING ARCHITECTURES FOR WSVC SYSTEMS

A. WSVC notation

In a multidimensional multilevel spatiotemporal decompo-sition there is a proliferation of signals and/or subbands.To support our discussion we have adopted a notation thatguarantees easy referencing to the generated signals in ageneral decomposition structure. We consider a generic spa-tiotemporal signal x = x(θ, t), with D spatial dimensionsθ = {θ1, . . . , θD}. Signal x which undergoes an n-levelmultiresolution spatial transform S(n) is indicated with xS(n).For non redundant transforms, such as the DWT, the spatiallytransformed signal actually consist of the subband set xS(n) ={xcS(n), x

d(n)S(n), . . . , x

d(1)S(n)}, where superscript c indicates the

7

signal representation at the coarsest spatial resolution (low-pass) while, for each spatial level n ∈ [1 . . . n], d(n) indicatesa critically sampled level of detail. In the case of separabletransforms (such as the DWT) each d(n) consists of 2D − 1directional subbands. For redundant transforms (such as theLaplacian pyramid) the transformed signal consists of thesignal set xS(n) = {xc

S(n), xd(n)S(n), . . . , x

d(1)S(n)} where d(n) is

an oversampled detail at level n. With a similar notationa spatiotemporal signal x = x(θ, t) which undergoes a m-level multiresolution temporal transform T (m) is indicated asxT (m). As before, superscripts c, d(·) and d(·) will be usedreferring to the temporal dimension to indicate coarsest tem-poral resolution, critically and not critically sampled temporaldetails respectively. In the figures of the present section we willgraphically represent temporally transformed data subbands byassociating a gray level to them: the white corresponds to cwhile grey-tones get darker as the detail level goes from thefinest to the coarsest. Moreover, x and x indicate original andquantized (reduced coefficient representation quality) signalsrespectively. Symbol “∗” will be used as wild card to refer tothe whole set of spatiotemporal subbands of a given spatial ortemporal transform.A decoded version of the original signal x extracted andreconstructed at given temporal resolution k, spatial resolutionl and reduced quality rate, will be indicated as l

kx.

Concerning ME/MC, given a certain motion model, a descrip-tion is produced consisting of a set of motion vector fieldsmv(l, k), where, l and k refer to various spatial and temporalresolution levels respectively. As mv(l, k) and mv(l + 1, k)are obviously not independent, mechanisms of up- or down-conversion, with or without refinement or approximation ofthe motion vector field, are usually implemented. Then, forexample, mv(l+1, k)=mv(l, k)d indicates that motion vectorsat a coarser resolution have been down-converted, by a suitablereduction of the block size and of the vector module, withoutapproximation. Otherwise, in case of down-conversion withapproximation the notation mv(l, k)d,a would have been used.Moreover, mv(l, k)=mv(l + 2, k)u(2),r indicates that motionvectors at a 2-nd layer of finer resolution have been up-converted twice with a refinement of the block partition and/orof the motion vector value.

B. Basic WSVC architectures

In Figs.9 and 10 the main multi-resolution decompositionstructures and corresponding WSVC architectures are shown.Each one will be presented in the following paragraphs.

1) t+2D: Perhaps the most intuitive way to build a WSVCsystem is to perform an MCTF in the original spatial domainfollowed by a spatial transform on each temporal subband.This is usually denoted as a t+2D scheme, it guaranteescritically sampled subbands and it is represented in Fig.9(a).Earlier wavelet based coding systems were based on thisscheme [16] [17]. Many other wavelet based SVC systemsare based on the t+2D spatiotemporal decomposition, amongwhich we can cite [18] [19] [20] [21] [31] [33] [51] [52] [54][55] [56] [57] [58]. Despite its conceptual simplicity, the t+2Dsolution presents some relevant drawbacks especially for spa-

tial scalability performance. When only temporal and qualityscalability (full spatial resolution decoding) are required, theprocess is reversed until reaching the desired frame-rate atresolution k (partial or complete MCTF inversion) and SNRquality: kx = xT (m)S(n)QS−1(n)T −1(m−k) where both directand inverse temporal transforms make use of the set of motionvectors mv(0, k), k = 0 . . . k. Instead, if a lower spatial resolu-tion l version is needed the inversion process is incoherent withrespect to the forward decomposition. For clarity, let us con-sider only spatial scalability, so that the spatially scaled signalis obtained by lx = xT (m)S(n)S−1(n−l)T −1(m). The problemis due to the fact that the inverse MCTF transform is obtainedusing motion vectors mv(l, k) that are typically deduced fromthe estimated ones by mv(l, k) = mv(0, k)d(l). 1) This cannotguarantee perfect reconstruction because spatial and motioncompensated temporal transforms cannot be inverted withoutmost likely changing the result (due to the shift-variant natureof the spatial transform) [15], and 2) these motion vectorshave not been estimated on data at the target resolution andtherefore they cannot be considered optimal for use, whenspatial sub-sampling has been made with non alias free waveletfilters. Moreover, the bit occupancy of full resolution motionvectors penalizes the performance of spatially scaled decodingespecially at lower bit-rates.The above issues can only be mitigated in such a criti-cally sampled t+2D scheme but not structurally resolved.To reduce the MV bit load at full resolution scalable MVcoding can be used [59] [60], to reduce it at intermediateand lower resolutions approximated MV down-conversionsmv(l, k) = mv(0, k)d(l),a can be adopted (e.g. from half-pel at spatial resolution l again to half-pel at resolution l + 1instead of the quarter-pel generated by the down-conversion).These two solutions have the disadvantage of introducingsome imprecisions in the inverse temporal transform andtheir possible adoption should be balanced in a R-D sense.The problem of the non optimality of the down-conversiondue to spatial aliasing, introduced by the decimation, can bereduced by using (possibly only for the needed resolutions)more selective wavelet filters [61] or locally adaptive spectralshaping by acting on the quantization parameters inside eachspatial subband [62]. However, such approaches can determinecoding performance loss at full resolution, because either thewavelet filters or coefficient quantization laws are moved awayfrom ideal conditions.

2) 2D+t: In order to solve the spatial scalabilityperformance issues of t+2D schemes, a natural approachcould be to consider a 2D+t scheme, where the spatialtransform is applied before the temporal one as shownin Fig.9(b). The resulting scheme is often called In-BandMCTF (IBMCTF) [63] because estimation of mv(l, k)is made and applied independently on each spatial levelin the spatial subband domain, leading to a structurallyscalable motion representation and coding. With IBMCTFspatial and temporal scalability are more decoupled withrespect to the t+2D case. Signal reconstruction after acombination of spatiotemporal and quality scalabilityis given by l

kx = xS(n)T (m)QT −1(m−k)S−1(n−l) where

the (partial) inverse temporal transform makes use of

8

S CT

mv(0,k)

x xT (3)

xT (3)S (2)

S CT

mv(l,k)

x xS (2)

xS (2)T (3)

(a)

(b)

Figure 9. Basic WSVC schemes in a signal perspective: t+2D (a), 2D+t (b).

mv(l, k), k = 0 . . . k, l = 0 . . . l, (a subset of) the same MVsused for the direct decomposition.Despite its conceptual simplicity the 2D+t approach suffers aswell from the shift-variant (periodically shift-invariant) natureof the DWT decomposition. In fact, motion estimation inthe critically sampled detail subbands is not reliable becausethe coefficient patterns of moving parts usually change fromone frame to consecutive or preceding ones. Solutions tothis issue have been proposed [64] [65]; they foresee aframe expansion formulation, at the expense of an increasedcomplexity, to perform motion estimation and compensationin an overcomplete (shift-invariant) spatial domain, whiletransmitting only decimated samples. Even so, 2D+t schemesdemonstrated lower coding efficiency compared to t+2D ones,especially at higher resolutions [66]. Motivations of this factare probably due to the reduced efficiency of MCTF whenapplied in the high-pass spatial subband domain. In addition,spatial inversion of detail subbands which contain block-basedMCTF coding artifacts visually smoothes blocking effects butprobably causes an overall visual quality degradation.Another possible explanation of a reduced coding efficiencyresides in the fact that parent-child spatial relationships,typically exploited during embedded wavelet coding orcontext based arithmetic coding, are partially broken byindependent temporal filtering on the spatial levels.

3) Adaptive architectures: Being characterized by differentadvantages and disadvantages, t+2D and 2D+t architecturesseem inadequate to give a final answer to the compositeissues arising in SVC applications. One possible approachcomes from schemes which try to combine the positive as-pects of each of them and propose adaptive spatiotemporaldecompositions optimized with respect to suitable criteria. In[67] [68] authors suggest that content-adaptive 2D+t versus

t+2D decomposition can improve coding performance. Inparticular, they show how various kind of disadvantages offixed transform structures can be more or less preponderantdepending on video data content, especially on the accuracyof the derived motion model. Therefore, local estimates ofthe motion accuracy allow to adapt the spatiotemporal datadecomposition structure in order to preserve scalability whileobtaining overall maximization of coding performance.Recently, another adaptive solution, named aceSVC [69] [70],has been proposed where the decomposition adaptation isdriven by scalability requirements. This solution tries toachieve maximum bit-stream flexibility and adaptability toa selected spatiotemporal and quality working point config-uration. This is obtained by arranging spatial and temporaldecompositions following the principles of a generalised spa-tiotemporal scalability (GSTS) [71]. In aceSVC encoded bit-stream, thanks to the adapted decomposition structure, everycoding layer is efficiently used (without waste of bits) for theconstruction of higher ones according to the predefined spatial,temporal and quality working point configurations.Another perspective to compensate for the t+2D vs 2D+t draw-backs can come from the pyramidal approaches, as discussedin the reminder of this section.

4) Multi-scale pyramids: From the above discussion it isclear that the spatial and temporal wavelet filtering cannotbe decoupled because of the motion compensation takingplace. As a consequence, it is not possible to encode differentspatial resolution levels at once (with only one MCTF) andthus, both lower and higher resolution frames or subbandsequences should be MCT-Filtered. In this perspective, apossibility for obtaining good performance in terms of bit-rate and scalability is to use coding schemes which are usuallyreferred as to 2D+t+2D or multi-scale pyramids [72]. In suchschemes spatially scaled versions of video signals are obtained

9

S

S

I

+

+

-

-

C

C

C

open

loop

closed

loop

I

open

loop

closed

loop

T

mv(0,k)

T

mv(1,k)

T

mv(2,k)

x

xS

c

xS (2)c

^ cxS

xS (2)

^ c

xS (2)

d(1)~

xS (2)

d(2)~

xS (2)T (3)c

xS (2)T (3)

d(2) ~

xS (2)T (3)

d(1) ~

(a)

(b)

S

S

C

C

CT

mv(2,k)

x

xS

c

xS (2)c x

S (2)T (3)c

T

mv(0,k)

T

mv(1,k)

I

+

+

-

-

open

loop

closed

loop

I

open

loop

closed

loop

xT (3)

xS T (3)

c

xS (2)T (3)

^ c

xS T (3)

^ c

0 T (3)

~

1T (3)

~

Figure 10. Pyramidal WSVC schemes with the pyramidal decomposition put before (a) and after (b) the temporal MCTF.

starting from the higher resolution, while the coding systemsencompass Inter-Scale Prediction (ISP) mechanisms in orderto exploit the multi-scale representation redundancy. Thesemulti-resolution decompositions derive naturally from the firsthierarchical representation technique introduced for images,namely the Laplacian pyramid [6] (see Sec. II-B). So, even iffrom an intuitive point of view 2D+t+2D schemes seems to bewell motivated, they have the typical disadvantage of overcom-plete transforms, namely those leading to a full size residualimage, so that coding efficiency is harder to achieve. As in thecase of the previously discussed t+2D and 2D+t schemes, twodifferent multi-scale pyramid based schemes are conceivable,depending where the pyramidal residue generation is imple-mented, i.e. before or after the MCTF decomposition. Thesetwo schemes are shown in Fig.10. In both schemes the lowestspatial resolution temporal subbands xc ∗

S(n)T (m) are the same,while higher resolution residuals are generated differently.The scheme of Fig.10(a) suffers from the problems of 2D+t

schemes, because motion estimation is operated on spatialresiduals, and to our knowledge did not lead to efficient SVCimplementations.In the scheme depicted in Fig.10(b) we first observe that thehigher resolution residuals ∆∗

l T (m), with l = 0 . . . n − 1,

cannot be called x∗ d(l+1)T (m)S(n) since the spatial transform is not

exactly a Laplacian one because the temporal transform stages,which are placed between the spatial down-sampling and theprediction with interpolation stages, operate independently (onseparate originals). This fact can cause prediction inefficien-cies which may be due to motion model discrepancies amongdifferent resolutions or simply to the non linear and shiftvariant nature of the motion compensation even in the presenceof similar motion models. It is worth noting that the effectof these inefficiencies affects the whole residue and then itworsens the problem of the increased number of coefficientsto code. Strategies to reduce these inefficiencies are workableand we will shortly return on these issues in the following

10

subsection. Note that the scheme shown in Fig.10(b) can beused to represent the fundamental coding principles underlyingthe SVC standard [2].It is also important to mention that some recent works demon-strate that using the frame theory [7] it is possible to improvethe coding performance associated to the Laplacian pyramidrepresentation [73] [74] and also to reduce its redundancy[75]. In the future these works may play a role in improvingthe spatial scalability performance of the coding schemesdescribed in this paragraph.

C. The STP-tool scheme

We here present a WSVC architecture that has been devel-oped [76] and also adopted as a possible configuration of theMPEG VidWav reference software [36] (see Sec.IV-B). Thisscheme is based on a multi-scale pyramid and differs from theone of Fig.10(b) in the ISP mechanism. The solution, depictedin Fig.11, is called STP-tool as it can be interpreted as aprediction tool (ISP) for spatiotemporal subbands. After therepresentation of the video at different scales, these undergoa normal MCTF. The resulting signals are then passed to afurther spatial transform stage (typically one level of a wavelettransform) before the ISP. This way, it is possible to implementa prediction between two signals which are likely to bearsimilar patterns in the spatiotemporal domain without the needto perform any interpolation. In particular, instead of the fullresolution residuals ∆∗

l T (m) of Fig.10(b) the spatiotemporal

subbands and residues {∆∗l+1 T (m), x

c ∗ d(l+1)S(l)T (m)S} are produced

for resolutions l = 0 . . . n− 1 where ∆∗l+1 T (m) are produced

by the coarse resolution adder (shown in Fig.11) according to:a) ∆∗

l+1 T (m) = xc ∗ cS(l)T (m)S − xc ∗

S(l+1)T (m) in a spatial openloop architecture or b) ∆∗

l+1 T (m) = xc ∗ cS(l)T (m)S−xc ∗

S(l+1)T (m)

in a spatial closed loop architecture.In an open-loop scheme and without the temporal transform,the ∆ residual would be zero (if the same pre- and post-spatial transforms were used). In this case, the scheme wouldbe reduced to a critically sampled one. With the presence ofthe temporal transform, things are more complicated due tothe data-dependent structure of the motion model and to thenon linear and time variant behavior of the MCTF. In general,it is not possible to move the spatial transform stages forwardand backward with respect to the MCTF without changingthe result. If the spatial transform used to produce spatial sub-bands are the same, the main factor that produces differencesbetween the xc ∗ c

S(l)T (m)S and the xc ∗S(l+1)T (m) signals can be

identified in the incoherences between the motion trajectoriesat different spatial resolutions. On one hand, independentmotion estimation at various spatial resolution could lead tointra-layer optimized motion compensation. On the other hand,a motion model that captures the physical motion in the sceneis certainly effective for the motion compensation task andreasonably should be self similar across scales. Therefore, adesirable objective is to have a hierarchical motion estimationwhere high-resolution motion fields are obtained by refinementof corresponding lower resolution ones (or viceversa). Indeedthis guarantees a certain degree of inter-scale motion field co-herence which makes the STP-tool prediction more effective.

A hierarchical motion model is a desired feature here and inaddition it helps to minimize the motion vector coding rate.The STP-tool architecture presents another relevant and pe-culiar aspect. Although the number of subband coefficientsto code is not reduced with respect to a pyramidal approach,different (prediction) error sources are separated into differentcritically sampled signals. This does not happen in a pyramidalschemes where only full resolution residuals ∆∗

l T (m) areproduced (see Fig.10(b)).To better understand these facts let us analyze sep-arately each residual or detail component in the set{∆∗

l+1 T (m), xc ∗ d(l+1)S(l)T (m)S}. In an open loop architecture,

• the residue signals ∆∗l+1 T (m) account for the above

mentioned inter-scale motion model discrepancies; while• the temporal high-pass, spatial high-pass subbands

xc d(k) d(l+1)S(l)T (m)S account for the intra-scale motion model

failure to completely absorb and represent the real motion(occluded and uncovered areas and motion feature nothandled by the motion model); finally

• the temporal low-pass, spatial high-pass subbandsx

c c d(l+1)S(l)T (m)S contain the detail information to increase the

resolution of low pass temporal frames.

Thus, maximizing the inter-scale motion coherence impliesminimizing the energy (and the coding cost) of ∆∗

l+1 T (m).Note that even if a perfect motion down-scaling is adopted,localized residual components would be due to the shift vari-ant properties of the motion compensation especially whereuncovered areas and occlusion handling must be adopted. Inother words, the coefficient increase, determined by the pres-ence of the residues ∆∗

l+1 T (m), with respect to the criticallysampled case is the price to pay for allowing a suitable motionestimation and compensation across different scales.In a closed-loop ISP architecture we also have the greatadvantage that the quantization errors related to resolution l areconfined to signals εc ∗

l+1 T (m) = xc ∗S(l+1)T (m)− xc ∗

S(l+1)T (m) ofresolution l + 1 that contribute in a more or less predominantway to the residues to code, that is ∆∗

l+1 T (m) = ∆∗l+1 T (m) +

εc ∗l+1 T (m). This is of great advantage for the reduced number of

coefficients on which the closed-loop error is spread upon. Be-sides, the εc ∗

l+1 T (m) signals are less structured, and thus more

difficult to code, with respect to xc ∗ d(l+1)S(l)T (m)S and ∆∗

l+1 T (m).Therefore the STP-tool scheme allows a useful separation ofthe above signals, with the advantage that suitable codingstrategies can be designed for each source.As for t+2D and 2D+t schemes, the STP-tool solution maysuffer from the presence of spatial aliasing on reduced spatialresolutions. If not controlled this may also create some inter-scale discrepancies between the motion models. Again, apartial solution can be found with the use of more selectivedecimation wavelet filters, at least for decomposition levelscorresponding to decodable spatial resolutions.The STP-tool solution has been tested on different platformswith satisfactory performance, as described in the followingsections.

11

S

S

T , mv(0,k)

S

S

+

+

-

-

open

loop

closed

loop

open

loop

closed

loop

C

C

C

x

xS

c

c

c

xS (2)c x

S (2)T (3)c

xS (2)T (3)

^ c

xS T (3)

^ c

xT (3)

xT (3)S

xS T (3)

c xS T (3)S

c

1T (3) , x

T (3)S

d(1)

2T (3), x

S T (3)S

c d(1)

T , mv(1,k)

T , mv(2,k)

Figure 11. STP-tool scheme in a generated signals/subbands perspective.

IV. WSVC REFERENCE PLATFORM IN MPEGIn the past years, the ISO/MPEG standardization group ini-

tiated an exploration activity to develop scalable video codingtools and systems mainly oriented to obtain spatiotemporalscalability. One of the key issues of the conducted work wasnot to sacrifice coding performance with respect to the state-of-the-art reference video coding standard H.264/AVC.In 2004, at Palma meeting, the ISO/MPEG group set upa formal evaluation of scalable video coding technologies.Several solutions [1] [66] [77] [3], either based on waveletspatiotemporal decomposition or H.264/AVC technologies,had been proposed and compared. The MPEG visual testingindicated that performance of an H.264/AVC pyramid [25]appeared the most competitive and it was selected as a startingpoint for the new standard. At the next meeting in Hong Kong,MPEG jointly with the IEC/ITU-T video group, started a jointstandardization initiative accordingly. A scalable referencemodel and software platform named JSVM and derived fromH.264/AVC technologies was adopted [2]. During the abovementioned competitive phases, among other wavelet basedsolutions, the platform submitted by Microsoft Research Asia(MSRA) was selected as the reference to continue experimentson wavelet technologies. The MPEG WSVC reference modeland software (RM/RS) [78] will be indicated, hereinafter, withthe acronym VidWav (Video Wavelet). The VidWav RM/RShas evolved seeing various components integrated, providedthey would have given sufficient performance improvementafter appropriate core experiments.In the paragraphs of the present section we mainly recall thecharacteristics of the VidWav RM. Document [78] should betaken as a reference to go into further details. In the nextsection, we will discuss the main differences with respect tothe SVC standard. We will also provide some comparisonresults between the two approaches.

A. VidWav: general framework and main modulesThe general framework supported by the VidWav RM is

shown in Fig.12. With Pre- and Post- spatial decomposition

Video

Frames

Temporal

WaveletDecomposition

Motion

Estimation

Post- 2D Spatial

WaveletDecomposition

MV & Mode

Coding

Entropy

Coding

...

...

...

Pre- 2D Spatial

WaveletDecomposition

Figure 12. VidWav general framework.

different WSVC configurations can be implemented: t+2D,2D+t and STP-tool. The main modules of the manageablearchitectures are briefly considered in the following.

Motion estimation and coding: A macroblock based motionmodel was adopted with H.264/AVC like partition patterns. Tosupport spatial scalability, macroblock size is scaled accordingto the decimation ratio. Interpolation accuracy is also scaled.Therefore, full pel estimation on 64x64 macroblocks at 4CIFresolution can be used at quarter-pel precision on a 16x16macroblock basis for QCIF reconstructions. Hence, motion isestimated at full resolution and kept coherent across scalesby simply scaling the motion field partition and the motionvector values. This poses the limitations already discussed inSec.III-B for the t+2D architecture.For each block there is the possibility to select a forward,backward or bidirectional motion model. This requires amode selection information. Mode selection also determinesthe block aspect ratio and informs about the possibility ofdetermining current MV values from already estimated ones.Motion vector modes and values are estimated using rate-constrained lagrangian optimizations and coded with variablelength and predictive coding, similarly to what happens inH.264/AVC. For motion compensation filtering a connected/disconnected flag is included to the motion information onunits of 4x4 pixels. For 4:2:0 YUV sequences, motion vectorsfor chroma components are half those of the luminance.

12

Temporal Transform: Temporal transform modules imple-ment a framewise motion compensated (or motion aligned)wavelet transform on a lifting structure. The block based tem-poral model is adopted for temporal filter alignment. Motionvectors are directly used for the motion aligned prediction(MAP) lifting step while the motion field should be invertedfor motion aligned update (MAU). To this end, Barbell updatelifting [31] is adopted which directly uses original motionvectors and bilinear interpolation to find values of updatesignals from non integer pixel values. In order to further reduceblocking artifacts, overlapped block motion compensation(OBMC) can be used during the MAP. This is accomplished bymultiplying the reference signals by a pyramidal window withsupport larger than 4×4 before performing the temporal filter-ing. The update signal is also clipped according to a suitablethreshold in order to limit the visibility of possible ghostingartifacts. Moreover, the implemented temporal transform isgenerally locally and adaptively a Haar or a 5/3-taps dependingwhether motion block modes are unidirectional (forward orbackward) or bidirectional.

Spatial transform: A simple syntax allows to configurePre- and Post- spatial transforms on each frame or previouslytransformed subband. This allows to implement both t+2D and2D+t schemes. Different spatial decompositions are possiblefor different temporal subbands for possible adaptation to thesignal properties of the temporal subbands.

Entropy Coding: After the spatiotemporal modules thecoefficients are coded with a 3D (spatiotemporal) extensionof the EBCOT algorithm, called 3D EBCOT. Each spatiotem-poral subband is divided into 3D blocks which are codedindependently. For each block, fractional bit-plane coding andspatiotemporal context based arithmetic coding are used.

B. VidWav STP-tool configuration

The VidWav reference software can be configured to followthe STP-tool architecture of Fig.11. A corresponding func-tional block diagram of the resulting system is shown inFig.13. The coding process starts from the lower resolutionwhich acts as a base layer for ISP. In particular, a closed loopISP is implemented as coded temporal subbands are saved andused for STP-tool prediction for the higher resolution. STP-tool prediction can be applied on all spatiotemporal subbandsor only to a subset of them. Decoded temporal subbandsat a suitable bit-rate and temporal resolution are saved andused for ISP. At present, no particular optimization exists inselecting the prediction point. Obviously, it should be not toofar from the maximum point for a certain temporal resolutionat lower spatial resolution in order to avoid bit-stream spoilage(lower resolution bit-stream portions are useless for higherresolution prediction) and prediction inefficiency (too largeprediction residual). Spatial transforms on the levels interestedby STP-tool ISP can be implemented either by using the threelifting steps (3LS) [61] or the 9x7 filters. In order to betterexploit inter scale redundancy these filters should be of thesame type of those used for the generation of the referencevideos for each of the considered spatial resolutions. In thisimplementation, some important limitations remain. These are

Pre-SpatialTransform(2D DWT)

ME/MC+MCTF

N Level2D DWT

3DEBCOT

ME/MC+MCTF

2DDWT

N Level2D DWT

3DEBCOT

Bitstram

Multiplexer

MotionEstimation

AdaptiveMCTF

PartitionModes& MVcoding

+

-

N Level2D DWT-1

Î

ME parameters

RD Optimization Control

LL

Operating points setting

I

scalablebitstream

Figure 13. VidWaw framework in STP-tool configuration. The lowestspatial resolution layer is equivalent to a t+2D scheme. Higher spatial layersimplement STP-tool ISP.

mainly related to software integration problems due to the datastructures used by the original MSRA software. Therefore, co-herent and refinable bottom up motion estimation and codingare not present in the current VidWav software. When STP-tool is used, three independent motion fields are estimatedindependently. This introduces some undesired redundancy inthe bit-stream which has a major impact for the lowest qualitypoint at higher spatial resolutions. In addition, it does notprovide the motion field coherence across scales which wasseen as a desirable feature for good STP-tool prediction.

C. VidWav additional modules

Other modules have been incorporated to the baselineVidWav reference platform after careful review of performanceimprovement by appropriate testing conditions. Such tools canbe turned on or off through configuration parameters. Moredetailed description of these tools can be found in [78].

Base layer: The minimum spatial, temporal and qualityresolution working point define a Base Layer on which addi-tional Enhancement Layers are built to provide scalability upto a desired maximum spatial, temporal and quality resolution.This tool enables the use of a Base Layer, in 2D+t subbandscalable video coding, which has been previously compressedwith a different technique. In particular, the H.264/AVC streamcan be used as Base Layer by implementing a hierarchicalB-frames decomposition (HBFD) structure. This allows eachB picture of the Base Layer to temporally match corre-sponding high-pass subbands in the Enhancement Layers. Ittherefore enables the frames to share similar MCTF structuredecomposition and this correlation can be efficiently exploited.Motion information derived from the Base Layer codec canalso be used as additional candidate predictors for estimatingmotion information of the blocks in the corresponding high-pass temporal subbands of the Enhancement Layers. Clearly,the Base Layer tool can significantly improve the codingperformances, at least of the lowest working point which isnow encoded with a non scalable method. A second but not

13

less relevant provided benefit, is the backward compatibilitywith preexisting non scalable H.264/AVC decoder.

In-band 2D+t: The 2D+t scheme is activated when anon identity pre-spatial transform in Fig.12 is set. In sucha configuration the MCTF is applied to each spatial subbandgenerated by the pre-spatial transform. In Section III-B2 ithas been already evidenced that in-band scheme suffers aloss in coding performance at full resolution when the MCTFis performed independently on each subband. Conversely, ifhigh resolution signals are involved in the prediction forlow resolution, drifting occurs when decoding low resolutionvideo, because in that case high resolution signals are notavailable at the decoder. The VidWav RM adopts a technique[79] based on the use on additional macroblock inter codingmodes in the MCTF of the spatial low-pass band. With respectto the pure 2D+t scheme, this method provides a better trade-off between the lower resolution drifting error and the highresolution coding efficiency and also improves the globalcoding performances.

Wavelet ringing reduction: In Wavelet based video com-pression, ringing caused by the loss of details in the highpass spatial subbands is a major concern for the resultingvisual quality [80]. Ringing is generated during inverse DWTbecause quantization-noise of lossily-coded wavelet coeffi-cients is spread by the reconstruction filters of the IDWT. Theoscillating pattern of these filter impulse responses at differentreconstruction scales becomes objectionable especially in flatluma and chroma regions. The ringing magnitude is relatedto the quantization step-size. In order to reduce these ringingpatterns, a 2D bilateral filter applied to each component of theoutput video is used.

Intra mode prediction: The intra-prediction mode exploitsthe spatial correlation within the frame in order to predictintra areas before entropy coding. Similarly to the H.264/AVCapproach, a macroblock can be predicted form the values ofits neighbors, i.e. by using spatial predictors.

V. WSVC AND SVC STANDARDIZATION REFERENCEPLATFORMS: A COMPARISON

A. Architecture and tools: similarities and differencesIn this paragraph some differences and similarities between

JSVM and VidWav RM (see Fig.13) frameworks will behighlighted and discussed.

1) Single layer coding tools: We will start to analyze thedifferences between the VidWav RM [36] and the JSVM [2]when they are used to encode a single operating point, i.e. noscalability features being considered. Broadly speaking thiscan be seen as a comparison between a t+2D WSVC (whichwill remain fully scalable at least for temporal and qualityresolution) and something similar to H.264/AVC (possiblysupporting temporal scalability thanks to HBFD).VidWav uses a block based motion model. Block mode typesare similar to those of JSVM with the exception of the Intra-mode which is not supported by VidWav. This is due to thefact that the Intra mode determines blocks rewriting from theoriginal and prediction signal (high-pass temporal subbands)producing a mixed (original+residue) source which is detri-mental for subsequent whole frame spatial transforms. This is

also one of the main differences for spatiotemporal redundancyreduction between the two approaches. JSVM (as well as allH.264/AVC and most of the best performing hybrid videocoding methods) operates in a local manner: single framesare divided into macro-blocks which are treated separately inall the coding phases (spatial transform included). Instead, theVidWav (as well as WSVC approaches) operates with a globalapproach since the spatiotemporal transform is directly appliedto a group of frames. The fact that in the VidWav frameworka block based motion model has been adopted is only dueto efficiency and the lack of effective implementations ofalternative solutions. However, this cannot be considered anoptimal approach. In fact, the presence of blockiness in thehigh-pass temporal subbands due to block based predictions isinconsistent and causes inefficiencies with respect to the sub-sequent spatial transform. The energy spreading around blockboundaries, generates artificial high frequency components anddisturbs both zerotree and even block based entropy codingmechanisms which act on spatiotemporal subbands. On theother hand blockiness is not a main issue when block-basedtransforms, such as the DCT, are instead applied, as in thecase of JSVM. OBMC techniques can reduce the blockinessin the high-pass temporal subbands but do not remove it.Contrarily to JSVM, the single layer (t+2D) VidWav frame-work only supports open loop encoding/decoding. Thanks tothe closed loop encoding, JSVM can exploit an in-loop de-blocking filter, which makes a similar job with respect tothe post-processing de-blocking(JSVM)/de-ringing(VidWav)filters with the additional benefit of improving the quality ofthe reconstructed signals used in closed-loop predictions.Closed-loop configurations within WSVC systems would bepossible by adopting an HBFD like decomposition with theaim of incorporating in-loop de-ringing strategies as well.

2) Scalable coding tools: We now consider spatial scalabil-ity activated in both solutions, looking in particular to JSVMand VidWav in STP-tool configuration. The latter can be con-sidered as to the WSVC implementation closest to the JSVMone. The main difference between the two systems in terms ofa spatial scalability mechanism is again related to the block-based vs frame-based dichotomy. In JSVM, a decision oneach single macro-block determines whether the macroblockis encoded by using the information of its homologous at alower spatial resolution or not. In the latter case, one of thecompatible modality available for a single layer encoding willbe used. The decision, among all possible modes is globallyregulated by R-D optimization. On the contrary, ISP insideSTP-tool architecture is forced to be made at a frame level(with possible decisions on which frames to involve).Similarly to JSVM, STP-tool can use both closed and openloop interlayer encoding while this is not true for t+2D or 2D+tconfigurations which can only operate in open loop mode.

B. Objective and visual result comparisons

The comparison between DCT based SVC and WSVCsystems is far from being an obvious task. Objective andsubjective comparisons are critical even when the most ”simi-lar” architectures are considered (e.g. JSVM and STP-tool),

14

especially when comparisons are performed using decodedvideo sequences at reduced spatial resolutions. In this case,since in general different codecs use different down-samplingfilters, the coding-decoding processes will have different re-construction reference sequences, i.e. different down-sampledversions of the same original. JSVM adopt the highly selectivehalf-band MPEG down-sampling filter, while WSVC systemsare more or less forced to use wavelet filter banks which ingeneral offer less selective half-band response and smoothertransition bands. In particular, in t+2D architectures waveletdown-sampling is a requisite while in STP-tool ones (seeSec.III-C) one could adopt a more free choice, at the ex-pense of a less efficient ISP. Visually the reference sequencesgenerated by wavelet filters are in general more detailed butsometimes, depending on data or visualization conditions,some spatial aliasing effects could be seen. Therefore visualbenefits in using wavelet filters is controversial while schemesadopting more selective filters can take advantage, in terms ofcoding efficiency, of the lower data entropy. Unfortunately theMPEG down-sampling filter is not a good low-pass waveletfilter because the corresponding high-pass filter have an un-desired behavior in the stop-band [81]. Attempts to designmore selective wavelet filters have been made [61] [81] withexpected benefits in terms of visual and coding performancefor WSVC schemes.Anyway, depending on the particular filter used for spatialdown-sampling, reduced spatial resolution decoded sequenceswill differ even at full quality, impairing fair objective com-parisons. For the above reasons PSNR curves at intermediatespatiotemporal resolution have been mainly used to evaluatethe coding performance of a single family of architectureswhile for a direct comparison between two different systemsmore time-demanding and costly visual comparisons havebeen preferred. In the following we summarize the latestobjective and subjective performance comparison issued bythe work on SVC systems as conducted in April 2006 byISO/MPEG. Finally, we will also return on the possibilityof increasing objective comparisons fairness by proposing acommon reference solution.

1) Objective comparison results: Experiments have beenconducted under testing conditions described in document[82]. Three classes of experiments have been completed withthe intention of evaluating coding performance of differentsystems and tool settings. The first class of experiments,named combined scalability, aimed to test the combined useof spatial, temporal and quality scalabilities according toshared scalability requirements [4]. The comparison has beenperformed using JSVM 4.0 and the VidWav RS 2.0 in STP-tool configuration. Fig.14 shows an example of the R-D curveswhere Y PSNR are only shown. The example considers differ-ent sequences, quality points and spatiotemporal resolutions,i.e. 4CIF-60Hz, CIF-7.5Hz, QCIF-15Hz (original resolution is4CIF-60Hz in all cases). The complete set of results can befound in [3, Annex1].As mentioned in Section IV-B, the VidWav STP-tool configu-ration implements an inefficient motion estimation and codingthat partially justifies the performance differences visible inthe above mentioned PSNR plots. The other two groups of

28

30

32

34

36

38

40

city

_4C

IF60

_102

4

city

_4C

IF60

_128

0

city

_4C

IF60

_153

6

city

_4C

IF60

_179

2

city

_4C

IF60

_204

8so

ccer

_4C

IF60

_153

6

socc

er_4

CIF

60_1

780

socc

er_4

CIF

60_2

048

socc

er_4

CIF

60_2

560

socc

er_4

CIF

60_3

072

crew

_CIF

7.5_

192

crew

_CIF

7.5_

224

crew

_CIF

7.5_

256

crew

_CIF

7.5_

320

crew

_CIF

7.5_

384

harb

our_

QC

IF15

_96

harb

our_

QC

IF15

_112

harb

our_

QC

IF15

_128

harb

our_

QC

IF15

_160

harb

our_

QC

IF15

_192

YP

SN

R (

dB

)

VidWav (STP-tool)

JSVM 4.0

Figure 14. Combined Scalability example results on 4 test sequences: City,Soccer, Crew and Harbour, at different spatiotemporal resolutions and bit-rates, using encapsulated bit-stream decoding as described in [82].

experiments, spatial scalability and SNR scalability, aimed totest the use and the R-D performance of single scalability toolswhen other forms of scalability are not requested. VidWav inSTP-tool and in t+2D configuration modes were tested. For theinterested reader complete results are reported in [3, Annex2and Annex3].Where PSNR comparisons did not suffer from the differentoriginal reference signal problems, mixed performance wereobserved with a predominance of higher values for sequencesobtained using JSVM.

2) Subjective comparison results: Visual tests conductedby ISO/MPEG included 12 expert viewers and have beenperformed according to guidelines specified in [83]. On thebasis of these comparisons between VidWav RS 2.0 and JSVM4.0 appear on average superior for JSVM 4.0, with marginalgains in SNR conditions, and superior gains in combinedscalability settings. A full description of such test can be foundin [3, Annex 4]. It should be said that, from a qualitative pointof view, a skilled viewer could infer from which system thedecoded image is generated. In fact, for motivations ascribableto spatial filtering aspects, VidWav decoded frames are ingeneral more detailed with respect to JSVM ones, but bits notspent in background details can be used by JSVM to betterpatch motion model failures and then to reduce the impact ofmore objectionable motion artifacts for the used test material.

3) Objective measures with a common reference: A rea-sonable way to approach a fair objective comparison, betweentwo systems that adopt different reference video sequences V1

and V2, generated from a same original sequence X , is tocreate a common weighted sequence V = α1V1 + α2V2. Inparticular, by selecting α1 = α2 = 1/2 it can be easily verifiedthat PSNR(V, V 1) = PSNR(V, V 2). This means that V1 andV2 are both equally disadvantaged by the creation of the meanreference V . Fig.15(a) shows the generation of the above threesignals using two different low pass-filters H1(f) and H2(f)which are assumed here to be a low-pass wavelet filter and theMPEG down-sampling filter respectively. As already discussedthe output V2 will have a more confined low-frequency contentwith respect to V1. Fig.15(b) is a projection of the decoding

15

V1

V2

V

high freq.

low freq.

2

)(1 fH

)(2 fH

2)()( 2121 fHfH

2

x

V1

V2

V

(a) (b)

Figure 15. Common reference generation (a) and representation (b).

32

33

34

35

36

37

38

39

40

41

42

harb

our_

qcif

15_1

92

harb

our_

qcif

15_1

60

harb

our_

qcif

15_1

28

harb

our_

qcif

15_1

12

harb

our_

qcif

15_9

6ci

ty_q

cif1

5_12

8ci

ty_q

cif1

5_11

2ci

ty_q

cif1

5_96

city

_qci

f15_

80ci

ty_q

cif1

5_64

crew

_qci

f15_

192

crew

_qci

f15_

160

crew

_qci

f15_

128

crew

_qci

f15_

112

crew

_qci

f15_

96

YP

SN

R [

dB

]

JSVM common ref.STP-tool common ref.JSVM own ref.STP-tool own ref.

Figure 16. PSNR comparison at QCIF resolution with the usage of own andcommon reference.

process which shows vertically the video frame frequencycontent and horizontally other kind of differences (accountingfor filter response discrepancies and spatial aliasing). Suchdifferences can be considered small since V1 and V2 aregenerated starting from the same reference X while targetingthe same down-sampling objective. Thus V can be consideredhalfway on the above projection being a linear average. Now,as transform based coding systems tend to reconstruct first thelower spectral components of the signal, it is plausible that,at a certain bit rate (represented in Fig.15(b) by the dashedbold line), the VidWav reconstructed signal lies nearer to theJSVM reference than to its own reference. The converse isnot possible and this gives to V the capability to compensatefor this disparity. Therefore, signal V can reasonably be usedas a common reference for PSNR comparisons of decodedsequences generated by different SVC systems. These conjec-tures find a clear confirmation in experimental measures as itcan be seen e.g. in Fig.16.

VI. FURTHER INVESTIGATIONS ON WSVC SCHEMES

In order to better identify the aspects or tools that couldsignificantly improve WSVC performances and to drive futureinvestigations and improvements more precisely, a series ofbasic experiments has been conducted comparing JSVM andits most similar WSVC codec, namely STP-tool.In a first experiment a set of generic video sequences and

20000 40000 60000 80000 100000 120000 140000

rate [kbps]

30

32

34

36

38

40

Y P

SN

R [

dB

]

1920x1024 JM

1920x1024 STP-Tool

960x512@25Hz (BL) STP-Tool

960x512 JM

960x512 STP-Tool

CrowdRun SNR Scalability

Figure 17. Quality (SNR) scalability results for the HD video sequenceCrowdRun.

still images were selected with the intent to compare the twocodecs only in terms of intra coding performance. In otherwords, we tested the ability of independently compressing astill image that can be either a still image or frame randomlyselected from a video sequence. Each test image was codedusing both codecs in a single layer configuration repeating theprocedure for a range of different rate values. These valueshave been selected in a way to have a reconstruction qualityranging from 30 to 40 dB. On the basis of the quality ofdecoded images the test set would be divided into two groups.The first group included all the images for which JSVMgives the best average coding performance while the secondincludes those where PSNR is in favor of the STP-tool codec.Not surprisingly, the first group contains only frame of videosequences such as those used for the MPEG comparisons andsequences acquired with low resolution cameras. Conversely,the second group was characterized by the presence of all thetested still images and high definition (HD) video sequencesacquired with devices featuring the latest digital technologies.Thus, most performing DCT-based technologies appear tooutperform wavelet-based ones for relatively smooth signalsand vice versa. This indicates that eligible applications forWSVC are those that produce or use high-definition/high-resolution content. The fact that wavelet-based image (intra)coding can offer comparable if not better performance whenapplied to HD content is compatible with the results reportedby other independent studies, e.g., [84] [85]. In [86] theSTP-tool has been used to test the performances of WSVCapproaches for in home distribution of HD video sequence.As it can be noticed from Fig.17 the SNR performance ofthe proposed codec are comparable with those offered bythe state of the art of single point encoding techniques, e.g.H.264/AVC. Another set of tests has been conducted withthe intent of analyzing the coding efficiency of the multi-resolution to compress high pass MC temporal subbands (orhierarchical B-frames residues). In this case coding efficiencywas evaluated by comparing the performance only on theresidual image by using the same predictor for both encodersand by changing the filters of the spatial wavelet transform.

16

The main consideration arising from the obtained resultsis that commonly used filters, such as the 9x7, 5x3 and2x2, are not as effective on high pass temporal subbands(or B residues) as much as they are on low pass temporalsubbands (or frames). This is mainly because of the natureof the signal being considered which contains the residueinformation produced by a local adaptation procedure whilethe wavelet filter operates as a global transformation of theconsidered temporal subband. This means, for example, thatif we want to encode a frame containing reconstruction errorsonly for one macro-block after the spatial transformationthese errors will be spread across all the spatial subbandsand the relative error support will also be extended becauseof filter lengths. A more detailed analysis of the energy ofspatial subbands revealed that, contrarily to what happens forlow pass temporal frames, most of the energy is located inthe highest spatial resolution subbands. This is in accordancewith the multi-resolution interpretation for the used spatialtransform but because of their size these subbands cannotbe efficiently encoded. Additionally, this prevents inter-bandprediction mechanisms to correctly work.

VII. PERSPECTIVES

Although considerable efforts have been made to improveWSVC solutions there are some reasons to say that we couldbe at the beginning of a prolific research field with manypossible applications. Present coding performances are nottoo far from that of JSVM, which implements the technologydefined in the forthcoming SVC standard. Hybrid video coding(especially based on block-based spatial DCT and block-basedtemporal motion estimation and compensation) has experi-mented a strong increase in performance as a consequenceof the introduction of many optimized tools and configurationparameters. While some of these solutions have not yet beentested on a common reference architecture such as VidWav,other were forcely discarded since not adequate to the wholeframe nature of the spatial wavelet transform. For exampleadvanced block-based predictions with multiple block-modeselection, advanced block-based motion estimation, compen-sation and coding can be fully adopted with block-based trans-forms while they are not natural with full frame transforms. Anexample for which an attempt has been made is represented bythe “intra-prediction” mode. This tool works remarkably wellfor JSVM but if simply imported on a VidWav architecture,it does not give the expected results. Wavelet based videocoding systems would require alternative prediction modes andmotion models. Existing solutions appear still preliminary and,in order to maximize their benefit, they need to be effectivelyintegrated in the complete system. When compared to JSVM,existing wavelet codecs lack several performance optimiza-tion tools. In particular R-D optimized solutions are hardlyimplementing those tools which are not fully compatible withthe whole frame approach used in WSVC systems. Therefore,both a) activities directed on finding new tools and solutionsand b) activities directed on shaping and optimizing the useof existing tools, seem to be important for the improvementand development of effective WSVC systems.

New application potentials

There are a number of functionalities and applications whichcan be effectively targeted by WSVC technology and seem toprospect comparable if not more natural solutions with respectto their hybrid DCT+ME/MC SVC counterparts. A brief list isreported here, while a more complete overview can be foundin [3].• HD material storage and distribution. Thanks to the built-

in scalability brought by wavelets, it is possible to encodea given content with non-predefined scalability range andat a quality up to near lossless. At the same time, avery low definition decoding is possible allowing, a quickpreview of the content, for example.

• The possibility of using non-dyadic wavelet decompo-sitions [87], which enables the support of multiple HDformats is an interesting feature not only for the afore-mentioned application but also for video surveillance andfor mobile video applications.

• WSVC with temporal filtering can also be adapted sothat some of the allowable operating points can bedecoded by J2K and MJ2K (MJ2K is only “intra” codedvideo using J2K; if J2K compatibility is preserved MJ2Kcompatibility will also remain).

• Enabling efficient similarity search in large video data-bases. Different methods based on wavelets exist. Insteadof searching full resolution videos, one can search lowquality videos (spatially, temporally, and reduced SNR),to accelerate the search operation. Starting from salientpoints, on low quality videos, similarities can be refinedin space and time.

• Multiple Description Coding which would lead to bettererror-resilience. By using the lifting scheme, it is easy toreplicate video data to transmit two similar bit streams.Using intelligent splitting, one can decode independentlyor jointly the two bit streams (spatially, temporally and/orSNR reduced).

• Space variant resolution adaptive decoding. When encod-ing the video material, it is possible to decode at a highspatial resolution only a certain region while keeping ata lower resolution the surrounding areas.

VIII. CONCLUSION

This paper has provided a picture of various tools that havebeen designed in recent years for scalable video compression.It has focused on multi-resolution representations with the useof the wavelet transform and its extensions to handle motion inimage sequences. A presentation of benefits of wavelet basedapproaches has also been suggested for various applicationperspectives. The paper has also shown how it is possible toorganize various multi-resolution tools into different architec-ture families depending on the (space/time) order in whichthe signal transformation is operated. All such architecturesmake it possible to generate embedded bit-streams that canbe parsed to handle combined quality, spatial and temporalscalability requirements. This has been extensively simulatedduring the exploration activity on wavelet video coding ascarried out effort by ISO/MPEG, demonstrating that credible

17

performance results could be achieved. Nevertheless, whencompared to the current SVC standardization effort, whichprovides a pyramidal extension of MPEG-4/AVC, it appearsthat though close, the performance of wavelet based videocodecs remains inferior. This seems partly due to:• the level of fine (perceptual and/or rate-distortion the-

oretic) optimization that can be achieved for hybrid(DCT+ME/MC) codecs because of the deep understand-ing on how all its components interact;

• the spectral characteristics of standard resolution videomaterial on which standard multiresolution (waveletbased) representation already exhibit lower performanceeven in intra-mode (some conducted preliminary experi-ments on high resolution material are showing increasingperformance for WT approaches) ;

• the inability of the experimented multiresolution (WT)approaches to effectively represent the information lo-cality in image sequences (block based transform can beinstead more easily adapted to handle the non stationarityof visual content, in particular by managing uncov-ered/covered areas through a local effective representationof intracoded blocks).

This last argument deserves probably some further consid-erations. For image compression, WT based approaches areshowing quite competitive performance due to the energycompaction ability of the WT to handle piecewise polynomialsthat are known to well describe many natural images. Invideo sequences, the adequacy of such model falls apartunless a precise alignment of moving object trajectories can beachieved. This might remain only a challenge, since as for anysegmentation problem, it is difficult to achieve it in a robustfashion, due to the complex information modeling which isoften necessary. As an alternative, new transform families[88] could also be studied so as to model the specificity ofspatiotemporal visual information.

IX. ACKNOWLEDGMENTS

This paper is an attempt to summarize the current under-standing in wavelet video coding as carried out collectivelyin the framework of the ISO/MPEG standardization. While ithas not yet been possible to establish wavelet video coding asan alternative video coding standard for SVC, its performanceappears on the rise when compared to previous attempts toestablish credibly competitive video coding solutions withrespect to hybrid motion-compensated transform based ap-proaches. We hope that the picture offered will help manyfuture researchers to undertake work in the field and shape thefuture of wavelet video compression. The authors would liketo acknowledge the efforts of colleagues and friends who havehelped to establish the state-of-the-art in the field as describedin this work. Particular thanks are meant for all those institu-tions (academia, funding organizations, research centers andindustries) which continue to support work in this field and inparticular the people who have participated in the ISO/MPEGwavelet video coding exploration activity during several years,e.g. Christine Guillemot, Woo-Jin Han, Jens Ohm, StephanePateux, Beatrice Pesquest-Popescu, Julien Reichel, Mihaela

Van der Shaar, David Taubmann, John Woods, Ji-Zheng Xu,just to name a few.

REFERENCES

[1] ISO/MPEG Video, “Registered responses to the call for proposalson scalable video coding,” ISO/IEC JTC1/SC29/WG11, 68th MPEGMeeting, Munich, Germany, Tech. Rep. M10569/S01-S24, Mar. 2004.

[2] ——, “Joint scalable video model (JSVM) 6.0,” ISO/IECJTC1/SC29/WG11, 76th MPEG Meeting, Montreux, Switzerland,Tech. Rep. N8015, Apr. 2006.

[3] R. Leonardi, T. Oelbaum, and J.-R. Ohm, “Status report on wavelet videocoding exploration,” ISO/IEC JTC1/SC29/WG11, 76th MPEG Meeting,Montreux, Switzerland, Tech. Rep. N8043, Apr. 2006.

[4] ISO/MPEG Requirements, “Applications and requirements for scalablevideo coding,” ISO/IEC JTC1/SC29/WG11, 71st MPEG Meeting, HongKong, China, Tech. Rep. N6880, Jan. 2005.

[5] ISO/MPEG Video, “Description of core experiments in MPEG-21 scal-able video coding,” ISO/IEC JTC1/SC29/WG11, 69th MPEG Meeting,Redmond, WA, USA, Tech. Rep. N6521, July 2004.

[6] P. Burt and E. Adelson, “The laplacian pyramid as a compact imagecode,” IEEE Trans. Commun., vol. COM-31, pp. 532–540, Apr. 1983.

[7] S. Mallat, A Wavelet Tour of Signal Processing. San Diego, CA, USA:Academic Press, 1998.

[8] P. Vaidyanathan, Multirate systems and filter banks. Prentice Hall PTR,1992.

[9] M. Vetterli and J. Kovacevic, Wavelets and Subband Coding. Engle-wood Cliffs, NJ: Prentice-Hall, 1995.

[10] I. Daubechies, Ten lectures on wavelets. Philadelphia, PA, USA: SIAM,1992.

[11] W. Sweldens, “The lifting scheme: a construction of second generationwavelets,” SIAM J. Math. Anal., vol. 29, no. 2, pp. 511–546, 1997.

[12] I. Daubechies and W. Sweldens, “Factoring wavelet transforms intolifting steps,” J. Fourier Anal. Appl., vol. 4, no. 3, pp. 247–269, 1998.

[13] A. Calderbank, I. Daubechies, W. Sweldensand, and B.-L. Yeo, “Wavelettransforms that map integers to integers,” Appl. Comput. Harmon. Anal.,vol. 5, no. 3, pp. 332–369, 1998.

[14] H. Heijmans and J. Goutsias, “Multiresolution signal decompositionschemes. part 2: Morphological wavelets,” CWI, Amsterdam, The Ned-erlands, Tech. Rep. PNA-R9905, June 1999.

[15] J.-R. Ohm, Multimedia Communication Technology. Springer Verlag,2003.

[16] J. Ohm, “Three-dimensional subband coding with motion compensa-tion,” IEEE Trans. Image Processing, vol. IP-3, pp. 559–571, Sept. 1994.

[17] S.-J. Choi and J. Woods, “Motion-compensated 3-D subband coding ofvideo,” IEEE Trans. Image Processing, vol. IP-8, no. 2, pp. 155–167,Feb. 1999.

[18] B.-J. Kim, Z. Xiong, and W. Pearlman, “Low bit–rate scalable videocoding with 3D set partitioning in hierarchical trees (3D SPIHT),” IEEETrans. Circuits Syst. Video Technol., vol. CSVT-10, pp. 1374–1387, Dec.2000.

[19] A. Secker and D. Taubman, “Lifting-based invertible motion adaptivetransform (LIMAT) framework for highly scalable video compression,”IEEE Trans. Image Processing, vol. IP-12, no. 12, pp. 1530–1542, Dec.2003.

[20] G. Pau, C. Tillier, B. Pesquet-Popescu, and H. Heijmans, “Motioncompensation and scalability in lifting–based video coding,” SignalProcess.: Image Commun., vol. 19, no. 7, pp. 577–600, Aug. 2004.

[21] B. Pesquet-Popescu and V. Bottreau, “Three dimensional lifting schemesfor motion compensated video compression,” in Proc. Int. Conf. onAcoust. Speech and Signal Process. (ICASSP), vol. 3, Salt Lake City,USA, May 2001, pp. 1793–1796.

[22] J. Ohm, “Motion-compensated wavelet lifting filters with flexible adap-tation,” in Proc. Thyrrenian Int. Workshop on Digital Commun. (IWDC),Capri, Italy, Sept. 2002.

[23] Y. Zhan, M. Picard, B. Pesquet-Popescu, and H. Heijmans, “Longtemporal filters in lifting schemes for scalable video coding,” ISO/IECJTC1/SC29/WG11, 61th MPEG Meeting, Klagenfurt, Austria, Tech.Rep. M8680, July 2002.

[24] J. Xu, Z. Xiong, S. Li, and Y.-Q. Zhang, “Three-dimensional embeddedsubband coding with optimal truncation (3D ESCOT),” Appl. Comput.Harmon. Anal., vol. 10, pp. 290–315, 2001.

[25] H. Schwarz, D. Marpe, and T. Wiegand, “Scalable extension ofH.264/AVC,” ISO/IEC JTC1/SC29/WG11, 68th MPEG Meeting, Mu-nich, Germany, Tech. Rep. M10569/S03, Mar. 2004.

18

[26] R. Xiong, F. Wu, S. Li, Z. Xiong, and Y.-Q. Zhang, “Exploiting temporalcorrelation with block-size adaptive motion alignment for 3D waveletcoding,” in Proc. Visual Commun. Image Process., vol. SPIE-5308, SanJose, CA, Jan. 2004, pp. 144–155.

[27] A. Secker and D. Taubman, “Highly scalable video compression usinga lifting-based 3D wavelet transform with deformable mesh motioncompensation,” in Proc. Int. Conf. on Image Process. (ICIP), Sept. 2002,pp. 749–752.

[28] L. Luo, F. Wu, S. Li, Z. Xiong, and Z. Zhuang, “Advanced motionthreading for 3D wavelet video coding,” Signal Process.: Image Com-mun., vol. 19, no. 7, pp. 601–616, 2004.

[29] D. Turaga, M. van der Schaar, and B. Pesquet-Popescu, “Temporalprediction and differential coding of motion vectors in the MCTFframework,” in Proc. Int. Conf. on Image Process. (ICIP), Barcelona,Spain, Sept. 2003.

[30] V. Valentin, M. Cagnazzo, M. Antonini, and M. Barlaud, “Scalablecontext-based motion vector coding for video compression,” in PictureCoding Symp. (PCS), Saint-Malo, F, Apr. 2003, pp. 63–68.

[31] J. Xu, R. Xiong, B. Feng, G. Sullivan, M.-C. Lee, F. Wu, andS. Li, “3D subband video coding using barbell lifting,” ISO/IECJTC1/SC29/WG11, 68th MPEG Meeting, Munich, Germany, Tech. Rep.M10569/S05, Mar. 2004.

[32] C. Tillier, B. Pesquet-Popescu, and M. van der Schaar, “Highly scalablevideo coding by bidirectional predict-update 3-band schemes,” in Proc.Int. Conf. on Acoust. Speech and Signal Process. (ICASSP), Montreal,Canada, May 2004.

[33] M. V. der Schaar and D. Turaga, “Unconstrained motion compensatedtemporal filtering (UMCTF) framework for wavelet video coding,” inProc. Int. Conf. on Acoust. Speech and Signal Process. (ICASSP), vol. 3,Hong-Kong, China, Apr. 2003, pp. 81–84.

[34] N. Mehrseresht and D. Taubman, “Adaptively weighted update steps inmotion compensated lifting based on scalable video compression,” inProc. Int. Conf. on Image Process. (ICIP), vol. 3, Barcelona, Spain,Sept. 2003, pp. 771–774.

[35] D. Maestroni, M. Tagliasacchi, and S. Tubaro, “In–band adaptive updatestep based on local content activity,” in Proc. Visual Commun. ImageProcess., vol. SPIE-5960, Beijing, China, July 2005, pp. 169–176.

[36] ISO/MPEG Video, “Wavelet codec reference document and softwaremanual,” ISO/IEC JTC1/SC29/WG11, 73th MPEG Meeting, Poznan,Poland, Tech. Rep. N7334, July 2005.

[37] H. Schwarz, D. Marpe, and T. Wiegand, “Analysis of hierarchical Bpictures and MCTF,” in Proc. Int. Conf. on Multimedia & Expo (ICME),Toronto, Canada, July 2006.

[38] D. Turaga, M. van der Schaar, and B. Pesquet-Popescu, “Complexityscalable motion compensated wavelet video encoding,” IEEE Trans.Circuits Syst. Video Technol., vol. CSVT-15, no. 8, pp. 982–993, Aug.2005.

[39] C. Parisot, M. Antonini, and M. Barlaud, “Motion-conpensated scanbased wavelet transform for video coding,” in Proc. Thyrrenian Int.Workshop on Digital Commun. (IWDC), Capri, Italy, Sept. 2002.

[40] G. Pau, B. Pesquet-Popescu, M. van der Schaar, and J. Vieron, “Delay-performance trade-offs in motion-compensated scalable subband videocompression,” in ACIVS’04, Brussels, Belgium, Sept. 2004.

[41] J. Shapiro, “Embedded image coding using zerotrees of wavelet coef-ficients,” IEEE Trans. Signal Processing, vol. SP-41, pp. 3445–3462,Dec. 1993.

[42] A. Said and W. A. Pearlman, “A new fast and efficient image codecbased on set partitioning in hierarchical trees,” IEEE Trans. CircuitsSyst. Video Technol., vol. CSVT-6, pp. 243–250, June 1996.

[43] S.-T. Hsiang and J. W. Woods, “Embedded image coding using ze-roblocks of subband/wavelet coefficients and context modeling,” inMPEG-4 Workshop and Exhibition at ISCAS 2000, Geneva, Switzerland,May 2000.

[44] S. D. Servetto, K. Ramchandran, and M. T. Orchard, “Image codingbased on a morphological representation of wavelet data,” IEEE Trans.Image Processing, vol. IP-8, pp. 1161–1174, Sept. 1999.

[45] F. Lazzaroni, R. Leonardi, and A. Signoroni, “High-performance em-bedded morphological wavelet coding,” IEEE Signal Processing Lett.,vol. SPL-10, no. 10, pp. 293–295, Oct. 2003.

[46] ——, “High-performance embedded morphological wavelet coding,” inProc. Thyrrenian Int. Workshop on Digital Commun. (IWDC), Capri,Italy, Sep. 2002, pp. 319–326.

[47] F. Lazzaroni, A. Signoroni, and R. Leonardi, “Embedded morphologicaldilation coding for 2D and 3D images,” in Proc. Visual Commun. ImageProcess., vol. SPIE-4671, San Jose, California, Jan. 2002, pp. 923–934.

[48] D. Taubman, “High performance scalable image compression withEBCOT,” IEEE Trans. Image Processing, vol. IP-9, pp. 1158–1170, July2000.

[49] D. Taubman, E. Ordentlich, M. Weinberger, and G. Seroussi, “Embeddedblock coding in JPEG 2000,” Signal Proc.: Image Comm., vol. 17, pp.49–72, Jan. 2002.

[50] J. Li and S. Lei, “Rate-distortion optimized embedding,” in PictureCoding Symp., Sept. 1997, pp. 201–206.

[51] S.-T. Hsiang and J. Woods, “Embedded video coding using invertiblemotion compensated 3-D subband/wavelet filter bank,” Signal Process.:Image Commun., vol. 16, no. 8, pp. 705–724, May 2001.

[52] P. Chen and J. W. Woods, “Bidirectional MC-EZBC with lifting imple-mentation,” IEEE Trans. Circuits Syst. Video Technol., vol. CSVT-14,pp. 1183–1194, Oct. 2004.

[53] N. Adami, M. Brescianini, M. Dalai, R. Leonardi, and A. Signoroni,“A fully scalable video coder with inter-scale wavelet prediction andmorphological coding,” in Proc. Visual Commun. Image Process., vol.SPIE-5960, Beijing, China, July 2005, pp. 535–546.

[54] V. Bottreau, M. Benetiere, B. Felts, and B. Pesquet-Popescu, “A fullyscalable 3D subband video codec,” in Proc. Int. Conf. on Image Process.(ICIP), vol. 2, Thessaloniki, Greece, Oct. 2001, pp. 1017–1020.

[55] K. Ho and D. Lun, “Efficient wavelet based temporally scalable videocoding,” in Proc. Int. Conf. on Image Process. (ICIP), vol. 1, New York,USA, Aug. 2002, pp. 881–884.

[56] N. Cammas, “Codage video scalable par maillages et ondelettes t+2D,”Ph.D. dissertation, Univ. of Rennes 1, IRISA, Rennes, France, Nov.2004. [Online]. Available: http://www.inria.fr/rrrt/tu-1122.html

[57] G. Pau, “Ondlettes et decompositions spatio–temporelles avancees;application au codage video scalable,” Ph.D. dissertation, Ecole nationalesuperieure de telecomunications, Paris, France, 2006.

[58] M. Flierl and B. Girod, “Video coding with motion-compensated liftedwavelet transforms,” Signal Process.: Image Commun., vol. 19, no. 7,pp. 561–575, Aug. 2004.

[59] J. Barbarien, A. Munteanu, F. Verdicchio, Y. Andreopoulos, J. Cornelis,and P. Schelkens, “Motion and texture rate–allocation for prediction–based scalable motion–vector coding,” Signal Process.: Image Commun.,vol. 20, no. 4, pp. 315–342, Apr. 2005.

[60] M. Mrak, N. Sprljan, G. Abhayaratne, and E. Izquierdo, “Scalablegeneration and coding of motion vectors for highly scalable videocoding,” in Picture Coding Symp. (PCS), San Francisco, CA, USA, 2004.

[61] V. Bottreau, C. Guillemot, R. Ansari, and E. Francois, “SVC CE5: spatialtransform using three lifting steps filters,” ISO/IEC JTC1/SC29/WG11,70th MPEG Meeting, Palma de Mallorca, Spain, Tech. Rep. M11328,Oct. 2004.

[62] Y. Wu and J. Woods, “Aliasing reduction for scalable subband/waveletvideo coding,” ISO/IEC JTC1/SC29/WG11, 73rd MPEG Meeting, Poz-nan, Poland, Tech. Rep. M12376, July 2005.

[63] Y. Andreopoulos, A. Munteanu, J. Barbarien, M. van der Schaar,J. Cornelis, and P. Schelkens, “Inband motion compensated temporalfiltering,” Signal Process.: Image Commun., vol. 19, no. 7, pp. 653–673, Aug. 2004.

[64] H.-W. Park and H.-S. Kim, “Motion estimation using low–band–shiftmethod for wavelet–based moving–picture coding,” IEEE Trans. ImageProcessing, vol. IP-9, pp. 577–587, 2000.

[65] Y. Andreopoulos, A. Munteanu, G. V. D. Auwera, J. Cornelis, andP. Schelkens, “Complete-to-overcomplete discrete wavelet transforms:theory and applications,” IEEE Trans. Signal Processing, vol. SP-53,no. 4, pp. 1398–1412, Apr. 2005.

[66] ISO/MPEG Video, “Subjective test results for the CfP on scalable videocoding technology,” ISO/IEC JTC1/SC29/WG11, 68th MPEG Meeting,Munich, Germany, Tech. Rep. N6383, Mar. 2004.

[67] N. Mehrseresht and D. Taubman, “An efficient content-adaptive MC 3D-DWT with enhanced spatial and temporal scalability,” in Proc. Int. Conf.on Image Process. (ICIP), Oct. 2004.

[68] D. Taubman and N. Mehrseresht and R. Leung, “SVC technical contribu-tion: Overview of recent technology developments at UNSW,” ISO/IECJTC1/SC29/WG11, 69th MPEG Meeting, Redmond, WA, USA, Tech.Rep. M10868, July 2004.

[69] N. Sprljan, M. Mrak, T. Zgaljic, and E. Izquierdo, “Software proposal forwavelet video coding exploration group,” ISO/IEC JTC1/SC29/WG11,75th MPEG Meeting, Bangkok, Thailand, Tech. Rep. M12941, Jan.2006.

[70] M. Mrak, N. Sprljan, T. Zgaljic, N. Ramzan, S. Wan, and E. Izquierdo,“Performance evidence of software proposal for wavelet video codingexploration group,” ISO/IEC JTC1/SC29/WG11, 6th MPEG Meeting,Montreux, Switzerland, Tech. Rep. M13146, Apr. 2006.

19

[71] C. Ong, S. Shen, M. Lee, , and Y. Honda, “Wavelet video coding - gener-alized spatial temporal scalability (GSTS),” ISO/IEC JTC1/SC29/WG11,72nd MPEG Meeting, Busan, Korea, Tech. Rep. M11952, Apr. 2005.

[72] ISO/MPEG Video, “Scalable video model v2.0,” ISO/IECJTC1/SC29/WG11, 69th MPEG Meeting, Redmond, WA, USA,Tech. Rep. N6520, July 2004.

[73] M. N. Do and M. Vetterli, “Framing pyramids,” IEEE Trans. SignalProcessing, vol. SP-51, no. 9, pp. 2329–2342, Sept. 2003.

[74] M. Flierl and P. Vandergheynst, “An improved pyramid for spatiallyscalable video coding,” in Proc. Int. Conf. on Image Process. (ICIP),Genova, Italy, Sept. 2005.

[75] G. Rath and C. Guillemot, “Representing laplacian pyramids withvarying amount of redundancy,” in Proc. European Signal Process. Conf.(EUSIPCO), Florence, Italy, Sept. 2006.

[76] N. Adami, M. Brescianini, R. Leonardi, and A. Signoroni, “SVCCE1: STool - a native spatially scalable approach to SVC,” ISO/IECJTC1/SC29/WG11, 70th MPEG Meeting, Palma de Mallorca, Spain,Tech. Rep. M11368, Oct. 2004.

[77] ISO/MPEG Video, “Report of the subjective quality evaluation for SVCCE1,” ISO/IEC JTC1/SC29/WG11, 70th MPEG Meeting, Palma deMallorca, Spain, Tech. Rep. N6736, Oct. 2004.

[78] R. Leonardi and S. Brangoulo, “Wavelet codec reference documentand software manual v2.0,” ISO/IEC JTC1/SC29/WG11, 74th MPEGMeeting, Nice, France, Tech. Rep. N7573, Oct. 2005.

[79] D. Zhang, S. Jiao, J. Xu, F. Wu, W. Zhang, and H. Xiong, “Mode-based temporal filtering for in-band wavelet video coding with spatialscalability,” in Proc. Visual Commun. Image Process., vol. SPIE-5960,Beijing, China, July 2005, pp. 355–363.

[80] M. Beermann and M. Wien, “Wavelet video coding, EE4: Joint reductionof ringing and blocking,” ISO/IEC JTC1/SC29/WG11, 74th MPEGMeeting, Nice, France, Tech. Rep. M12640, Oct. 2005.

[81] M. Li and T. Nguyen, “Optimal wavelet filter design in scalable videocoding,” in Proc. Int. Conf. on Image Process. (ICIP), Genova, Italy,Sept. 2005.

[82] ISO/MPEG Video, “Description of testing in wavelet video coding,”ISO/IEC JTC1/SC29/WG11, 75th MPEG Meeting, Bangkok, Thailand,Tech. Rep. N7823, Jan. 2006.

[83] C. Fenimore, V. Baroncini, T. Oelbaum, and T. K. Tan, “Subjectivetesting methodology in MPEG video verification,” in Proc. Applicationsof Digital Image Process. XXVII, vol. SPIE-5558, aug 2004, pp. 503–511.

[84] M. Ouaret, F. Dufaux, and T. Ebrahimi, “On comparing JPEG2000 andintraframe AVC,” in Proc. Applications of Digital Image ProcessingXXIX, vol. SPIE-6312, Genova, Italy, Aug. 2006.

[85] D. Marpe, S. Gordon, and T. Wiegand, “H.264/MPEG4-AVC fidelityrange extensions: Tools, profiles, performance, and application areas,”in Proc. Int. Conf. on Image Process. (ICIP), vol. 1, Genova, Italy, Sept.2005, pp. 593–596.

[86] L. Lima, F. Manerba, N. Adami, A. Signoroni, and R. Leonardi,“Wavelet-based encoding for HD applications,” in Proc. Int. Conf. onMultimedia & Expo (ICME), Beijing, China, July 2007.

[87] G. Pau and B. Pesquet-Popescu, “Image coding with rational spatial scal-ability,” in Proc. European Signal Process. Conf. (EUSIPCO), Florence,Italy, Sept. 2006.

[88] B. Wang, Y. Wang, I. Selesnick, and A. Vetro, “An investigation of 3Ddual-tree wavelet transform for video coding,” in Proc. Int. Conf. onImage Process. (ICIP), Singapore, Oct. 2004.

Nicola Adami has received the Laurea degree inElectronic Engineering in 1998 and the Ph.D de-gree in Information Engineering in 2002, from theUniversity of Brescia, Italy. From 2002 till the endof 2003 he has been fellow at the same institute.From the 2003 he is holding the position AssistantProfessor at the Engineering Faculty of Universityof Brescia. His research interests are signal process-ing, multimedia content analysis and representation,image and video coding. He has been involvedin several European granted projects and has been

participating to ISO/MPEG and JPEG standardization works.

Alberto Signoroni (M’03) has received the Laureadegree in Electronic Engineering and the Ph.D. in In-formation Engineering from the University of Bres-cia, Brescia, Italy, in 1997 and 2001, respectively.Since March 2002, he has been an Assistant Profes-sor in Signal Processing and Telecommunications atthe Department of Electronics for the Automation ofthe University of Brescia. His research interests arein image processing, image and video coding sys-tems, medical image coding, medical image analysis.

Riccardo Leonardi has obtained his Diploma(1984) and Ph.D. (1987) degrees in Electrical En-gineering from the Swiss Federal Institute of Tech-nology in Lausanne. He spent one year (1987-88) asa post-doctoral fellow with the Information ResearchLaboratory at the University of California, SantaBarbara (USA). From 1988 to 1991, he was a Mem-ber of Technical Staff at AT&T Bell Laboratories,performing research activities on image communi-cation systems. In 1991, he returned briefly to theSwiss Federal Institute of Technology in Lausanne to

coordinate the research activities of the Signal Processing Laboratory. SinceFebruary 1992, he has been appointed at the University of Brescia to leadresearch and teaching in the field of Telecommunication. His main researchinterests cover the field of Digital Signal Processing applications, with aspecific expertise on visual communications, and content-based analysis ofaudio-visual information. He has published more than 100 papers on thesetopics. Since 1997, he acts also as an evaluator and auditor for the EuropeanUnion IST and COST programmes.