1

Inter-Layer FEC Decoded Multi-Layer VideoStreaming

Yongkai Huo, Xin Zuo, Robert G. Maunder and Lajos HanzoSchool of ECS, University of Southampton, SO17 1BJ, United Kingdom.

Tel: +44-23-8059 3125, Fax: +44-23-8059 4508Email: {yh3g09, xz3g08, rm, lh}@ecs.soton.ac.uk, http://www-mobile.ecs.soton.ac.uk

Abstract— Layered video coding creates multiple layers ofunequal importance, where the enhancement layers will beaffected when the base layer is corrupted. In this treatise, a novelinter-layer FEC scheme is investigated, where the information ofthe base layer1is incorporated into the systematic bits of theenhancement layers with the aid of an exclusive-OR operation.When the base layer can be recovered independently, the softinformation of the enhancement layers can be deduced by flippingthe sign of the check information received. In this case, theprotection strength of the enhancement layer is unaffected andno extra protection bits are required. Otherwise, the inter-layerFEC decoding philosophy related to the base layer and theenhancement layer will be activated to assist in decoding thebase layer, where the protection of the enhancement layer isadditionally exploited to protect the base layer. Data partitioningbased experiments show that our proposed scheme outperformsthe traditional unequal error protection FEC aided transmissionsystem by about 1.8 dB of channel SNR or 7.7 dB of PSNR atan acceptable complexity.

I. INTRODUCTION

Layered video coding [1] was proposed for video streamingrelying on multiple layers of unequal importance, namelyon the base layer and enhancement layers, which have beenwidely employed in existing video standards. The movingpicture expert group (MPEG) [2] developed a multiview profile(MVP) [3] based stereoscopic video compression technique aspart of the MPEG-2 standard, where the left view is encodedinto the base layer and the right view is encoded into anenhancement layer. Furthermore, the related technique of scal-able video coding (SVC) [4], [5] employed in H.264/AVC [5]provides multiple layered bitstreams, which may be adaptedaccording to the preferences of the users. For example, depend-ing on the preferences of specific users, a subset of layers inthe encoded bitstream may either be incorporated or discarded,depending on the user’s requirements. A different applicationof layered video-streaming may be found in networks, wherethe less important enhancement layers may be discarded incase of network-congestion or buffer overflow [6].

In general, when some of the layers are lost or corruptedduring transmission, other layers which depend on them cannotbe decoded by the video decoder, even if the dependent layersare perfectly received [4]. Hence, unequal error protection(UEP) [7] was proposed for strongly protecting the base-layer

The financial support of EU’s concerto project, of the EPSRC under theauspices of the China-UK Science Bridge and that of the RC-UK under theIndia-UK Advanced Technology Centre (IU-ATC) is gratefully acknowledged.

—————-1In layered video bitstreams, an enhancement layer may also be protected

by its dependent enhancement layers in the same way as the base layer isprotected by the enhancement layers. In this treatise, we only describe thedependency between the base layer and the enhancement layers for simplicity,but more complex dependency scenarios may be readily applied.

for the sake of improving the overall system performance. Theauthors of [8] proposed the UEP density evolution (UDE)principle for transmission over the binary erasure channel(BEC). As a further advance, unequal error protection basedturbo coded modulation was investigated in [9] and both thechannel capacity as well as the cutoff rates of the differentprotection levels were determined. Cross-layer operation aidedSVC coded video streaming was proposed in [10], whichensures robust delivery of scalable video over error-pronechannels. More specifically, the transmitter estimated the totaldistortion of the reconstructed video frame conditioned onboth the bandwidth, as well as on the packet loss ratio (PLR),and invoked a powerful error concealment method. Based onthe estimated distortion, content-aware rate allocation and anoptimized bit detection technique was conceived.

In traditional UEP, forward error correction (FEC) wasinvoked separately for each layer. However, when for examplethe base-layer is corrupted, the enhancement layers dependingon it must be dropped. Hence the transmission power assignedto the enhancement layers was wasted. To circumvent thisproblem, the authors of [11], [12] proposed the so-called layer-aware forward error correction (LA-FEC) philosophy relyingon Raptor coded protection of the enhancement layer fortransmission over the BEC. At the transmitter, the parity bitsof the less important enhancement layers may also be used forprotecting the base layer. When the physical layer FEC fails tocorrectly decode the more important base layer, the LA-FECassigned to the less important enhancement layers may also beinvoked for correcting the remaining errors in the base layer.

Motivated by these advances, in this treatise we developan inter-layer operation aided FEC scheme relying on asystematic FEC code in the physical layer, where the originalsystematic video bits of the base layer are also incorporatedinto the systematic bits of the enhancement layers using theXOR operation. When the decoder fails to correct the channel-induced errors residing in the base layer, our inter-layer FEC(IL-FEC) principle will be activated to assist in decodingthe base layer with the aid of the classic sum-product [13]algorithm. The proposed technique may be readily combinedwith any existing systematic unequal/equal error protectioncodecs used in the enhancement layer.

This rest of this paper is organized as follows. In SectionII, we review the state-of-the-art in layered video techniques.Section III presents our IL-FEC encoding technique and itscorresponding decoding algorithm. The performance of IL-FEC using a classic turbo code and a recursive systematicconvolutional (RSC) code is characterized in Section IV fortwo video sequences of different motion activity. Finally, weoffer our conclusions in Section V.

2

II. LAYERED VIDEO STREAMING

layer 0

layer 1

⋮

layer L-1

layer 0

layer 1

⋮

layer L-1

⊕

⊕

Fig. 1. Architecture of a layered video scheme [1], where the video qualityis refined gradually.

Again, in layered video compression [1], [4], [12], thevideo is encoded into several discrete layers, which are ofunequal importance and are capable of progressively refiningthe video resolution/quality. The most important layer is thebase layer, while the remaining layers are referred to asenhancement layers. An enhancement layer may in fact befurther relied upon by a number of enhancement layers, po-tentially introducing multiple encapsulated dependent layers.Each enhancement layer requires the availability of all of itsunderlying base layers for decoding. The architecture of alayered video scheme is displayed in Fig. 1.

SVC [4] has been an active research field for more thantwo decades. This terminology is also used in the Annex Gextension of the H.264/AVC video compression standard [5].Indeed, SVC is capable of generating several bitstreams thatmay be decoded at a similar quality and compression ratio tothat of the existing H.264/AVC codec. When for example low-cost, low-quality streaming is required by the users, some ofthe enhancement layers may be removed from the compressedvideo stream, which facilitates flexible bitrate-control basedon the specific preferences of the users. SVC supports threedifferent scalability techniques, namely temporal scalability,spatial scalability and direct quality-based scalability.

Recently, the Joint Video Team (JVT) proposed multiviewvideo coding (MVC) as an amendment to the H.264/AVCstandard [5]. Apart from the classic techniques employed insingle-view coding, multiview video coding invokes the so-called inter-view correction technique by jointly processingthe different views for the sake of reducing the bitrate. Hence,the first encoded view may be termed as the base layer, whilethe remaining views may be treated as the enhancement layers.

Another technique referred to as data partitioning (DP) [5],[14], [15] is also supported by the H.264/AVC codec, whichcreates multiple bitstreams per video slice. In the DP mode, allsymbols of a video slice having similar semantic importanceare allocated to one of the partitions. There are at most threedifferent partition types [16], namely the type A partition,type B partition and type C partition. The information carriedby the type A partition includes the header information,such as macroblock (MB) types, quantization parameters andmotion vectors. The type B partition is also termed as theintra-partition and contains the so-called coded block patterns(CBPs) and intra-frame coded coefficients. Since the type Bpartition carries intra-frame-coded information, it is capable ofcurtailing error propagation in the reconstructed video, whenerrors corrupt the reference frame of the current picture. Incontrast to the type B partition, the type C partition carries theinter-frame-coded information, including the inter-CBPs andthe inter-frame coded coefficients. In the absence of partitionA, the information in partition B and C cannot be used.

DEMUX

L0

L1

EncoderH.264

Concatenate

DecoderH.264

Deconcatenate

MUX

y0Le(x0)

x1,p

s

s x

x

x1

x0

x0,p

n

y0,p

y01Le(x0)/x0

x1

x0

y1,p

Encoder 0

Encoder 1

Decoder 0

Decoder 1

x01

Le(x1)L(x1)

FEC

FEC

FEC

FEC

Fig. 2. IL-FEC encoding architecture of layered video with two layers.

However, given the availability of partition A, partition B andpartition C can be used independently of each other. Hence,the type A partition may be treated as the base layer, whilethe type B and type C partitions may be interpreted as twoenhancement layers.

III. INTER-LAYER FORWARD ERROR CORRECTION CODE

In this section, we will introduce the architecture of theinter-layer FEC scheme. Firstly, the encoding scheme willbe presented in Section III-A. Then the corresponding de-coding algorithm using the sum-product algorithm [13] willbe introduced in Section III-B. Before elaborating further,we stipulate the following scenarios. There are two layersin the coded bitstream, namely L0 and L1, where L0, L1

are the base layer and the enhancement layer, respectively.The corresponding architecture of the transmission scheme isshown in Fig. 2. Although we consider two layers only, the IL-FEC may also be readily applied to more complex scenarios,such as SVC. Furthermore, arbitrary systematic FEC codecssuch as systematic LDPCs [17], turbo codes and RSC codesmay be readily applied as our FEC code.

A. Inter-Layer FEC Encoding

At the transmitter, the H.264 encoder encodes the rawvideo signal s into the bitstream x, which is split into thebase layer L0 and the enhancement layer L1 by the DEMUXcomponent2of Fig. 2. Then the bit sequence x0 of the baselayer will be encoded into the systematic bits x0 and parity bitsx0,p by the FEC encoder 0 of Fig. 2, while the bit sequence ofthe enhancement layer x1 will be encoded into the systematicbits x1 and the parity bits x1,p by the FEC encoder 1. The XORoperation will be performed on the bit sequences x0 and x1 togenerate the check bits x01, thereby implanting the informationof x0 into the bit sequence x01. Finally, the bit sequences x0,x0,p, x01 and x1,p will be concatenated into a joint bitstreamfor transmission. Note that x0 is interleaved before the XORoperation. However, we neglected this operation in Fig. 2 forsimplicity.

Furthermore, if the bit sequences x0 and x1 have an equallength, then the XOR operation may be readily performed bit

2In the AnnexB format of the H.264 bitstream, each slice may be encodedinto multiple NALUs, hence creating multiple layers, which can be readilyseparated by the DEMUX. The MUX component combines different layerscontained in different NALUs into a valid H.264 bitstream.

3

xj10 xk10x0

X0

xjn10 x

kn10

Xn1

0X10

(a)

x10

Xv10

x0 xn0

0

Xu10X0 X

un00 X

vn00

(b)Fig. 3. Definition of X0 when the base layer sequence x0 and theenhancement layer sequence x1 carry unequal length of bits. n0 and n1

indicate length of bit sequences x0 and x1, respectively.

by bit. However, in reality, the bit sequences x0 and x1 mayhave an unequal length. Hence, in the following we will detailthe XOR operation, when x0 and x1 have unequal length.Firstly, we divide x0 into n1 sets of bits, namely X1

0 ,· · · , Xn10 ,

where n1 indicates the length of the x1 sequence. When x0

carries more bits than x1, the set Xi0 is defined in Fig. 3 (a),

while if x1 carries more bits than x0, the set Xi0 is defined in

Fig. 3 (b). The set Xi0 consists of all the bits connected to it

and each unique bit xi1 of Xi

0 creates the check bit xi01. For

the scenario shown in Fig. 3 (a), each bit xi0 only belongs to

one of the sets X10 ,· · · , Xn1

0 . The XOR operation performedon Xi

0 and xi1 to generate xi

01 is expressed as follows

xi01 =

∑⊕

xr0∈Xi

0

xr0 ⊕ xi

1, 0 < i ≤ n1, (1)

where ⊕ denotes the binary XOR operation. Note that anumber of codecs, such as low-density parity-check (LDPC)codes [13] and Luby transform (LT) [18] codes may beemployed for generating the sets X1

0 ,· · · , Xn10 . However, they

may impose error-propagation in this specific scenario. Hence,we employ the above method to prevent error-propagation.

B. Inter-Layer FEC Decoding

At the receiver, the error-infested version of the transmittedsignal is received, which includes the systematic informationof the base layer y0, the parity information of the base layery0,p, the systematic information of the check bits y01 and theparity information of the enhancement layer y1,p. Providedthat the base layer can be successfully decoded 3, only thedecoder link indicated by the solid line will be employed fordecoding at the receiver side of Fig. 2. The perfectly decodedbit sequence x0 of the base layer, which is generated bydecoder 0 will be input to decoder 1. Then the correspondingsystematic information of y1 may be recovered by processingx0 and the received signal y01 of Fig. 2. In this case, onlysign flipping is performed on y01 with the aid of the equationL(u1 ⊕ u2) = L(u1)� L(u2) [19], where u1, u2 are randombinary variables and the boxplus operation � is defined as

3In video communication, typically cyclic redundancy check (CRC) codesare used to detect, whether a bitstream is error-free or not.

follows [20]

L(u1)� L(u2) = log1 + eL(u1)eL(u2)

eL(u1) + eL(u2)

=sign (L(u1)) · sign (L(u2)) ·min (|L(u1)|, |L(u2)|)

+ log(1 + e−|L(u1)+L(u2)|

)− log

(1 + e−|L(u1)−L(u2)|

).

Specifically, the corresponding process of recovering y1 maybe formulated as

yi1 =∏

xr0∈Xi

0

sign (xr0) · L

(xi01

), 0 < i ≤ n1, (2)

where xr0 is the modulated version of xr

0 and the log-likelihoodratio (LLR) L

(xi01

)may be deduced from the received signal

yi01. Since only sign flipping is performed on y01 to recovery1, the protection of x1 remains unaffected.

When the base layer cannot be successfully decoded, theinter-layer FEC decoding principle will be activated, whichutilizes the extrinsic information of x0 generated by decoder 1for recovering the base layer. In this case, the decoding processindicated by both the solid line and the dashed line of Fig. 2will be activated. The extrinsic information Le

(xi0

)extracted

from decoder 0 is utilized to recover the LLR information ofthe systematic bit xi

1 as follows

L(xi1) = L

∑⊕xr0∈Xi

0

xr0 ⊕ xi

01

=∑

�xr0∈Xi

0

Le(xr0)� L

(xi01

).

(3)

Then decoder 1 processes both the systematic informationL(xi

1) and the parity information y1,p as its input for decoding,and generates the extrinsic information of the enhancementlayer L1, namely Le

(xi1

)of Fig. 2. Again, Le

(xi1

)corre-

sponds to a unique set Xi0 and a unique check information

L(xi01

). Then the corresponding extrinsic information of any

xi0 ∈ Xi

0 may be derived as follows

Lie(x

i0) = L

∑⊕

xr0∈Xi

0\xi0

xr0 ⊕ xi

1 ⊕ xi01

=

∑�

xr0∈Xi

0\xi0

Le(xr0)� L

(xi1

)� L

(xi01

).

(4)

Furthermore, since xi0 belong to multiple sets of X1

0 ,· · · , Xn10 ,

the final extrinsic information of xi0 is expressed as the sum

Le(xi0) =

n1∑r=1

Lre(x

r0). (5)

Finally, decoder 0 processes Le(xi0) as the a-priori informa-

tion of the bit xi0 in order to perform decoding. Following

the affordable number of decoding iterations, the iterativedecoding process will terminate4.

Again, the base layer does not rely on the enhancement layerfor decoding its information and the systematic informationof the enhancement layer can be extracted from the perfectlydecoded base layer. However, when the base layer cannotbe decoded independently, the iterative decoding philosophy

4The CRC check may be performed to terminate the decoding procedure.

4

exchanging extrinsic information between the base layer andthe enhancement layer will be activated, where the checkinformation y01 will be utilized to correct the errors inflictedupon yi0. Note that when the base layer cannot be decoded cor-rectly, the enhancement layer must be dropped. Furthermore,since the transmitted signal x01 has the same length as thelength of the bit sequence x1, we do not require any extraparity bits. Furthermore, the iterative decoding philosophyis activated only when the base layer cannot be decodedcorrectly, hence we impose an acceptable complexity anddelay5. If the decoded bit sequence x0 of the base layer iscorrupted after the IL-FEC decoding stage in Fig. 2, it will bedropped together with the enhancement layer x1. Otherwisethey will be forwarded to the H.264 decoder for generatingthe reconstructed video signal s.

IV. SIMULATION RESULTS

In this section, we characterize the achievable performanceof our proposed IL-FEC scheme. Furthermore, two classicFEC codes, namely a near-capacity turbo code and an RSCcode were considered for providing more comprehensive in-sights. In all the simulations the 30-frame video sequencesrepresented in (352× 288)-pixel common intermediate for-mat (CIF) and 4:2:0 YUV format were encoded using theJM/AVC 15.1 H.264 reference video codec operated in thedata partitioning mode. The motion-copy error concealmenttool of H.264 was employed. Furthermore, the encoder wasconfigured to operate in the fixed-byte slice mode defined in[5]. Each slice was partitioned into at most three types ofpartitions, namely the type A, type B and type C partitions.The encoded H.264 bitstream consists of an intra-coded (I)frame, followed by 29 predicted (P) frames. These parametersjointly determine the bitrate. For benchmarking the perfor-mance, two sequences associated with different motion activitywere considered, namely the Foreman and Football clips. Thevideo scanning rates expressed in frame per second (FPS)were 30 and 15 for the Foreman and Football sequences,respectively. The Foreman sequence was coded into a bitrate of655 kbps and an error-free peak-signal to noise ratio (PSNR)of 38.4 dB, while the coded Football bitstream had a bitrateof 1522 kbps and a error-free PSNR of 37.6 dB.

Error Protection Code RatesArrangements Type A Type B Type C AverageEEP 1/2 1/2 1/2 1/2UEP1 5/12 1/2 7/12 1/2UEP2 9/20 21/40 21/40 1/2UEP3 2/3 4/9 4/9 1/2UEP4 5/6 5/12 5/12 1/2

TABLE ICODE RATES FOR DIFFERENT ERROR PROTECTION ARRANGEMENTS.

The bitstream was encoded and transmitted on a networkabstract layer unit (NALU) [5] basis. For the IL-FEC schemesusing the RSC codes, the generator polynomials of G =[1, 1101/1011, 1111/1011] were employed. Moreover, we em-ployed two identical RSC codes relying on the generatorpolynomials G = [1, 101/111] in our IL-turbo scheme. Inthe IL-FEC schemes, both the type B and type C parti-tions were utilized for protecting the type A partition. We

5According to our experiments, it is sufficient to use a single iteration whichresults in a low complexity.

10-5

10-4

10-3

10-2

10-1

BE

R

2 4 6 8 10 12 14

Eb/N0 (dB)

30-frame-Football sequence, FPS=15, CIF

EEP-RSC

EEP-IL-RSC

UEP1-RSC

UEP1-IL-RSC

UEP3-RSC

UEP3-IL-RSC

UEP4-RSC

UEP4-IL-RSC

Fig. 4. BER versus channel SNR performance for the type A partition ofthe Football sequence using the RSC related coding schemes of Table I.

chose the traditional turbo-coded and RSC-coded schemesas our benchmarker. Note that all NALUs in the I framewere encoded using the traditional turbo and RSC codesfor both the benchmarkers and the inter-layer FEC scheme.All the bitstreams were transmitted using binary phase shiftkeying (BPSK) modulation over an uncorrelated Rayleighfading channel. The experiments were repeated 100 times forgenerating the results of all simulation scenarios.

Since the three types of H.264 partitions have unequalimportance, different-protection FECs should be allocated tothe different partition types. In our experiments, we consideredfive possible FEC arrangements for the three types of parti-tions, whose parameters are listed in Table I. Specifically, inthe EEP schemes, the three types of partitions were allocatedequal protection. The type A partition was allocated thestrongest protection in the UEP1 and UEP2 schemes, and thelowest protection in UEP3 and UEP4. Note that we alwaysmake the overall code-rate of all partitions to be the samefor all the arrangements of Table I. Moreover, the intra-framecoded information carried by the type B partition is moreimportant than the inter-frame coded information of the typeC partition [16], [21]. Hence stronger protection was arrangedfor the type B partition than for the type C partition in theUEP1 related regimes. Furthermore, the six error protectionarrangements of Table I can be readily combined with any ofthe four benchmark encoders, namely the classic turbo, theIL-turbo, the RSC and the IL-RSC arrangements.

Below, we will firstly characterize the attainable BER versuschannel SNR performance and PSNR versus channel SNRperformance in Section IV-A. Then in Section IV-B we willquantify the computational complexity of the benchmarkersby counting the number of executed decoding operations.

A. Performance Results

Firstly, we present our simulation results for the scenarioemploying the RSC code using the Football sequence, notingthat we observed similar trends for the Foreman sequence.The BER versus channel SNR performance results for thetype A partition6is characterized in Fig. 4 for the Footballsequence. Observe in Fig. 4 that the schemes using the IL-RSC regime achieved a reduced BER in comparison to their

6All types of partitions were transmitted through the Rayleigh channel inall simulations. However, the BER versus SNR curves of different partitionsare different, since they were unequally protected by the RSC code.

5

10-5

10-4

10-3

10-2

10-1

BE

R

2 4 6 8 10

Eb/N0 (dB)

30-frame-Football sequence, FPS=15, CIF

EEP-RSC

EEP-IL-RSC

UEP1-RSC

UEP1-IL-RSC

UEP3-RSC

UEP3-IL-RSC

UEP4-RSC

UEP4-IL-RSC

Fig. 5. BER versus channel SNR performance for the type B partition ofthe Football sequence using the RSC related coding schemes of Table I.

20

22

24

26

28

30

32

34

36

38

Y-P

SN

R(d

B)

6 7 8 9 10 11 12 13 14

Eb/N0 (dB)

30-frame-Football sequence, FPS=15, CIF

EEP-RSC

EEP-IL-RSC

UEP1-RSC

UEP1-IL-RSC

UEP3-RSC

UEP3-IL-RSC

UEP4-RSC

UEP4-IL-RSC

Fig. 6. PSNR versus channel SNR performance for the Football sequenceusing the RSC related coding schemes of Table I.

benchmarkers. Specifically, the EEP-IL-RSC scheme of TableI outperforms the EEP-RSC benchmarker by about 5.9 dB ata BER of 10−5. Moreover, due to the fact that the type Apartition has the strongest protection in the UEP1 scheme, theUEP1-IL-RSC scheme has the best BER performance amongstall the schemes. Hence, we may conclude that the IL-RSCalgorithm is capable of providing a stronger protection thanthe traditional RSC scheme.

The BER versus channel SNR performance of the type Bpartition of the Football sequence is presented in Fig. 5. Forthe type C partition, we have also observed similar trends inFig. 5. It is shown in Fig. 5 that the performance of the IL-RSCscheme is always worse than that of its benchmarkers. This isattributable to the fact that provided the type A partition cannotbe corrected in its own right, the errors residing in the type Apartition propagated further to the type B partition. In otherwords, the type A partition may introduce errors into the typeB partition, when it cannot remove all the errors in its ownright. However, the errors are introduced only when decodingof the type A partition fails, when the type B partition mustbe dropped in the traditional half-rate RSC code scenarios.Hence, the BER degradation of the type B partition does notaggravate the situation any further.

Let us now present the PSNR versus channel SNR perfor-mance in Fig. 6 for the Football sequence. Observe in Fig. 6that among all the schemes using the traditional RSC scheme,the EEP-RSC arrangement exhibits the best performance. Thismay be attributable to the fact that the Football sequence

26

28

30

32

34

36

38

Y-P

SN

R(d

B)

3.0 3.5 4.0 4.5 5.0 5.5 6.0

Eb/N0 (dB)

30-frame-Foreman sequence, FPS=30, CIF

EEP-turbo

EEP-IL-turbo

UEP1-turbo

UEP1-IL-turbo

UEP2-turbo

UEP2-IL-turbo

Fig. 7. PSNR versus channel SNR performance for the Foreman sequenceusing the turbo related coding schemes of Table I.

exhibits a high motion-activity. When the type A partitionis strongly protected by sacrificing the type B and type Cpartition’s protection at a fixed total throughput as in UEP1,the type A partition may be correctly decoded at the cost ofthe type B and type C partitions. The type A partition itselfcannot alone help the H.264 decoder in concealing all theerrors in high-motion video clips. Otherwise, when the typeA partition is weakly protected as in the UEP3 and UEP4schemes of Table I, the successfully decoded type B and Cpartitions cannot be utilized for improving the reconstructedvideo quality, since the type A partition is more likely tobe corrupted. Moreover, all traditional UEP/EEP schemes arecapable of achieving some power reduction by employing ourproposed IL-RSC. We achieve a gradually improved videoquality, when weakening the protection of the type A partitionin the IL-RSC schemes. Specifically, a power reduction of1.8 dB can be achieved by employing the UEP4-IL-RSCarrangement at a PSNR of 35 dB. Alternatively, about 7.7dB PSNR video quality improvement may be attained at achannel SNR of 9 dB.

For providing further insights, the PSNR versus channelSNR performance of the IL-turbo code is presented in Fig.7 for the Foreman sequence, while the performance for theFootball sequence also exhibited similar trends in Fig. 7.Observe in Fig. 7 that among all the schemes using thetraditional turbo scheme, the UEP2-turbo arrangement exhibitsthe best performance. The UEP1-turbo codec only succeeds inoutperforming the EEP-turbo scheme beyond a channel SNRof 4.5 dB. This may be due to the fact that according to TableI the type C partition is relatively weakly protected, whichdegrades the video quality at low channel SNRs. Moreover,all traditional UEP/EEP schemes are capable of achievingan improved video quality by employing our proposed IL-turbo scheme. Specifically, about 3 dB PSNR video qualityimprovement may be attained at a channel SNR of 3.5 dB.A subjective comparison of the EEP-turbo and EEP-IL-turboarrangements is presented in Fig. 8.

B. Complexity Analysis

For the sake of providing insights into the complexityimposed by our scheme, we benchmark the complexity of ourproposed IL-FEC using the RSC code, noting that the turbocoded system exhibits similar trends in Fig. 9. Note that, ifa type A partition was corrupted, the decoding complexity ofthe type B and type C partitions did not contribute to the total

6

Fig. 8. Comparison of frames at channel SNR of 3.5 dB for the Foremanand Football sequences. The first column indicates the original frames. Thesecond column represents the EEP-turbo decoded frames. The third columnindicates the EEP-IL-turbo decoded frames.

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Aver

age

nu

mb

ero

fR

SC

dec

od

ing

per

NA

LU

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Eb/N0 (dB)

30-frame-Football sequence, FPS=15, CIF

EEP-RSC

EEP-IL-RSC

UEP1-RSC

UEP1-IL-RSC

UEP3-RSC

UEP3-IL-RSC

UEP4-RSC

UEP4-IL-RSC

Fig. 9. Complexity comparison of the IL-RSC and the classic RSC for someof the the error protection arrangements of Table I.

complexity, since they cannot be used by the H.264 decoderin this scenario. Hence, the complexity of the IL-RSC andRSC codes depends on the channel SNR. Furthermore, they-axis of the curves in Fig. 9 indicates the averaged numberof RSC decoding processes per NALU that were executed inour experiments, which we repeated 100 times for reasons ofstatistical relevance. Specifically, there are 2221 NALUs inthe H.264 encoded bitstream considered. Again, each NALUwas encoded into a single packet by the RSC/IL-RSC codes.Observe from Fig. 9 that the complexity imposed by the IL-RSC regime reduces upon increasing the channel SNR. Thisis due to the fact that the activation frequency of the inter-layer decoding technique gradually reduces, since the RSCis more likely to successfully remove all the errors in thetype A partition in its own right upon increasing the channelSNR. Specifically, the UEP4-IL-RSC scheme gains 1.8 dB ofchannel SNR by imposing about 30% higher complexity at aPSNR of 35 dB. Alternatively, the UEP4-IL-RSC has a PSNRgain of 7.7 dB upon imposing 32% higher complexity at achannel SNR of 9 dB.

V. CONCLUSIONS

A novel inter-layer FEC coded video scheme was proposed,where the information of the base layer was also incorporatedinto the systematic bits of the enhancement layers with theaid of an XOR operation. When the base layer can besuccessfully decoded in its own right, the systematic bitsof the enhancement layers can be extracted by flipping thesign of the check information received without introducingany degradation, where the check information is generated byperforming inter-layer XOR operation on the base layer and theenhancement layers. However, when the base layer cannot becorrectly decoded without the assistance of the enhancement

layers, the inter-layer FEC decoding philosophy exchanginginformation between the base layer and the enhancementlayers will be activated to assist in decoding the base layer.Our experiments show that the proposed scheme is capable ofoutperforming the traditional UEP FEC codecs.

In our future work, we will incorporate the IL-FEC schemeinto SVC and multiview video coding.

REFERENCES

[1] T. Zhang and Y. Xu, “Unequal packet loss protection for layered videotransmission,” IEEE Transactions on Broadcasting, vol. 45, pp. 243–252, June 1999.

[2] L. Hanzo, P. Cherriman, and J. Streit, Video Compression and Commu-nications: From Basics to H.261, H.263, H.264, MPEG2, MPEG4 forDVB and HSDPA-Style Adaptive Turbo-Transceivers. New York: JohnWiley, 2007.

[3] H. Imaizumi and A. Luthra, Three-Dimensional Television, Video, andDisplay Technologies, ch. MPEG-2 Multiview Profile, pp. 169–181.Berlin, Heidelberg, and New York: Springer Verlag, 2002.

[4] H. Schwarz, D. Marpe, and T. Wiegand, “Overview of the scalable videocoding extension of the H.264/AVC standard,” IEEE Transactions onCircuits and Systems for Video Technology, vol. 17, pp. 1103–1120,September 2007.

[5] Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, ITU-T Rec. H.264/ISO/IEC 14496-10 AVC: Advanced Video Coding forGeneric Audiovisual Services, March 2010.

[6] F. Yang, Q. Zhang, W. Zhu, and Y.-Q. Zhang, “End-to-end TCP-friendlystreaming protocol and bit allocation for scalable video over wirelessInternet,” IEEE Journal on Selected Areas in Communications, vol. 22,pp. 777–790, May 2004.

[7] B. Masnick and J. Wolf, “On linear unequal error protection codes,”IEEE Transactions on Information Theory, vol. 13, pp. 600–607, Octo-ber 1967.

[8] N. Rahnavard and F. Fekri, “New results on unequal error protectionusing LDPC codes,” IEEE Communications Letters, vol. 10, pp. 43–45,January 2006.

[9] M. Aydinlik and M. Salehi, “Turbo coded modulation for unequal errorprotection,” IEEE Transactions on Communications, vol. 56, pp. 555–564, April 2008.

[10] E. Maani and A. Katsaggelos, “Unequal error protection for robuststreaming of scalable video over packet lossy networks,” IEEE Trans-actions on Circuits and Systems for Video Technology, vol. 20, pp. 407–416, March 2010.

[11] C. Hellge, T. Schierl, and T. Wiegand, “Multidimensional layeredforward error correction using rateless codes,” in IEEE InternationalConference on Communications, 2008, pp. 480–484, May 2008.

[12] C. Hellge, D. Gomez-Barquero, T. Schierl, and T. Wiegand, “Layer-aware forward error correction for mobile broadcast of layered media,”IEEE Transactions on Multimedia, vol. 13, pp. 551–562, June 2011.

[13] R. Gallager, “Low-density parity-check codes,” IEEE Transactions onInformation Theory, pp. 21–28, 1962.

[14] Nasruminallah and L. Hanzo, “EXIT-chart optimized short block codesfor iterative joint source and channel decoding in H.264 video tele-phony,” IEEE Transactions on Vehicular Technology, vol. 58, pp. 4306–4315, October 2009.

[15] Nasruminallah and L. Hanzo, “Near-capacity H.264 multimedia commu-nications using iterative joint source-channel decoding,” IEEE Commu-nications Surveys and Tutorials, vol. 14, pp. 538–564, Second Quarter2012.

[16] S. Wenger, “H.264/AVC over IP,” IEEE Transactions on Circuits andSystems for Video Technology, vol. 13, pp. 645–656, July 2003.

[17] N. Miladinovic and M. Fossorier, “Systematic recursive construction ofLDPC codes,” IEEE Communications Letters, vol. 8, pp. 302–304, May2004.

[18] M. Luby, “LT codes,” in Proc. 43rd Annual IEEE Symposium Founda-tions Computer Science (FOCS), pp. 271–280, 2002.

[19] J. Hagenauer, E. Offer, and L. Papke, “Iterative decoding of binary blockand convolutional codes,” IEEE Transactions on Information Theory,vol. 42, pp. 429–445, March 1996.

[20] J. Chen, A. Dholakia, E. Eleftheriou, M. Fossorier, and X.-Y. Hu,“Reduced-complexity decoding of LDPC codes,” IEEE Transactions onCommunications, vol. 53, pp. 1288–1299, August 2005.

[21] Nasruminallah, M. El-Hajjar, N. Othman, A. Quang, and L. Hanzo,“Over-complete mapping aided, soft-bit assisted iterative unequal errorprotection H.264 joint source and channel decoding,” Vehicular Technol-ogy Conference, 2008. VTC 2008-Fall. IEEE 68th, pp. 1–5, September2008.