Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


OverviewLecture 5

• Low-density parity-check (LDPC) codes

• Iterative decoding on the factor graph

• Density evolution

Lecture 6: Turbo Codes

1 Overview2 Turbo Encoder

StructureGeneralization

3 Turbo DecoderOptimal DecoderApproximation: The Turbo DecoderExtrinsic InformationRealization of the Turbo DecoderStopping Criteria

4 BCJR Algorithm5 Performance Analysis

BER SimulationUnion Bound

2 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


Overview

Information theory

• Random codes with infinite block length achieve the channelcapacity.

Traditional block codes

• Fixed block length• Algebraic code designs• Often difficult to decode

Convolutional codes

• Encoding and decoding of sequences, no (or not necessarily) fixedblock length.

• Efficient decoding with trellis based decoding (Viterbi or BCJRalgorithm).

• Strong codes only with high constraint length (→ high decodingcomplexity)

Traditional design goals

• Design codes with good distance properties(e.g., large minimum distance).

• Establish dependencies between a large number of bits(large memory).

3 / 25

Lecture 6


Overview

Turbo Encoder

Structure

Generalization

Turbo Decoder

BCJR Algorithm


Turbo Encoder– Structure

EncoderRSC

EncoderRSC

EncoderRSC

EncoderRSC

classical description alternative description

π

B

X

U

MX

R1 = 1

R1 = 1/2

X2

X1

X1

B

X2

π

MUX

B

X

R2 = 1R2 = 1

Components

Idea: construct a strong code from simple component codes

• Code rate: 1RTC

= 1R1

+ 1R2

(classical Turbo code RTC = 1/3)

• Component encoders• Must be recursive → RSC codes→ necessary for good distance properties

• Impact on convergence, position of waterfall region, error floor

• Interleaver• “Random” permutation of information symbols• Necessary for good distance properties• Decorrelates mesages exchanged during iterative decoding• Block interleaver, random interleaver, s-random interleaver

4 / 25

Lecture 6


Overview

Turbo Encoder

Structure

Generalization

Turbo Decoder

BCJR Algorithm


Turbo Encoder– Generalization

EncoderRSC

π

π2

EncoderRSC

EncoderRSC

B X1

X2

XM

XUX

M

πM−1

Multiple Turbo code

• M parallel concatenated codes (RSC codes)

• M − 1 individual interleavers

• Code rate: 1RMTC

=M∑

i=1

1Ri

→ Very strong code with low rate

5 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder

Extrinsic Information

Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Optimal Decoder

Encoder 1RSC

Encoder 2RSC

TransmissionChannel

TransmissionChannel

π

X1

X2 Y 2

Y 1B

L(Bn|Y 1,Y 2)Log-APP

Decoder

B̃

Optimal solution

• Jointly considering the constraints from both codesand the interleaver

• Highly complex since the interleaver introduces large memory

→ In general not applicable

6 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Approximation: The Turbo Decoder

Encoder 1RSC

Encoder 2RSC

TransmissionChannel

TransmissionChannel

π

X1

X2 Y 2

Y 1

Decoder 2SISO

Decoder 1SISO L(Bn|Y 1,A1)

L(B̃n|Y 2,A2)

A2

A1

B

B̃

General idea

Approximate L(Bn|Y1,Y2) by separating the decoderinto 2 cooperating SISO decoders (e.g., Log-APP decoders)

• Decoder 1• Exploits code constraints from the RSC Code 1• Generates L(Bn|A1,Y1), with additional a priori information

A1 = f (Y2,A2), provided by Decoder 2• Returns a priori information A2 = f (Y1,A1) to Decoder 2

• Decoder 2• Exploits code constraints from the RSC Code 2• Generates L(B̃n|A2,Y2), with additional a priori information

A2 = f (Y1,A1), provided by Decoder 1• Returns a priori information A1 = f (Y2,A2) to Decoder 1

7 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Approximation: The Turbo Decoder

Encoder 1RSC

Encoder 2RSC

TransmissionChannel

TransmissionChannel

π

X1

X2 Y 2

Y 1

Decoder 2SISO

Decoder 1SISO L(Bn|Y 1,A1)

L(B̃n|Y 2,A2)

A2

A1

B

B̃

Decoding procedure

• Initial step: Decoder 1 performs decoding solely based on the observationY1 (no a priori information available, A1 = 0)

• 2. step: Decoder 2 performs decoding based on the observation Y2 andthe updated a priori vector A2; returns the updated a priori vector A1

• 3. step: Decoder 1 performs decoding based on the observation Y1 andthe updated a priori vector A1; returns the updated a priori vector A2

• The steps 2. and 3. are repeated until convergence is reached or astopping criterion is fulfilled

8 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Example

1 0 1 0 00 1 1 1 10 1 1 0 01 0 0 0 10 0 0 1

→ BEC →(binary erasure

channel)7/24 bits erased

C = 0.71

x 0 1 0 0x 1 x 1 10 x x 0 01 0 x x 10 0 0 1

Parallel concatenated single-parity-check codes

• information word B = [1010 0111 0110 1000] is written in a matrix→ block interleaver

• 2 single-parity-check codes (SPCCs) are applied to the columns androws

• iterative decoding between the 2 SPCC decoders

1. horiz. decoder

1 0 1 0 0x 1 x 1 10 x x 0 01 0 x x 10 0 0 1

2. vert. decoder

1 0 1 0 00 1 x 1 10 1 x 0 01 0 x 0 10 0 0 1

3. horiz. decoder

1 0 1 0 00 1 1 1 10 1 1 0 01 0 0 0 10 0 0 1

9 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Extrinsic Information

Y 2

Y 1

Decoder 2SISO

Decoder 1SISO

L(B̃n|Y 2,A2) = A2n + E2n

A2

L(Bn|Y 1,A1)

A1 = ?

How to choose the a priori information?

Example: passing a priori information from Decoder 2 to Decoder 1

• The a posteriori LLRs L(B̃|Y2,A2) include both the a priori LLRs A2n

and the extrinsic LLRs E2n

• the a priori LLRs A2n are provided by Decoder 1; they are known to

Decoder 1 and need not to be fed back!

• The extrinsic LLRs E2n combine the observations Y2 and the a priorivalues A2\n in a way which is unknown to Decoder 1

→ Only the extrinsic LLRs are exchanged between the decoders

A1n = π−1 (E2n) and A2n = π (E1n)

10 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Extrinsic Information

Extrinsic LLRs

• Factorization of the APPs Pr(Bn|A,Y)

Pr(Bn|A,Y) =1

CPr(An|Bn) · Pr(Yn|Bn) · Pr(A\n,Y\n|Bn) · Pr(Bn),

with A\n = [An−11 ,AN

n+1] and Y\n = [Yn−11 ,Yn,YN

n+1]

• Expressed as LLRs

L(Bn|A,Y) = L(An|Bn)︸︷︷︸additional

a priori LLR

+ L(Yn|Bn)︸︷︷︸channel LLR

+ L(A\n,Y\n|Bn)︸︷︷︸extrinsic LLR

+ L(Bn)︸︷︷︸a priori

distribution

,

• For Turbo decoding relevant: extrinsic LLRs w.r.t. An

En := L(A\n,Y|Bn) = L(Bn|A,Y)− An

= L(Yn|Bn)︸︷︷︸channel LLR

+ L(A\n,Y\n|Bn)︸︷︷︸extrinsic LLR

Note: for LLRs An it holds L(An|Bn) + L(Bn) = L(Bn|An) = An

11 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Realization of the Turbo Decoder

Decoder 2SISO

π

π−1

Decoder 1SISOY 1

Y 2

A2

A1

L(Bn|Y 1,A1)

L(B̃n|Y 2,A2)

E2

E1

12 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Realization of the Turbo Decoder

Decoder 2SISO

π

π−1

Decoder 1SISOY 1

Y 2

A2

A1

L(Bn|Y 1,A1)

L(B̃n|Y 2,A2)

E2

E1

Comments

• BCJR algorithm expects the observations An,Y1n,Y2n at its inputsto be statistically independent

• Due to the iterations independence of the channel observations inYi and the a priori LLRs in Ai cannot be maintained (cycles!)

• Interleaver mitigates this effect; local dependencies are spread

• Dependencies become a problem for a high number of iterationsand/or short block length (→ increased error floor)

13 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

Optimal Decoder

Turbo Decoder


Realization

Stopping Criteria

BCJR Algorithm


Turbo Decoder– Stopping Criteria

• Stop if the hard-decision estimates of both decoders agree.

• Cross entropy (Kullback-Leibler Distance)

• Many more...

14 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BCJR Algorithm

bk qk qk−1qk−2D D

xk

bk

00

01

10

11

s′ = (qk−1, qk−2) s = (qk , qk−1)

00

01

10

11

Output bk xk

00

11

00

11

01

10

01

10

• Named after the inventors (Bahl, Cocke, Jelinek, and Raviv).• Soft-input/soft-output decoding algorithm for trellis codes;

trellis based derivation of the a posteriori probabilities Pr(u|y).

• Observation: each state transition sk → sk+1 defines uniquely acode symbol vk and an information symbol uk .

• APPs for the symbols uk , vk can be derived from the APPs of thestate transitions s ′ → s (branch APPs)

Pr(Bk = b|y) =1

Pr(y)

∑(s′,s)∈Ub

Pr(s′, s, y) and Pr(xk = x|y) =1

Pr(y)

∑(s′,s)∈Vx

Pr(s′, s, y)

with the sets Ub,Vx of state transitions (s ′, s) associated with therealization (b, x) of the respective bits.

15 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BCJR Algorithm

• Factorization of the branch probability

Pr(s′, s, y) = Pr(Sk =s′, Sk+1 =s, yk−11 , yk , y

Kk+1)

= Pr(yKk+1|Sk+1 =s)︸︷︷︸

=:βk (s)

Pr(yk , Sk+1 =s|Sk =s′)︸︷︷︸=:γk (s′,s)

Pr(Sk =s′, yk−11 )︸︷︷︸

=:αk−1(s′)

→ βk (s) considers the future observations yKk+1.

→ γk (s′, s) corresponds to the observation yk = [ybk , yx k ] of thecurrent state transition (s′, s).

→ αk−1(s′) considers the past observations yk−11 .

• Forward recursion for deriving αk (s)

αk (s) = Pr(Sk+1 = s, yk1 ) =

∑s′

Pr(Sk+1 = s, Sk = s′, yk , yk−11 )

=∑

s′Pr(yk , Sk+1 = s|Sk = s′) Pr(Sk = s′, yk−1

1 )

=∑

s′γk (s′, s)αk−1(s′)

with the initialization α0(s ′) = 1 for s ′ = 0 and α0(s ′) = 0 else.

16 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BCJR Algorithm

• Backward recursion for deriving βk−1(s ′) follows in a similar way:

βk−1(s ′) =∑

s

βk (s)γk (s ′, s)

with the initialization βK (s) = 1 for s = 0 and α0(s) = 0 else.

• The γ-term can be derived as

γk (s ′, s) = Pr(yk ,Sk+1 =s|Sk =s ′)(a)=

Pr(yk |Sk+1 =s, Sk =s ′) Pr(Sk+1 =s|Sk =s ′)

(b)=

Pr(yk |(bk , xk )) Pr(bk )1(s′,s)

(c)=

Pr(ybk |bk ) Pr(yx k |xk ) Pr(bk )1(s′,s)

with 1(s′,s) = 1 if (s ′, s) is a valid state transition and 1(s′,s) = 0else.

• (a) follows from Bayes’ rule,• (b) holds since (s′, s) uniquely defines (uk , vk ) and since for a given

state s′ the transition to s is driven by the information bit uk .• (c) holds due to the conditional independence of channel

observations

17 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BCJR AlgorithmImplementation

• BCJR is implemented in log-domain to avoid numerical instabilitiesdue to low numbers (log-APP decoding).

• Jacobian logarithm and max∗ operation:

log(ea + eb) = max(a, b) + log(1 + e−|a−b|) =: max∗(a, b)

Generalization: max∗(x1, . . . , xN ) = max∗(x1,max∗(x2, . . . , xN )).

• Then,

L(Bk |y) = max(s′,s)∈U0

∗ log Pr(s′, s, y)− max(s′,s)∈U1

∗ log Pr(s′, s, y)

log Pr(s′, s, y) = log βk (s) + log γk (s′, s) + logαk−1(s′)

logαk (s) = maxs′∗ log γk (s′, s) + logαk−1(s′)

log βk (s′) = maxs

∗ log βk (s) + log γk (s′, s)

log γk (s′, s) =1− 2bk

2(L(Bk ) + L(Ybk |Bk )) +

1− 2xk

2L(Yx k |Xk )

where we ignore some normalization constants in the log γ termthat cancel out in the calculation of L(Bk |y).

• If applied for Turbo decoding, replace L(Bk ) by L(Bk ) + L(Ak |Bk ).

18 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound

Performance Analysis– Simulation

[U. Madhow, Fundamentals of Dig. Comm., 2008]

Example

• Rate-1/3 Turbo code with recursive convolutional codes (5, 7)8

• Waterfall region: can be predicted by analyzing the convergence ofthe iterative decoding (e.g., density evolution, EXIT charts).

• Error-floor region (at high SNR): analysis of the distance propertiesand union bound.

19 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound

Performance Analysis– Union Bound

An approximation of the union bound for Turbo codes is given by

Pe∼≤

K∑w=wmin

wW w w !

(nmax !)2K 2nmax−w−1A(P|w , nmax )2

∣∣∣∣∣∣W =P=e

− Eb RN0

,

where

• w is the information weight of the error events

• nmax ≤ w is the maximum number of error events that can lead toinformation weight w

• A(P|w , n) is the parity weight enumeration function conditioned onthe information weight w , assuming of n error events

• K is the interleaver length

Performance is influenced by the terms in the exponents of K , W , and P.

Dominating term at high SNR: w = wmin.

20 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound



Pe∼≤

K∑w=wmin

wW w w !


∣∣∣∣∣∣W =P=e

− Eb RN0

.

Recursive codes

• We have wmin = 2, nmax = bw/2c• Case 1: w =2k → nmax =k→ dominating term decays with K−1; interleaver gain!

Furthermore, A(P|w , nmax ) = A(P|2k, k) = A(P|2, 1)k .

• Case 2: w =2k + 1 → nmax =k→ odd input weights decay with K−2 and can be neglected.

• Case 1 gives us the effective free distance peff = 2 + 2pmin.

• Conclusion: Turbo codes have a low minimum free distance but theinterleaver gain (Pe ∼ K−1) guarantees error probabilities around10−5 . . . 10−6 for sufficiently large K .

21 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound



Pe∼≤

K∑w=wmin

wW w w !


∣∣∣∣∣∣W =P=e

− Eb RN0

.

Non-recursive codes

• We have wmin = 1, nmax = w , and A(P|w , nmax ) = A(P|1, 1)w , andit follows that

Pe∼≤

K∑w=1

K w−1

(w − 1)!W w A(P|1, 1)2w

∣∣∣∣∣W =P=e

− Eb RN0

.

→ The dominant term at high SNR is w = 1.

• Observation: the performance does not improve with increasingblock length K , no interleaver gain!

22 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound

Performance Analysis– Derivation of the Union Bound

• With the input/parity-weight enumerator function Aturbo(w , p):

Pe ≤∑

w

∑p

w

KAturbo(w , p)Q

(√2EbR(w + p)

N0

)(a)≤

∑w

∑p

w

KAturbo(w , p)e

− Eb RN0

we− Eb R

N0p

(b)≤

∑w

w

KW wAturbo(P|w)

∣∣∣∣∣W =P=e

− Eb RN0

and by using

(a) the upper bound Q(x) ≤ e−x2

2 ;(b) the conditional parity weight enumerator function

Aturbo (P|w) =∑

p

Aturbo (w , p)Pp .

→ How to derive Aturbo (P|w)?

23 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound


• Assumption: random interleaver that maps one weight-w info wordinto another weight-w info word with uniform probability 1/

(Kw

).

• With the conditional weight enumerator functionsA1(P|w),A2(P|w) of the component encoders we get:

Aturbo(P|w) =A1(P|w)A2(P|w)(

Kw

)• Approximation of the conditional parity weight enumerator function

for convolutional codes:

A(P|w) ≈nmax∑n=1

A(P|w , n)

(K

n

), nmax ≤ w ≤ K

where A(P|w , n) is the parity weight enumerator function for inputweight w and codewords consisting of n error events.

24 / 25

Lecture 6


Overview

Turbo Encoder

Turbo Decoder

BCJR Algorithm


BER Simulation

Union Bound


• By combining the results and using the approximation(

Kl

)≈ K l/l!

for large K , we get:

Aturbo(P|w) ≈nmax∑n1=1

nmax∑n2=1

(Kn1

)(Kn2

)(

Kw

) A1(P|w , n1)A2(P|w , n2)

≈nmax∑n1=1

nmax∑n2=1

w !

n1!n2!K n1+n2−wA1(P|w , n1)A2(P|w , n2)

≈ w !

(nmax !)2K 2nmax−wA1(P|w , nmax )A2(P|w , nmax )

and with A1(P|w , n) = A2(P|w , n) = A(P|w , n)

Pe∼≤

K∑w=wmin

wW w w !


∣∣∣∣∣∣W =P=e

− Eb RN0

25 / 25

Advanced Channel Coding Turbo Codes and Iterative Decoding

Documents

Transcript of Advanced Channel Coding Turbo Codes and Iterative Decoding