Bayes decoding for discrete linear channels with semantic value ofinformationM. Firer, L. Panek, and L. L. R. Rifo Citation: AIP Conf. Proc. 1490, 126 (2012); doi: 10.1063/1.4759596 View online: http://dx.doi.org/10.1063/1.4759596 View Table of Contents: http://proceedings.aip.org/dbt/dbt.jsp?KEY=APCPCS&Volume=1490&Issue=1 Published by the American Institute of Physics. Related ArticlesBayesian uncertainty quantification and propagation in molecular dynamics simulations: A high performancecomputing framework J. Chem. Phys. 137, 144103 (2012) Geometric percolation in polydisperse systems of finite-diameter rods: Effects due to particle clustering and inter-particle correlations J. Chem. Phys. 137, 134903 (2012) Wigner surmises and the two-dimensional Poisson-Voronoi tessellation J. Math. Phys. 53, 103507 (2012) Bulk universality of general β-ensembles with non-convex potential J. Math. Phys. 53, 095221 (2012) Fully synchronous solutions and the synchronization phase transition for the finite-N Kuramoto model Chaos 22, 033133 (2012) Additional information on AIP Conf. Proc.Journal Homepage: http://proceedings.aip.org/ Journal Information: http://proceedings.aip.org/about/about_the_proceedings Top downloads: http://proceedings.aip.org/dbt/most_downloaded.jsp?KEY=APCPCS Information for Authors: http://proceedings.aip.org/authors/information_for_authors

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

Bayes decoding for discrete linear channels withsemantic value of information

M. Firer∗, L. Panek† and L.L.R. Rifo∗∗

∗Department of Mathematics, Imecc - Unicamp†CECE - Unioeste

∗∗Department of Statistics, Imecc - Unicamp

Abstract. In this work we explore the decoding process when information words have differentsemantic values. In that setting, we introduce a convenient loss function that allows us to expressthe overall performance of a decoding scheme for discrete channels, switching the usual goal ofminimizing the error probability to that of minimizing the expected loss.

Keywords: BAYESDECODING,MAXIMUMLIKELIHOODDECODING, NEARESTNEIGH-BOR DECODING, POSET METRIC, INFORMATIONS WITH VALUESPACS: 02.50.Le, 89.70.-a

INTRODUCTION

Since the mid 1990s, some new metrics were introduced in the study of error-correctingcodes, mainly metrics determined by a partial order in the set of positions coordinatesof linear codes, called for simplicity just poset metrics (see [? , Jerry, Barg, Brualdi,Felix, Hyun, Hyun-MDS, Jang, Kim, Allan, PFKH, KR]. In this work we assume theuse of linear codes over discrete channels (DC), introducing a new parameter, the valueof the information, that turns poset metrics into a valuable tool for getting decoders witha good performance.More specifically, we consider the semantic value of information, something that

should be defined by the expert knowledge about the information being communicated.Such a value function, that associates to each information word a non-negative realnumber, defines a natural loss function, allowing us to make a slight but relevant changein one of the main questions driving coding theory: instead of looking for a code (withgiven properties, such as dimension and length) that minimizes the expectation of thenumber of errors after decoding, we are actually looking for a triple, consisting of acode, the way information is mapped into the code and a decoder that minimizes theexpected loss.To deal with such a larger and difficult goal, we bring into the picture a family of

decoders that are in some sense more manageable, decoders that are nearest-neighbordecoders, according to a family of metrics called poset metrics.

XI Brazilian Meeting on Bayesian StatisticsAIP Conf. Proc. 1490, 126-134 (2012); doi: 10.1063/1.4759596

© 2012 American Institute of Physics 978-0-7354-1102-9/$30.00

126

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

LINEAR CODES UNDER A DECISION-THEORETIC POINT OFVIEW

Let Fnq be the linear space of n-tuples over a finite field Fq, with cardinality q. An [n;k]q

linear code, C ⊆ Fnq, is a linear subspace with dimension k. Let dH (·, ·) be the usual

Hamming distance: dH (x,y) is the number of coordinates in which x and y differ. Thefield Fq represents our alphabet and F

nq the set of words of length n. The code is the

subset of words that we have to transmit through a channel.A discrete channel over Fq is characterized by the set of conditional probabilities{

P(b|a) : a,b ∈ Fq

}where P(b|a) represents the probability of receiving the symbol b

given that the symbol a has been transmitted. We assume that the channel is memoryless(DMC), that is

P(y|c) =n

∏i=1

P(yi|ci)

where c = (c1, . . . ,cn) and y = (y1, . . . ,yn) represent n consecutive transmitted andreceived symbols, respectively. A DMC over Fq is called symmetric (DSMC) withcrossover probability p if

P(y|c) = (1− p)n−dH(c,y)

(p

q−1

)dH(c,y)

.

Our goal consists in estimating the value c from the received value Y = y, insidethe Bayesian context, addressing decision theoretical criteria (see Parmigiani and Inoue,2009).We might suppose that some values of c ∈C are more probable to be transmitted than

others, denoting by P(c) this prior probability function on C. After observing y, P(y|c)is the likelihood function of c. Intuitively, after observing y, some values of c are morelikely to have generated the reading y.Both, prior probability and likelihood function, can be combined to give the posterior

probability of c given the observation y, P(c|y), by Bayes’ formula

P(c|y) = 1z

P(c)P(y|c),

where z = ∑τ P(τ)P(y|τ) is the normalization constant.An estimate a(y) for c, given the observed value Y = y, can be defined as a decision

rule a : Fnq →D , where D is called a decision set.

When the purpose is to obtain a point estimate for c, we can considerD =C, with a(y)belonging to the code. In this context, this decision rule a is called a decoding scheme,or just a decoder.It is reasonable to require that a(c) = c for all c ∈ C and in this situation we call a

an ordinary or reasonable decoder. Let D(c) be the decision region of c relative to thedecoding scheme a:

D(c) := a−1 (c) ={

y ∈ Fnq : a(y) = c

}.

127

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

The decision regions D(c) of a decoder a determine a partition of Fnq. Given a decoding

scheme, an error occurs if c is sent and the received codeword lies in some decisionregion D(c′), with c′ �= c. The probability of error is therefore

Pe (c) = 1− ∑y∈D(c)

P(y|c)

where the sum runs over all y ∈D(c). Assuming that the probability distribution onC isuniform, the decoding error probability of C is the average

Pe (C) =1M

∑c∈C

Pe (c) .

Let R+ be the set of non-negative real numbers and consider the map

μ0-1 :C→ R+

given by

μ0-1 (c) =

{0 if c = 01 if c �= 0 .

It follows thatPe (C) =

1M

∑c∈C

∑y∈Fn

q

μ0-1 (a(y)− c)P(y|c) .

We remark that it is essential to consider C to be a linear code, in order to ensure thata(y)− c ∈ C. The function μ0-1 is a characteristic function that only detect decodingerrors, but do not distinguish different decoding.In many real situations, it is reasonable to attribute different values to different code-

words, and this is what will be done in this work, considering instead of μ0-1 valuefunctions that may assume any (non-negative) real value. A typical example of this sit-uation is the transmission of digital images: small variations in the color values of eachpixel does not affect the quality perception of the image. This work is inspired by thisvery common kind of situation (see [? , Berlekamp, BNZ, Cover].

VALUE FUNCTIONS AND EXPECTED LOSS

In the decision theoretic framework, we define a loss function associated to a decisionrule as

l :C×C→ R+,

such that l(c,a) gives a measure of the loss when the information c ∈ C was sent andthe information y ∈ F

nq was received and decoded as a(y) = a ∈C. This function might

depend on the distance between a(y) and c, l(c,a) = r(dH(c,a)).We say that the decoder a∗ is a Bayes decoder for C relative to the loss function l if,

for each received information y ∈ Fnq, it minimizes the posterior expected loss

E(l (c,a(y)) |y) = ∑c∈C

l (c,a(y))P(c|y).

128

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

Let us consider some particular cases of loss functions.A 0-1 loss function is defined as l(c,a) = 1 if a �= c, and 0 otherwise. The posterior

expected value for this loss attains its minimum at a∗(y) = cy, where cy is a codewordsuch that

P(

cy

∣∣y)=max

c∈CP(c|y) .

This criterion is called maximum a posteriori decoder (MAP).In particular, if the prior distribution is uniform in C, that is, each codeword c is

transmitted with probability P(c) = 1M , with M = qk, then a∗(y) = cy also satisfies

P(y|cy) =maxc∈C

P(y|c)

and is named maximum likelihood decoder (ML).In that case, assuming that the channel is DSMC, both ML and MAP decoders

coincide with the nearest-neighbor decoder (NN), such that

dH (y,cy) =minc∈C

dH (y,c) .

A value function for a linear code C is a map μ that associates to each codeword anon-negative real number

μ :C→ R+.

We define the loss function related to μ , for a given reading y, as l (c,a(y)) =μ (a(y)− c). We remark that, since C is linear, the difference a(y)− c is actually acodeword hence it makes sense to consider the value μ (a(y)− c). By doing so, we areevaluating the errors that may occur during the process consisting of encoding, transmit-ting and decoding information (see [? , Kolmogorov, Shannon, Shannon-Distortion]. Insuch a situation it is reasonable to require that μ (0) = 0 (we should not loose anythingif everything was right along the process). If this happens, we say the value functionis reasonable. We make this distinction since we will use some “unreasonable” valuefunctions that proved to be valuable for proving Theorem 1.Given a linear code C, a decoder a for C, and a value function μ : C → R+, we

determine the expected loss of a relative to μ and to a received information y to be theaverage

Ly (a,μ) = E(l (c,a(y))) = ∑c∈C

μ (a(y)− c)P(c|y) .

We define the overall expected loss of a as the average of the expected loss for allpossible informations y ∈ F

nq,

Ea (μ) := E(L (a,μ)) = ∑y∈Fn

q

Ly (a,μ)P(y),

where P(y) = ∑c P(c)P(y|c) is the probability of receiving y. That expression can berewritten as

Ea (μ) = ∑c∈C

∑y∈Fn

q

μ (a(y)− c)P(y|c)P(c).

129

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

Making the change of variable τ = a(y)− c we have that

Ea (μ) = ∑τ∈C

Ga (τ)μ (τ)

whereGa (τ) = ∑

y∈Fnq

P(y|a(y)− τ)P(a(y)− τ) . (1)

We remark that Ea (μ) actually depends onC and should be denoted as Ea,C (μ), but thisdependence on C will be omitted when it should not cause any confusion.Since we are considering the value of information, the total expected loss depends not

only on the code itself but also on the way the information is mapped into the code. Inother words, we are actually considering a value function μ : I→R+ where I is a sourcecode. When we say that a codeC is given we are assuming that it is given an embeddingg : I →C ⊆ F

nq and the function μ :C→ R+ is the unique function such that μ ◦g = μ .

Given a DSMC with crossover probability p and an [n;k]q linear code C such thatP(c) = 1

M for all c ∈C, we have that from expression (1)

Ga (τ) = z ∑y∈Fn

q

sdH(y,a(y)−τ)

where

z = z(p) :=(1− p)n

Mand s = s(p) :=

p

(1− p)(q−1).

MAIN RESULTS

Let [n] := {1,2, . . . ,n} be a finite set with n elements and let be a partial order on [n].We call the pair P = ([n] ,P) a poset.

An ideal in P is a subset I satisfying the following condition: if j ∈ I and i j, theni ∈ I. Given a subset X in P, we denote by 〈X〉 the smallest ideal containing X , calledthe ideal generated by X . If x = (x1, . . . ,xn) and y = (y1, . . . ,yn) are two vectors in F

nq,

then their P-distance dP (x,y) is defined by

dP (x,y) = |〈{i : xi �= yi}〉| ,where |A| denotes the cardinality of A.An ordinary decoder aP of an [n;k]q linear code C is called a nearest-neighbor P-

decoder (P-NN) ifdP (y,aP (y)) =min

c∈CdP (y,c)

for every y ∈ Fnq. The set of all P-NN decoders associated with C, for all poset metrics

dP in Fnq, will be denoted by O (C). Those kind of decoders are called poset decoders.

A Q-NN decoder a∗Q ∈ O (C) such that

E(a∗Q,μ

)= min

aP∈O(C)E(aP,μ)

130

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

is said to be a Poset-Bayes decoder for C relative to the value function μ .

Definition 1. Given an [n;k]q linear code C, a value function μ for C, and decoders

aP,aQ ∈ O (C), we say that aP is better than aQ relative to μ if

E(aP,aQ) (μ) := EaP (μ)−EaQ(μ)< 0.

The next result states that for any binary linear code and any given decoder, thereis a different decoder such that each of them may be relevant, depending on the valuefunctions to be considered.

Theorem 1. Let C be a binary linear code and a : Fn2→C a decoder. Then, there exist

a decoder b : Fn2→C, b �= a, and value functions for which a is a better decoder than b,

and value functions for which b is better than a.

With some aditional conditions, if we consider an ML decoder a, we can ensure thatthe decoder b is a P-NN decoder. Let us define arg(dH (C,y)) as the set of all codewordsof C closest to y in the usual Hamming metric,

arg(dH (C,y)) =

{c ∈C : dH (y,c) =min

θ∈CdH (y,θ)

},

where C is an [n;k]q linear code and y ∈ Fnq.

Theorem 2. Let C be an [n;k]2 binary linear code and y = (1,1, . . . ,1). If|arg(dH (C, y))| > 1, there are ML decoders aH and aH of C such that there arevalue functions for which aH is a better decoder than aH, and value functions for whichaH is better than aH.

Let us state a important class of linear codes that satisfies the condition of the Theorem2.

Corollary 1. Let dH be the Hamming metric on Fn2 and C be an [n;k]2 binary linear

code. If k > 1, then there is a P-NN decoder aP and an H-NN decoder aH of C such thatthere are value functions for which aP is better than aH, and value functions for whichaH is better than aP.

With these results, given a fixed code, there are value functions for which it is better(in the sense of minimizing the expected loss) to decode using a non-Hamming decoder.

FINAL COMMENTS

The introduction of value functions in the context of Coding Theory gives rise to tworelevant questions: (1) Are the “value functions” indeed a new parameter that cannotbe incorporated into the existing ones? Is it possible to define a theoretical channel forwhich minimizing the error probability is equivalent to minimize the expected loss?(2) What is the actual relevance of the “value functions”? What kind of problems is itpossible to address using this approach?

131

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

For answering the first question, about the novelty of value functions, we consideran extreme situation: let us consider a code with absolutely no redundancy, that is, an[k;k]q-code C. Under this circumstance, and considering that the channel is reasonable,in the sense that, for any τ ∈C = F

kq, P(τ|τ) > P(y|τ) for any τ �= y ∈C, we have that

the unique ML decoder is the trivial decoder a(y) = y,∀y ∈C.Consider now a value function μ : C → R

+ that is reasonable, in the sense thatμ (0) = 0.The expected loss is given by

Ea (μ) = ∑τ∈C

Ga (τ) μ (τ)

= ∑τ∈C

⎛⎝ ∑

y∈Fkq

P(y|a(y)− τ)P(a(y)− τ)

⎞⎠ μ (τ)

= ∑τ∈Fk

q

⎛⎝ ∑

y∈Fkq

P(y|y− τ)P(y− τ)

⎞⎠ μ (τ) .

The question posed is the following: Can we define another channel (the transitionmatrix (P∗ (y|x)) and the frequency of messages P∗ (y)) in such a way that minimizingthe expected loss function is equivalent to using a ML-decoder for this new channel?Can we reduce this optimization problem to the usual one by defining an appropriate(even if artificial) channel model?The answer is not, for a simple reason. Let us consider the 0-1 loss function, so that

Ea (μ0-1) = ∑τ∈C

Ga (τ)μ0-1 (τ)

= ∑τ∈C

⎛⎝ ∑

y∈Fkq

P∗ (y|a(y)− τ)P∗ (a(y)− τ)

⎞⎠μ0-1 (τ) .

Since we are assuming that k = n, and our channel is assumed to be reasonable, we musthave a(y) = y, forall y ∈ F

kk, and then

Ea (μ0-1) = ∑τ∈Fk

q

⎛⎝ ∑

y∈Fkq

P∗ (y|y− τ)P∗ (y− τ)

⎞⎠μ0-1 (τ) .

Looking at Ea (μ), we see that, despite the fact we have a unique possible choice for acode (C = F

kq) and for a decoder (a(y) = y), there is still some work to be done.

We need to understand how the information is placed into the code. To be moreexplicit, let S|qk| be the symmetric group acting as a permutation group of Fk

q and S0|qk| ={g ∈ S|qk||g(0) = 0

}. Each g∈ S0|qk| determines another way to place information inside

our block code, hence determining a value function μ ◦g. We remark that since g(0) = 0we have that μ ◦g(0) = 0, hence whenever μ is a reasonable value function so is μ ◦g.

132

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

Considering the characteristic function μ0-1, we have that μ0-1 ◦ g(y) = μ0-1 (y) forevery y ∈ F

kq so that, as expected,

Ea (μ0-1 ◦g) = Ea (μ0-1) ,

that is, the error probability function depends only on the channel, the code and theassumed decoder. However, considering a value function other than the characteristicfunction μ0-1, we still have to workout an optimization problem: since μ �= μ0-1, the

subgroup Hμ =

{g ∈ S0|qk||μ ◦g = μ

}is not all S0|qk|, hence we still have to look at the

equivalence classes of S0|qk|/Hμ looking for representatives g ∈ S∗n such that

Ea (μ ◦g) = ∑τ∈C

Ga (τ) μ (g(τ))

= ∑τ∈Fk

q

⎛⎝ ∑

y∈Fkq

P(y|y− τ)P(y− τ)

⎞⎠ μ (g(τ))

is minimal. To put it shortly: even in the most trivial case that can be considered, usinga code with no redundancy and no error correction detection, non trivial better resultsmay be attained when the semantic value of errors is taken into consideration.To address the second question, about the relevance of value functions, we stress that

this kind of relevance is dependent on the application and on the models to be developedin different areas. In a picture decoding, for instance, the value can be determined by ourvisual perception of different scale-of-gray colours. In this sense, it takes into consid-eration the semantic value of information in the development of coding schemes, withdifferent schemes for transmition of different information, as for example transmitionof pictures, sound, experimental data and so on, or even considering different schemeswithin subcategories, such as black&white versus coloured images, speech and music,and so on.

ACKNOWLEDGMENTS

Significant part of this work was developed during the summer of 2011, when M. Firerwas at the Centre Interfacultaire Bernoulli at EPFL, Lausanne, Switzerland, supportedby the Swiss National Science Foundation, to which he deeply thanks. The second authorwas at that time pursuing the Ph.D. degree in the Department of Mathematics, UEM,Brazil. During 2011, L. Rifo was visiting professor at CIMFAV, at the University ofValparaiso, Chile, supported by Conicyt-MEL Grant.

REFERENCES

1. J. Anderson and M. Firer - Classification of poset-block spaces admitting MacWilliams-type identity -preprint (2011).

2. R. Ash - Information Theory - John Wiley & Sons, Inc. (1965).

133

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions

3. A. Barg and P. Purkayastha - Bounds on ordered codes and orthogonal arrays - MoscowMathematicalJournal, vol. 9, No. 2 (2009) 211-243.

4. E. R. Berlekamp, R. J. McEliece and H. C. A. van Tilborg - On the Inherent Intractability of CertainCoding Problems - IEEE Transactions on Information Theory, Vol. 24, n. 3 (1978).

5. S. Borade, B Nakiboglu and L. Zheng - Unequal error protection: an information-theoretic perspective- IEEE Transactions on Information Theory, vol. 55, No. 12 (2009) 5511-5539.

6. R. Brualdi, J. S. Graves and M. Lawrence - Codes with a poset metric - Discrete Mathematics 147(1995) 57-72.

7. T. Cover and J. A. Thomas - Elements of Information Theory - John Wiley & Sons (2006).8. L. V. Felix and M. Firer - Canonical-Systematic Form of Hierarchical Codes - pre-print (2011).9. K. Po Ho and J. M. Kahn - Imagen Transmission over Noisy Channels Using Multicarrier Modulation

- Signal Processing: Image Comunication 9 (1997) 159-169.10. J. Y. Hyun and H. K. Kim - The poset structures admitting the extended binary Hamming code to be

a perfect code - Discrete Mathematics 288 (2004) 37-47.11. J. Y. Hyun and H. K. Kim - Maximum distance separable poset codes - Designs, Codes and Cryptog-

raphy, vol. 48, No. 3 (2008) 247-261.12. C. Jang, H. K. Kim, D. Y. Oh and Y. Rho - The poset structures admitting the extended binary Golay

code to be a perfect code - Discrete Mathematics 308 (2008) 4057-4068.13. B. Juba and M. Sudan - Universal semantics of Communication I - STOC’08 Proceedings of the 40th

annual ACM symposium on Theory of computing (2008) 123-132.14. H. K. Kim and D. Y. Oh - A Classification of Poset Admitting the MacWilliams Identity - IEEE

Transactions on Information Theory, vol. 51, No. 4 (2005) 1424-1431.15. A. N. Kolmogorov - On the Shannon theory of information in the case of continuous signals - IEEE

Transactions on Information Theory, September (1956).16. B. Masnick and J. Wolf - On linear unequal error protection codes - IEEE Transactions on Informa-

tion Theory, vol. 3, No. 4 (1967) 600-607.17. A. de Oliveira Moura and M. Firer - Duality for poset codes - IEEE Transactions on Information

Theory, vol. 56, No. 7 (2010) 3180-3186.18. L. Panek, M. Firer, H. Kwang Kim and J. Yoon Hyun - Groups of linear isometries on poset structures

- Discrete Mathematics 308 (2008) 4116-4123.19. L. Panek, M. Firer and M. M. S. Alves - Symmetry groups of Rosenbloom-Tsfasman spaces - Discrete

Mathematics 309 (2009) 763-771.20. L. Panek, M. Firer and M. M. S. Alves - Classification of Niederreiter-Rosenbloom-Tsfasman Block

Codes - IEEE Transactions on Information Theory, vol. 56, No. 10 (2010) 5207-5216.21. Parmigiani, G. and Inoue, L.Y. (2009) Decision Theory, Principles and Approaches. Wiley, Cornwall,

UK.22. M. Yu Rosenbloom and M. A. Tsfasman - Codes for the m-metric - Problems of Information

Transmission 33 (1997) 45-52.23. I. Sason and S. Shamai - Performance Analysis of Linear Codes under Maximum-Likelihood Decod-

ing: A Tutorial - Foundations and Trends in Communication and Information Theory, vol. 3, No. 1-2(2006) 1-222.

24. C. E. Shannon - A mathematical theory of communication - Bell System Tech. J. 27 (1948) 379-423.25. C. E. Shannon - Coding theorems for a discrete source with a fidelity criterion - IRE Nat. Conv. Rec.,

march (1959).26. B. S. W. Schröder - Ordered Sets - An Introduction - Birkhäuser (2003).27. S. Tavildar and P. Viswanath - Approximately universal codes over slow-fading channels - IEEE

Transactions on Information Theory, vol. 52, No. 7 (2006) 3233-3258.

134

Downloaded 19 Oct 2012 to 187.106.22.116. Redistribution subject to AIP license or copyright; see http://proceedings.aip.org/about/rights_permissions