210 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-33, NO. 2, MARCH 1987

New Results on the Real-T ime Transm ission Problem

SEBASTIAN ENGELL, MEMBER, IEEE

Ahstrucf-A new concept is presented for the treatment of real-time trausmission problems. It basically consists of a modification of the flow of information. The resulting quantity, which we call the forward flow of information, is smaller than the flow of information according to the usual definition except for special cases. We derive various negative coding theorems in which the forward flow of information is used. These bounds are sharper than those previously known for transmission with finite de!ay.

I. INTRODUCTION

T HE CLASSICAL results of information theory as presented, e.g., in [l] and [2] give necessary conditions

for the transmission of a source signal with prescribed reproduction accuracy. If the transmission may be per- formed with an arbitrary delay, these bounds can actually be reached if the delay approaches infinity. Often, how- ever, the reproduction of the source has to take place without delay or with a certain maximal delay, e.g., in control applications. One m ight suspect that in this case stronger requirements for the quality of the transmission channel result from the lim itation of the possible delay. One approach to deal with this problem is the introduction of a modified rate-distortion function as proposed by Gorbunov and Pinsker [3], [4]. The rate of a source without anticipation (or with respect to a finite delay) is defined as the m inimal flow of information subject to a fidelity constraint and the restriction that the output process is causally related to the input process (resp. that the shifted output process is causally related to the input process). Then necessary conditions for the reproduction of the source arise from the requirement that the channel capac- ity in the classical sense exceeds the rate without anticipa- tion (or the rate with respect to a finite delay) of the source. The resulting bounds on the possible transmission accuracy are, however, not tight [5], and one possible way to improve things could be to replace the channel capacity by a quantity which in itself reflects the real-time aspect of the transmission problem. This is the program of the following sections. In Section II, after some preliminaries

Manuscript received March 28, 1985; revised June 18, 1986. This work was supported in part by the Deutsche Forschungsgemeinschaft under Grant DFG 152/1-l and by the Fritz-Thyssen-Stiftung.

The author was with the Measurement and Control Group, Fachbereich 7, University of Duisburg, Duisburg, W. Germany. He is-now with the Fraunhofer-Institut IITB. Sebastian-Kneinn-Strasse 12-14. D-7500 Karlsruhe 1, W. Germany.’

.A

IEEE Log Number 8610799.

we start with a definition of the forward flow of informa- tion from heuristic arguments and give some basic proper- ties of this quantity. The main idea behind the definition of the forward flow of information is to discard that part of the flow of information which is related to those points of the input process whose estimate cannot be further improved because of the prescription of a finite delay. As Section III reveals, this rather naive thovght proves to be useful for the treatment of real-time transmission problems because both in the strong donverse (on the m inimal ertor probability) and in the negative coding theorems for a general fidelity criterion the flow of information in the classical sense may be replaced, without changing the other quantities involved, by the forward flow if a finite delay is prescribed. Moreover, the forward flow and the nonantic- ipatory rate in the sense of [3] and [4] can be combined to give a necessary condition for the delay-free transmission of Markov sources. Therefore, the concept of the forward flow of information leads one step further in the derivation of necessary condition? for real-time data transmission.

The treatment here is throughout for discrete-time sta- tionary processes with finite or infinite alphabets. Exten- sions to the continuous-time case have been proposed in [8] and 191.

The processes {x(k)} and {y(k)} are assumed to be defiQed on the measurable product space (a,, gX) x

(Q,, q

We use the abbreviation x! = { xk, i < k I j} and, simi- larly, a,, for the finite or semi-infinite subspace of at,. Moreove;, by gXf we abbreviate the u-algebra of sets ExI such that (E,, x Ox, X QXY) E gX. xi and yi denote thk processes at “time” k” = i.

II. THEFOPWARDFLOWOFINFORMATION

A. Preliminaries

The usual definition of the flow af information per unit of time is [6], [7]

001%9448/87/0300-0210$01.00 01987 IEEE

ENGELL: REAL-TIME TRANSMISSION PROBLEM 211

for jointly stationary processes {x(k)} and {y(k)}, k E Z, (compare [7, sec. 7.41). From (6) where h is the (average) time between successive samples. Using Kolmogorov’s formula I($, yO,lxo_,) = 0, VNEN

I({ a, b}, c> = I(% 4 + m , cla), (2) Or (la) may be rewritten as I(-$, { yo,, x!L}) = I(-+, x0-J;

hence (lb) I( x:, y!,) 2 I(#, x”J.

A variant of the flow of information is

or, equivalently, [7] for jointly stationary processes,

which gives the additional gain of information about x provided by the observation of Y, when yr: is already known.

The relation between y( x, y) and J(x, y) and other variants of the flow of information is thoroughly discussed in [7]. For this study we need the following results.

Proposit ion 1 [7]: W e have the following: 4

J(x, Y) 2 J(x, JJ>; (4) b) if

‘(Yip Y?A) < m> (5) equality holds in (4).

In contrast to J(x, y), .$x, y) is not symmetric in general.

A transmission x + y is characterized by the condi- tional probability P(EY,mlxoO,) for EYE, E yy. If for jointly stationary processes

for all EYk E CQ transmissioc cau~al.~

with probability one, we call the

Proposit ion 2: Let the transmission x + y be causal. Then

&7(x, y) = ;qx1,, YllYLJ7

and if {x(k)} satisfies

1(x,, iv?;) < 00, (84

Now

+,N, Y,NI 2 +f, Y”,) = I( 47, Y”,> + ~bgN~ Yc?lYL)

5 +,N, Y”,) + I(x”o,, YgNIYooo)

I z(x,N, x”J + 1(x”“,, ygNlyOJ where the last inequality follows from (10). W e therefore have

which together with (4) proves our assertion.

An important class of channels is the class of causal channels with finite memory. W e call the transmission x + y causal with finite memory m (m E Z, m > 0) if

P( EY,I y!T;, x”“, ) = P( EY,Iy:lA-l, xj-.,-r), (lla)

and we call the transmission memoryless if

P( Ey,I y:-2, &) = P( E,jxi) Wb) for all Ey, E C+$, with probability one.

B. Definition of the Forward F low of Information

W e define the forward flow of information for a transmission delay d E N, as

for jointly stationary discrete-time processes {x(k)}, { y(k)}. The introduction of this quantity is motivated by the following consideration. The mutual information of two random variables u and u, def ined on space (Q2,, gu,> x (a,, 9”) is the expected function

a measurable value of the J7-6 Y> = ax, Y>.

Proofi From (6) it follows that

1(x?, yk,lxk,) = 0, (9) i(E,, u) = -log WG)

PU,“bw) and this implies (7). For stationary discrete-time processes @a) is equivalent to for E, E gu,, u E G2,. This gives a measure of the average

change from P, to P,,, due to the observation of a particu-

(8’3) lar value of u. Therefore, y(x, y) gives a measure of the average change from the conditional probability measure

212 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-33, NO. 2, MARCH 1987

P( E,, I y! ,) to P(E,, 1~5,) due to the additional ob- servati& of yi. In a tra&nission problem where (x(k)} represents the source signal to be transmitted and {y(k)} represents the observed signal, any estimation of x from y at time k is based on the conditional probability W,,_l~k).

If the transmission is required to be performed with a maximal delay d, the estimate of ~:d, has to be performed at or before time k = 0. The observation of yr does not have any influence on these estimates. Thus only the change from P(EXFJ y!,) to P(E,,dI y? ,) is relevant in this situation and g!,(x, y) gives an appropriate measure of this change in the same sense as 1(x, y) gives an appropriate measure_ in the situation where an arbitrary delay is permitted. J(x, y) characterizes the gain (per unit of time) of information about the whole process {x(k)} (- cc I k I cc) due to t$e additional observation of one value of {y(k)}, say yr. K,(x, y) characterizes the gain of information (per unit of time) about the process {x(k)} from the value k = d + 1 on into the future (hence for- wa;d in time) due to the observation of yi.

J(x, y) can be written as

where the second term gives the amount of information which yr delivers about XT, but which cannot be used in a finitely delayed transmission. Based on this reasoning, we also define

which is the counterpart of J(x, y) for the finite delay d.

III. BASIC PROPERTIES OF~$(X,Y) AND FJx, y)

The next pfpposition- summarizes some elementary properties of K, and K, which are immediate conse- quences of the properties of mutual information.

Proposition 3: a) Let 0 I d, I d,, d,, d, E RJ. Then

0 5 d,,(x, Y> 5 I,(x, Y) S J(x, Y> (144 and

b) We have

and

lim Zd(x, y) = J(x, y) d-cc (154

jimmC(x, Y) = J(x, Y). (15b)

c) Let {z(k)} be a measurable causal transformation of {y(k)), i.e.,

zk =f( Y”,), f being measurable with respect to (a,, CC$), then

&(x, Y) 2 &(x, z). (16)

Part c) of Proposition 3 is the weak data-pr_ocessing theorem for K,(x, y). Next we investigate how K,(x, y) and K,(x, y) are related.

Proposition 4: a) Generally,

FAX, y) 2 ax, y), Vd> 0. (17) b) If equality holds in (4) i.e., &x, y) = ?(x, y), then

both sides of (17) are equal for all d 2 0.

Proof: See Appendix I.

From Proposition 4 we conclude that Ed(x, y) and gd(x, y) are equal if the transmission x -+ y is causal and x satisfies the regularity condition (8a) resp. (8b). There- fore, for this case, which is the most interesting one in real-time transmission, Kd and K, can be replaced by each other without restrictions. If x + y is a causal transmission and {x(k)} has special properties, the ex- pression for the forward flow of information can be sim- plified.

Proposition 5: Let the transmission x + y be causal, and let {x(k)} be a pth order autoregressive process, i.e.,

P(E,$x’,) = P(E,+‘,) (18) for all E,,, E 5YXlrlm with probability one. Then for all drp-1

K,(x, y) = :1(x’,, YIIYLL (19)

and especially,

for p = 1 (Markov source).

Proof: From the causality of the channel,

P( E,,L,~%,) = P(E,l,I&o)

with probability one. We claim that this together with (18) implies

P( E,,~, x +1x’_,) = P(E,lmlx’,) * J&+‘,) (21)

for all EYE, E 3’Y~,, Exr E 9x? with probability one. This directly leads to

I( y’,, xImlxlp) = 0

which is sufficient for (19) resp. (20). To show (21) we simply evaluate the left side of (21):

p(EYL x E,&J = p(EYi, x E,? x !&Ix!~)

= j& P(Ey+““,,P( dx,” x W&‘,) x-p,

= LxolL _ P( E~+L)P( dC2x~,)P(WWm) X-L

ENGELL: REAL-TIME TRANSMISSION PROBLEM 213

with probability one from causality,

with probability one from (18),

= P( Egrn x f&+‘,)P( E,,-lx’,)

= P( Ey~ml-+)~(E~pl~‘,) with probability one as claimed.

If, for example, {x(k)} is a Gauss-Markov process which is transmitted over a Gaussian channel, zo(x, y) can be evaluated as

zl<X> Y) = ; log cfnin(k + IlkI

&n@lk) (22)

where eii,, (il j) denotes the m inimal expected squared error for the estimation of xi from y’,. These quantities can easily be calculated for given noise processes using the methods of Snyders [lo] or in simple cases the approach proposed by Yao [ll].

F in_ally, we investigate under which condit ions ~Jx, y) and J(x, y) are equal.

Proposit ion 6: Let the transmission x + y be causal with finite memory m 2 0. Then if

d>m, (23) g,(x, y) and f(x, y) have the same value.

Proof: This is almost immediate from the definition of the causal channel with finite memory. Remember that

f(x, y) = sip ;qx:,> rIlYQ

where 9t denotes a partition of the range of y, into a finite number of intervals and jr the resulting discretized random variable. However, from (lla) and (lib) we get

= &(x, Y) if (23) is satisfied, which together with (14a) proves the proposition.

This proposit ion shows that the m_emory of the_channel is responsible for the difference of J(x, y) and K,(x, y) and that the restriction to a causal transmission’essentially implies that al though the channel has a certain memory, this cannot be exploited fully in the coding-decoding

lnoise

(k Transmission y(k) I Estimation x it(k)

Fig. 1. Transmission problem.

procedure. If the channel is of the additive noise type

y(k) = x(k) + r(k) (24) with x and r independent processes, condit ion (23) is satisfied if the noise {r(k)} is autoregressive of an order 15s~ than or eq_ual to d. If r has independent samples, K,(x, y) and J(x, y) are equal for d = 0 and all positive d. This fact will become more understandable in connec- t&m with the negative coding results in the next section. As K,(x, y) and J(x, y) are invariant under causal transfor- mations of {y(k)}, they are also equal if the output is a causal transformation of {[x + r](k)}, and r has the respective property. In the following sections we derive the counterparts of the standard negative coding theorems using K,(x, y) or K,(x, y) instead of J(x, y); thus we obtain sharper bounds for the m inimal transmission error in real-time situations than those given by the classical theory.

IV. A STRONG CONVERSE

W e consider the situation shown in F ig. 1. The processes x, y, and 2 are assumed to be jointly stationary discrete- time processes which assume only a finite number of distinct values. W e are at first interested in error-free transmission. Let Fe denote the average probability that xk and )lk+d are different. The classical strong converse cod- ing theorem [l] then says that

&‘(pe) 2 he [f?(x) - J(x, Y)]

where g(x) = J(x, x) is the entropy flow of x and .%‘( Fe) is given by

x(FJ = P,log(M- 1) - p,log(p,)

-(l - PJlog(1 - P,) (25) (M-denotes the number of dis_tinct values of xk). & .%?( P,) is strictly increasing in P, and is zero only if P, vanishes,

H(x) I qx, y) is necessary for error-free transmission; if the entropy flow of the source exceeds j(x, y), FC > 0. O f course, Jean be replaced by J because the variables are discrete. Our stronger version of this result is Theorem 1.

Theorem I: Let the transformation y + 2 be causal. Then

z( p,) 2 h . [H(X) - 13,(x, y)].

In particular,

zd(x, y) 2 H(x)

(26)

* (27)

214 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-33, NO. 2, MARCH 1987

is necessary for error-free transmission with delay d.

For the proof we need the following lemma which is proven in Appendix II.

Lemma 2: Let {x(k)} and { u(k)} be jointly stationary discrete processes. Then for all L > d, L E N,

H(XilUi+d) 2 & I H(x,L-d)

- ~ I( xlf;i;O,i-d-l)y UilU~-’ )I . (28) i=l

Proof of Theorem 1: In [l, theorems 4.3.1 and 4.3.21 it is shown that

.x( Fe) 2 H(Xilfi+d). (29)

Lemma 2 states that the right side of (29) is greater than or equal to the right side of (28); therefore,

3r( Fe) 2 ii; [ H(x,L-d)

- i I( xtf;LTO, i-d-l), Z?ili?~-,-l i=l )I

for all L > d and, therefore,

= h [ H(x) - &(x, .?)I

2 h . [H(x) - &(x, y,]

from Propositions 4 and 3.

Thus the strong_converse coding-theorem holds if .?( x, y ) i: replaced by K,(x, y). As K,(x, y) cannot exceed J(x, y), the bound is sharper than the usual version of Theorem 1. Error-free transmission is impossible with de- lay d if the entropy flow of the source exceeds the forward flow of information for a transmission delay d over the channel x + y.

V. NEGATIVE CODING THEOREMS FOR DISCRETE SIGNALS WITH RESPECT TO A GENERAL FIDELITY

CRITERION

The fact that the strong converse for finite delay and causal decoding can be sharpened using the forward flow of information leads immediately to the conjecture that the same m ight be true for transmission under arbitrary fidel- ity constraints. In this section we show that this conjecture is true for the transmission of discrete signals. The key to this generalization is the following proposition.

P&position 7: Let { u(k)}, { u(k)} be discrete processes. Then for any joint distribution P,(u,$, u,“) (i.e., the prob-

ability of the discrete events in tit,; x GO,) which satisfies

Pl(Uo”> = PUbo”> i fl &pl [St”iY ui>l =

r=l

(c?~‘,, denotes expectation with respect Pl)

(30)

CO (31)

to the distribution

g i$lI,,( ukl, Uib$‘) = K,, (32)

a joint distribution P,(u,“, u,“) exists which satisfies

p*(&) = p&o”) = ehoL) (33)

i i$lgp2 [ZtUi, ‘i>l = ‘0 (34)

i i$lI,,( u:, uil$‘) = K,. (35)

Proof: The distribution P2 of the discrete variables ut, ~0” may be written as

p*(uk qf) = ii P*( u$Y-1, uo”) Pz( uo”). i=l

We now put

p*(&) = PIho”> and

Pz( uJ$‘, 240”) = P,( q&-l, UiL_l), i=l... L.

(36) It is then sufficient for (34) and (35) to hold that

P*(utCL_l, 0;) = P,(u,L_,, f$)), i= I... L (37) because this implies the equality of P2(ui, ui) and P1( ui, u,) as well as the equality of P,(q$) and PI($). From the latter, (35) is immediate because then

Hp2( qltg’) = I$( qltg’)

and from (36)

Hp*( u;~tgl, uo”) = HPI( Uil$‘, &).

Equation (37) can be proved by induction: for i = 1, it is obviously true. Assuming that (37) holds for i, we show that it is true for i + 1 as follows:

because of the assumption that (37) holds for i, and this proves that it holds also for i + 1.

ENGELL: REAL-TIME TRANShfISSION PROBLEM 215

From this result we get the following negative coding theorem.

Theorem 2: Let {x(k)} be a discrete stationary source and 1(x,, yk) be a fidelity criterion. Let R,(e) denote the rate of x with respect to { as follows:

R,(e) = lim 1 inf I( x,L, y,L) J5-m L P(y(fIx(f)

(38)

under the restriction

fzr each L. Then no discrete stationary transmission with K,(x, y) < R,(c) can achieve &[[(x;, y,)] S E.

Proof: Assume the following transmission:

and

From Proposit ion 7 and the fact that {y(k)} is discrete, a sequence of pairs (x,“, y,“) exists with

which fulfills the fidelity constraint, but this contradicts the definition of R,(c), and thus no such transmission can exist.

Pz( Uj, Uh) = p,( ui? ‘k).

Corollary: Consider the transmission problem depicted in F ig. 1 with discrete signals. Let the desired transmission have delay d 2 0 and fidelity e, i.e.,

8[l(xj, $j+d)] 2 c,

The first equat ion follows directly from the definition of P2 and summation over $. The second equat ion can be shown by induction as in the proof of Proposit ion 7, using the fact that

pl( ui+l, 4) = Cpl(‘i~ u~)pu(ui+Ilui) %

and let the decoder be causal. Then a necessary condit ion to achieve this is

k(4 2 Rib, Y). (40)

which follows from (41) and (42).

Proof: For d = 0 the claim follows from Theorem 2 and the fact that causal transformations cannot increase zO(x, y). For d > 0 consider the transformed output pro- cess

Theorem 3: Let {x(k)} b e a discrete stationary Markov source and {(x,, yk) be a fidelity criterion. Let R:(c) denote the nonanticipatory rate of x with respect to 2 as follows:

Clearly, u(k) = 2(k + d).

20(x, u) = &(x, a>.

As we know from Theorem 2,

&(x7 0) s R,(4

is necessary for the desired fidelity, and hence (40) because zd is nonincreasing under causal transformations.

A natural question which arises at this point is whether the aforement ioned results remain true if the transmission

is restricted to be causal and the rate-distortion function R,(E) is replaced by the nonanticipatory rate in the sense of Gorbunov and Pinsker [3], [4]. It turns out that the answer to this is affirmative if the source has the Markov property. This relies on the following result.

Proposit ion 8: Let {u(k)} and {u(k)} be discrete processes, and let { u(k)} be a Markov process

pu( % llljk--l) = c4bkl%1)~ Vj < k - 1. (41)

Let P,(uk, 0,“) be a causal joint distribution, i.e.,

for which (30)-(32) hold. Then a joint distribution Pz( u;, u,“) exists which satisfies the causality condit ion (42) such that (33)-(35) hold.

Proof: W e define P,(u,“, v,“) by

Therefore,

P*( u;luy, I4g = P,( UJuy, 2.4;).

It is then sufficient to prove

p2(& u;) = P&, u~)pu(uLIui) and

RO,(c) = liTm i inf 1(x:, ygL) P(Y6w)

(43)

under the restriction (39) and the causality condit ion

p( JQ$) = p( Yolxb), i=l... L. (44)

Then no discrete stationary causal transmission exists with K&x, y) < R:(e) such that S[{(x,, yi)] I E.

The proof of this is completely similar to the proof of Theorem 2. Generally, it is not possible to postulate this result because the distribution P2 in Proposit ion 7 is not necessari ly causal in general, and the construction of P2 used in Proposit ion 8 does not necessari ly yield the desired properties if x is not a Markov process.

216 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-33, NO. 2, MARCH 1987

VI. GENERALIZATIONTO SIGNALS WITH ARBITRARY ALPHABETS

The arguments developed in the previous section can easily be generalized to the case of arbitrary instead of finite alphabets. The counterpart of Proposition 7 is as follows.

Proposition 7’: Let {u(k)} and {u(k)} be discrete-time processes defined on the measurable space

Let gu, xol denote the smallest sub-a algebra with respect to which ‘[u!, ZJ/] is measurable. Let P, be a probability distribution on (9,k X S&i, g”k,,,t) such that

P,(E,OL x QJ = PU(E”& (45) for all Eu; E ??+ and (31) and (32) are satisfied. Then a distribution P2 exists on the same measurable space such that

P,(E”~ x Q”$) = PI@“& (46) and (34) and (35) are satisfied. The proof is outlined in Appendix III. Proposition 7’ then gives us the following negative coding theorem.

Theorem 2’: Let {x(k)} b e a discrete-time stationary source. Then a necessary condition for the stationary transmission of x under the restriction &[[(xi, ri)] I E is

I R,(4 s G(x, Y) w>: where R,(E) is defined as in Theorem 2. If the source satisfies 1(x,, &A) < cc, then

Rx(4 5 &Ax, v> .(48) is necessary for causal transmission of x under the restric- tion E[{(xi, y,)] I c.

Proof: Equation (47) is immediate from the argu- ments which were used to prove Theorem 2. If the source satisfies the foregosg regularity condition and the trans- m ission is causal, K,(x, y) = K,(x, v) and thus (48) fol- lows. Note that, generally, (48) cannot be obtained because the “channels” used in the proof of Proposition 7’ are not necessarily causal.

Corollary: Consider the transmission problem shown in Fig. 1, and assume I(xi, ~5:) < cc (regular source). Let the transmission have an admissible delay d 2 0, and let the required fidelity be b[l(xi, Zi+d)] I z. Then for causal transmission and estimation,

&(d 5 13,(x, Y) (49) is necessary. The arguments to prove this are similar to those for the discrete case. Finally, the following generali- zation of Theorem 3 also can be stated.

Theorem 3’: Let {x(k)} be a discrete-time stationary regular Markov source. Then

(50)

where R:(c) is defined as in Theorem 3 (with the neces- sary technical modifications) is necessary for causal nonde- layed transmission of x with fidelity &[S(x,, y,)] I c.

Proof: Using the same procedure as in the proof of Proposition 7’, Proposition 8 can be generalized to processes with arbitrary alphabets. The theorem then fol- lows from the same arguments as those used to prove Theorems 2, 3, and 2’.

Thus we have established in this section and the previ- ous ones the counterparts of the classical negative coding theorems with respect to the forw_ard flow of information. F_r those channels for which K,(x, y) is smaller than J(x, y) this gives sharper bounds on the transmission error which can be achieved for the given channel. If the source is Markovian or memoryless, the forward flow of informa- tion must exceed the nonanticipatory rate; hence for this case the ioss of performance due to a finite delay can be taken into account in both the characterization of the transmission and of the source.

VII. SUMMARYANDCONCLUDINGREMARKS

We have defined a new quantity for the measurement of the capability of a channel to transmit information with finite delay-the forward flow of information. We derived basic properties of this quantity, including conditions for the two variants of the forward flow of information to be equal and conditions under which the forward flow of information equals the flow of information in the classical sense. It turned out that for memoryless channels the two quantities do not differ; if, the memory of the channel exceeds the admissible delay, however, the forward flow of information is, in general, smaller than the (unrestricted) flow of information. Almost by definition, the forward flow of information is invariant under causal transforma- tions of the received signal. We then established negative coding results involving the forward flow of information.

Generally speaking,, under the restriction of causal trans- m ission with a finite delay, the forward flow of informa- tion rather than the flow of information must not be smaller than the rate of the source with respect to the desired fidelity. For Markov sources it is even possible to combine this result with the sharpening introduced by Gorbunov and Pinsker, who introduced a rate with respect to transmission with a finite delay as the m inimal flow of information in the standard sense which is necessary under this additional restriction. Therefore, for Markov processes (and also for memoryless sources) and channels with mem- ory a stronger result than those known previously follows.

For memoryless channels, no stronger bound is ob- tained. This is not surprising, because, e.g., for a Gauss-Markov source the optimal nonanticipatory chan- nel in the calculation of the nonanticipatory rate is the additive white noise channel; hence for this channel the bound resulting from R:(e) is tight and cannot be shar- pened. As R!J(_E) and R,(E) do not differ for memoryless sources, but K,(x, y) is less than J(x, y) for channels

ENGELL: REAL-TIME TRANSMISSION PROBLEM 217

with memory, sharper bounds also result for this combina- tion than those obtained from the use of R:(c) alone.

In this paper we have not studied the influence of a possible signal transformation prior to the transmission (encoding). Contrary to causal decoding, even causal en- coding pfpcedures do not leave ~Jx, y) invariant. In this respect K,(x, y) and 3(x, y) behave differently in princi- ple. Therefore, it is not possible to calculate zd(x, y) using the encoded source instead of x. If one is interested in an analogue to the channel capacity for the forward flow of information, it is not adequate to ask for the maximal forward flow for_arbitrary source and encoding. Instead, the maximum of K,(x, y) for a given source and a class of encoders may be determined, e.g., for a Gaussian memoryless source and linear encoders, to obtain a for- ward capacity with respect to this source. Another possible approach, e.g., to measurement problems, is to ask for the maximal possible forward flow of information without encoding, which would give a measure for the quality of a certain sensing configuration if the estimation of the mea- sured variable has to be performed with finite delay as in control problems. A more detailed study of these questions remains a field for future research.

ACKNOWLEDGMENT

Part of the research reported here was done while the author was visiting the Department of Electrical Engineer- ing, McGill University, Montreal, PQ, Canada. The sup- port obtained there is very gratefully acknowledged.

APPENDIXI

We shall repeatedly use the following lemma which is stated in 171.

Lemma Al: Let {a, } (n 2 1) be a sequence of nonnegative numbers, and

a, = ,tlanj

where Osa,,,IM<co, Qn,j<n

and lim a,,=a.

,Z, J’,n-J.-CC

Then

lim an = lim 1 i a,, = a. n-m n n-cc n ._ J-1

Proof of Proposit ion 4: From, e.g., [7, eq. (7.41)],

(AI.1)

where 5 is the discrete variable resulting from a partition of the range of a. Consequently,

where 9, denotes the partition of the range of y,, and y1 is the corresponding discrete variable.

On the other hand,

from stationarity, and

from (AI.l) where the 9; may be different partitions and j$‘) is the associated discrete variable. Therefore,

using Lemma Al, and the last expression equals gd(x, y). So a) is proved. To show b), we prove

K,( x, y) - &(x7 Y) 5 J(x, Y) - Jtx, Y), O<dlcn,

which obviously implies the claim that, equivalently,

qx,y> - Kd(X,Y) 2J(X,Y) - ~d(X,Y). (AI.4

From (AI.l) we have

J(x,y) - z-(/(x,y) = sup;l(xTv,Rlx’“d~Yo,)

and

J( X, Y> - Ed(.5 Y> = Nf imm i

* f I( x6-d-1, y;lyg-l, x;N_d-I). i=d+2

Repeating the arguments used earlier, we get

j(x,y) - Kd(x,y)k SUP bm -5 9, N+m Nh

and by invoking Lemma Al again,

J(x,y) - K,(x,y) 2 SUP~l(XT'1,,Y,lYO,~~"d) 91

which proves (AI.2).

218

We have

APPENDIX II PROOFOFLEMMA~

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-33, NO. 2, MARCH 1987

Equation (AIII.l) can again be proved by induction. It is by definition true for i = 1. Therefore, we assume that (AIII.l) holds for some i, and we determine

As

i=d+Z

i=d+2

= H( x~-dlUgd+l ) - H( x,-+4,+q L-l

+ c H(X,&& x,L=dd)

i=d+2

+f+,-dIU,L) L-d-l

= ;F1 ff( x+od+‘, XFd) + ff( XhlU~)~

5 (L - d) ff(X;lUd+i) which yields (28).

-P@2,b x du; x dub x Q)

from the assumption that (AIII.1) holds for i, and this equals

APPENDIX III P1(Q,b X EU; X EO, X ... XE,,+, X QO;+,) which proves (AIII.l).

We follow the proof of Proposition 7, making only technical modifications. Pl being given, we can write

Pl( EUt X E”, x ... xEUL) PI

We define P2 on the sets EUt X E”, X . . . XEUL by the same formula 141

P,(Eu; x 4, x ... xE,sL) r 151

putting [71

P2( E,,,lu;,v~-;‘) = P,(E,,Iu,+~-~).

If PI

P,(f+ x Eufml x Eul X *a* XEu8 X a,,)

P,(L+ x EUfel x E,, x ... xEui x Q,,), 191

= PO1

i=l... L, (AIII.l)

then Pl and P2 agree for all [a,,,~ x E”f-_, x En& x Q,,] E 1111

99, ,,x,;, i = 1 ... L, and therefore (46), (34), and (35) hold.

REFERENCES

R. G. Gallager, Information Theory and Reliable Communichtion. New York: Wiley, 1968. T. Berger, Rate-distortion Theory. Englewood Cliffs, NJ: Prentice-Hall, 1971. A. K. Gorbunov and M. S. Pinsker, “Nonanticipatory and prog- nostic epsilon entropies and message generation rates,” Probl. Inform. T?ansmission, vol. 9, pp. 184-191, 1973. -, “Prognostic epsilon entropy of a Gaussian message and a Gaussian source,” Probl. Inform. Transmission, vol. 10, pp. 93-109, 1974. G. A. Munson, “Causal information transmission with feedback,” Ph.D. dissertation, Cornell Univ., Ithaca, NY, 1981. I. M. Gelfand and A. M. Jaglom, Calculation df the Amount of Information. Berlin: Deutscher Verlag der Wissenschaften, 1958 (Am. Math. Sot. Trarisl., 1959). M. S. Pinsker, Information and Information Stability of Random Vuriables and Procksses. San Francisco, CA: Holden Day, 1960; Berlin: Deutscher Verlag der Wissenschaften, 1963. S. Engell, “Processing of measurements with respect to optimal transmission of information,” Ph.D. dissertation, Univ. Duisburg, Duisburg, FRG, 1981 (in German). -, “Transmission of information with finite delay,” Univ. Duis- burg, D&burg, W. Germany, Res. Rep. FB7, FGS, 1982. J. Snyders, “Error expressions for optimal iinear filtering of sta- tionary processes,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 574-582, 1972. K. Yao, “On the direct calculations of MMSE of linear realizable estimators by Toeplitz form method,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 95-97, 1971.