The Levinson recurrence and fast algorithms for solving Toeplitz systems of linear equations

IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-35, NO. 6, JUNE 1987 839

The Levinson Recurrence and Fast Algorithms for Solving Toeplitz Systems of Linear Equations

Abstract-This work brings together classical polynomial theory as it relates to the Levinson recurrence for a Hermitian Toeplitz operator and matrix theory as it relates to the class of Hermitian centro-Her- mitian matrices. A new computationally efficient alternative is presented to the Levinson recurrence on either the Hermitian or skew- Hermitian polynomial spaces. This approach also leads to an entirely new algorithm for solving systems of linear equations when the coefficient matrix is Hermitian Toeplitz or real symmetric Toeplitz. Analysis of the computational complexity of the algorithms presented is also performed, and it is shown that these algorithms lead to significant improvements in the computational complexity as compared to the previously best-known recursive algorithms. They also provide further in- sight into the mathematical properties of the structurally rich Toeplitz matrices.

B I. INTRODUCTION

ECAUSE of their wide applications in statistical’ signal processing and other related areas, the algebraic

structure associated with Toeplitz matrices has been studied extensively, and several papers [1]-[5] have been published on fast algorithms that can be used for solving a linear system when the coefficient matrix is a Toeplitz matrix. The fast algorithms are recursive in nature and are intimately related to the recurrence relations for the computation of polynomials orthogonal on the unit circle, termed Levinson (or Szego) polynomials (LP’s) [4]. The arithmetic complexity associated with these algorithms is O( 2 m’ ) operations for the complete solution, m being the number of unknowns in the system. We say that the arithmetic complexity c is O ( um2 ) if ( c/m2 ) -+ u for large m. More recently, fast algorithms have been reported that require o( am log’ m ) operations [6]-[8]. However, the methodology and techniques employed to obtain these fast procedures are quite diverse.

In a recent paper, Delsarte et al. [9] have shown that the conventional recursive algorithm for computing the predictor (Levinson) polynomials relative to a real positive definite Toeplitz matrix is redundant in complexity. The algorithm derived by Delsarte et al. is termed the “split Levinson algorithm” and is based on the correspondence between the set of real polynomials orthogonal

Manuscript received April 16, 1986; revised January 8, 1987. This research was supported in part by a Senate Fellowship from Syracuse Uni- versity and in part by the Canada Natural Sciences and Engineering Re- search Council under NSERC Grants A0912 and A6824.

H. Krishna is with the Department of Electrical and Computer Engi- neering, Syracuse University, Syracuse, NY 13244-1240.

S . D. Morgera is with the Department of Electrical Engineering, McGill University, Montdal, P.Q., H3A 2A7 Canada.

IEEE Log Number 8613869.

on the unit circle and the set of symmetric and/or skew- symmetric real polynomials orthogonal on the real interval [ - 1, 1 ). Recently, in an altogether different but entirely relevant context, sequences of symmetric polynomials have been applied for zero location with respect to the unit circle [ 113.

In this work, we treat the complex case for a Hermitian Toeplitz operator, For a particular LP, the Hermitian Lev- inson polynomial (HLP) and skew-Hermitian Levinson polynomial (SHLP) are defined, and a computationally efficient alternative to the Levinson recurrence on either the Hermitian or skew-Hermitian polynomial spaces is presented. At any stage in the recursion, the conventional LP may be recovered from either the HLP’s or the SHLP’s. Up to this point, the results are interesting, but the vital question arises of whether either the HLP’s or the SHLP’s can be employed directly to solve the associated Toeplitz general system of linear equations. This is examined in detail, and new recursive algorithms are derived that require 0 ( 1 .25m2 ) complex multiplications (MULT), as compared to 0 ( 2 m2 ) complex MULT for the previously known Levinson algorithm. The number of complex additions (ADD) for the new algorithms is 0 ( 2.75m’ ). In this work, knowledge of the structure inherent in Toeplitz matrices and the underlying polynomical systems is further refined and exploited in order to develop systematic algorithms having the lowest known computational complexity. Within this classical framework, we do not be- lieve it is possible to further reduce the complexity of solving complex systems using recursive algorithms.

This paper is organized as follows. Section I1 sets up the notation for the sequel and develops the necessary mathematical background for the work. In Section 111, based on the LP’s, we define the HLP’s and SHLP’s, and present computationally efficient algorithms for their recursion. Based on the HLP’s or the SHLP’s, a recursive algorithm for solving the Hermitian Toeplitz general system of linear equations is presented in Section 1V. In Sec- tion IV, we also describe algorithms for solving a general system of linear equations based on the symmetric Lev- inson polynomials (SLP’s) and the skew-symmetric Lev- inson polynomials (SSLP’s), when the Toeplitz matrix is a real symmetric matrix. All the algorithms presented in Sections I11 and IV are new. Finally, in Section V, the computational complexity of the procedures described in Sections I11 and IV is analyzed. It is worthwhile to mention here that the problem considered in this work is con-

0096-3518/87/0600-0839$01 .OO @ 1987 IEEE

840 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-35, NO. 6, JUNE 1987

siderably more complicated than, and easily specializes to, the real case studied in [9].

11. MATHEMATICAL PRELIMINARIES The mathematical preliminaries fundamental to the

content of this work are presented in this section. The various notations are defined and certain well-known results on the LP recurrence and its connection to Hermitian Toeplitz matrices is described [12].

A . Notation Let $T denote the field of complex numbers and T be a

Hermitian Toeplitz matrix of order ( n + 1 ), specified in terms of its elements in the first column, i.e., ci E $T, i = 0, 1, * , n. Also, since Tis Hermitian, c - ~ = cT, i = 0, 1, * , n, where the elements in the first row are de- noted by c P i , i = 0, l , * * * , n. For a given matrix T, let Tk denote the principal submatrix of T of order ( k + 1 ). Note that T,, = T. A ( k + 1)-dimensional column vector xk = (xk,oxk, - * X k , k ) T can equivalently be expressed as a polynomial x k ( z ) = Ci=o xk, iz of degree k , and vice versa. The equivalence between the vector notation X k and polynomial notation xk ( z ) is used throughout this work in order to simplify the form of the expressions. Let J denote the exchange matrix having ones on the antidiagonal and zeros elsewhere. The order of the matrix J is understood to be conformable with the context in which it appears. For a given polynomial .xk ( z = ci = xk, iz ', let ik ( z ) denote the reciprocal polynomial, i k ( z ) = zkxx,* (z- ' ) = E!=, Xk,k-iz . In vector notation, we have .tk = JX;. A vector xk (polynomial xk (2) ) is called Hermitian if xk = . tk(xk(z) = & ( z ) ) , and skew-Hermitian if x k = -Jk(xk(z) = - i k ( z ) ) . The matrix Tbelongs to a larger class of matrices called Hermitian centro-Hermitian matrices and satisfies the properties l ) TH = Tand 2) JTJ = T*, where Tu = (T* ) T . A similar set of definitions and properties may also be stated for the real case. For ex- ample, a real vector xk (polynomial xk(z)) is called symmetric i fxk = . t k ( x k ( z ) = i k ( z ) ) , and skew-symmetric if x, = - fk(Xk(z) = - i k ( z ) ) , where ik = Jxk(ik(z) = zkxk(z - ' = C$=Oxk,k - i z i ). A real symmetric Toeplitz matrix belongs to the larger class of matrices called symmetric-centrosymmetric matrices and satisfies the properties 1) TT = T and 2) JTJ = T [ 101. It is worthwhile mentioning here that, unless stated otherwise, we deal with the complex case in this work.

B. The Levinson Recurrence

k i

k

* i

For the Hermitian Toeplitz operator T, the LP's satisfy the one-step recurrence relation

&(z) = & - 1 ( ~ ) + Z P ~ & - ~ ( Z ) , k = 1, 2, . , n

( 1 4 where

k - 1

Pk = - (Dk- l /Dk-2) - ' ( jz C k - i 4 k - 1,i) ( I b )

(DklDk-1) = ( D k - 1 / D k - 2 ) ( l - ( P k l 2 ) ( I c ) and D- = 1, &,(z) = 1. Here Dk is the determinant of the matrix Tk. The polynomials & ( z ) , k = 0 , 1 * - > 12, are orthogonal on the unit circle. If the Hermitian Toe- plitz matrix T is positive definite, the parameter P k E has modulus less than unity and is known as the Schur- Szego parameter or the partial autocorrelation/reflection coefficient 141. The coefficients of +k ( z ) are complex, except for that of zo, which is unity, as it is clear from (la) that & ( O ) = $ k - l ( 0 ) = - = & ( O ) = 1 , k = 1 ,

requires 0 ( n2 ) complex MULT and ADD. One of our objectives is to reduce this complexity by

exploiting the structure inherent in T. First, we recall that any polynomial p ( z ) E P,,, the space of complex polynomials in z of degree n or less, can be written as the sum of a Hermitian polynomial s(z) E P, and a skew-Hermi- tian polynomial a ( z ) E P,. In regard to ( z ) we have

2, * ' . , n. Computation of & ( z ) for k = n, therefore,

S k ( 4 = i[dk(Z) + & < z ) ] ,

ak(z) = ;[ $k(z) - & k ( z ) ] . (2)

The polynomial s k ( z ) and a k ( z ) are termed HLP and SHLP, respectively. We note that neither sk ( z ) nor ak ( z ) as defined in (2) lead to simple recurrences in sk( z ) or a k ( z ) , a point that makes the specific construction unde- sirable; however, they will serve for the point to be made here.

The LP & ( z ) satisfies the linear system of equations

TkCpk = (Dk/Dk-l) [ I 0 * ' * 0IT,

k = 1 , 2 , 3 n. ( 3 4

Therefore, the polynomials sk ( z ) and a k ( z ) satisfy, respectively,

T k S k $ (Dk /Dk- l ) [ I O ' * 11'3

k = 1 , 2 , * a * , n

and

TkUk = i(Dk/Dk-l ) [ 1 0 ' * 0 - l f ,

k = 1 , 2 , - 0 . 9 n. ( 3 4 The results of (3b) and (3c)'may be directly verified

using (2) and (3a). We note in this case that the right- hand sides (RHS's) of (3) are real, that of (3b) being symmetric and that of (3c) being skew-symmetric. In the next section, using a more appropriate order recursive context, we present a new computationally efficient algorithm for the Levinson recurrence when Tis Hermitian Toeplitz.

111. THE HERMITIAN LEVINSON RECURRENCE In this section, we develop the recurrence for the HLP

sk( z ) . The development of the recurrence for the SHLP

KRISHNA AND MORGERA: LEVINSON RECURRENCE AND FAST ALGORITHMS 84 1

ak ( Z ) is along identical lines and is summarized at the end of this section. One of the most general representations possible for sk ( z ) in terms of 6 k - 1 ( z ) is

sk(z) Xk-l($k-l(Z) + zx:- l ( f ;k-I(z) ’ (4)

where xk - is a complex scalar quantity. The necessity for introducing two real degrees of freedom via a complex scalar such as xk- in (4) will become obvious in the sequel. Clearly, sk ( z ) = & ( z ) , and using (la) and (4), we readily verify that

sk(z) = ( Y k & ( Z ) + ( Y : & ( Z ) , k = 1, 2, * * ’

( 5 4 where

a k = ( x k - 1 - x:-lP:)/(l - I P k ( ) . (5b) 2

This expression relates the polynomials sk( z ) and &( z ) of the same degree. It is easy to verify that the use of (5 ) as in (3b), leads to ci!k(Dk/Dk-l) and ( Y Z ( & / & - l ) in the first and last positions of the RHS vector.

To develop a recurrence for sk ( z ) , we proceed by iterating (5a) backward one step, solving the resulting expression and (4) for +k - ( z ) and ( f ;k - ( z ) and substituting these iterates along with & ( z ) , expressed in terms of S k + 1 ( z ) and sk(z ) into (la). The result is

sk+ l ( z ) - zv:sk(z) - k z ) - zv:-Isk-l(z) xk - Z X : P k - [ ; * ( , - I - Z X : - I P k - l ) 1

( 6 4

where

P k : = (Yk/aZ, vk : = x k / ( Y k . (6b)

We now choose the complex scalars as x k = (Yk , k = 1, 2, - - ; thus, (6) reduces to

From (5b), the above choice for xk also forces the rela- tionship

x k = ( x k - 1 -XZ-lP:)/(l - l P k l ) - ( 8 ) 2

Solving (7) for sk + ( z ) and simplifying using (8), we obtain a two-step recurrence for sk( z ) :

sk+l (z ) = (pk + p:z) sk(z) - qkZsk-l(Z) (sa) where

P k := x k / x k - 1

and 2 2

qk := ( . I C k I x k - 1 ( ( l - ( P k \ ) * (9b) We note that whereas the coefficient Pk associated with sk ( z ) is complex, the coefficient qk associated with sk - ( z ) is real (and positive, if the matrix Tk is positive definite). It is interesting to ,observe that the recurrence (9) does not require the introduction of an “auxiliary polynomial” such as & - ( z ) in the recurrence (1) for $k ( z ). The fundamental reason for this lies in the fact that the polynomials { sk ( z ) 1 are not orthogonal on the unit circle, but are more classical in the sense that degree-balanced versions of them are orthogonal on the interval [ - 1, + 1 ) of the real line as shown in the Appendix. The reader is also referred to [ 12, sec. 11.51 and [ 13, sec. 9.111; the correspondence there between real-line orthogonal polynomials and the polynomials orthogonal on the unit circle is in the same spirit as that employed here. It is interesting to note, however, that the condition in [12], [13] that the construction involve polynomials of formal degree 2 k may be relaxed. Further, the results of the Ap- pendix are new and much more general, not requiring the usual assumption that the spectral density function be even on [O, 2 ~ ) , i.e., that the coefficients of & ( Z ) be real.

It is possible to further simplify the form of (9) by de- fining the polynomials sk ( z ) = l & ( Z ) , k = l, 2, * - * , where the ik are real scalars. The recurrence (9) now be- comes

s ; + l ( z ) = ( l k / l k + l ) ( P k + P t z ) s i ( z )

- ( l k - l / l k + l ) qkzSi-l(Z). (lo)

We select l k - l k , and l k + in such a way as to force the real coefficient of si- ( z ) in (10) to equal unity, viz.,

1 - 1 k + l - - ( lk- lqk) - 1 = l i - ! I ( (xk -1 /xk (2 ) ( l - ( P k ( 2 ) -1 -

(11)

Substituting (1 1) into (lo), we obtain

s i + l ( z ) = ( l k / l k - l ) ( l - ) P k I 2 ) - l

(pZ- l + p i l z ) s ; ( z ) - z s ; - l ( z ) . (12)

Now define the complex scalar Tk via

Tksk = [Tk 0 ’ . * 0 T Z ] T

( 1 3 4

and recall the relation ( D k - l / D k - 2 ) ( 1 - = (Dk / & - ) in addition to the fact that

Tk = ci!k(Dk/Dk- 1 )

= ( x k - l - x Z - l P : ) (Dk lDk- l ) ( l - ( P k ( ) *

2 - 1

(13b)

Substitution of (9b) and (13b) into (12) combined with some algebraic manipulation results in the recurrence


where

since s k ( z ) = l k S h ( z ) implies from (13a) that r k = l k r i .

The expression (14) is our key recurrence for the HLP’s; it has been put in the form shown because of the fact that the required coefficients ri- and rh may easily be computed from inner-product relations. After computation of (14), we may obtain the unscaled polynomials as s k ( z ) = Zksi ( z ) , where l k may be computed using (1 1). Before presenting the complete algorithm for the key recurrence (14) and the auxiliary scalar recurrences, we now discuss the vital issue of recovery of the LP’s dk(z) from

To recover & ( z ) from sk ( z ) , the proper relation may be obtained by iterating (4) one step forward and subtract- ing from an appropriately scaled version of (5 ) . The result is

s k ( z ) .

+ k ( z ) = [ s k + l ( z ) - z S k ( z ) ] / ( l - z>7 (15)

which is, in fact, the left-hand side (LHS) of (7). The above particularly simple form for $ k ( z ) obtains primar- ily because of the introduction of the complex scalar x k - in (4) and the fortuitous choice xk = c y k , k = 1, 2 , * , n. Recovery of &( z ) is seen to require s k ( z ) at two steps in the recurrence. From (15), we note that s k + (1) =

tion. Another interesting relation for the choice x k = a k ,

k = 1,2, e - * , is obtained by combining (8) and (13b), 1.e. ,

S k ( l ) , k = 0, 1, * + , a most interesting side observa-

A. The Hermitian Levinson Algorithm

Before outlining the new algorithm, some preliminaries are in order regarding the auxiliary scalar recurrences. Al- though we may desire recovery of C$k ( z ) using (15) only at termination of the recurrence (14), it is highly desirable in many applications to compute the partial autocorrela- tiodreflection coefficients P k at each step of the recursion. We first express P k in terms of ri and rL- 1; from (13b) and (14b), we have

The recursion for xk may also be put in terms of ri and rh- , although precomputation of (17) permits the computation of xk directly as in (8). In similar fashion, given x k , xk - 1, and P k , l k + may be directly computed as in (1 1). The new algorithm for the Hermitian Levinson recurrence up to precise degree n may be outlined as follows.

The Algorithm Step I : (Initialization)

so(z) = 1 = [ s h ( z ) ] , s1(z) = 0 . q 1 + z ) = [ 4 < 4 3 .ro = ~ 0 / 2 , X O = 0.5, lo = 11 = 1

Fork = 1, 2, * * 7 n7 DO Step 2: (Coefficient Computation)

k

T i = c i = O

rk = ‘ T L - l / T L

Step 3: (Polynomial Update)

S h + l ( Z ) = (r: + rkz) sL(z) - ZSL- l (Z)

STORE S; + 1 ( z), ( Z )

Step 4: (Auxiliary Recursions)

P k = (“) [ 1 - y - (partial autocorrelation / x k - 1 ri ‘ k - “ I 1 reflection coefficient)

x k = ( x k - 1 - XZ-1 P : ) 6 k

1-1 - 2 k + l - l i ! l ( l x k - l / x k ( ) 6 k

If k < y1 GO TO Step 2; OTHERWISE GO TO Step 5 (OPTIONAL) OR EXIT.

Recovery of q5n ( z ), if required, can be accomplished as a step terminal to the above algorithm, defined in the following manner.

Step 5: (Levinson Recovery)

+ n ( z > = ( 1 / x n ) [ l n + l s A + l ( Z ) - Z L ~ A ( Z ) ] / ( ~ - z ) .

Due to the Hermitian symmetry of s k ( z ) , computation of s k ( z 1, including Levinson recovery, requires o ( n2 / 2 complex MULT and O ( 1 .5n2 ) complex ADD, as opposed to the most efficient methods requiring 0 ( n2 ) complex operations of both types. Note that the Hermitian symmetry of s k ( z ) can be directly employed to reduce the computational complexity of both Steps 2 and 3. The analysis of the complexity aspect of the algorithm is presented in Section V. In the following, we establish that if the computation of reflection coefficients P k , k = 1, 2, - , is not required, then the auxiliary recursions are also not required for the computation of q5n ( z ) . Such is the case when the objective is to solve a general system of linear equations. Since s k + l ( l ) = s k ( 1 ) = - * * = s o ( l ) , we have

4+1sA+*(1) = InsA(1) = 1 or

I n f l = l/~;+~(l) and I, = l/s;(l). (18)

Consequently, In + and In may be computed using the

KRISHNA AND MORGERA: LEVINSOW RECURRENCE AND FAST ALGORITHMS 843

coefficients of the polynomials SA + ( z ) and sA(z) in n real ADD and 2 real MULT. Also using (15), we can write

+n(z) = ( l n + l / x n ) + L ( z ) where

Recall that the coefficient of the smallest power z o in cbn ( z ) is 1. Therefore, once +A ( z ) is computed using (19), we can compute x , in one complex MULT, thereby de- termining & ( z ) completely. With minor changes to the above procedure, a Hermitian polynomial algorithm can be devised for the related Szego recursion. The reader is referred to [ 141 for a detailed analysis.

B. The Skew-Hermitian Levinson Algorithm In the following, we summarize the definitions and the

various expressions for the skew-Hermitian Levinson algorithm. The analysis is very similar to that of the Her- mitian Levinson algorithm. In this case, we have the following relations:

ak(Z) = X k - l + k - l ( Z ) - zX:-l$k-l'(Z)

= a k $ k ( Z ) - ( ; Y t $ k ( Z ) = -&(z ) (20a)

where

ak := ( x k - 1 + x:-lp;)/(l - [ P k l ) . (20b) 2

We observe that degree-balanced versions of the SHLP's { ak ( z ) are also orthogonal on the interval [ - 1 , + 1 ). Once again, we choose x k = ayk, k = 1 , 2, - , and obtain a two-step recurrence relation

where

The LP's & ( z ) , k = 1, 2, - - , may be recovered by

+ k ( Z ) = ( 1 / x k ) [ak+l(Z) - Zak(z) ] / ( l - z ) . (22) The auxiliary scalar recursions are found to be

6k = ( l - ( P k l ) 2 - 1

xk = ( X i - 1 f X t - l p : ) 6 k 1 - 1 - 2

k + l - 1;:l ( ( X k - l I X k l ) 6k (23) Also, we have the relation ak + (1) = ak( 1 ) = . - = a. ( 1 ), which may be employed to determine 4,, ( z ) without performing the auxiliary recursions. Furthermore, (1 6)

is satisfied for this case. The initialization is given by

ao(z) = j = [aA(z>] ,

a l ( z ) = 0.5( 1 + j ) - 0.5(1 - j ) z = [ u ; ( z ) ]

T~ = ( 1 + j ) c0/2, x0 = 0.5(1 + j ) , lo = 11 = 1. Finally, we end this section by stating that the compu-

tational complexity of the Hermitian and skew-Hermitian recurrences is approximately the same.

IV. ALGORITHMS FOR SOLVING GENERAL SYSTEMS OF LINEAR EQUATIONS

In this section, we study the problem of solving an ( n + 1)st-order system of linear equations Tx = b where b = (bobl * - bn ) , x = (xox l * x , ) T and T is a Her- mitian Toeplitz matfix. The algorithms derived in this section are based on the recurrences for either the HLP's or the SHLP's described in the previous section. We also describe algorithms when T is a real symmetric matrix. These algorithms are based on the recurrences for either the SLP's or the SSLP's [SI. A. Algorithm Based on HLP's

The methodology adopted to obtain the new algorithm is similar to the already-known recursive algorithm [5] in the sense that it consists in recursively computing the solution of the linear system of order k + l based on the solution of the linear system of order k , k = 1, 2, * * , n. However, there are two major differences that lead to computational savings over the previous algorithm: first, it computes the solution at each stage using the HLP's as compared to the classical LP's; the second difference will become clear in the sequel. The objective is to solve the system Tx = b. At the kth step, we are given the solution

T

to T k - l X k - 1 = bk-1 where X k - 1 - ( x k - l , o X k - l , l ' ' X k - l , k - l ) T and bk-1 (bLk-"b 1 ' *

-

bk- )T. Note that, in general, the first element of the vector b k - l , i.e., bLk-') # bo. This is the second major difference between the present technique and the previous technique. At the kth step, we compute the solution vector

and bk = (bLk)bl - bk )T. The kth step begins with ex- tending the system Tk- 1 xk - = , b k - I as

to the System T k X k = bk, where X k = ( X k , o X k , 1 * ' X k , k )T

844 lEEE TRANSACTIONS ON ACOUSTIC X , SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-35, NO. 6, JUNE 1987

Since the HLP’s s i ( z ) satisfies the linear system Tks; = [ T; 0 * * 0 T L * I T , we form the next iteration as

where tk is a scalar. Premultiplying both sides of (25) by the submatrix Tk, we obtain

The scalar tk is chosen so as to ensure that the last element of the RHS vector is equal to bk. This leads to the following recursive procedure:

k

(27) At the end of the above recursive process, we have solved for the system T,x, = b, wherex, = ( X , , ~ X , , ~ - - x,,, )T

and b, = (bt’b, * * b,)T. In general, bp) # bo. How- ever, we may express b as the sum b = b, + b‘ , where b’ is a vector all of whose elements are zero, except for the first element, which is equal to bo - bg). Thus, we see that the solution vector corresponding to b is the sum of two solution vectors, one corresponding to b, ( = x, ) and the second corresponding to the vector b’ . Recall that the LP 6, ( z ) satisfies the system of linear equations Tq, = ( D , / D , - ) [ 1 0 . - 0IT. Therefore, the solution vector corresponding to b‘ is given by (bo - bc’ ) (D, - /D, ) q,. Consequently, the overall solution polynomial can be written as

(28)

The initial values for the recursions given in (27) are x. ( z ) = bo/co and bh0’ = bo. It is interesting to observe here that the above-described algorithm for solving the system of linear equations does not require the auxiliary recursions of Step 4 in the algorithm for computing the HLP’s. This is due to the fact that the scalar (D , /D , - I ) and the LP 6, ( z ) can be computed using (16) and (19), respectively, without involving the auxiliary recursions.

B. Algorithm Based on SHLP’s A similar algorithm may be derived that employs the

SHLP’s, a k ( z ) , to recursively compute the solution vector. Such a procedure can be summarized as follows:

and finally

The initial values for the procedure are x. ( z ) = bo /co and bi0) = bo. Once again, the auxiliary recursions are not required for solving the system.

C. The Real Case In this section, we study the case when Tis a real sym-

metric Toeplitz matrix. All the matrices, vectors, and polynomials, therefore, are real. For the case when the matrix is real symmetric Toeplitz, Delsarte et al. have obtained a similar recursion for the Levinson algorithm and refer to it as the split-Levinson algorithm [9]. In [9], two-step recurrence relations are obtained for the SLP defined as s k ( ~ ) = & - I ( z ) + z ~ & - ~ ( z ) and the SSLP u k ( z ) = + k - , ( z ) - z&] ( z ) . The polynomials s k ( z ) and a k ( z ) are termed singular predictor polynomials and satisfy the system of linear equations T k p = [ Tk 0 * * 0 rklT and Tkak = [ rk 0 0 - T ~ ] , respectively.

A comparison of the recurrences for the real case and the more general complex case is in order here. We observe that if all the quantities involved in (4) were forced to be real, then (4) would become

Here, xk - serves only as a scalar multiplier. Setting xk - I

= 1, k = 1, 2, * * . , n f 1 , leads to the recurrences derived in [9]. This also explains the reason why the analysis for the real symmetric case cannot simply be extended to the Hermitian case. Also, for the SSLP a k ( z ) defined in (20), ak( 1 ) = 0 where 8 is pure imaginary. Therefore, z = 1 is not a root of the SHLP’s a k ( z ) (complex case) as opposed to the SSLP’s u k ( z ) (real case). In both’ real and complex cases, the computational savings over the previously best-known procedures are approximately the same [ 181.

When the coefficient matrix Tis a real symmetric Toe- plitz matrix, the algorithm for solving the system of linear equations may be based on either the SLP’s sk(z) or the SSLP’s a k ( z ) , k = 1, 2, * - - , n. These algorithms may be derived using arguments analogous to those of the Her- mitian case. Based on the recurrences for the polynomials

KRISHNA AND MORGERA: LEVINSON RECURRENCE AND FAST ALGORITHMS 845

s k ( z ) and ak( z ) given in [9], we present a summary of the algorithms in the following.

1) Algorithm Based on SLP: k

v k = C i X k - 1 , k - i i = 1

t k = ( b k - vk ) / 7 k

X k b ) = x k - l ( z ) + t k s k ( z )

( 3 0 ) The initial values for the procedure are xo ( z ) = bo / c o and bho) = b 0.

2) Algorithm Based on SSLP:

The initial values for the procedure are x0 ( z ) = bo /co and bho) = b 0.

V. COMPUTATIONAL COMPLEXITY CONSIDERATIONS

In this section, we analyze the computational complexity of the algorithms for computing the HLP's and SHLP's, the algorithms for solving a general system of linear equations based on these polynomials, and the SLP- and SSLP-based algorithms for the real case.

Using the property that s i ( z ) is Hermitian, we can write the computation for s i ( z ) as (refer to the algorithm in Sec- tion 111-A) i:' i = O (c,*-,s;,T + ci"s;,i) for k = 21 - 1

r; =

I c ( Ck?iSi,T + + CIYpSi,l i = O

L for k = 21

i = 1 , 2 , - . , 1,

k = 1 , 2 , 9 n. (32)

In the above expressions, the computational savings stem from efficient computation of terms of the form ef + gf * where e , f, g E p. The computation of ef + gf * may be performed in 6 real ADD and 4 real MULT. In comparison, the computation of a term of the form ef + gh where e , f , g , h E e, requires 6 real ADD and 8 real MULT or 12 real ADD and 6 real MULT (a term of the form ef + gh would appear in the standard Levinson algorithm). Let us define a complex MULT as being equivalent to 4 real MULT and 2 real ADD and a complex ADD as being equivalent to 2 real ADD. The computational complexity of the expressions in (32)., therefore, is (0.5n2 + 2.5n) MULT and ( 1.5n2 + 0.511) ADD (it is clear from the context that all the operations are complex, unless stated otherwise). Since l,,/l,,+, = s A + l ( l ) / s A ( l ) , it can be computed in ( n + 2 ) real ADD and 1 real MULT. Fi- nally, (19) may be written as

4;,0 = SA + 1,o

+A,i = 4 A , i - 1 S ; + l , i - ( Z n / l n + ~ ) s A , i - l ,

i = 1 , 2 , , n , (34 )

which determines +,, ( z ) to within a scalar multiple of ( ln+ l / x , , ) in 0.5n MULT and 2n ADD. Note that (1, + / x , ) = 1 /+A,o . The overall complexity of the HLP computation is (0.5n2 + 3n) MULT and ( 1.5n2 -I- 3.52) ADD. Similar expressions may be written for the SHLP's. The number of operations is the same as the HLP case.

Noting that s;(z) (a i ( z ) ) i s Hermitian (skew-Hermi- tian), the polynomial x,, ( z ) may be computed using (27) [ (29)] in ( 0 . 7 5 1 ~ ~ + 3.25n) MULT and (1.25n2 + 3n) ADD. Finally, the solution vector x is obtained from x, in ri MULT and n ADD. The computational complexity of the complete algorithm for solving the system of linear equations using the HLP's (SHLP's), s k ( z ) ( a , ( z ) ) , is ( 1.25n2 + 7;25n) MULT and (2.75n2 + 7.5n) ADD. In terms of m, the order of the system, the complexity is ( 1.25m2 + 4.75m) MULT and (2.75m2 + 2m) ADD. The computational complexity of the previously best- known recursive algorithm is O( 2m2 ) [ 5 ] . The algorithms presented in this paper therefore represent sign@- cunt improvements in the multiplicative complexity, while the overall additive complexity (in terms of the total number of real additions) remains approximately the same. A similar analysis may also be performed for the real case, and it can be easily shown that the computational complexity of the SLP-based algorithm given in (30) for solving the general system is ( 1.25m2 + 3.75m) real MULT and (2m2 + 4.5m) real ADD, while the complexity of the SSLP-based algorithm given in (31) is ( 1.25m2 + 5.75m) real MULT and (2m2 + 5.5m) real ADD. These algorithms ,are also computationally superior to the SLP or


SSLP-based algorithms described in [15], having complexity of O( 1 .25m2 ) real MULT and O( 3 .25m2 ) real ADD. (Note that only the real case is studied in [15].) A preliminary investigation also suggests that all of the algorithms possess numerical stability properties similar to those of the standard Levinson technique [ 161, [17]. However, details of such an analysis are beyond the scope of this paper and will be the subject of another paper to appear shortly.

In recent years, several algorithms have been reported that require U ( om log’ m) arithmetic operations for solving a system of linear equations [6]-[8]. However, we feel that recursive techniques, such as those presented here, will continue to play a dominant role in many statistical signal processing applications because of their ap- pealing and classically based formulation and well-understood features. Also, it is worthwhile to mention here that even though asymptotically 0 ( om log2 m) complexity algorithms are computationally superior to the 0 ( l .25m2 ) complexity algorithms presented here, the crossover point between the performance of the two algorithms lies at a reasonably high value of m (m - 175).

VI. CONCLUSIONS Our goal has been to bring together the richness of re-

sults available in the theories of orthogonal polynomials and matrices as they relate to the Levinson recurrence and the underlying Hermitian centro-Hermitian structure of the Toeplitz operator. A new recurrence is presented for polynomials possessing Hermitian (or skew-Hermitian) coefficient symmetries. It is shown that the conventional Lev- inson polynomial may be easily recovered from the new recurrence at the nth step. As compared to the most efficient known methods, the new algorithms are twice as economical in terms of the number of multiplications, while the overall number of additions remains the same. Using the Hermitian (skew-Hermitian) Levinson polynomials, a new procedure for finding the solution to a Toe- plitz general system of linear equations that is also computationally efficient is presented. A new computationally efficient procedure for finding the solution to a Toeplitz system of linear equations for the real symmetric case is also presented. Our hope now is that this work may represent a small contribution to classical theory, as well as a computationally efficient tool for the many and varied present-day applications borne of that theory.

APPENDIX ORTHOGONALITY RELATIONS ASSOCIATED WITH

HERMITIAN LEVINSON POLYNOMIALS In this Appendix, we demonstrate the manner in which

the HLP’s { s k ( z ) 1 for z = eJe 6 E [0, 2 a ) may be associated with two polynomial families, each orthogonal on the real interval x E [ - 1, 1 ) with respect to the dis- tributions of the Stieltjes type derived from df ( e ) , the distribution for the LP’s { r$k (z) 1. The results are estab- lished by “balancing” the polynomial s k ( z ) , followed by

the application of the conformal mappings x = $ (z ’ I2 + z-’ / ’ )andy= 1 / 2 j ( z ’ / ’ - ~ - ~ / ’ ) f o r z = e j ~ , t h e r e b y establishing a one-to-one correspondence between the unit circle I z J = 1 andx = cos 8 / 2 , y = sin 6 /2 . We have yet to discover results in the literature as general as those presented here.

We define the (degree-) balanced polynomial uk ( z ) obtained from sk ( z ) as

u&) := z -k /2sk (Z) . (AI )

Clearly, since s k ( z ) is Hermitian, i.e., s k ( z ) = &(z), the polynomial uk ( z ) has pure real coefficients for z = e j s

and, with certain caveats, the LHS of (AI) may be written as uk (x), a polynomial of degree k in the indeterminate x = cos 6 /2. In the sequel, we assume 1 z I = 1.

From (Al) and (9) it is possible to obtain the following explicit representation for uk (x) in terms of a trigonometric sum, viz.,

k

u k w = 2 ;c &,?k-l j /2 cos ( W ) l = O

k

+ 2 z SL,(k- / j /2 sin ( W 2 ) l = O

:= u&) + y W k - & ) ( A2a )

where

and the notation i indicates an index I ranging over even (odd) integers only, depending on whether k is even (odd). Association of the first trigonometric sum of (A2a) with uk (x), a polynomial of precise degree k in x = cos 8 / 2, follows by induction on the identity

COS [ ( z + 1 ) 8/21 = 2 x COS (ze/2)

- cos [ ( I - 1) 8/21.

In similar fashion, using

sin [(I + 1) 8/21 = 2y cos (16/2) + sin [(I - 1) 8/21,

we can associate the second trigonometric sum of (A2a) with a product of y = sin 6 / 2 and a polynomial of precise degree k - 1 in x = cos 8/2. It is clear that uk( -x) = ( - l ) k ~ k ( x ) and wk(x) = ( - l ) k ~ k ( x ) , consistent with the notation i?.

With respect to the measure do (e), the LP’s satisfy

- 5 dk(e’s> ~ : ( e - j s > ~ 6 ) = ( ~ ~ k - 1 ) 8krn 1 2?r

2 a 0

where the elements ck = cZk of Tk are the Fourier Stieltjes coefficients, i.e. ,

KRISHNA AND MORGERA: LEVINSON RECURRENCE AND FAST ALGORITHMS 847

The spectral distribution function a( e ) is assumed to be a bounded nondecreasing function absolutely convergent on [ 0, 2n) . The derivative do( 0 ) /dB exists (almost everywhere) on [0, 2n ) and equals the seectral density function or trigonometric weightf( e ) . In general, we can decomposef(8) as f ( 0 ) = fete) + & ( e ) , from which the corresponding spectral distribution functions can be defined via du,(O)/dB = fete) , do,(e)/dts = f o ( e ) . From (A4), using ck = cTk, the symmetry properties of a, ( e ) and a, ( e ) follow easily, viz.,

.,(e> + ue(2a - e ) = 4 2 a )

uo(e) - @,(2n - e> = 0. (A5)

Associated with the above will be the real-line measure @(x) and the respective weights $,(X) and $,(x) for x = COS 8/2. Noting that da,(B) = -&,(x) and doo(8) = -d$, ( x ) leads to the following expressions for the real-line weight functions:

$,(x) = oe(2a) - .,(e) $,(x) = -%(e ) . (A61

It is now easy to verify that the real-line weights enjoy symmetry properties equivalent to those of (A5), i.e.,

$e (x) + +e( - x ) = 2ce,(2n)

$,(x) - $ o b 4 = 0. (A7) Further, we have d$,(x) /dx = [ d a , ( O ) / d e ] wl(x) and d$,(x)/dx = [da,(B)/dO] wl(x), where the weight wl(x) = (1 - x 2 )-1’2. Now, returning to (A3), it is straightforward, with some algebra, to show that uk (cos 0 / 2 ) is orthogonal with respect t o da ( e ) to an arbitrary balanced Hermitian polynomial. Specifically choosing the polynomial as urn (cos 8 /2 ) , we obtain the equivalent real- line relation

1 s U d X ) &(x) = 2 Re { X k - l T k } 6 h - 1

(A8)

Rather general considerations, stemming from (A3), in addition to the symmetry properties (A7), lead to the orthogonality relations imbedded in (A8), viz.,

1 j -1 %(X) %(X> d$&) 1

u k ( X ) urn- 1 (x) ”%(x) &’,(x) = 2sk,O7k ah R R -1

( A9a 1 and

1 1 V k - l ( x ) Urn-l(x) Wi(x> drC/e(x) - 1

1

+ J %dx) V k - l ( X ) @ 2 ( 4 d$o(x) = -2sk,OTkskm I I -I

( A9b )

where w z ( x ) = w;’(x), S k , o = s k ( 0 ) = x? + xk& = x$- using (5) and Re { T~ ) : = T ~ , R Im { T ~ ) : = T;, with

as defined in the algorithm of Section 111-A. Summation of (A9a) and (A9b) in accord with (A2a) provides the result of (A8).

ACKNOWLEDGMENT

The authors would like to thank the referees for bring- ing references [15] and [17] to their attention and C. Dausman for the fine work in typing this manuscript.

REFERENCES

N. Levinson, “The Wiener RMS (root-mean-square) error criterion in filter design and prediction,” J . Math. Phys., vol. 25, pp. 261- 278, 1947. W. F. Trench, “An algorithm for the inversion of finite Toeplitz matrices,” J . Sac. Ind. Appl. Math., vol. 12, no. 3, pp. 515-523, 1964. S. Zohar, “The solution of a Toeplitz set of linear equations,” J . ACM, vol. 21, no. 2, pp. 272-276, Apr. 1974. T. Kailath, A. Vieira, and M. Morf, “Inverses of Toeplitz operators, innovations, and orthogonal polynomials,” SIAM Rev. , vol. 20, no. 1, pp. 106-119, Jan. 1978. G. Carayannis, N. Kalouptoidis, and D. G. Manolakis, “Fast recursive algorithms for a class of linear equations,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, no. 2, pp. 227-239, Apr. 1982. F. G. Gustavson and D. Y. Y. Yun, “Fast algorithms for rational hermite approximation and solution of Toeplitz systems,” IEEE Trans. Circuits Syst., vol. CAS-26, no. 9, pp. 750-755, Sept. 1979. R. Kumar, “A fast algorithm for solving a Toeplitz system of equations,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP- 33, no. 1, pp. 254-267, Feb. 1985. G. Ammar and W. Gragg, “Asymptotically superfast solution of positive definite Toeplitz systems,” paper presented at the SIAM Con$ Linear Algebra in Signals, Syst. Contr., Boston, MA, Aug. 12-14, 1986. P. Delsarte and Y. Genin, “The split Levinson algorithm,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, no. 3, pp. 470-478, June 1986. S. D. Morgera, “On the reducibility of finite Toeplitz matrices-Ap- plications in speech analysis and pattern recognition,” Signal Pro- cessing, vol. 15, nos. 5, 6 , pp. 425-443, Oct. 1982. Y. Bistritz, “Zero location with respect to the unit circle of discrete time linear system polynomials,” Proc. IEEE, vol. 27, pp. 1131- 1142, Sept. 1984. G. Szego, Orthogonal polynomials, 3rd ed. (Colloquium Publ. No. 23, American Math SOC.), Providence, RI, 1967. L. Ya. Geronimus, Orthogonal Polynomials. New York: Consul- tants Bureau, 1961. S. D. Morgera and H. Krishna, “The Szego recurrence and the role of Hermitian and skew-Hermitian polynomial spaces,” SIAMJ. Appl. Math., submitted for publication. A. Kok and D. Manolakis, “Efficient algorithms for linear phase structures with applications in signal processing,” in Proc. EUSIPCO Con$, The Hague, The Netherlands, Sept. 2-5, 1986. G. Cybenko, “The numerical stability of the Levinson-Durbin algorithm for Toeplitz systems of equations,” SIAM J . Sci. Stat. Com- put., vol. 1, no. 3, pp. 303-319, Sept. 1980. S. T. O’Donnell and I. S. F. Ispen, “Stability analysis for the split Levinson algorithm for SPD Toeplitz systems,” paper presented at the SIAM Con$ Linear Algebra in Signals, Syst. Contr., Boston, MA,

H. Krishna and S. D. Morgera, “A fast algorithm for Toeplitz system of linear equations,” paper presented at the SIAM Conf. Linear Al- gebra in Signals, Syst. Contr., Boston, MA, Aug. 12-14, 1986.

Aug. 12-14, 1986.


Hari Krishna (S184-M’86) was born in Roorkee, India, on September 30, 1960. He received the B.Tech. degree in electrical engineering (first class with distinction) from the Indian Institute of Technology, Delhi, India, in 1981 and the M.Eng. and Ph.D. degrees in electrical engineering from Concordia University, Montrkal, P.Q., Canada, in 1982 and 1985, respectively.

Since 1985 he has been an Assistant Professor in the Department of Electrical and Computer En- gineering, Syracuse University, Syracuse, NY.

His research interests are in the areas of digital communications, digital signal and image processing, coding theory, and fast algorithms for computational complexity problems arising in these and other related areas.

Dr. Krishna was a recipient of the F. A. Gerard Prize, awarded to the most deserving graduate of the M.Eng. programs at Concordia University.

Salvatore D. Morgera (S’70-M’72-SM’84) was born in Providence, RI, on August 5 , 1946. He received the B.Sc. degree in physics (honors) and the M.Sc. and Ph.D. degrees in electrical engineering from Brown University, Providence, in 1968, 1970, and 1975, respectively.

From 1968 to 1978 he was a Senior Research Scientist with the Submarine Signal Division, Raytheon Company, Portsmouth, RI, and Princi- pal Investigator and Technical Director for a number of research and development programs in the

areas of detection and estimation, acoustic communications, adaptive spa- tial-temporal processing, and distributed processing architectures. During this period he also served as a consultant to the National Security Industrial Agency (NSIA) on command, control, and communications. From 1978 to 1985 he was a Professor in the Department of Electrical Engineering, Con- cordia University, Montrial, P.Q., Canada; the administrator responsible for the FCAR research group dealing with communications, circuits, and systems; and a Fellow at the Science College. He is presently an Adjunct Professor at Concordia University and a Professor in the Department of Electrical Engineering at McGill University, Montrkal, and Head of the Communications Systems Group. He is a scientific consultant to a number of North American companies. His recent interests include information theory, applied stochastic processes, pattern recognition, digital signal and image processing, and algorithms and computational complexity.

Dr. Morgera is an IEEE Information Theory Group representative to the Society of Ocean Engineering and served as a Special Session Chairman and Technical Program Committee Member, 1983 IEEE ISIT. He was Co- chairman of the session on Statistical Pattern Recognition, Pattern Rec- ognition in Practice I1 in 1985, and Invited Session Organizer for the special session on Computer-Aided Decision Making, COMPINT’85. Most recently, he has served as Chairman of a session on Detection and Esti- mation, EUSIPC0’86; Chairman and Organizer of the session on Speech and Image Processing, IEEE MONTECH’86; and Technical Program Committee member, IEEE ICC’86.

The Levinson recurrence and fast algorithms for solving Toeplitz systems of linear equations

Documents

Transcript of The Levinson recurrence and fast algorithms for solving Toeplitz systems of linear equations