Terminal attractors in neural networks

ORIGINAL CONTRIBUTION

Terminal Attractors in Neural Networks

1. INTRODUCTION

Over the past decade, there has been an increasing intercat in artificial neural networks. defined as adap

tivc dq’namical systems model4 on the general t’c;t- ture\ of biologicit networks. that can carry out useful

intormation procchsing by means of their state rc-

sponsc to initial or continuous input. One motivation

fat- this interest is found in the seminal works of

Grossberg (1968). Cohen and Grossberg (19%). C‘ar-

pcntcr and Grossberg (1987) and Hirsch (I’#?. I’M) in the areas of global stability, adaptive pattern rec-

ognition. content-addressable memories (CAM). and cooperative-competitive decision making bq

neural networks. Hopfield (19X2. 19X-l) has stressed

the applicability of neural networks for content-ad-

dressable memories. In this direction. sOme new re-

sults were published by Lapcdes and Farber (l%(7).

who have approached the problem of programming

fixed points into a neural net by formulating the con-

ditions that the synaptic connection matrix must sat-

ictj if the desired fixed points are to be present.

l’hl\ r~\carch ua carriccl OLII at the Jet Propukion Lalwrator!.

(‘alilornia Imtitutc 01‘ Twhnolop~. under Contract No. NAS7- VS. ‘l’hc worh wt\ supported by the sponsorship ot the Innw;ltl\c

Scicnw and Technology ol’fice 01 the Strategic Dcfcnw Initiative

C)rganir;ition.

The author wiahcs to ackno\+lcdge his very truitl’ul cliscuwon

Lvith Dr. Jacob Bnrhen

Requots lor reprints \houlrl he sent to Michall Zah. Jet Pro-

pukion Laboratory. Calilornia Institute ol’Technology. Pasaden;~.

<‘A 01 125.

Keeler ( 19X(,) has introduced ;I ncu algorithm for

ani~l~zing basins of attractions in neural network>

used ;I\ CAM. Suez. Protopopsccu. and Barhen

( 19Xx) have dtxloped sufficient conditions for the

asymptotic stability in the neighborhood of each member of ;III arbitrary set of state vectors that arc

\pecificallv designed to equilibrium point\ of the

neural network.

Notwithstanding these succcssc~. it is rapidly be-

coming evident that curt-at models arc characterized

by ;I number of limitation\: existcncc of spurious

equilibria. infinite time required (in principle) to

rexh an attractor. and low storage capacity. Still

unavailable is ;I solution to the problem: find such ;I

synaptic connection matrix which pro\,ides an!’ de-

sired set of attractors with prc\cribcd basins.

The main purpose of this work i\ to remove some of these limitations and. thercbq. to improve the dy-

namical characteristics of the neural network models

as well as to develop an effective methodology for

their design. The approach is based upon ;I new type

of static attractors-terminal attractors for 2 contcnt-

addressable memoy. associati\,c memory. and pat-

tern recognition in neural networks operating in con-

tinuous time. The idea of storing memory states 01

the patterns to be recognized as static attractors of the neural network dynamic\ implies that initial con-

figurations of neurons in some neighborhood of a memory state will be attracted to it. So fur. the kit-

tractors considered in nonlinear dyamics, ;IS well as

in neurodynamics. have represented regular solutions. that is, solutions which never intersect the tran-

30

Gents. This is why the theoretical relaxation time ot the system to such attractors is infinite. The idea of ;I terminal attractor is based upon :I violation of the Lipschitz condition at ;I fixed point. As ;I result. the fixed point becomes ;I singular solution which en- velopes the family of regular solutions, while each regular solution approaches such a terminal attractor in finite time (Figures 1. 7). In addition. terminal

attractors are characterized by an “infinite” (local)

stability. Both of these properties appear to IX of ma,jor significance for neural network dynamics.

It is worth mentioning that the terminal attractor as a mathematical concept has a very clear physical interpretation. It was first introduced and analyzed

in mechanical systems (Zak. 1970. 1981. 1982a. 1982b, 1983a, 1983b). As shown is these works, terminal attractors are associated with energy-cumulation effects (snap of a whip or jump of the shear strain energy at the soil surface as a response to underground excitations). Recently M. Zak (1988) introduced the concept of terminal attractors for con-

tent-addressable memory in neurodynamics. The purpose of this paper is to show that ax ;I

result of an appropriate incorporation of terminal attractors into neural net models. one can t’ind ;I

synaptic connection weight matrix such that i+il the desired equilibrium points within the prescribed basins will be stable, and no false static attractors will appear. This paper also shows that utilization of tcr- minal attractors in neural net models enhances their dynamical properties: for instance. by appropriate combinations of such attractors one can crcatc at-

tracting trajectories which will be “rcmembercd” by the dynamical model. The concluding sections will illustrate applications of terminal attractors in neural nets for content-addressable and associative mcmo- ries. pattern recognition. self-organization. illld dy-

namical training.

2. TERMINAL ATTRACTORS

This section introduces a new type of attractor in dynamical systems-terminal attractors.

u

t u = 0 - REGULA R AlTRACTOR

u3

0

Ul

11, t2, t3 - - u2

FIGURE 1. Convergence to regular attractor.

FIGURE 2. Convergence to terminal attractor.

Static Termi& Attractors aad RepeHers

It should be recalled that the equilibrium points p of ;I dynamical system

jl = f,(.X,. X2 . X,,), i =z I, z, . . n (11

are defined as its constant solutions

.\ 7 XI’ = Const. i = ! I Ft. (2)

If the real parts of the eigenvalues of the matrix

hf = d f 1, I G(P)

are iIll negative. that is,

Re R, c 0

then these points are asymptotically stable. Such points are called static attractors:since each sc&tion that gets close enough to p (i.e., enters a so-+ed basin of attraction) approaches the corresponding constant value as a limit as t -+ x. AR equifibritim point represents a repeller if all the eigenvalues of the matrix (3) have positive real parts. Nonlinear

Terminul Attract0r.s in Neural Networks 261

dynamical systems can have any number of attractors and repellers. whereas repellers separate basins of the neighboring attractors. Such a property is utilized in neural networks for CAM, pattern recognition, etc.

LJsuallc.. nonlinear dynamics and neural networks deal only- with such systems which satisfy the Lip- schitL condition

This condition guarantees the existence of a unique solution for each of initial conditions. This is whv a transient solution cannot intersect the corresponding constant solution to which it tends and,

therefore. the theoretical time of approaching the

attractors is always infinite. Such attractors we will call regular (Figure 1).

NON, WC will introduce a terminal attractor which is approached by transients in finite time. The cx- istence of terminal attractors is illustrated by the fol- lovving example

_i : --s’ :, ($1

This equation has an equilibrium point at s = 0 at which the Lipschitz condition (4) is violated

di 1 -= --x dx 3

: ’ - --7; at x + 0. (6)

Since here the condition (3) is satisfied,

Rei--+ -m<O, (7)

this point is an attractor of “infinite” stability. The relaxation time for a solution with the initial

condition x = X, < 0 to this attractor is finite

.r 1-1, dx 3 f,, = - - =I - Xi’ < s

j/j s” 2

Consequently, this attractor becomes terminal. It represents a singular solution which is intersected by all the attracted transients (Figure 2).

For the equation

.i: XZ 1’3 (9)

the equilibrium point x = 0 becomes a terminal repeller

di ---_t.rm”+xat*-O, dx

i.e., Re i - x > 0. (10)

If the initial condition is infinitely close to this repeller, the transient solution will escape the repeller during a finite time period

while for a regular repeller, the time would be infinite.

Instead of eqs (5) and (c)), one can consider a more general case

jl = FS”, k > 0

for which the relaxation time (for the attractor) or the escaping time (for the repeller) is

r + % ifk? I

As shown in the theory of differential equations, singular solutions in equations

F(x. JJ. 1”) = I) (12)

are found by eliminating _v’ from the system

b-(x, y, y’) = 0 $ = 0. (13)

Hence, static terminal attractors (if they exist in eqn (12)) must be among the solutions to the system

(lj). Let us introduce a polynomial

P(s) = (.u - .Y,)(.V - _V,) (,\ - _I>,,)

J,f = n (.Y - _\-; ) . .Y, . . \-

4

and consider a differential equation

.i- = [ P(.I )I’ ‘.

This equation has 2n equilibrium points

.Y = _Y~, k = I. 2 211.

It is easily verifiable that

s = s.~ ,(k = I. 2. II)

are terminal attractors. and

.Y = .t-Jr(k = I. 2. 12)

are terminal repellers (Figure 3).

(1.4

(15)

(16)

(17)

(18)

o- TERMINAL Al-TRACTOR

0 - TERMINAL REPELLER

FIGURE 3. Array of terminal attractors and repellers,

XL?

Indeed, in small neighborhoods of (17) and (18). the polynomial (14) can be linearized and represented xi

in which

and eqn (15) is reduced to eqn (5) and eqn (9) for the neighborhoods (17) and (18). respectively.

Physical Interpretation of Terminal Attractors

We will now demonstrate that the terminal attractor as a mathematical concept has a very clear physical interpretation: it describes energy-cumulation effects. As an example, consider a propagation of an isolated pulse in an elastic continuum along the x- axis. In general, the speed of propagation f = L depends on X. Suppose there exists such a point x’” where %(x*) = 0. Then the time t* during which the leading front of the propagating pulse will approach this point is expressed via the folIowing integral

[:s = I

!~a, d.*-

1 , i(s)’

If i can be presented in the form

i = (X* - x)~, 0 < k < 1,

then this integral converges and, therefore, the time t” is finite. It is easily verifiable that in this case the differential equation

i = (XC - x)”

describing the dynamics of the pulse propagation has a terminal attractor at x = x*. But if the leading and the trailing fronts of the propagating pulse approach the same point x* during finite time, then eventually the width of the pulse will shrink to zero, and all the energy transported by the pulse will be distributed over a vanishingly smaI1 length. Hence, the existence of a terminal attractor in such models leads to an unbounded concentration of energy in the neighborhood of the attractor.

Based upon this model, Zak (1970,1983a, 1983b), explained and described the formation of a super- sonic snap at a free end of a filament suspended in a gravity and a centrifugal force field, as well as the cumulation of the shear strain energy at the soil surface as a response to an underground explosion. In these models, a free end of the filament and a free surface of the soil serve as terminal attractors.

!M. Zuk

Periodic Tenaincrl L&t Sets

So far, this paper has concentrated on static terminal attractors. It will now demonstrate tfie existence of periodic terminal attractors. For that purpose, let us consider a dynamical system separable in polar coordinates r. 0

Here, diidr - --r; at r + R (cf. eqn (bj) and. therefore, the solutions r = R, II := tot + II,, i> a

terminal limit cycle. Its basin is defined by the condition: r > 0. For the solution with the initial condition r, < 0 the relaxation time is finite

-K dr p s dr t, = .I ,, r(R - r)“,

< ._r,, $? - rj’ ’ I

= $- (R - r,,)? ’ < x. ,I

It is easily verifiable that a periodic terminal repeller can be obtained by changing the sign in the right-hand side of eqn (19).

3. NEW PPPPKTS IN NON-tIPSCHTTZIAIfi DYNAlV@ZS

lnstead of continuing the analogy between the classical and “non-Lipschitzian” dynamics, the next section will introduce some new effects resulting from the failure of the Lipschitz condition at the limit sets.

Static TermiN At&actors With Terminal ‘Jhjectories

As shown in nonlinear dynamics, different types of regular attractors (or repellers) can be introduced based upon the second order dynamical system linearized with respect to the origin x = 0, y = 0

(21a)

or

dx ax + by -=p dy cx + d.y

(21b)

Depending upon the eigenvalues of the matrix

the attractors (or repellers) x = 0, y = 0 can be a node, a star, a spiral point, or an improper node- (Percival & Richards, 1982).

If instead of eqn (21) one introduces the fohowing system

i = (ax + by)”

j, = (cx + dy)‘*’ (23aj

Terminul Attructors in Nrurul Networks

then the equilibrium point x = 0, y = 0 represents a terminal attractor (or repeller): the Lipschitz condition is violated at this point. Nevertheless, the differential equation of trajectories in configuration space x. .v

(23b)

satisfies the Lipschitz condition and it does not have an> singular solutions. This means that both variahlcs .V and \I are “simultaneously” approaching the terminal attractor. as in the case of a regular attractor (SW eqn (2 I b)). Moreover, a similar classification ot terminal attractors of the type (231) can bc performed. based upon the coefficients N. h. I’, and tl.

This section introduces a more “pathological” situation. when for the differential equations of trajectories in configuration space. the Lipschitz condition is also violated. AL will be shown below, such a violation uill lend to the loss of the uniqueness of the solutions in the configuration space: the trajectories will merge before approaching the terminal attractor.

Let u\ start with the following dynamical system

\ - _ (,- ~ .\’ )I : (2-1)

j’ - [VI,\- - _r’)]‘:. (25)

It is easily verifiable that the Lipschitz condition here is violated at x = x’“, y = 0.

The differential equation of trajectories in configuration space X. y can be written as

For this equation, the Lipschitz condition is violated at y = 0. This means that y = 0 is a singular solution. and all the trajectories in configuration space X, y first tlow to the x-axis, that is, y = 0, and then approach the terminal attractor x = x”, _V = 0. together (Figure 4).

Indeed. as follows from eqn (26)

FIGURE 4. Terminal trajectory y = o.

263

while x = x(t) follows from eqn (24)

Here x,, and Y,~ are the initial conditions. The time of approaching the singular solution

y = 0 by the variable y follows from eqn (27). if x(r) is substituted from eqn (28)

3 t, = j (I,, - .\‘.’ )? i. (29)

The time t, of convergence of the solution to the terminal attractor follows from Eqn (28)

Obviously.

t: < I,. (31)

This means that the trajectory of the motion of the original dynamical system (24). (25) in the configuration space X, y first flows into the trajectory v = 0, and only then does it approach the terminal attractor x = x’“, y = 0. Such a trajectory as y = 0 we will call a terminal trajectory.

The situation described above can be generalized to the case where a terminal trajectory is a prescribed curve. Indeed. turning again to the system (24). (25) let us transfer to a new system of coordinates

s = !;I,. !‘ = .f($. &) (321

assuming that f is a differentiable, and ?~f/a,Y, # 0. Then eqns (24) and (25) read

,;j, = - (3, - 2;) a (33)

The terminal trajectory y = 0 is converted into a curve

f(S,. ‘3:) = 0. (35)

Hence. for a desired terminal trajectory (39, the corresponding dynamical system is (33. 34) (Figure

5). Applying the same idea to an n-dimensional dy-

namical system, one can “store” not only a terminal trajectory, but a set of hyperplanes of trajectories which ends by a terminal attractor.

Unpredictable Dynamical Systems

The concept of unpredictability in deterministic classical dynamics was introduced in connection with the

M. Zlik 26-l

92’

* 01

FIGURE 5. Terminal trajectory y = f(x).

discovery of chaotic motions in nonlinear systems. Such motions are caused by the Lyapunov instabihty, which is characterized by a violation of a continuous dependence of solutions on the initial conditions during an unbounded time interval (t-+ x). That is why the unpredictability in these systems develops grad- ually. Indeed, if two initially close trajectories di- verge exponentially

i: = c,, exp it, 0 < X < x

then for an infinitesimal initial distance c,, -+ o, the current distance I becomes finite only at t -+ 2. For this reason, the Lyapunov exponents (the mean exponential rate of divergence) are defined in an unbounded time interval

t---+ %.

In distributed dynamical systems, described by partial differential equations, there~exists a stronger instability discovered by Hadamard. In the course of this instability, a continuous dependence of a solution on the initial conditions is violated during an arbitrarily small time period. Such a blow-up instability is caused by a failure of hyperbohcity and tran- sition to ellipticity (Zak, 1982a, 1982b).

This section will show that a similar type of a blow- up instability leading to “discrete pulses” of unpredictability can occur in dynamical systems which contain terminal repellers (Zak, 1989a).

Let us analyze the transient escape from the terminal repeller in the equation

k = X”3, X,, = x(0)

assuming that lx,\ + 0. The solution to eqn (5) reduces to the following

x = rct3” x # 0 (37)

Hence, two different solutions are possible for “almost the same” initial conditions. The most es- sential property of this result is that the divergence

of the solutions (11) is characterized by an unbounded Lyapunov exponent

in which t,, is an arbitrarily small (but finite) positive quantity. In contrast to eqn (36), here the tyapunov exponent can be defined in an arbitrarily small time interval, since during this interval the initial infinitesimal distance between the solutions becomes finite. Thus, a terminal repeller represents a vanishingly short but infinitely powerful “pulse of unpredictability” which is “pumped” into the dynamical system.

In order to illustrate the unpredictability in such non-Lipschitzian dynamics, we will turn to the following equation

X - yx’, 1 = I). while (391

_v = cos mt. (40)

Assuming that x --, 0 at t -+ 0. one obtains regular solutions

and a singular solution (an equihbrium point)

X = 0. (42-1

During the first time period

0 <r( t < $[;; t43>

the equihbrium point (42) is a terminal repeller (since y > 0). Therefore, within this period, the-solutions (41) have the same property as the solutions (37): their divergence is characterized ~by an unbounded Lyapunov exponent.

During the next time period

the equilibrium point (42) becomes a terminal attractor (since y < 0). and the system which approaches this attractor at t = mc) remains motionless until t > 3~12~. After that, the terminal attractor converts into the terminal repeller, and the system escapes again, etc.

It is important to notice that each time the system escapes the terminal repeller, the solution Splits into two symmetric branches, so that the total trajectory can be combined from 2” pieces, where n isthenum- ber of cycles, that is, it is the integer part of the quantity (t/2no). As one can see, here the nature of the unprec#ictability is significantly differemf~om the unpredictability in chaotic systems. This differ-

Terminal Attractors in Neural Networks ‘65 _ _

ence will be emphasized even more by ample: Let us replace eqns (39) and following

the next ex- (40) by the

neural networks would be their ability to be activated not by external inputs. but rather by internal rhythms (see eqn (30) or eqn (45)).

.i- - y(sin x)’ ’ = 0. y = - 1 + 2e 1 cos (pt. (44)

assuming again that x * 0 at t + 0. Since y > 0 at t = 0, the equilibrium point x = (I initially is a terminal repeller. Hence, the regular solution will consist of two possible (positive and negative) escaping branches which will approach the neighboring terminal attractors at .Y = X, or x = -z. respectively. The system will be at rest in one of these two attractors until y becomes negative, i.e.. until these terminal attractors become terminal repellers. After that, the solution will split again into two possible escaping branches. while the system can continue to escape the equilibrium point x = 0 or return to it. However, because y --j - 1 at t - m. all the equilibrium points

x = +2nk. k = 0. 1, 2 . etc., (35)

will eventually become permanent terminal attractors. and the system will relax at one of them. But because of branching of the solution, it is impossible to predict which one of the competing attractors (J-5) will he finally approached by the system. Hence, here the unpredictability is represented not by a chaotic attractor. but rather by a set of competing static attractors.

Thus. a new type of unpredictability is introduced in dynamical systems caused by failure of the Lip- schitz condition at equilibrium points. Unlike the chaotic systems, the non-Lipschitizian dynamics may exhibit an unpredictability characterized by unbounded Lyapunov exponents. The sources of these unbounded exponents are terminal repellers which “pump the unpredictability” in the form of vanishingly short. but infinitely powerful “pulses.” That is why a set of possible trajectories in phase space is not a Cantor set (as in a chaotic system), but rather a countable set of a combinatorial nature. In this respect. the Non-Lipschitizian dynamics has some connections with the “digital world.”

It is important to emphasize that in the chaotic systems the unpredictability is caused by a supersen- sitivity to the initial conditions, while the uniqueness of the solution for fixed initial conditions is guaranteed by the Lipschitz condition. In contrast to that, in non-Lipschitzian dynamics presented here, the unpredictability is caused by the failure of the uniqueness of the solution at some of the equilibrium points.

Thus, the non-Lipschitzian dynamics introduces systems with a multiple-choice response to an initial deterministic input. Such models can become an un-

derlying idealized framework for dynamical systems with “creativity.” whose response is based upon a .‘hidden logic.” The most significant property of such

4. TERMINAL ATTRACTORS FOR CONTENT-ADDRESSABLE MEMORY

IN NEURAL NETS

The idea of storing memot-y states as static attractors of the neural network dynamics implies that initial configurations of neurons in some neighborhood of a memory state will be attracted to it. This section will show that incorporation of terminal attractors in neural nets allows one to solve the following problem: Given IZ vectors to be stored. Find such ;I neural network which has II (and onl! II ) \tablc equilibrium points representing these stored vectors.

We will start with the simplest case of a single neuron in which a terminal attractor can be incorporated as follows

ii + u = TV - (u - P’)z . II .y 0. (46)

where u(r) is the mean soma potential of the neuron. T is a constant, V(u) is a sigmoid function (for instance, V = tanh u), U(‘) is a desired static terminal attractor if T is selected as follows

T : [jl’~(~([j”~,I ‘, (17)

It is easily verifiable that the last term in the right- hand side of eqn (46) does not affect the location of the fixed point (see Figure 6). but it significantly changes the degree of its stability. Indeed. linearizing eqn (46) with respect to a point I/ > o one obtains

in which

FIGURE 6. Location and stability of terminal attractor.

2h6 M. Zak

neighborhoods of the corresponding equilibrium point read

in which

(58)

-1+ T

N,, = L cash’ tij”’

- *jl’) f E 2 3 - i.

2;,‘9;” e :v, (59) i

Now the stability of the equilibrium points u)“’ depends on the eigenvalues A,, of the matrices al:), respectively. In order to find these eigenvalues, let us multiply the matrices a!,‘), by ~““(a ---$ o)

u *Cki -_ c? “u’“‘, II ‘l (60)

Then, as follows from eqns (58) and (59), respectively

a:‘k’(i # j) ----+ 0, uf’“’ ---+ -f a’;” at c ---+ 0, ~(61)

that is, the matrix udTCk) is a diagonal at E -+ 0, and its eigenvalues are

2.” - 1 1. - --a

3 1”‘; i, k = 1. 5 . n. (62)

Hence,

i, = I: 1 -?‘?A* = -- +2/3 _, - 5 at c ----+ 0. k 3 (63)

Thus, all the eigenvalues of the matrices ur’ are negative and unbounded at the equilibrium points hjk’, which means that ah these desired equilibrium points are terminal attractors. Their basins can be controlled by appropriate selection of the coefficients r$” and $’ (see eqn (54).

Obviously, the system (54) has additional equilibrium points (repellers) which do not satisfyeqn (52). In order to demonstrate this, let us consider eqn (54) in a small neighborhood of its fixed points Af” and Ai*’ which were converted into terminal attractors from an attractor and a repeller, respec$ively (Figure

7)

in which i(‘) < 0 and i(Z) > 0. In the neighborhood of the ,point A)“, this equa-

tion has two additional fixed points (repellers) R’ and R” which eoordinates ~:(%re within-the region

Obviously.

i, + -x if ~1 - lit”. (SO)

Thus, incorporation of a terminal attractor into an equilibrium point of eqn (46) makes this point “infinitely” stable and supresses all possible insta- bilities coming from other terms.

Let us turn now to a n-neuron network

ii, + I(, = C T,, V,, i = 1. 2 II (Slj

in which T,, are constants and V, = V(q) are linearly independent and singlevalued vectors.

Its equilibrium points are defined by the following algebraic equations

Jl

lip = 2: T,,V(lij”‘) (52)

However, so far, the stability of the equilibrium points tilk) is not yet guaranteed. In order to make all the desired equilibrium points stable, we will incorporate a terminal attractor into each fixed point by modifying eqn (51) as follows

(54)

in which

i~~i”‘(~~” - a”‘)1 + 1, G:” # u”’ if k # 1 (55)

while yjk’ and of”’ are positive constants. The exponential multipliers are introduced into

eqn (54) in order to localize effects of terminal attractors (see the conditions (55)). It is easily verifiable that all the equilibrium points of the original eqn (51) are among the equilibrium points of eqn (54). Indeed, substituting u = fiik) into eqn (54) one arrives at eqn (52). But now all these equilibrium points are “infinitely” stable. In order to prove that, let us linearize the system (54) with respect to points ii,*lk) which are sufficiently close to the equilibrium points @I, respectively, so that

I? (k) _ al”‘/ = & - 0. (56)

The linearized versions of the system (54) in the

Terminul Attructors in Neural Networks 267

b

Gi

FIGURE 7. Conversion of repellers into terminal attractors.

while in the neighborhood of Aj” there are no additional equilibrium points.

Thus, all the vectors ~i)~‘(i, k = 1, 2. , PZ) are stored as terminal attractors. But, as follows from eqn (53), these tz vectors uniquely define the weight matrix T,, since the vectors V, = V(l4,) are linearly independent and singlevalued. Hence, ci,” are the only vectors which are stored as static attractors. All the additional equilibrium points, as has been demonstrated above, are repellers. It means that this model does not have false static attractors. However, in general. if the weight matrix is not symmetric (T,, f T,!). the existence of periodic or chaotic attractors cannot be excluded, and only in a particular case of a symmetric weight matrix (T(, = T,,) such a model will not have any false attractors, since periodic or chaotic attractors do not exist in gradient systems.

Thus by incorporating terminal attractors in a neural network. one can store n desired vectors as stable equilibrium points with no false static attractors. It is worth mentioning that, in general. the corresponding weight matrix T,, is not symmetric.

One of the advantages of the terminal attractor approach is in the fact that the weight matrix T,, corresponding to the desired fixed points is defined by eqn (52) and, therefore, can be found by con- ventional methods (Lapedes & Farber, 1986). while the stability of these fixed points is provided by the incorporation of terminal attractors in each equilibrium point. The last procedure can be represented by additional (variable) diagonal terms Tz in the weight matrix r,,

It is worth emphasizing that the dynamical system (54) obtained as a result of this approach is characterized by a failure of the Lipschitz conditions at the desired equilibrium points. This leads to a loss of the uniqueness of the solution at these points: a family of regular solutions (representing transient motions of the system) is intersected by a singular solution (representing the equilibrium of the system); and as

a result of that, the theoretical time of approaching these equilibrium points becomes finite.

5. TERMINAL ATTRACTORS FOR PATTERN RECOGNITION

Pattern recognition is based upon a selection of a category (from N given categories) into which a certain pattern must be stored. As shown by Hopfield (1982). this problem can be attacked with a neural network in which patterns arc stored in stable equilibrium points, and the decision surface coincides with the basins’ boundaries. Hopfield also gave a partial solution to the problem in the case of a symmetric weight matrix. However. one of the most se- rious limitations of his approach is a very small degree of control over the basin\ of attraction. In this section, we will demonstrate the usefulness of terminal attractors in Hopfield’s approach to pattern recognition by solving the following problem: Given II linearly independent vectors tij”. find such a neural network which has n/2 stable equilibrium point with prescribed basins.

In order to solve this problem. we will modify the

neural net dynamical model given by eqn (51) as

follows

1,

(1, + 11, = 2 T,,V, + [P(u,)]’ ‘. i r 1. 2. II (64)

in which P(u)) is a polynomial (see eqn (14))

If the weight matrix T,, is found from eqn (53), then all the vectors ~1”’ represent equilibrium points of eqn (60), since

P(tij”‘) = 0. (66)

Now we will show that

II, = til?’ ‘I. I’ = 1. ? . rli?

are terminal attractors and

(67)

II, = li:“‘. I’ = 1. L . rr/2 (68)

are terminal repellers, if II is an even number. First of all. we will recall that similar results were

already obtained for a simplified version of eqn (64)

li, = (P(l$,]’ 3

see eqns (15)-(1X).

(69)

The proof of the same result for the original version of eqn (64) is based upon the fact that in a sufficiently small neighborhood of an equilibrium point a terminal attractor dominates over “regular”

terms, and since eqn (64) can be obtained from eqn (69) by addition of “regular” terms, the stability (or instability) of the equilibrium point will persist.

268 M. Zak

Indeed, turning to the linearized version of the system (64) in the neighborhoods of the corresponding equilibrium points which can be written in the form (57), one finds that

-1 + T 1

a,, = // _ _ ajiil. ? 1 cash? [iii’ 3 ’ (71)

Here E is given by eqn (56)) and aik) is found from the linearization of the polynomial P(u, - tilk’) in the neighborhood of u:“’

P(uJ = @-)(Gjk’ - u,), that is

Following the same procedure as performed in the previous section (see eqns (60-62)), one obtains the expression for the eigenvalues of the matrices ui:’ (cf. eqn (63))

1 &tE:.? 1 -zifk = 211 - & 1 = _ _*

3 ’ m if k = 2v (73)

which proves that all the equilibrium points (67) are “infinitely” stable, that is, they are terminal attractors, and all the equilibrium points (68) are “infinitely” unstable, that is, they are terminal repellers.

The basins of attractions can be characterized by the distances between the neighboring repellers; for instance, the basin of the attractor 1?~2’-1) can be defined by its “projections” hf”‘-”

V’ I, = rij”’ _ $‘I X, im

Hence, if the desired attractors bij”-‘) as well as their basins bj2’-” are prescribed, then the corresponding repellers can be found from eqn (74). This allows one to constract the polynomial (65), which when incorporated into the original neural net model (see eqn (64)) will solve the problem.

So far it has been assumed that the number of neurons n is an even number. If n is an odd number, then all the equilibrium points tic”-” become repellers and LP’) become attractors.

6. MODELS WITH HiERARCHY OF TERMiNAL AlTRACTORS

This section introduces a sketch of more complex neural nets which consist of a hierarchy of coupled terminal attractors. Such models can perform complex associative memory as well as some useful self- organization properties.

ConspIex Associative Memory

We will start with the model described by eqn (54), which has n terminal attractors tiik’, k = 1, 2 . . . n

if the weight matrix Ti, is defined by eqn (56). Let us introduce n “replicas” of this model such that all the replicas are functionally identical- to the original model, but each of which is driven by only one of the terminal attractors. The difference in dynamical evolution of the replicas is due only to the difference in their initial conditions, which are assigned to the basins of attraction of the corresponding vectors. Hence, instead of eqn (54) we will consider the following n x n system

+_ (‘ij”’ - ‘q’)’ 3. i, k -=’ 1. L II (75)

in which u!“’ are the variables of the kth rephca, while VI’“’ = V(uj”)).

So far, the coordinates of the terminal attractors Sik) which enter eqn (75) not only explicitly, but also implicitly (via the matrix Tij) have-been considered as prescribed parameters. Let us now introduce a secondary (a higher level) neurai net having the terminal attractor coordinates lip’ of the system (75) as new variables. Such a model will describe a dynamical interaction between the terminaf attractors a?’ if one presents it in the form

i, k = 1, 2 II.

Here x is an activation coefficient defined as

It can be presented in the following anafyti@form

x = lim c .\:’ !‘7_ i(“l” li:“)‘, (78) **

Hence, due to the activation coefficient x, the secondary neural net (76) becomes active only after all the transients uik)( t) of the originaf neuralnet (75) approach the corresponding terminal attractors al”‘. (it should be recalled that the theoreti@ time for this is finite.) The weight coefficients Ty form a tensor of the fourth rank. Since the dimension of a vector tilk’ is n2, while the number of the weight coefficient is nJ, generally speaking. eqn (76) has n’ equilibrium points. For the sake of simplicity we will confine our analysis by the case

T:]’ = 0 if k f I, or i f j.

Then eqn (76) reduces to

[i:” + 0;” = TYV -:“‘x.

(79)

(80)

Equation (SO) has only one equilibrium point fif, while Ty is found from fan equation similar to

eqn (47)

Terminal Attractors in Neural Network3 269

In order to provide stability of this equilibrium

point, one should incorporate a terminal attractor

into eqn (X0) (cf. eqn (46))

$“1 + $“I = [ Tf)Pj”’ + (Ej”’ - $j”‘)]’ I;(. (82)

Thus. as soon as all the transients uii’(f) in the

original neural net approach the corresponding terminal attractors lij”‘. the secondary neural net (X2) becomes active and drives the original terminal attractors lij”’ In a new (prescribed by eqn (81)) position l?:“. In other words, the neural nets (75) and (X2) perform associative memory.

If one eliminates the restriction (79). then the original terminal attractors Uj”’ driven by eqn (82) with

full weight tensor r:,’ may occupy 17: different positions. depending upon their initial dislocations.

In the same way one can introduce the third, forth. etc.. neural nets and obtain a chain of associations

Active Terminal Attractors

The concept of an attractor (including a terminal

attractor) implies that the attractor passively “waits” for a transient solution which is supposed to ap-

proach it. In order to improve the convergence, we will introduce a concept of an active attractor which temporarily “leaves” its position in order to “search” for the corresponding transient solution; after

“catching” this solution. it returns to the original position. (Strictly speaking, such an active attractor is not a true static attractor any more.) In order to trigger a mobility of terminal attractors of the model (75). we will slightly modify eqn (76)

p Y 2 i T:+l;‘l( v;” _ y)] ‘1 /

+ (l! :,‘i’ - ti:“‘)’ :. (r(A)

in which 14:‘“” describe the original positions of the terminal attractors ii, “I of the system (76).

As follows from eqn (84). the activity of the secondary neural net is triggered by inputs to the original neural net

uj” # 0. (85)

In the course of the transient process, the coordinates of the original terminal attractors become functions of time

Cl”’ = Cl”‘(t) (86)

until all the attractors “catch” the corresponding inputs u,“‘, that is.

14:” = C:“. that is. V]” = ri{“. (87)

After this, the double sum in eqn (X4) becomes

zero. and the “wandering” attractors 6:“’ return to their original static positions 14;‘(~). The “safe” return

is guaranteed by the last term in the right-side of eqn (X4). which makes u:‘“‘) infinitely stable. Obviously, such a mobility of terminal attractors may change their actual basins.

From the viewpoint of control theory, the secondary neural net performs a parametric control of the original neural net. The usefulness of such a con-

trol is supposed to be provided by an appropriate selection of the weight tensor rj’; for instance. the strategy of this control can include a temporary sym-

metrization of the original weight matrix r,, by minimizing (in dynamics) its antisymmetric terms (see

eqn (53))

in which (Vi”) ’ are the elements of the matrix

Such a symmetrization may destroy possible periodic

or chaotic attractors, where the transient solution can be trapped and, thereby. control actual basins of attraction.

Certainly. this example is not a proposal for an actual control strategy, nor any claim for a vigor- ousness in it. The only purpose of the example is to illustrate the concept of an active terminal attrac-

tor.

7. TERMINAL ATTRACTORS IN NEURAL NETS WITH PROGRAMMED DELAYS

The previous sections of this paper dealt only with

such neural nets which are described by ordinary differential equations. The solutions to these systems

u,(t,,, UY, t) can be viewed as continuous transformations of the space of initial points u, E E” defined on t = t, to points in E” representing the state vari-

ables. In other words. the pattern to be stored or to be recognized is represented by an initial vector u;‘. while the time of its exposure is not perceived by the neural net. Hence, neural nets based upon ordinary differential equations cannot be utilized for such a pattern which are characterized not only by the vector of the state variables ~1;‘. but also by durations of the exposure sp of its components.

This section introduces neural nets described by delay-differential equations which can store as terminal attractors not only vectors, but also the durations of their exposures. For this purpose, we will consider a neural network with programmed delays

T, in the response of the outputs of changes in the Now we will prove that this equilibrium point is inputs a terminal attractor.

First of all. one has to notice that the concepts of stability or instability in delay-differential equation< are different from those in ordinary differential equations, since here one should analyrc the effect ot disturbances not at the initial point ot a trajectory. but rather of an initial function. Nevertheless. under certain reasonable assumptions. the classical Lya- punov theory of stability developed lor- ordinary differential equations after some modifications can aIs be applied to delay-differential equations. For instance. the $~bal stability can bc ;midyzed using ;t Lyapunov functional (instead of ;I Lyapunov function).

O,(f) t [c,(r) = i 7’,,V[U,(l - T,))

+ CY,[ti. - u,( I - f )]’ *. u, = 0. (00)

while the delays r, are variables of another dynamical subsystem

i, + T, = c A,,I/‘(T,) + (5, -. T,)" V1) I I

in which T,, and A ;, are constant weight matrices. The delay 5, will be selected later from the stability con- siderations. The dynamical subsystem (9 1) is not coupled with eqn (90) and, therefore, can be considered independently.

If A,, selected such that

then 5, = T( is a terminal attractor, which is approached by transient solutions during a theoretically finite period t’“.

One can conclude that for t < t”, the first group of equations, that is, (90). represents a system of nonlinear differential equations with deviating arguments, that is, a system of functional difference equations. However, for

1 ‘5 t ”

all the delays become constant

5, := 7 = (‘onst

(9.3)

and, consequently, eqn (90) reduces to the following

IrAt) + u,(t) = i T,,V(u,(t - ?,)I I,

+ a,(ri,(t) -

which is a system of nonlinear equations.

11,(1 - f,,]’ :. (94)

delay-differential

Unlike ordinary differential equations, for eqn (94) we require not a constant initial vector LL:‘, but rather a vector function defined over the lag periods

C+J, = u,*( -5:’ 5 t ‘= 0). W)

As shown above, all these lags ~7 and, therefure, the corresponding periods of exposures of the input patterns u; are stored at the terminal attractors i, of the subsystem (91).

Equation (94) has an equilibrium point u, = ic, if the weight matrix T,, is selected such that

ti, = i T,,V(r;,), since U,(t) /- 1

= ti,(t - i,) if C(t) = 0 . (96)

For local stability, one can apply a linearization procedure (with respect to an equilibrium point to he considered) which is followed by the Laplace transform. After that, the stability analysis is reduced to evaluation of the eigenvalues /., of the matnx (.V whose element\ will contain additional multipliers (’ coming from the terms with Jags f,. Following the same procedure which was performed for cqn (-14). one obtains instead of eqns (-IX) and (,49) for the coefficients of the matrix (3) corresponding to cqn (84) linearized with respect ta the equilibrium point c,) the foliowing formulas

i it j

in which E -+ 0 (see eqn (56)). This yields (cf. eqn

(62))

./, ‘Y, ,., Yz --“t> , ;‘, = ;;7 _.___, 7; I/ 081

while

I i, = X, f IL’,, i 2. \ -I. (Y9)

Separating eqn (98) into real and imaginary parts yields

X, + ;‘,e $‘I cos i,y, = 0, i,y, -- ;‘,v ‘jlfl sin ?!y, =. 0

whence

5,X, = - S, y, cotan f,y,,

ej’, mtari~ ,, sin ?,y, _ __ _I %.Y,

i, [ - 0. (ma)

As follows from the second equation in .(lOO)

f,Y, = nk - 8, 13~ ----+ 0, k = 1, 2 . . . etc (101)

and consequently

T,X, = -(irk - P)cotan(nk - ii’, --+ ~0 if ii’ ---+ U.

Ttvminal Attractors in Neural Networks 271

that is,

x,--3 --z if i < 0.. . ( 102)

Hence. all the real parts of the eigenvalues i., of the matrix (3) are negative, while their moduli are unbounded. This means that the equilibrium point II, = ri, of the system (Y4) is a terminal attractor. if 7, arc selected to be negative, for instance ?, =

IFI. Thus. the neural net described by eqns (YO) and

(c)l ) allows one to store and recognize such patterns bvhich arc characterized by a vector M” as well as by

durations 5;’ of the exposure of its components II;‘. Generally speaking. this model can be utilized for

cvcn more complex problems: to store ii vector func-

tion C,(r) defined for o 5 t 5 f,. The reason for such ;I possibilit! is due to the fact that delay-differcnti~tl

equations require initial conditions in the form of

function of time (see eqn (YS)). Hence, if this initial condition, that is. the functions I{;’ (t) are sufficiently

“close” to the corresponding stored functions [i,(r).

then they may bc attracted to these stored functions. Thcreforc. the weight matrix T,, must be selected

such that [i,(r) represent ;I periodic terminal attractor (the example of such an attractor is given by cqn (20)). Howc\,er. practical realization of this proce- durc i\ much more difficult than in the previous USC and. thcreforc. requires a special analysis. Wc M,ill not go into further details of the problem. confining ourselves to this brief discussion only.

8. DYNAMICAL TRAINING USING TERMINAL ATTRACTORS

The weight coefficients 7;, or Th’ considered in the

previous sections were defined by the corresponding analytical formulas (53) and (Xl). but we have not yet specified how they will be incorporated into

neural nets. Because of ;I very large number of neurons and their interconnections in future generations of artificial neural nets. it does not seem possible to program the weight coefficients according to the analytical formulas mentioned above. In order to avoid

such a programming we will develop a dynamical training procedure by introducing a dynamical equation for the weight matrix which governs its relaxa-

tion to the corresponding stable equilibrium point. This point will be represented by a terminal attractor which guarantees its local stability and fast convergence (Zak, lY8Yb).

Let us introduce a strength energy

-/I 1

as a measure of interconnections between neurons, requiring that for each prescribed performance of the neural network this energy as a function of T,,

must be minimum. Each prescribed performance im-

poses additional limitations on the weight coeffi-

cients T,,. In case of content-addressable memories. these limitations are expressed by the following conditions

,q,i == i 7;,v(q”‘) ~ [j’“’ = ,)

i = I. Il. x = I. . tt1. (104)

Utilizing the Lagrange multiplier technique. one

has to minimize a modified strength energy

S( ‘I‘... j.,) = S + 1 x /.,.g,, (IUS)

in which i,, arc constants to bc tound.

Since the energy 3 is ;I quadratic form of the vari-

ables T,, and i,i (see cqns (103) and (103)). its gradient TVs is a linear form of its arguments

(106)

(107)

It can be verified that the determinant of the linear

system

(108)

(II V”’ V’,“” V”’ IllI ’

vi,”

/Ii

2 A r det v(,,

, V’“‘, V’“” I, ., ,, I/‘,“’ “. ‘(. I,

(109)

is non-zero if the vectors Vl”’ are linearly independent. In this case, the system (108) defines a unique

solution which is the extremum of the energy s. The extremum is a minimum, since

;,2$

___ = (i - j)l > 0 if i = CY. i = 1,. i # j

a T,,;I T,,, () if i f a, or j f /j ( 1 10)

It is worth mentioning that the system (10X) gives ;I solution to the problem not only when vz = II. but also when ~7 < II. In the last case. the redundant weights are found from minimization of the strength

energy. Obviously. the problem does not have any solutions if m > II: in this case. the vectors I/‘“’ are linearly dependent and the non-hornogeneous’equa-

tions (10X) become incompatible. In order to use eqn (10X) for dynamical learning,

it will be assumed that aII the variables 7;,. and i,, are time-dependent. Then

.$ = r,$ ,i (111)

in which x denotes the vector with components T,,

and j.,n. and CS is the vector gradient of the strength energy 3 with the components (106) and (107).

272 M. Zak

The dynamical system

,f Z -&VS (112)

represents a gradient system which converges asymptotically to the minimum defined by eqns (106) and (107), while the strength energy s plays the role of the Lyapunov function, since

S = -cy2(VS)Z < 0 if VS # 0. (113)

So far, the multiplier CY’ was not specified. Now we will show that by an appropriate selection of this multiplier the convergence of the dynamical system (112) can be significantly improved. Indeed, let us seek cr’ in the form

cc = lVS/ . )’ > 0. (114)

Since VS is a linear function of its arguments, it

has the order

VS-xatVS-0 (115)

and consequently

VS-fatVS--+O. (116)

Hence, with reference to eqns (112) and (I 14)

jVSl - JVSI’+: at OS - 0,

that is,

djVS( - = -fl*(VSl’-~ at OS- 0

dt

in which ,!Y is a constant. Now we can evaluate the relaxation time

t= I

IP.$-O dlV,$l

IPSI,, (VS(‘+:

if y 2 0 =

<zifp<O’

Thus, for y 1 0, the relaxation time is infinite, while for y < 0 it is finite. The differential equation (117) suffers a qualitative change at y < 0: it loses the uniqueness of the solution, while the equilibrium point lV$l = 0 becomes a singular solution being intersected by all the transients.

It can be seen that at the equilibrium point IV’s/ = 0 the Lipschitz condition is violated

and, consequently, this point is a terminal attractor. In order to be consistent with the results presented in the previous sections, we will set

i ;‘= -- 3

providing in (118) the following relaxation time

r z ;’ !V,$iil, I <’ 7. ill9j

Now, the dynamical systemgoverning the learning process can be written in the following final form

L

!,i F,, = & (i - j)‘T,, + 2 &Pj”’ . i, ,f -= 1. 2 . ii

i I

I,,, = - a2 i

k- l.“...rn5l7 1-i

(120)

an d

As shown above, this system has a unique stable solution which is approached during a finite time, It defines the weight Coefficients~T,, by minimizing the strength energy 3 while satisfying the conditions (104) required for a prescribed content-addressable memory.

Instead of conditions (104), other types of limitations can be used, such as continuous mapper conditions

where Ijk) is a~constant vector representing an external input. It is easy to verify that the uniqueness and stability of the solution as well as finite relaxation time can be provided by using the system (120) after replacement of the conditions (104) by the conditions (122).

Thus, it has been demonstrated that the learning procedure based upon minimization ~of the strength energy leads to a dynamics in the weight space along a learning “trajectory” which converges to a terminal attractor. In conclusion, we will give an interesting physical interpretation of this approach, comparing it with the principle of the least constraints~f~rmu- lated by Gauss for mechanical systems wit-h _geo- metrical constraints. According to this principle, actual motion of such a system corresponds to the least constraints, that is, to the least &&xence from the free (unconstrained) motion. If by a free motion in neurodynamics one understands the motion of un- coupled neurons (T,, = 0 if i f j), then the weights TiJ can be associated with constraints imposed~upon the free motions, while the m&i~_~of the strength energy (103) realizes the least constraints. Now our approach can be formulated as foNows: for each pre-

Terminal Attractors in Neural Networks 273

scribed performance, a neural net will “select” such

interconnections T,, which impose the least con-

straints upon the free motions of its neurons if these

constraints are measured by the strength energy 3. In other words. neurons “try” to act with the least “disagreement” with their neighbors.

9. TERMINAL ATTRACTORS IN DISCRETE MODELS OF NEURAL NETS

In the previous sections. terminal attractors were as- sociatcd with continuous-time operating neural nets.

In this connection. it is interesting to learn what will happen if a continuous-time model with a terminal

attractor is converted into a discrete-time model. In this section. we will analyze how a discrete-time

model. whose underlying continuous-time model has a terminal attractor behaves in a small neighborhood of this attractor.

Let us consider a simple differential equation

ic = f(u) - IL’~% ,f(O) = 0 (123)

and assume that the derivative df/du exists and is bounded. Then one can easily verify the l/ = o is a

terminal attractor. Rewriting eqn (123) in the form

let us introduce the following discrete analog of this differential equation

= A(W ‘)U” ( 126)

The algorithm (126) converges if A is a contracting operator. that is.

IiAIl < 1 (127)

while the less this norm, the faster the convergence process.

We will now emphasize the role of the terminal attractor of the original equation (123) at u = 0: as one can easily verify:

[[4”‘] 2 ’ --+ cc if u(” - 0

and. therefore.

(128)

l/AJl - 0, if u”) - 0 (129)

Thus. in a sufficiently close neighborhood of the terminal attractor. the norm of the contracting operator becomes infinitesimal and, therefore. the convergence process becomes “infinitely” fast.

The same result can be obtained for a general neural net

with terminal attractors u, = li,. Hence, despite the fact that terminal attractors

were introduced for a continuous-time operating sys-

tem. they exhibit similar properties in the corresponding discrete-time analogs.

10. DISCUSSION AND CONCLUSIONS

This paper has introduced a new approach to nonlinear dynamics phenomenology for the purpose of

neural computations. The approach exploits a novel paradigm in nonlinear dynamics based upon the terminal attractor concept. Incorporation of terminal

attractors in neurodynamics requires some revision of the classical theory of dynamical systems because of a violation of the Lipschitz condition at the equilibrium points. Such a revision is systematically per-

formed in the first two sections of this paper. It has been demonstrated that non-Lipschizian dynamics

based upon a failure of the Lipschitz conditions ex- hibits new qualitative effects. One of them is the existence of attracting trajectories which can be “re-

membered” by the dynamical system. This effect follows from the violation of the Lipschitz condition for differential equations of trajectories in phase space. Another effect is a multiple-choice response to ex-

ternal excitations. The dynamical systems which pos- sess such a property can serve as an underlying idealized framework for neural nets with “creativ-

ity.” The most significant property of such systems would be their ability to be activated not only by

external inputs. but also by internal rhythms. The next four sections presented a systematic

analysis of applications of terminal attractors to ac-

tivation dynamics of neural networks. The starting

point is the content-addressable memory and pattern recognition. Two basic properties of terminal attractors are exploited here: the finite time of transient dynamics and the infinite local stability of terminal attractors. It has been demonstrated that due to the

last property. terminal attractors can be incorporated into neural networks such that any desired set of these attractors with prescribed basins is provided by

an appropriate selection of the synaptic weights. It has been shown that based upon the finite time of transient dynamics, more advanced performances, such as complex associative memory and control of the basins of attractions by using active terminal attractors. can be achieved. The results are generalized to neural networks which are modeled by delay-differential equations. By incorporating terminal attractors in such neural networks, one can store

274 !%I. ZllX

patterns which are characterized not only by the vector of the state variables, but also by durations of the exposure of its components.

The last section has been devoted to development of a dynamical training by introducing a dynamical system for the weight matrix which governs its relaxation to the corresponding stable equilibrium point represented by a terminal attractor.

In conclusion, we would like to summarize some results of practical applications of terminal attractors in neural networks performed at the Jet Propulsion Laboratory (JPL). There are two lines of activities in this direction. The first line is associated with learning dynamics. The starting point is a complex and important problem in robotic research: the inverse kinematic problem for redundant manipulators (Barhen, Gulati, & Zak, in press). In this work. topographically mapped terminal attractors were used to define a neural network whose synaptic elements can rapidly encapture the inverse kinematic transformations using a priori generated examples and, subsequently, generalize to compute the joint- space coordinates required to achieve arbitrary end- effector configurations. Simulation on 3-DOF and 7- DOF redundant manipulators demonstrated that the terminal attractor approach is computationally competitive with iterative methods currently used in ro- botics to solve the inverse kinematics of redundant manipulators.

The second line of research is associated with the problem of global optimization. Based upon the system (75), Barhen and Zak developed a model which exploits dynamical interaction between terminal attractors, while the interattractor coupling conditions are treated as constraints appended to the original optimization problem. Preliminary numerical simu- lations demonstrated that this method is competitive with existing methods. Some of the benchmark problems selected from the standard nonlinear optimization literature were solved.

REFERENCES

Barhen, .I.. Gulati. S.. & Zak, M. (in press). Self-organizing neu-

romorphic architectures for manipulator inverse kinematics.

NATO ASI. WI.

Hirsch. M. W. (1982). Systems of difl’erentlai equations that ~I-C

competitive or cooperative, I. Limit SC!> .\//I,%{ lorrr& oi

Marhrmaticoi Anrtlysis. 23, lt17- 170 Hirsch, M. W. (1Y85). II. Convergence almoht cverywhcre. SIAM

Journul of’ Mathemuricul Ann1ysi.c. 16. 3??--4Ys.

Hopfield, J. J. (1YX2). Neural networks and physical systems with

emergent collcctivc computational abilities. Procct&~r ofrlfc

iVuiiorirt1 Acudrrnv o/’ Scienw\. 79. 15.53. 75% Hopfield. J. J. (10X4). Neurons with grade.1 response have cr+

lcctive computational properties like those of two-state neu-

rons. Procwdin,yc or t/w N~lrrviu~ Aw~fcmb c-91’ Scicrtws. 81. 3058-3001,.

Keelcr. J. 11. (IYXh). Basins 01 attraction <It neural network

mod&. A II’ ~‘onfrrmcr ProceedinCq.~. Et. 159-265. Lapedcs. A.. & Farber. R. ( 10X6). Programming a massively par-

allel. computation universal system: Static behavior. AIP C’on-

f?rolc~, I’r0cecding.r. 151. 28%2St(.

Percival. I.. Xc Richards. D. ( 1Y82). 1ntrodu( mm to dynumir~:) (pp.

34--X). New York: Cambridge UniverGty Press.

Zak. M. ( 1070). Llniqueness and qtability oi the solution IO a

filament with a free end. A/$ictf Marh. anr! Mech. 34(G) 104X--

1057

Zak, M. ( IC)XZal. On the Pallure o1 hy(xxhaliclty in elaatictty.

Joruriui oj Elust~cify. 12. 219-229. Zilk. M. (IYX2h). Post-instability in c‘onllnuou> systems. S&if

~M&anic., Archrver, 7. 467-503.

Zak. M. (1983a). Cumulative effect at the ho11 surface due to

shear wave propagation. Journul of’ Appkd Mechanics. 5U.

22:-2x.

Zak. M. ( 1983h). Wrinkling phenomenon in structures. Solid Me-

chunic‘c Arthiwjs. 8. 181-316, 170---31 1 Zak, M. ( 1988). Terminal attractors tar addressable memory in

neural networks. Physics Letters~ ,4. 133. l&22.

Zak. M. (1989a). Non-iipschitzian dynamics for neural net mod-

elling. Applied Mathemiltical Letfrrv. 2. 69-E.

Zak. M. (IY89h). The least constraint principle for learning In

neurodynamics. Physic.s Lcslters /I_ 135. 25-28.

Terminal attractors in neural networks

Documents

Transcript of Terminal attractors in neural networks