IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-8, NO. 5, MAY 1978

ACKNOWLEDGMENTThe author wishes to express his gratitude for guidance

and encouragement received from Prof. T. lijima of theTokyo Institute ofTechnology. He also wishes to express histhanks to Dr. H. Nishino, the Division Chief of the Informa-tion Science Division, the Electrotechnical Laboratory, forhis assistance in this work, and to Dr. Y. Isomichi, a SeniorScientist of the Electrotechnical Laboratory, for suggestingthe method of statistical calculation.

REFERENCES[1] ISO Recommendation R1831, "Printing specifications for optical

character recognition," Ref. No. ISO/R1831-1971(E), Nov. 1971.[2] T. lijima and I. Yamasaki, "A method for print quality evaluation of

a large number of data," ISO/TC97/SC3/WGI/N173, Feb. 1972.[3] "Comments by the Japanese member body on the revision of

ISO/R1831," ISO/TC97/SC3/N80, Jan. 1973.[4] J. L. Crawford, "Comments received on document N106, PIDAS

print quality analysis by COL grid overlay," ISO/TC97/SC3/N115,Feb. 1974.

[5] M. Bohner, M. Sties, and K. H. Bers, "An automatic measurementdevice for the evaluation of the print quality of printed characters,"Pattern Recognition, vol. 9, pp. 11-19, Jan. 1977.

[6] W. R. Throssel and P. R. Fryer, "The measurement of print qualityfor optical character recognition systems," Pattern Recognition, vol.6, pp. 141-147, 1974.

[7] I. Yamasaki and T. lijima, "Sampling mechanism of characterimage," Trans. Inst. Electronics and Communication Engineers ofJapan, vol. 55-D, pp. 15-22, Jan. 1972; available in English inSystems Computers Controls, vol. 3, pp. 60-67, Jan.-Feb. 1972.

[8] "A method for print quality evaluation of a large number ofdata," J. Information Processing Society of Japan, vol. 13, pp.225-231, Apr. 1972; available in English in Information Processing inJapan, vol. 12, pp. 119-125, 1972.

[9] , "Print quality evaluation of a large number of data," J. Infor-mation Processing Society of Japan, vol. 13, pp. 525-532, Aug. 1972;available in English in Information Processing in Japan, vol. 13, pp.7-12, 1973.

[10] I. Yamasaki, "A method for two-valuing of printed images," J. Infor-mation Processing Society of Japan, vol. 16, pp. 419-425, May 1975;available in English in Information Processing in Japan, vol. 15, pp.152-157, 1975.

[11] "A quantitative representation of quality for a set of printedcharacters," J. Information Processing Society of Japan, vol. 18, pp.253-256, Mar. 1977; available in English in Information Processing inJapan, vol. 17, 1978.

[12] H. C. Andrews, "Multidimensional rotations in feature selection,"IEEE Trans. Comput., vol. C-20, pp. 1045-1051, Sept. 1971.

[13] "A method for assessing the information value of a character set (theC.O.M. method)," ISO/TC97/SC3/GT1 (ECMA-5)/N38, Dec. 1964.

A Sentence-to-Sentence Clustering Procedurefor Pattern Analysis

SHIN-YEE LU, MEMBER, IEEE, AND KING SUN FU, FELLOW, IEEE

Abstract-Cluster analysis for patterns represented by sentencesis investigated. The similarity between patterns is expressed in termsof the distance between their corresponding sentences. A weighteddistance between two strings is defined and its probabilistic inter-pretation given. The class membership of an input pattern (sentence)is determined according to the nearest neighbor or k-nearest neighborrule. A clustering procedure on a sentence-to-sentence basis isproposed. A set ofEnglish characters is used to illustrate the proposedmetric and clustering procedure.

I. INTRODUCTIONIN A PREVIOUS paper [1], we proposed a syntactic

clustering procedure, in which each formed cluster ischaracterized by a pattern grammar. Therefore, theprocedure yields not only the clustering results but also a

Manuscript received August 24, 1977; revised December 5, 1977. Thiswork was supported by the AFOSR under Grant 74-2661.

S. Y. Lu was with the School of Electrical Engineering, Purdue Univer-sity, West Lafayette, IN 47907. She is now with the Department of Electri-cal and Computer Engineering, Syracuse University, Syracuse, NY.

K. S. Fu is with the School of Electrical Engineering, Purdue University,West Lafayette, IN 47907.

grammar for each cluster. In order to do so, a grammar mustbe inferred when a new cluster is initiated, and later it isupdated whenever an input pattern is added to the samecluster. Error-correcting parsers are employed to measurethe distance between an input pattern and the languagesgenerated from the inferred grammars. The input pattern isthen classified according to the nearest neighbor syntacticrecognition rule. The emphasis of the syntactic clusteringprocedure is the use of grammar in which the hierarchy ofthe structure of patterns is described. In this paper, we shallpropose a clustering procedure on a pattern-to-patternbasis.

In statistical pattern recognition [9], a pattern is repre-sented by a vector called a feature vector. The similaritybetween two patterns is often expressed by a metric in thefeature space. The selection of features and the metric has afairly strong influence on the results of cluster analysis [2],[3]. In syntactic pattern recognition [10], a pattern is repre-sented by a linguistic notion called a sentence. The sentencecould be a string, a tree, or a graph of pattern primitives andrelations. We have proposed the use of a distance between

0018-9472/78/0500-0381$00.75 @ 1978 IEEE

381

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-8, NO. 5, MAY 1978

two sentences to express the similarity of their correspond-ing patterns. The distance between strings and the distancebetween trees have been studied [4], [5]. A set of Englishcharacters is used to illustrate the proposed clusteringprocedure. In the example, patterns are described by stringsin PDL-like representations [6]. It is also our purpose todemonstrate the consistency ofthe distance defined betweensentences and the similarity between the correspondingpatterns.The proposed sentence-to-sentence clustering algorithm

is described in Section II. In Section III, the definition ofdistance between two strings is briefly reviewed and ex-tended to the definition of weighted distance. An algorithmthat computes the proposed weighted distance on strings ispresented. This algorithm is an extension of Wagner andFisher's algorithm [4]. A probabilistic interpretation of theweighted metric is also discussed. The stochastic deforma-tion model described in Section Ill-B could be used as alinguistic decoder for speech recognition or a communica-tion system [11]-[13]. Finally, an illustrative example onclustering is presented in Section IV.

II. A SENTENCE-TO-SENTENCE CLUSTERING ALGORITHM

A. A Nearest Neighbor Recognition Rule

Suppose that C, and C2 are two clusters of syntacticpatterns. Let patterns in Cl and C2 be represented by twosets of sentences Xl = {xl, xi, *, x1} and X2 = {x2,x2, I ,x2}, respectively. For an unknown pattern repre-sented by sentence y, the nearest neighbor recognition ruleassigns y to cluster C1 if

min d(xJ, y) < min d(x,y),i I

and assigns y to cluster C2 if

(1)min d(xJ, y) > min d(x', y)i I

where d(x, y) denotes the distance between sentences x and y.In order to determine mind d(x', y), for some i, the

distance between y and every element in the set Xi has to becomputed individually. For the case that patterns are repre-sented by strings, the algorithm of computing string-to-string distance is given in Section III.The nearest neighbor rule can be easily extended to a

k-nearest neighbor rule. Let Xi-={xl, x2, , xn} be areordered set of Xi such that d(R', y) < d(54, y) ifj < 1, for all1 Kj, I < ni, then

K I K 1

decideyecif E- d(xJ,y)g I d(f,y). (2)j= K j=K

B. The Main AlgorithmWe shall describe a clustering procedure in which the

classification of an input pattern is based on the nearest (ork-nearest) neighbor rule.

Algorithm I: Input: A set of samples X = {x l, X2, n,xand a threshold t. Output: A partition of X into m clusters,Cl, C2 .. Cm.

Method:Step I: Let j =1, m =1. Assign xi to cm.Step 2: Increase j by one. Compute

Di= min d(x', xj), for all i 1 < i < m.

If Dk is the minimum among all Di, and i)Dk < t, then assignx to Ck; or ii) Dk > t, then initiate a new cluster for x ,, andincrease m by one.

Step 3: Repeat Step 2 until every element in X has beenassigned to a cluster.

In Algorithm 1, a design parameter t, which has a stronginfluence on the results of cluster analysis, is required.Usually, the number of formed clusters decreases as thres-hold t increases. If the total number of clusters is known tcan be adjusted until the same number of clusters isgenerated.'We refer to Algorithm 1 as clustering based on the nearest

neighbor rule. When the Di in Algorithm 1 is the averagedistance between xi and the k nearest sentences in Ci, we callit the clustering based on the k-nearest neighbor rule.

III. DISTANCE ON STRINGS

In this section, Aho and Peterson's notion that defines adistance between two strings in terms of language transfor-mations is briefly reviewed [7]. We shall extend thedefinition to that of a weighted metric. A probabilisticinterpretation of the proposed weighted metric is discussedin Section III-B.

Definition 1: For two strings, x and y in L*, we can definean error transformation T: E* -* such that y E T(x). Thefollowing three error transformations are introduced.

1) Substitution transformation:

co) a 02 T (/)1 h(ob 2, for all a, bCeL.

2) Deletion transformation:

o1aC2I TD a2)10)2, for all a E E.

3) Insertion transformation:

O)1W)2; wI a( (1)2, for all a e E

where wo , 0)2 E L*-Definition 2: The distance between two strings x, Y' E L,

d(x, y), is defined as the smallest number of error transforma-tions required to derive y from x.

A. A Weighted Metric

The metric defined in Definition 2 yields exactly theLevenshtein distance between two strings [8]. A weightedLevenshtein distance can be defined by assigning threenonnegative numbers to transformations T-5, TD, and T,.respectively. We have proposed a weighted distance thatwould reflect the difference of the same type of errortransformation made on different terminals.

Definition 3: Let the weights associated with transforma-

Of course, there is still a possibility that the total number of cluisters is

correct, but the grouping of patterns in each cluster is not.

382

LU AND FU: CLtJSTERING PROCEDURE FOR PATTERN ANALYSIS

y b- b2 b3B | (alb1 ) l(al,b2) i(al,b3)

aI

x = a2

a3

l \sfi substitution

deletion

Fig. 1. Graphic interpretation of metric W.

tions on terminal a in a string w-)I a O2, where a E X, o)l and()2 E *, and Y. is a set of terminals, be defined as follows:

1) (02 a Ts,S(ab) t b)2,

where S(a, b) is the cost of substituting a for b;2) ) a-) 2 iTDD(a) Ol O2,

where D(a) is the cost of deleting a;

3) wo a (2 -oIl('- b a 2,

where I(a, b) is the cost of inserting b in front of a;

4) x Tj,I(b) X b

where I'(b) is the cost of inserting b at the end of astring.

Definition 4: Let x and y be two strings, and J be asequence of error transformations used to derive y from x.Let J be defined as the sum of the weights associated withthe transformations in J. Then the weighted distance be-tween x and y is

d,((x, y)= min{iJ}J

Definition 4 can be illustrated graphically. From point Bto point E, each path in the lattice shown in Fig. 1corresponds to a sequence of transformations used to derivey from x. A horizontal branch indicates an insertion trans-formation, a vertical branch indicates a deletion transforma-tion, and a diagonal branch indicates a substitutiontransformation. The weight assigned to a particular type oferror transformation on a particular symbol in x is labeled atits corresponding branch. Let J be a path in the lattice, andthen J is the sum ofweight associated with each branch in

J. d,t,(x, y) is the weight associated with the minimum-weightpath.

f insertion The following algorithm, which is an extension ofWagnerand Fisher's algorithm, computes the weighted distancebetween two strings.

Algorithm 2: Input: i) two strings x = a,a2 a,, andy = b1 b2 ... bm where ai, bi E E for all i. ii) a table ofweightsassociated with transformations on terminals in S. Output:the weighted distance d,,(x, y).Method:

Step 1: 6(0,0)=0.Step 2: DOi= 1,n.

6(i, 0)=(i- 1, 0) + D(aj).Step 3: DOj= 1,rm.

((0, j) = 6(0, j - 1) + I(a1, ba).Step4: DOi=1,n.

DOj= l,m.

El = (i - 1, j-1) + S(a1, b3).£2-=( (i - 1 j) + D(aj)83 = (6(i, j - 1) + I(ai+ 1, bj) if i < n or

83 = 6(i,i -1) + I'(bj) if i = n.

b(i, j') = min (Ei, 82, 83).

Step 5: d,,(x, y) = 6(n, m) exit.

B. A Probabilistic Interpretation of the Proposed MetricA stochastic model for the three types oferror transforma-

tions described in Section 111-A has been proposed [15]. Weshall briefly review it and then give its interpretation withrespect to the weighted metric.

1) A Stochastic Deformation Model: Following the nota-tions introduced in Section 111-A, the probabilities asso-ciated with the transformations Ts, TD, and T, with respect toterminal a are defined to be

1) qs(b a), the probability of substituting terminal aby b,

2) qD(a), the probability of deleting terminal a,3) q1(b a), the probability of inserting b in front of

terminal a, and4) q,(a), the probability of inserting terminal a at the

end of a string.The deformation probabilities of a single transformation

on terminal a is consistent iff

E qs(bIa)+ E qI(bja)+qD(a)= 1bei bEY

(3)

where I is a set of terminals.Let a E E* be a substring. The probability that terminal a

is transformed to a, denoted q(a a), is defined as follows:

I qD(a), if a = Aq(a a) = max {qs(b a), q1(b a)qD(a)}, if a = b

q1(b, a) ... q1(b,- 1 a) max [qs(b1 a), qI(b1 a)qD(a)], if a = b b2 ..., bl, 1 > 1.(4)

383

34JILL TRANSACTIO)NS ONN SYSTEMS, NIAN, AND (I YBERNETlCS.0(N\!.S0'-S,MN' \i.NV1tY J9 /

The consistency of this multiple-transformation model and the weights associateddefined in (4) can be proved from (3).2 Therefore, we have transformations, D(a) and

E q(x|Ia)= 1. (5) - log [qD(a)/qs(a a)] andZ Ia 1(* Furthermore, we define

The probability of inserting a, a c- L*, at the end of a stringis

q ( I(1-qi)q,(bl)q(b -qq(bl),

where

q = Z qj(a).I I

with deletion and insertionI(a, /), can be (defined as

log q,(h a). respectivelv.

I(b)-=-log gz(b).

when 2= /wwhen-=b1b2 ...l>l

Consequently, we have

d(,)(x, y) -d,jaj, i2) + d(1((a2.2 ) +*-f

C()(a,n Yn)a j (/i+ a, I )

(6)

Furthermore,

Eqt(a) (1- q') + Z (1-q)q,(ai)

+ i L (1 q') + q(ai)q'(bj) +-ij

=(I - q')(l + q' + q2+ )=1

Assume that a sequence of error transformations made ona terminal in a string is independent from its context. Thenfor two strings x and y the probability of transforming x to y,where x = a, a2 ** an iS

n

q(y x) = max fl q(xj aj)q'(ot + l)Ij-l (7)

where ' a2' ' an + l, J2 0, is a partition of y intoni + 1 substrings, and 1 <i < r. r is the number of differentways of partitioning y into n + 1 substrings.The graph shown in Fig. 1 can also be used to interpret the

proposed stochastic model after the label on each branch ofthe lattice is changed from the weight to the probability of itscorresponding error transformation. To derive y from x, thesequence of error transformations that has the highestassociated probability is the one defined by (7).Fung and Fu have defined a distance for a, b E 1, 6(a, b),

as a function ofq(b a), the probability ofsubstituting a for b,in their length preserved deformation model [14]. That is,

6(a, b) =-log q(b a) (8)

We can interpret the function 6(a, b) as the weight associatedwith the substitution of a for b. Using our notation, we have

S(a, b) = -log q,(b a)

2 The proof is given in [15].

q(21 la,)= -log q(aa) log i, Ian)q a,I)a*. o_ 108

qs ( 4111 (1t1 )

q (10Xn + 1I)log 1 q,

alog q(J ) + 4q(x Ix) (9)

where x, y are two strings, A is a constant, and 21x 2.7°C+ 1iS a partition of v such that the summation

n

Z dc((ai, 2) + dc,(X In)

is the minimum among all the possible partitions of y.Therefore, the stochastic deformation model could be

regarded as a special case of the weighted metric defined inSection 111-A. It provides a probabilistic interpretation forthe metric.

2) A Maximum-Likelihood Recognitioni Rule:a) Single-class case: Let X = {x 1, x, x,1 be a

sample set, where x,i = xj if i Jj. The rate of occurrence ofxi, < i < n is represented by an a priori probabilityp(I).Let y be a string not in X. Using the Bayes rule, the a:posteriori probability that the true representation of v is -xi is

q1 (s, x)p(x ),jiY - 1

(IU)

8 The maximum-likelihood recognition rule assigns v to be .Iif

p(xiIy) = max p(xjIy). (11)

b) Multiple-class case: Assume that there are twoclasses C1 and C2 represented by sample sets XI1- x

xI, ,' x,l} and X2 = {x2, x2,. x2}! respectively. Eachelement x1 in Xl has an a priori probability p(x'jCI1). The a

priori probabilities P(C1), I 1 2. are also known. Thedeformation probability that a string is deformed from some

384

LU AND FU: CLUSTERING PROCEDURE FOR PATTERN ANALYSIS

1 2 3 4 25 26 27 28

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ....................... ... ..... ........ ...........

g~~a 9 0 16.CAC"CCC ,a

~~~~~S m . .AC to CAcc Ca. C

~~~~~~.c

* . . . . . . . C~~

5 ~~~6 78 29 30 31 32

C......... .... ........... ... .................CCCCC . . . . . . . . . C ~~~~~~~~~~~~~~~~~~~~~:-111111 ....A.....A. .............

CCC . C. . . . . . C . .cc~

CACC CACCCCCCS .c

AC as.. .. . C'C.... .S....aA CCC........**.CCCCC...~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sc

SCCCC C C C C . . . . . . CC C C C C 555 CCCCCCCC CCC~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C~~~~~~~~~~~~~~~~~ICAC.CCC.

~~~~~~~~~~~~~~~~~~~~~~........... ............. ..... C....... C.......

9 10 11 12 33 34 35 36

CCaCC

I.CC, A

ICCACC St' A CCCCC0

C

.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

C~~~~~~~~~~~~~~~~~~~~~~ ... .. .. ... .. C. ... . C.. . . C.. . . . . . . . . .... ....C. .C ..C . ...

...C...C.. .C. ... .. ....... .. .. ..C . .. . C. C.....C .. .. .. . C. .. .C .. . C.. ..C C..C.

.~~cbeC laStla~~~~~~~~~~~~~~~~~~~~~~~~~~~~~CC

9s4

CCC C C C C C C CCC CC~~~~~~~~~~~aca CCCCo.C aC

....... .....C..C....C ....C ... .C..C...C C.41CC. 4243CC 4C4C~~~~~~~........C.C. .... C... C...... C....C.... C.... C... C.................C.

a 0* DCCAC CI

CCA CO

C~~~~~~~~~~~~~~~~~~~~~~~~~CAC'CSCtCa

212223.441 42 43 44

CSC C'C

S C~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~SCCCC

C~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~A:ICCaC~~~~~~~~~~~~~~~~~~~~~~C

C~~~~~~~~~~~~~~~~~~~~~~~~~~cA a:0 00 SC~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~AS *

C~~~~~~~Aasoc

C~~~~~~~~~~~~~~~~.... .. .. .. .. .. .....C . ... .. .. . .. .. . .. . ... .. .. ... ... .. . .. .. . . .. . .. .. . .. .. . C... . C.. . . C. CC....C.C..C.C..C.CC..CC.CC.

C49505C~~~~~~. ... .. ... .. .. ..C.C.C..C.C.C.C. C.C..

@

* Cs*

*

.C *

C

C

Fig. 2. 51 character patterns.

385

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNIETICS. \VI. SMC-8, \&) \\lXA 197 \

string in class Cl is q(y x', Cl). Then the probability that x isthe true representation of y is

p(I, C Y) =- 2 n) (12)

Z Z q(y Ij, Ck)P(xI Ck)P(Ck))k=1 j=1

The maximum-likelihood recognition rule assigns v to be xlif

p(x,, C,JI y) = max p(xJ e Ck Iv))- (13)

Again, the maximum-likelihood recognition rule can beconsidered as a probabilistic interpretation of the proposednearest neighbor rule.

IV. AN ILLUSTRATIVE EXAMPLE

A set of 51 English character samples is used to illustratethe proposed clustering procedure. All the computationswere carried out on a CDC 6500 computer with Fortran IVprogramming language. The same sample set has been usedin a previous paper [1]. The characters are from ninedifferent classes: D, F, H, K, P, U, V, X, and Y. Eachcharacter is a line pattern on a 20 x 20 grid. Starting from itslower left corner, each input pattern is initially chain-coded[18] cell by cell. After three consecutive cells have beencoded, a pattern primitive of this line segment or branch is

extracted.Four pattern primitives which are line segments with

different orientations

a b

TABLE 151 CHARACTER PATTERNS AND REPRESENTATIONS

PatternNo.

2

3

4

5

6

7

8

910

11

12

13

14

i)

16

17

18

19

20

21

22

23

24

25

26

String Representation

bbb+ (b+dddbdd) *

aaa+cddxaaxcbb

lbbb+ddcbaa) *

cbbbxdabbbb

bb+(b+dd) xd

bbbb+cc xbb

(bbb+dxddcbbaa) *

ba+bbxaax cc

aaa+cxaaxb

b+bbxdddd+bbxb

(bbb+dxddbbad) *

bb+(bb+dd)xd

bbbbbxabaa

bb+bbxaax cc

cbbxda+bbxb

bb+bbbxaaxbcc

bb+ (bbb+dddbaa) *

b+bbbxd+bxdxb

aa+cbxaax cc

bbb+ (bbadcbad) *

bbbxddabb

bbb+bcxaa

b+(bb+ddcbad) *

b+bbbxaaxbc

cbbbxabbba

PatternNo.

28293031

32

33

34

35

36

3738

39

40

4 I

424344454647

48

4950

51

String Representat

bbb+ (dx (b+dd) ) xdd

bb+c bxaaa

bb+bbxddd+axbb

baa+bbxaaxc,

ba+abxdd+axbbb

aa+Acxax-

bb+(bb+dxddcaad) *

bb+bbxa+axcu b

bb ( bbtd )x dd

cbbxdaabb

ba+bc xua

aa+ccxaaxcc

bb+ (bb+dd) xdd

(bbbbb+dddcbaad *

dabxcbxbabb+(b+ddd)xxJbbb+cxa

b+dxabxdd+bxbb

(bb+ddcbiJd *

aa+cxa2axcw,.

bbb+ddaabc dd,

bbbxda+bbxt)

baa+ccxAbxcbcba+bxa+abxbbb

cbbxdabbbb

C

are selected. Following Shaw's PDL [6], three concatenationrelations, +, x, *, and the parentheses ( and ) are used.However, * is used here primarily for the situation of a "selfloop," that is, a branch of which the head and tail coincide.The 51 sample patterns and their string representations aregiven in Fig. 2 and Table I, respectively, where the patternnumber indicates the input sequence used in the clusteringprocedure. For Pattern 1 Character P in Fig. 2, it has threebranches. The concatenations of branches are expressed asBRANCH 1 + (BRANCH 2 + BRANCH 3) * and the corre-

sponding string representation of the pattern isbbb + (b + dddbdd)*.

A. The DistanceThe proposed distance measure is performed on the

hinguistic representations of patterns (sentences), ratherthan on the patterns themselves. Consequently, whether ornot the model yields a good measure is a matter ofchoosingappropriate representations. Fig. 3 illustrates a couple ofexamples where the unweighted distance between the twoK's or between the two X's is smaller than that between a Kand an X

' In this example, the substitution transformation between a primitivesymbol a, b. c, and d, and a relational symbol +, x, *, (and ) are notconsidered.

Pattern 38

* C* C A* BIC A

CAc

* 6£|

* e*soaSc* AS

c

aa+ccxaaxcc

Pattern 46

*A* A

* 0

* AAeC

* BA* SB¢* AC

* A C* A C

* #A C -

* a

aa+ccxaaxcc

Pattern 14

* A

* a

* A£0

4 *

* AC* a c* A C* A C*

bb+bbxaaxcc

26

Pattern 34

A A

* A

SAS6 BA C

S C*1 5 5

A A C

* b £aa* S SA G

bb+bbxa+axcb

Fig. 3. Distances between similar and dissimilar pattern's

386

387LU AND FU: CLUSTERING PROCEDURE FOR PATTERN ANALYSIS

47

(a)Fig. 4. Minimum spanning tree of 51 character patterns.

clusters

20

18

16

14

12

1098

6

4

2

o k-i

X k-3

x

0 o

x o

0

x

4 4.5 5 5.5 6 6.5 7

Fig. 5. Number of clusters versus threshold.

B. Minimum Spanning Tree

The unweighted distances between every pair of thesample patterns is computed by using Algorithm 2. Aminimum spanning tree [3] for the 51 sample patterns can beconstructed and is shown in Fig. 4. The true clusters arecircled on the tree. We shall use it as a reference for our

cluster analysis.

C. Clustering Using Unweighted Distance

According to the clustering procedure described in Sec-tion II, the first step is to search for an appropriate thresholdvalue, then the cluster analysis based on this threshold isperformed. Fig. 5 shows the relation between the thresholds

(b)Fig. 6. Clustering results of using unweighted distance. (a) Using nearest

neighbor rule with threshold 6. (b) Using 3-nearest neighbor rule withthreshold 6.5.

used and the number of clusters obtained. Here, two experi-ments are conducted: one is the clustering based on thenearest neighbor rule, and the other is that based on the3-nearest neighbor rule. The thresholds used in these twoexperiments are t = 6 and t = 6.5, respectively. The finalclustering results are given in Fig. 6. When the nearest

Threshold

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. SMC-8, NO. 5. MAY 1978

TABLE IIWEIGHTS ASSOCIATED WITH SUBSTITUTION TRANSFORMATION

WHERE j IS SUBSTITUTED FOR

Si,j) a b c d x + * t )

a 0 1.3 2 1

b 1.8 0 1.2 1 - - - - -

c 1.2 1.3 0 1 - - - - -

d 1.0 1.0 1.0 0 - - - - -

x _ _ _ - 0 3.0 3.0 3.0 3.0

+ _ _ _ _ 3.0 0 3.0 3.0 3.0

* _ _ _ - 3.0 3.0 0 3.0 3.0

- _ _ _ - 3.0 3.0 3.0 0 3.0

--- - 3.0 3.0 3.0 3.0 0

TABLE IIIWEIGHTS ASSOCIATED WITH DELETION TRANSFORMATIONS

i D(i)

a 0.6

b 0.7

c 1.3

d 1.2

x 1

*

1.2

0.9

0.9

) 0.9

TABLE IVWEIGHTS ASSOCIATED WITH INSERTION TRANSFORMATIONS

WHERE j IS INSERTED IN FRONT OF i

I(i,j) a b c d x + * I )

a 0 1.0 1.0 1.0 1.0 1.0 3.0 3.0 3.0

b 1.0 0 1.0 1.0 1.0 2.5 3.0 3.0 3.0

c 1.0 1.0 0 1.0 1.0 1.0 3.0 3.0 3.0

d 1.0 1.0 1.0 0 1.0 1.0 3.0 3.0 3.0

x 1.0 1.0 1.0 1.0 3.0 3.0 3.0 3.0 3.0

+ 1.0 1.0 1.0 1.0 3.0 3.0 3.0 3.0 3.0

* 1.0 1.0 1.0 1.0 3.0 3.0 3.0 3.0 3.0

2.0 2.0 2.0 1.0 1.0 2.0 2.0 0.5 2.0

1.0 1.0 1.0 1.0 2.0 2.0 3.0 3.0 0.5

TABLE VWEIGHTS ASSOCIATED WITH ERROR TRANSFORMATIONS

TABLE VIRESULTS OF CLUSTERING USING WEIGHTED DISTANCE

Pattern true the nearestno. class pattern1 p _

18 P 1

21 Cp 1

24 P 21

33 P 24

2 X 1

9 X 2

20 X 9

32 X 20

38 X 20

46 X 32

49 X 38

3 D 1

7 D 3

11 D 7

40 D 3

45 D 3

47 D 45

4 U 2

16 U 4

22 U 4

36 U 4

48 U 16

51 U 36

5 F 1

12 F 5

27 F 5

35 F 12

39 F 5

42 F 5

6 Y 5

15 Y 6

23 Y 15

28 Y 15

37 Y 23

41 y 28

43 y 15

8 K 2

14 K 8

17 K 14

25 K 8

30 K 8

34 K 14

10 H 4

19 H 10

29 H 19

31 H 29

44 H 31

SI H 31

13 V 4

26 V 13

d istance f romthe nearest pattern

( )

2.7

4.2

3.0

1.7

(17.4)

4.4

2.4

1.3

0.0

0.7

3.6

(6.3)

1.0

2.31 .02.7

5.9

(9.3)

4.0

2.7

2.1

1.3

0.6

(9.6)

0

3.01.20.00.0(7.0)

5.4

1.0

1.7

2.4

3.9

0.6

(6.6)

0.6

1.01.9

0.0

2.3(7.1)

5.6

3.4

4.6

3.9

3.8

(6.1)

2.5

clusterresul t

1

2

2

3

3

3

4

4

4

4

4

5

5

6

6

5

6

6

6

8

8

6

8

88

9

9

I'(j) 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0

388

j a b c d x + * ( )

_ _

LU AND FU: CLUSTERING PROCEDURE FOR PATTERN ANALYSIS

neighbor rule is used, the chain effect results in four X's,pattern 32, 38, 46, and 49, being clustered together with theK's. This situation is improved when the 3-nearest neighborrule is used. However, both experiments failed to assign thetwo V's, pattern 13 and 26, in the same cluster, or to assignthe distorted D, pattern 47, together with other D's.

D. Clustering Using a Weighted Distance

The result of clustering presented in Section IV-C can beimproved by using a weighted distance. In this example, it ispossible to find a set of weights that can correctly cluster allthe patterns in the sample set. Tables II-V suggest such a setof weights. The result of cluster analysis is given in Table VI,where the numbers in the first column indicated the inputsequences. The true character of each pattern is given in thesecond column. The nearest pattern for an input pattern isobtained by computing the distances between the input andall the preceding patterns, respectively, and picking thesmallest. The actual distance between an input and itsnearest pattern is listed in column four. If the distance isgreater than the threshold, which is six in this experiment, anew cluster is initiated. The final result is shown in columnfive.

V. SUMMARY AND REMARK

Cluster analysis for patterns represented by sentences isstudied. Different from statistical pattern recognition, wherea cluster analysis is performed on a set of vectors, a set ofsentences is to be analyzed. However, once the sentences areextracted from patterns and the distance on sentences isdefined, any conventional clustering criterion can be used.The proposed clustering procedure, though only applied toan example of patterns represented by strings, can be easilyextended to that of patterns represented by trees using thetree distance defined in [5].The use of a weighted distance can improve clustering

results. In the example given in Section IV, the set ofweightsused is determined by a trial and error procedure. Thecomputer time required for the cluster analysis is 46 s. Tocomplete the analysis, it is necessary to formulate an algor-

ithm that searches for a set of suitable weights from a set oftraining samples. The stochastic deformation model pro-vides a theoretical basis for assigning weights as a function oferror frequency when a large data base is available.

REFERENCES[1] K. S. Fu and S. Y. Lu, "A clustering procedure for syntactic patterns,"

IEEE Trans. Syst., Man, Cybern., vol. SMC-7, Oct. 1977.[2] R. 0. Duda and P. E. Hart, Pattern Classification and Scene Analysis.

New York: Wiley, 1972.[3] E. Diday and J. C. Simon, "Clustering analysis," Digital Pattern

Recognition, K. S. Fu, Ed. New York: Springer-Verlag, 1976.[4] R. A. Wagner and M. J. Fisher, "The string to string correction

problem," J. Ass. Comput. Mach., vol. 21, Jan. 1974.[5] S. Y. Lu and K. S. Fu, "Error-correcting tree automata for syntactic

pattern recognition," in Proc. IEEE Computer Society Conf on Pat-tern Recognition and Image Processing, June 6-8, 1977, RPI, Troy,NY.

[6] A. C. Shaw, "A formal picture description scheme as a basis forpicture processing systems," Inform. Contr., vol. 14, 1969.

[7] A. V. Aho and T. G. Peterson, "A minimum distance error-correctingparser for context-free languages," SIAM J. Comput., vol. 4, Dec.1972.

[8] A. Levenshtein, "Binary codes capable of correcting deletions, inser-tions and reversals," Sov. Phy. Dokl., vol. 10, pp. 707-710, Feb. 1966.

[9] H. C. Andrews, Introduction to Mathematical Techniques in PatternRecognition. New York: Wiley, 1972.

[10] K. S. Fu, Syntactic Methods in Pattern Recognition. New York:Academic, 1974.

[11] L. R. Bahl and G. F. Jelinek, "Decoding for channels with insertions,deletions, and substitutions with applications to speech recognition,"IEEE Trans. Inform. Theory, vol. IT-21, July 1975.

[12] L. W. Fung and K. S. Fu, "Maximum-likelihood syntactic decoding,"IEEE Trans. Inform. Theory, vol. IT-21, July 1975.

[13] R. L. Kashyap and M. C. Mittal, "A new method for error correctionin strings with applications to spoken word recognition," in Proc.IEEE Conf Pattern Recognition and Image Processing, June 6-8,1977, Troy, NY.

[14] L. W. Fung and K. S. Fu, "Stochastic syntactic decoding for patternclassification," IEEE Trans. Comput., vol. C-24, June 1976.

[15] S. Y. Lu and K. S. Fu, "Stochastic error-correcting syntax analysis forrecognition of noisy pattern," IEEE Trans. Comput., vol. C-26. Dec.1977.

[16] E. Tanaka and T. Kasai, "Synchronization and substitution error-correction codes for the Levenshtein metric," IEEE Trans. Inform.Theory, vol. IT-22, Mar. 1976.

[17] V. A. Kovalevsky, "Sequential optimization in pattern recognitionand pattern description," in Proc. IFIP Congress 68, North-HollandPub]. Co., Amsterdam, 1968.

[18] H. Freeman, "On the encoding of arbitrary geometric configuration,"IEEE Trans. Electron. Comput., vol. EC-10, pp. 260-268, 1961.

389