Semigroup Forum OF1–OF17c© 2003 Springer-Verlag New York Inc.

DOI: 10.1007/s00233-002-0012-6

RESEARCH ARTICLE

Irreducible Semigroups of Matrices withEigenvalue One

J. Bernik, R. Drnovsek, T. Kosir, M. Omladic, and H. Radjavi

Communicated by Jimmie D. Lawson

Abstract

We study groups and semigroups of n×n matrices with the property that eachmatrix has a fixed point, i.e., 1 is an eigenvalue of each matrix. We show that forn = 3 and n ≥ 5 there are irreducible matrix groups and irreducible semigroupsof nonnegative matrices with this property. In fact, for n = 3 we determinethe structure of any such semigroup. We also present additional hypothesesimplying reducibility.

Key words and phrases: Matrices with eigenvalue one, fixed points, groupsand semigroups of matrices, irreducibility, nonnegative matrices, symmetric pow-ers of matrices.

2000 Mathematics Subject Classification: Primary. 15A30, 20M20, 47D03.Secondary. 15A18, 15A48, 20G20, 20H20.

1. Introduction

Let S be a (multiplicative) semigroup of n × n complex matrices with theproperty that every member of S has a fixed point, i.e., 1 is an element ofthe spectrum σ(S) for every S ∈ S . It is not hard to see that this does notimply the existence of a common fixed point for all members of S . But doesit at least imply reducibility of S , i.e., the existence of a common, nontrivial,invariant subspace for all the operators represented by S ? The answer is yesfor the simple case n = 2. We show that for n = 3 and n ≥ 5 the answer isno. For the case n = 4 the affirmative answer is much harder to show. Usingresults from algebraic geometry the proof will be given in a separate article [3].

The structure of irreducible semigroups or groups whose members all havefixed points seems to be interesting and intricate. We offer some evidence for thevalidity of this claim by completely characterizing such semigroups in the casen = 3: We prove that S is then the second symmetric power of an irreduciblesemigroup in Sl2(C).

What stronger hypotheses guarantee reducibility? There are some knownanswers. For example, if the (algebraic) multiplicity of 1 in σ(S) is at least n−1for each S , we have not just reducibility, but simultaneous triangularizability ofthe semigroup [10]. Another known result [10] is that if 1 is dominant in σ(S)for every S ∈ S , then S is reducible. Dominance of 1 in σ(S) means that every

OF2 Bernik et al.

other member of σ(S) has modulus less than 1. We prove an extension of thisresult by requiring only that a particular member of S satisfy the dominancecondition. We also give other sufficient conditions for reducibility.

2. General groups and semigroups of matrices

Let S be a semigroup of n× n matrices with n ≥ 2. If the spectrum σ(S) ofevery member S in S is {1} , then S is triangularizable by Kolchin’s The-orem. The following slight extension of Kolchin’s Theorem was proved in[10, Cor. 2.2.9].

Proposition 2.1. Let S be a semigroup of n × n matrices such that, forevery S ∈ S , the algebraic multiplicity of 1 in σ(S) is at least n− 1 . Then Sis triangularizable.

It is natural to consider a weaker spectral assumption: 1 ∈ σ(S) for everyS ∈ S .

We shall need the following preliminary result from [10, Cor. 2.1.6]:

Lemma 2.2. Let S be a semigroup of n× n matrices. Let ϕ be a nonzerolinear functional on Mn(C) whose restriction to S is constant. Then S isreducible.

Proposition 2.3. Let S be a semigroup of n×n matrices such that 1 ∈ σ(S)for every S ∈ S . Assume that the closure (in the operator norm topology) of Scontains a matrix of rank at most 2 . Then S is reducible.

Proof. By the continuity of the spectrum, we may assume that S is closed.Denote by I the semigroup ideal in S of all matrices with minimal positive rank,say r . By the assumption, r ∈ {1, 2} . If r = 1, then the trace is constant on I ,namely tr(S) = 1 for all S ∈ I . By Kaplansky’s Theorem [10, Cor. 2.2.3], weconclude that I is reducible. Now [10, Lem. 2.1.10] implies that S is reducibleas well.

Consider the case r = 2. If for every S ∈ I the multiplicity of 1 inσ(S) is 2, then tr(S) = 2 for all S ∈ I which completes the proof as before.Otherwise, pick A ∈ I with σ(A) ⊇ {1, α} , where α �= 0, 1. Without lossof generality we may assume that A is a diagonal matrix diag (1, α, 0, . . . , 0).Given S = (sij )

ni,j=1 ∈ S and k ∈ {1, 2} , the characteristic polynomial of AkS

is equal to

(λ2 − (s11 + αks22)λ+ α

k(s11s22 − s12s21))λn−2 = 0.

Since 1 ∈ σ(AkS) we have

1− s11 + αk(s11s22 − s12s21 − s22) = 0

for k = 1, 2. Because α �= 1 we conclude that s11 = 1 for every S ∈ S . ThenS is reducible by Lemma 2.2.

Bernik et al. OF3

As a consequence of either Proposition 2.1 or Proposition 2.3 we obtain:

Corollary 2.4. Let S be a semigroup of 2× 2 matrices such that 1 ∈ σ(S)for every S ∈ S . Then S is triangularizable.

We now present counterexamples in all cases except n = 2 and n = 4.

Proposition 2.5. For any odd integer n ≥ 3 there exists an irreducible finitegroup G of n× n matrices such that 1 is in σ(G) for every G ∈ G .

Proof. Let G be the finite group generated by the permutation matrix

U =

0 1 0 · · · 0

0 0 1 · · · 0

0 0 0 · · · 0

......

......

1 0 0 · · · 0

together with all diagonal matrices D with ±1 on the diagonal and detD =1. Observe that the algebra in Mn(C) generated by G contains all diagonalmatrices, whose only invariant subspaces are spans of some standard basisvectors. Since none of them is invariant under U (except trivial ones), weconclude that G is an irreducible group.

Proposition 2.6. Let S be an irreducible semigroup (resp. group, finitegroup) of n×n matrices such that 1 ∈ σ(S) for every S ∈ S . Then there existsan irreducible semigroup (resp. group, finite group) of 2n × 2n matrices withthe same spectral property.

Proof. Let T be the semigroup (resp. group, finite group) of all 2n × 2nmatrices of the form (

A 0

0 B

)and

(0 C

D 0

),

where A , B , C and D are from S . Clearly, every matrix of the first form has1 in its spectrum. To show the same is true for the second form as well, letx �= 0 be an eigenvector of CD corresponding to 1, and note that

(0 C

D 0

)(x

Dx

)=

(CDx

Dx

)=

(x

Dx

).

Using Burnside’s Theorem [10, Thm. 1.2.2.] one can see easily that T isirreducible, since S is.

The last two propositions and an easy induction give the following result.

OF4 Bernik et al.

Corollary 2.7. For any integer n ≥ 3 that is not a power of 2 , there existsan irreducible finite group G of n × n matrices such that 1 ∈ σ(G) for everyG ∈ G .

The following example shows that there is an irreducible group of 8× 8matrices such that 1 is an eigenvalue of each matrix.

Example 2.8. Let ρ be the adjoint representation of Sl3(C) on its Liealgebra sl3(C). Here ρ(A)X = AXA−1 and sl3(C) is the eight-dimensionalvector space of all matrices in M3(C) with trace 0. Representation ρ isabsolutely irreducible [5, Thm. 2.3.8, p. 80]. Therefore the image G of ρ isan irreducible group of matrices in M8(C). Observe that X is an eigenvector ofρ(A) at eigenvalue 1 if and only if AX = XA . Since the commutant of A (inM3(C)) has dimension equal to at least 3, we conclude that 1 is an eigenvalueof ρ(A) with multiplicity at least 2.

After the paper was submitted, a construction as in the above examplebut with a finite group has been found [2].

The above example combined with Corollary 2.7 and Proposition 2.6imply the following:

Corollary 2.9. For any integer n , except n = 2 and n = 4 there exists anirreducible group G of n× n matrices such that 1 ∈ σ(G) for every G ∈ G .

It can be checked that the sum of all the elements of the group G con-structed in the proof of Proposition 2.5 is equal to 0. The following simple result,which is needed in the sequel, implies that this is true in every irreducible finitematrix group G .

Proposition 2.10. Let G be a compact group of n×n matrices, and denoteby µ its Haar measure. If ∫

GGdµ(G) �= 0,

then G has a common fixed point.

Proof. Assume that∫G Gdµ(G) �= 0 and pick an element x ∈ C

n such that

y =

(∫GGdµ(G)

)x �= 0.

Then, for any G0 ∈ G , we have

G0y =

(∫GG0Gdµ(G)

)x =

(∫GH dµ(H)

)x = y.

In what follows, m(T ) will denote the (algebraic) multiplicity of 1 inσ(T ). Recall that a group G of n × n matrices is bounded if it is bounded asa subset of Mn(C).

Bernik et al. OF5

Theorem 2.11. Let G be a bounded group of n × n matrices such thatm(G) ≥ n/2 for any G ∈ G . Then G has a common fixed point.

Proof. With no loss of generality we may assume that G is compact. Letµ be the Haar measure of G . Boundedness of G implies that σ(G) ⊆ {z ∈C; |z| = 1} for every G ∈ G . Since m(G) ≥ n/2 the real part of the traceRe(tr(G)) is nonnegative for every G ∈ G . Since I ∈ G and tr(I) = n , thereexists a neighborhood U of I such that Re(tr(G)) > 0 for all G ∈ U . By thecompactness of G there exist G1, . . . , Gm ∈ G such that G = ∪m

j=1GjU . Sinceµ(G) = 1 and µ is translation invariant, we conclude that µ(U) > 0. It followsthat

Re

(tr

(∫GGdµ(G)

))=

∫GRe (tr(G)) dµ(G) ≥

∫URe (tr(G)) dµ(G) > 0.

Now Proposition 2.10 can be applied to complete the proof.

Remark 2.12. One may ask what is the minimal k = f(n) such thatm(G) ≥ k , for all G in a bounded group G of n×n matrices, implies that G hasa common fixed point (or is at least reducible). The obvious generalization ofExample 2.8 to the adjoint representation of Slm(C) on its Lie algebra slm(C)shows that for many values of n , namely those of the form m2 − 1, we havek ≥

√n+ 1.

3. Groups and semigroups of nonnegative matrices

In this section we consider matrices whose entries are all nonnegative. Forcollections of such matrices a much stronger notion of reducibility, that ofdecomposability is also natural to consider: a collection C in Mn(R

+) is said tobe decomposable (called ‘reducible’ by some authors working in the theory ofnonnegative matrices; also, not to be confused with the decomposability usingin the representation theory) if there exists a permutation matrix P such that{P−1AP ; A ∈ C} has a simultaneous 2×2 block-triangular form. Equivalently,if the members of C are considered as matrices of operators relative to thestandard basis of C

n , then decomposability is the condition that the linearspan of some nonempty, proper subset of the basis vectors is invariant underevery member of C .

Although there are a few affirmative results in what follows, negativeresults analogous to the general case above prevail in the case of nonnegativematrices.

Remark 3.1. If S in Proposition 2.6 is a semigroup of nonnegative matricesthen the semigroup T constructed in the proof consists of nonnegative matricestoo.

Theorem 3.2. Let G be an indecomposable group of nonnegative n × nmatrices such that 1 ∈ σ(G) for every G ∈ G . Then G is diagonally similar toa permutation group.

OF6 Bernik et al.

Proof. By [10, Lem. 5.1.11], G is monomial, i.e., each row and column inevery member has precisely one nonzero entry. If G were proved to be finite,then the theorem would follow by the second part of [10, Lem. 5.1.11].

Let us show that indecomposability and the fixed point condition implyfiniteness. We shall show that the set D of all diagonal matrices of G consistsonly of {I} . Let r be the minimal multiplicity of 1 in σ(D) among all D ∈ D .The fixed point condition implies that r ≥ 1. Assume that r < n and chooseD ∈ D with 1 ∈ σ(D) of multiplicity r . With no loss of generality we mayassume that

D = diag (1, 1, . . . , 1, ar+1, ar+2, . . . , an)

for some positive numbers ar+1, ar+2, . . . , an , other than 1. Since G is inde-composable, by [10, Lem. 5.1.5] there is a G0 in G whose (r + 1, 1) entry isnonzero. Then E = G−1

0 DG0 ∈ G has the form

E = diag (ar+1, b2, b3, . . . , bn),

where b2, b3, . . . , bn are some positive numbers. Note that amj �= 1 for allnonzero integers m , and so am1

j �= am2j for all distinct integers m1 , m2 . It

follows that there is an integer m such that amj bj �= 1 for all j = r + 1, . . . , n .Then 1 ∈ σ(DmE) has multiplicity at most r−1, which is a contradiction withthe choice of r . This shows that D = {I} . If G1 , G2 ∈ G are any (monomial)matrices with the same pattern of nonzero entries then G−1

1 G2 ∈ D , whichyields G1 = G2 . Since there are only finitely many patterns we conclude thatG is a finite group.

Corollary 3.3. Let G be an indecomposable group of nonnegative n × nmatrices. Then 1 ∈ σ(G) for every G ∈ G if and only if G is diagonallysimilar to a permutation group.

Corollary 3.4. Let G be a group of nonnegative n × n matrices such that1 ∈ σ(G) for every G ∈ G . Then G is reducible.

Proof. A decomposable group is obviously reducible. If G is indecomposablethen it is simultaneously similar to a permutation group which is reduciblehaving a joint fixed point ( 1 1 · · · 1 )T .

Proposition 3.5. For any odd integer n ≥ 3 there exists an irreduciblesemigroup Sn of n× n nonnegative matrices such that 1 is in σ(G) for everyG ∈ Sn .

Proof. The set

S =

{(a b

c d

); ad − bc = 1, a, b, c, d ≥ 0

}

Bernik et al. OF7

is a semigroup of nonnegative matrices in Sl2(R). Its subset

S◦ =

{(a b

c d

); ad − bc = 1, a, b, c, d > 0

}

is open (in Euclidean topology) in Sl2(R). Then S◦ and thus also S aredense subsets in the Zariski topology of Sl2(C) [9, Prop. 2.5.3, p. 56]. Hence, ifρ: Sl2(C) →Mn(C) is an irreducible rational representation then the restrictionof ρ to S is also irreducible. It is known [5, Prop. 2.2.3, pp. 64–65] that for apositive integer k the 2k -th symmetric power of Sl2(C) is the only irreduciblerational representation in M2k+1(C). Let S2k+1 be the image of the restrictionof this representation to S . If e1, e2 are the standard basis vectors for C

2

and f1, f2, . . . , f2k+1 the standard basis vectors for C2k+1 then we identify fj

with the symmetric product (∨j−1e1) ∨ (∨2k+1−je2), j = 1, 2, . . . , 2k + 1. Asymmetric power of a nonnegative matrix is again a nonnegative matrix. Ifλ, λ−1 are the eigenvalues of A ∈ Sl2(C) then the eigenvalues of the symmetricpower ∨2kA are λ2(k−j) , j = 0, 1, . . . , 2k . This is proved in [5, Prop. 2.2.3,pp. 64–65] for the case of semisimple A . Since in general A is a product ofa semisimple element and a commuting unipotent element, the general casefollows easily. Thus 1 ∈ σ(∨2kA) for any A ∈ Sl2(C). It follows that S2k+1

has all the properties required.

Remark 3.1 and Proposition 3.5 combined yield the following result.

Corollary 3.6. For any integer n ≥ 3 that is not a power of 2 , there existsan irreducible semigroup S of nonnegative n × n matrices such that 1 ∈ σ(S)for every S ∈ S .

Example 3.7. Let T be the set of all totally positive matrices in Sl3(R).A matrix is totally positive if its determinant and all of its minors (of all sizes)are positive [1, 8]. It is an immediate consequence of the Cauchy-Binet formulathat T is a semigroup. If

D =

1 0 0

0 −1 0

0 0 1

and A ∈ M3(R) is totally positive, then it is easy to see that DA−1D is apositive matrix, i.e., a matrix with all entries positive.

Irreducibility will follow by the same argument as in the proof of Propo-sition 3.5: Since T is an open set (in Euclidean topology) in Sl3(R), it is adense subset in the Zariski topology of Sl3(C). If ρ: Sl3(C) → Mn(C) is anirreducible rational representation, then so is also the restriction of ρ to T .

Let ρ be the adjoint representation of Sl3(R) on its Lie algebra sl3(R),as in Example 2.8. In sl3(R) we define a positive cone

P+ = {X ∈ sl3(R); XD ≥ 0} .

OF8 Bernik et al.

For X ∈ P+ and A ∈ T it follows that

ρ(A)X = AXA−1 = A(XD)(DA−1D)D.

Therefore ρ(A)X ∈ P+ and ρ(T ) is an irreducible semigroup of nonnegativematrices in M8(R). Just as in Example 2.8, it follows that 1 is an eigenvalueof ρ(A) with multiplicity at least 2.

Example 3.7 together with Corollary 3.6 imply the following:

Corollary 3.8. For any integer n , except n = 2 and n = 4 there exists anirreducible semigroup S of nonnegative n× n matrices such that 1 ∈ σ(S) forevery S ∈ S .

4. Semigroups of 3× 3 matrices

In this section we give a complete treatment of 3 × 3 case. We begin with aresult that will be improved later in Theorem 4.5.

Lemma 4.1. Let S be an irreducible semigroup of 3× 3 matrices such that1 ∈ σ(S) and detS = 1 for every S ∈ S . Then S is similar to a subsemigroupS0 of the group of all matrices of the form

a2 2ab b2

ac ad + bc bd

c2 2cd d2

,

where complex numbers a , b , c and d satisfy ad − bc = 1 . Additionally, it canbe achieved that S0 contains a diagonal matrix of the form diag (α, 1, 1/α) withα2 �= 1 .

Proof. If we had σ(S) ⊆ {−1, 1} for all S ∈ S , then S would be reducible(see [7] and also [4]). Therefore, we may assume that S contains a diagonalmatrix A = diag (α, 1, 1/α) with α2 �= 1. Pick any S = (sij )

3i,j=1 ∈ S and

denote the principal 2× 2 minors of S by

D1 =

∣∣∣∣∣s22 s23

s32 s33

∣∣∣∣∣ , D2 =

∣∣∣∣∣s11 s13

s31 s33

∣∣∣∣∣ , D3 =

∣∣∣∣∣s11 s12

s21 s22

∣∣∣∣∣ .Since 1 ∈ σ(AkS) for nonnegative integers k , we have

1− (αks11 + s22 + α−ks33) + (α−kD1 +D2 + α

kD3)− 1 = 0,

so

(D1 − s33) + αk(D2 − s22) + α2k(D3 − s11) = 0.

Bernik et al. OF9

Since the determinant ∣∣∣∣∣∣∣1 1 1

1 α α2

1 α2 α4

∣∣∣∣∣∣∣is nonzero, we conclude that

s33 = D1 = s22s33 − s23s32, (1)

s11 = D3 = s11s22 − s12s21, (2)

s22 = D2 = s11s33 − s13s31. (3)

The irreducibility of S will be used as follows. The transformation S �→ (S−1)T

is a trace-preserving homomorphism of irreducible semigroups S and

(S−1)T := {(S−1)T ; S ∈ S}.

By [6, Thm. 4.2], there is a matrix V such that (S−1)T = VSV−1 , and soSTVS = V for all S ∈ S . Since ATVA = V , we obtain that

V =

0 0 u

0 w 0

v 0 0

for some nonzero complex numbers u , v and w . Clearly, we may assume thatw = 1. Now the matrix equation STVS = V gives nine scalar equations

(u+ v)s11s31 + s221 = 0 (4)

us11s32 + s21s22 + vs31s12 = 0 (5)

us11s33 + s21s23 + vs13s31 = u (6)

us12s31 + s22s21 + vs11s32 = 0 (7)

us12s32 + s222 + vs32s12 = 1 (8)

us12s33 + s22s23 + vs32s13 = 0 (9)

us13s31 + s23s21 + vs33s11 = v (10)

us13s32 + s23s22 + vs33s12 = 0 (11)

(u+ v)s13s33 + s223 = 0 (12)

Subtracting (10) from (6) we get

(u− v)(s11s33 − s13s31) = u− v,

and so, by (3),(u− v)(s22 − 1) = 0.

OF10 Bernik et al.

If we had u �= v , then we would conclude that s22 = 1 for every S ∈ S whichwould imply that S is reducible by Lemma 2.2. Therefore, u = v . Writes11 = a

2 , s13 = b2 , s31 = c

2 , s33 = d2 , where the signs of the square roots a ,

b , c and d will be chosen later. From (4) and (12) we get

s221 = −2ua2c2 and s223 = −2ub2d2.

Choose γ such that γ2 = −2u , and choose the signs of the products ac and bdso that

s21 = γac and s23 = γbd.

Now, (6) yieldsua2d2 − 2uabcd+ ub2c2 = u,

so that (ad − bc)2 = 1. If necessary, change the signs of both a and c to getad − bc = 1. From (3) we obtain that s22 = a2d2 − b2c2 = ad + bc . Theequation (2) gives that

s12s21 = s11(s22 − 1) = a2(ad+ bc− (ad− bc)) = 2a2bc.

Similarly, from (1) we conclude that s23s32 = 2d2bc . If abcd �= 0, then it followsthat

s12 =2ab

γand s32 =

2cd

γ, (13)

and so

S =

a2 2ab

γ b2

γac ad+ bc γbd

c2 2cdγ d2

.

We must show that the same is valid if abcd = 0. Consider first the case whenb = 0. Then ad = 1, s22 = 1, and s23 = 0. From (11) we obtain that s33s12 =d2s12 = 0, so that s12 = 0. From (7) it follows that s22s21 + us11s32 = 0, soγac+ ua2s32 = 0 which gives that

s32 = − γcua

=2cd

γ.

So, (13) holds in this case. The case c = 0 can be treated similarly. It remainsto show that (13) is true also in the case when a = 0 (and similarly, if d = 0).In this case we have bc = −1, s22 = −1, and s21 = 0. From (5) we getuc2s12 = 0, so that s12 = 0. From (9) it follows that s22s23 + us13s32 = 0, so−γbd+ ub2s32 = 0 which gives that

s32 =γd

ub=

2cd

γ.

Bernik et al. OF11

Therefore, we have shown that for some γ �= 0 the semigroup S is similarto a subsemigroup S1 of the group of all matrices

a2 2ab

γ b2

γac ad+ bc γbd

c2 2cdγ d2

,

where complex numbers a , b , c and d satisfy ad− bc = 1, and, in addition, S1

contains a diagonal matrix of the form diag (α, 1, 1/α) with α2 �= 1. Finally,define D = diag (γ, 1, γ) and S0 = {DSD−1: S ∈ S1} to complete the proof.

Let S be a semigroup of 3 × 3 matrices such that 1 ∈ σ(S) for everyS ∈ S . By Proposition 2.3 we know that if S contains an element S such thatdetS = 0, then S is reducible. So from now on we assume that detS �= 0for every S ∈ S . We want to show that irreducibility of S actually impliesdetS = 1 for every S ∈ S , so that this condition can be removed fromLemma 4.1.

Lemma 4.2. Let S be a semigroup of 3 × 3 matrices such that 1 ∈ σ(S)for every S ∈ S . Assume that there exists an element A ∈ S that is similar todiag (1, α, β) with α �= 1 , β �= 1 and αβ �= 1 . Then S is reducible.

Proof. We may assume that A = diag (1, α, β). Pick any S = (sij )3i,j=1 ∈ S ,

and denote the principal 2× 2 minors of S by

D1 =

∣∣∣∣∣s22 s23

s32 s33

∣∣∣∣∣ , D2 =

∣∣∣∣∣s11 s13

s31 s33

∣∣∣∣∣ and D3 =

∣∣∣∣∣s11 s12

s21 s22

∣∣∣∣∣ .Since 1 ∈ σ(AkS) for k = 0, 1, 2, 3, we have

1− (s11 + αks22 + β

ks33) + (αkβkD1 + αkD3 + β

kD2)− αk βk detS = 0,

so

(1− s11) + αk(D2 − s22) + βk(D3 − s33) + αkβk(D1 − detS) = 0.

Consider first the case that α �= β . By the assumptions it follows that thedeterminant ∣∣∣∣∣∣∣∣∣

1 1 1 1

1 α β αβ

1 α2 β2 α2β2

1 α3 β3 α3β3

∣∣∣∣∣∣∣∣∣is nonzero. Therefore, s11 = 1 for every S ∈ S , and so S is reducible byLemma 2.2.

OF12 Bernik et al.

If α = β , then again the determinant

∣∣∣∣∣∣∣1 1 1

1 α α2

1 α2 α4

∣∣∣∣∣∣∣is nonzero and the same argument as above applies in this case as well.

Next, we consider the possibility that there exists an element A in Ssuch that detA �= 1 and A cannot be diagonalized.

Lemma 4.3. Let S be a semigroup of 3×3 matrices such that 1 ∈ σ(S) forevery S ∈ S . Assume that there exists an element A ∈ S such that detA �= 1and A cannot be diagonalized. Then S is reducible.

Proof. We may assume that A is equal either to

1 1 0

0 1 0

0 0 α

or to 1 0 0

0 α 1

0 0 α

where α �= 1 in the first case and α2 �= 1 in the second.

We consider the first case and let S ∈ S be arbitrary. Keeping thenotation from the proof of Lemma 4.2 and bearing in mind that 1 ∈ σ(AkS)for k ∈ N we have

1− (s11 + ks21 + s22 + αks33)

+ (D3 + αkD2 + α

kD1 + kαk(s21s33 − s23s31))− αk detS = 0.

Assume now that |α| ≤ 1. After dividing this equation by k and taking the limitas k → ∞ we see that s21 = 0 for every S ∈ S , so S is reducible by Lemma 2.2.If |α| > 1 we replace the semigroup S by the semigroup S−1 = {S−1; S ∈ S}and A by its inverse A−1 to see that S−1 and thus S is reducible in this caseas well.

Since the proof in the second case proceeds in a similar fashion, weomit it.

Finally, we eliminate the possibility that an irreducible semigroup S suchthat 1 ∈ σ(S) for every S ∈ S contains an element similar to diag (1, 1, α),where α2 �= 1.

Bernik et al. OF13

Lemma 4.4. Let S be an irreducible semigroup of 3× 3 matrices such that1 ∈ σ(S) for every S ∈ S . Then detS = ±1 for every S ∈ S . Furthermore, ifdetS = −1 then S is similar to diag (1, 1,−1) .

Proof. Let A ∈ S be an element such that detA = α �= 1. Then A issimilar to diag (1, 1, α), since otherwise S is reducible by Lemmas 4.2 or 4.3.We may assume that A = diag (1, 1, α).

Suppose α2 �= 1 and let S ∈ S be such that detS = 1. Then det(AkS) �=1 for k = 1, 2 and the same argument as above shows that AkS is similar todiag (1, 1, αk) for k = 1, 2. Therefore 1 is a root of the characteristic polynomialof AkS , k = 1, 2, and its derivative. Using the notation from the proof ofLemma 4.2 this yields

(1− s11 − s22 +D3) + αk(−s33 +D1 +D2 − 1) = 0

for k = 1, 2 and

(3− 2s11 − 2s22 +D3) + αk(−2s33 +D1 +D2) = 0

for k = 1, 2. Since the determinant

∣∣∣∣∣1 α

1 α2

∣∣∣∣∣is nonzero, we obtain four equations

s11 + s22 −D3 = 1 (14)

−s33 +D1 +D2 = 1 (15)

2s11 + 2s22 −D3 = 3 (16)

−2s33 +D1 +D2 = 0 (17)

Subtracting (14) from (16) we get s11 + s22 = 2 and similarly subtracting (17)from (15) we get s33 = 1. It follows that for S ∈ S such that detS = 1 we havetr(S) = 3, therefore σ(S) = {1} . This implies that the algebraic multiplicity of1 is at least two for every S ∈ S , but then S is reducible by Proposition 2.1; acontradiction.

We are finally in the position to prove the following result announcedabove.

Theorem 4.5. Let S be an irreducible semigroup of 3 × 3 matrices suchthat 1 ∈ σ(S) for every S ∈ S . Then detS = 1 for every S ∈ S . Moreover, Sis similar to the second symmetric power S2(T ) of some irreducible semigroupT in Sl2(C) .

OF14 Bernik et al.

Proof. By the previous lemma we know that detS = ±1 for every S ∈ Sand if detS = −1, then S is similar to diag (1, 1,−1). Let S0 = {S ∈S; detS = 1} . We now construct a new semigroup R by

R := S0 ∪ {−S; S ∈ S,detS = −1}.

It is easy to see that R is indeed a semigroup. Furthermore, it is irreducibleand for every T ∈ R we have 1 ∈ σ(T ) and detT = 1. By Lemma 4.1 we maytherefore assume that there exists an A ∈ R such that A = diag (α, 1, 1/α),α2 �= 1 and that every T ∈ R is of the form

a2 2ab b2

ac ad+ bc bd

c2 2cd d2

with ad− bc = 1. Observe that A ∈ S0 . Let S ∈ S be such that detS = −1.Then −S ∈ R is similar to diag (−1,−1, 1) and

−S =

a2 2ab b2

ac ad+ bc bd

c2 2cd d2

for some complex numbers a, b, c, d with ad−bc = 1. Since tr(−S) = (a+d)2−1 = −1 we have a2 = d2 . But det(AS) = −1 as well, and the same argumentyields αa2 = 1

αa2 . Since α2 �= 1 we have a2 = 0. Therefore we have shown

that every S ∈ S such that detS = −1 is necessarily of the form

0 0 −b2

0 1 0

−c2 0 0

with bc = −1. Assume now that such an element S ∈ S exists. Since R isirreducible, there exists an element T = (tij ) ∈ R such that t12 �= 0. It followsthat t11, t12 and t13 are all nonzero. Therefore in the product TS the (1, 1)entry is nonzero. But on the other hand det(TS ) = −1, so by the discussionabove this entry is zero. This contradiction shows that detS = 1 for everyS ∈ S .

Now, by Lemma 4.1 we may assume that S is a subsemigroup of thegroup of all matrices of the form

a2 2ab b2

ac ad+ bc bd

c2 2cd d2

,

Bernik et al. OF15

where a , b , c and d satisfy ad− bc = 1. Let {e1, e2} be the standard basis of

C2 . If A =

(a b

c d

)then

a2 2ab b2

ac ad+ bc bd

c2 2cd d2

is the matrix of S2(A) in the basis {e21, e1e2, e22} of S2(C2). Let T ⊂ Sl2(C)be the set of all matrices A such that S2(A) ∈ S . Since S2 is a homomorphismand S an irreducible semigroup it follows that T is an irreducible semigroupin Sl2(C). This completes the proof.

5. An affirmative result under a weak dominance condition

In this section we prove a reducibility result assuming that the semigroup underconsideration contains an element such that the eigenvalue 1 has minimal mul-tiplicity and is dominant. As before, m(S) denotes the (algebraic) multiplicityof 1 in σ(S).

Theorem 5.1. Le S be a semigroup of n×n matrices with m = min{m(S) :S ∈ S} ≥ 1 . If 1 is dominant in σ(A) for some A ∈ S with m(A) = m , thenS is reducible.

Proof. With no loss of generality, we may assume that

A =

(Im +N 0

0 C

),

where c := ‖C‖ < 1 and N is an m×m nilpotent matrix. For a general S ∈ Sin the corresponding block form,

S =

(S11 S12

S21 S22

),

we consider the product

SAk =

(S11(Im +N)k S12C

k

S21(Im +N)k S22Ck

).

We treat two cases separately. Suppose first that N = 0. Then

(S11 0

S21 0

)

belongs to the closure S in the operator norm topology of Mn(C). Since wecan assume with no loss of generality that S = S (because Sn → S implies that

OF16 Bernik et al.

the multiplicity of 1 in σ(S) is no less than the minimum of the correspondingmultiplicity in σ(Sn)), we conclude that

σ(S11) = {1}.

Hence the trace tr(S11) is equal to m , or tr(SP) = m for all S ∈ S , where

P =

(Im 0

0 0

). Thus S is reducible by Lemma 2.2.

Next assume that N �= 0. Let r be such that Nr �= 0, Nr+1 = 0. Weshall show that S11N

r is nilpotent for all S ∈ S . For each positive integer kwrite

(SAk − I)n =

(Pk Qk

Rk Sk

),

and prove by induction on n that

‖Pk‖ =(k

r

)n

‖(S11Nr)n‖+O(knr−1),

‖Qk‖ = O(k(n−1)rck) , ‖Rk‖ = O(knr),‖Sk − (−I)n‖ = O(k(n−1)rck).

It follows from the last equation that there exists k0 such that Sk is invertiblefor all k ≥ k0 . Since m(SAk) ≥ m for all k , the rank of (SAk − I)n is at mostn−m . For any k ≥ k0 this rank is equal to the rank of the matrix

(Pk Qk

Rk Sk

)(Im 0

−RkS−1k S−1

k

)=

(Pk −QkS

−1k Rk QkS

−1k

0 In−m

).

It follows that Pk=QkS−1k Rk for all k≥k0 . Since ‖QkS

−1k Rk‖=O(k(2n−1)rck),

we conclude that (S11Nr)n = 0 as desired.

Setting

M =

(Nr 0

0 0

)

we now have tr(SM ) = 0 for all S ∈ S , so that S is reducible again byLemma 2.2.

Acknowledgment

R. Drnovsek, T. Kosir and M. Omladic were supported in part by the Ministryof Education, Science, and Sport of Slovenia and H. Radjavi by the NSERC ofCanada.

Bernik et al. OF17

References

[1] Ando, T., Totally positive matrices, Lin. Alg. Appl. 90 (1987), 165–219.

[2] Bernik, J., R. Drnovsek, T. Kosir, T. Laffey, G. MacDonald, R. Meshulam,M. Omladic, and H. Radjavi, Groups and semigroups of matrices witheigenvalue one, in preparation.

[3] Bernik, J. and J. Okninski, On semigroups of matrices with eigenvalue onein small dimensions, preprint.

[4] Cigler, G., On matrix groups with finite spectrum, Lin. Alg. Appl. 286(1999), 287–295.

[5] Goodman, R. and N. R. Wallach, “Representations and Invariants of theClassical Groups”, Cambridge University Press, 1998.

[6] Hladnik, M., M. Omladic, and H. Radjavi, Trace-preserving homomor-phisms of semigroups, J. Funct. Anal. (to appear).

[7] Kaplansky, I., The Engel-Kolchin theorem revisited, in H. Bass, P. J. Cas-sidy, and J. Kovacic (eds.), “Contributions to Algebra”, Academic Press,New York, 1977, 233–237.

[8] Karlin, S., “Total Positivity”, Vol. 1, Stanford University Press, 1968.

[9] Margulis, G. A., “Discrete Subgroups of Semisimple Lie Groups”, Springer-Verlag, Berlin, Heidelberg, New York, 1991.

[10] Radjavi, H. and P. Rosenthal, “Simultaneous Triangularization”, Springer-Verlag, Berlin, Heidelberg, New York, 2000.

Department of MathematicsUniversity of LjubljanaJadranska 19, 1000 Ljubljana, Sloveniajanez.bernik@fmf.uni-lj.siroman.drnovsek@fmf.uni-lj.sitomaz.kosir@fmf.uni-lj.simatjaz.omladic@fmf.uni-lj.si

Department of Mathematics and StatisticsDalhousie UniversityHalifax, Nova Scotia, Canada B3H 3J5radjavi@mathstat.dal.ca

Received August 26, 2002and in final form March 7, 2003Online publication April 22, 2003