Transformation Groups, Vol. 5, No. 1, 2000, pp. 1{15 c Birkh�auser Boston (2000)DEPTH OF MODULAR INVARIANT RINGSH. E. A. CAMPBELLDepartment of Mathematicsand StatisticsQueen's UniversityKingston, OntarioCanada K7L 3N6eddy@mast.queensu.caI.P. HUGHESDepartment of Mathematicsand StatisticsQueen's UniversityKingston, OntarioCanada K7L 3N6hughesi@mast.queensu.caG. KEMPERIWRUniversit�at HeidelbergIm Neuenheimer Feld 36869 120 HeidelbergGermanyGregor.Kemper@iwr.uni-heidelberg.deR.J. SHANKDepartment of Mathematicsand StatisticsQueen's UniversityKingston, OntarioCanada K7L 3N6shank@mast.queensu.caD.L. WEHLAUDepartment of Mathematics andComputer ScienceRoyal Military CollegeKingston, OntarioCanada K7K 5L0wehlau@rmc.caAbstract. It is well-known that the ring of invariants associated to a non-modular repre-sentation of a �nite group is Cohen{Macaulay and hence has depth equal to the dimensionof the representation. For modular representations the ring of invariants usually fails to beCohen{Macaulay and computing the depth is often very di�cult. In this paper1 we obtain asimple formula for the depth of the ring of invariants for a family of modular representations.This family includes all modular representations of cyclic groups. In particular, we obtain anelementary proof of the celebrated theorem of Ellingsrud and Skjelbred [6].IntroductionWe consider a faithful representation of a �nite group G on a vector space V ofdimension n over a �eldK. The representation is said to be modular if the characteristicof K divides the order of the group G. Otherwise it is called non-modular. The actionReceived July 28, 1998. Accepted May 10, 1999.1Research partially supported by grants from ARP and NSERC

2 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAUof G on V induces an action of G on the dual V � given by �(f) = f ���1 for f 2 V � and� 2 G. This action extends to an action by algebra automorphisms on the symmetricalgebra R = S(V �). The ring of invariants isRG = ff 2 S(V �) j �(f) = f 8� 2 Gg:Let RG+ denote the augmentation ideal of RG, i.e., the ideal generated by the homoge-neous invariants of positive degree. A homogeneous sequence of elements a1; a2; : : : ; arin RG+ is regular on RG if for every i � r and every m 2 RG, m � ai 2 (a1; : : : ; ai�1)RGimplies m 2 (a1; : : : ; ai�1)RG. Here (a1; : : : ; ai�1)RG denotes the ideal in RG generatedby a1; : : : ; ai�1. For an ideal I in RG with I � RG+, we write depthI(RG) for the lengthof a maximal regular sequence on RG all of whose elements lie in I . By depth(RG)we mean depthRG+ (RG). The ring RG is Cohen{Macaulay if depth(RG) = dim(V ). Ahomogeneous sequence of elements a1; a2; : : : ; an in RG+ is a homogeneous system of pa-rameters for RG if RG is a �nitely generated module over the subalgebra generated byfa1; : : : ; ang. The ring RG is Cohen{Macaulay if and only if every homogeneous systemof parameters is a regular sequence. We write ht(I) for the height of an ideal I of RG.We note that, since RG is a �nitely generated subalgebra of R, the height of I is thedi�erence between the Krull dimension of RG and the Krull dimension of RG=I (seeBruns and Herzog [4, Theorem A.16]). We refer the reader to Bruns and Herzog [4] forstandard results on height, depth and Cohen{Macaulay rings.For any non-modular representation the ring of invariants is a Cohen{Macaulay ringand hence has depth n. For almost all modular representations the ring of invariantsis not Cohen{Macaulay (see Campbell et al. [5] and Kemper [10]). There has beensigni�cant recent interest in the depth of RG for a modular representation (see, forexample, Bourguiba and Zarati [3] and Smith [16]).Throughout the paper we will assume that K is a �eld of positive characteristic p.For � 2 G, we write V � for the set of vectors v 2 V such that �(v) = v. For a subgroupH � G we de�ne V H = fv 2 V j �v = v 8� 2 Hg.Choose a Sylow p-subgroup P � G and consider the subgroup G0p � G generatedby P and the commutator subgroup G0 of G. Since G0 � G0p, G0p is normal. ThereforeG0p contains every Sylow p-subgroup of G. In fact G0p is the smallest normal subgroupwhose quotient group is abelian and of order coprime to p. We say that G is shallow ifthere exists a normal subgroup N of G with index p such that f� 2 G0p j dim(V �) >dim(V G0p)g � N: If N is a subgroup satisfying the required hypotheses, we say that Gis shallow with respect to N . Note that shallowness is a property of a linear group, i.e.,a subgroup of a general linear group GL(V ), and not of an abstract group. Given anabstract group, we shall also speak of shallow representations of this group. We shalllook at examples of shallow groups below and in Section 1.The main goal of this paper is to prove the following theorem.Theorem 1. Suppose that G is shallow and dim(V P )+2�dim(V ). Then depth(RG)=dim(V P ) + 2:Moreover, we will show that the following statement holds without the assumptionof shallowness.

DEPTH OF MODULAR INVARIANT RINGS 3Theorem 2. If dim(V P )+2 � dim(V ), then depth(RG) � dim(V P )+2: On the otherhand, if dim(V P ) + 2 > dim(V ), then depth(RG) = dim(V ).We note that since the representation of G is faithful, dim(V P )<n and if dim(V P )=n� 1, then Theorem 2 applies and depth(RG) = n.If G is an abelian group with a non-trivial cyclic Sylow p-subgroup P , then G0p = P ,and there is only one subgroup N of index p. In fact, in this case, each element of Pwhich does not generate P , lies in N . Hence G is shallow. This proves the followingcorollary.Corollary 3. Let G be an abelian group with a cyclic Sylow p-subgroup P . Supposethat dim(V P ) + 2 � dim(V ). Then depth(RG) = dim(V P ) + 2:The result of Ellingsrud and Skjelbred [6] on cyclic p-groups is a special case of thecorollary. Notice that the abelian groups with cyclic Sylow p-subgroup provide a classof groups for which every linear representation in characteristic p is shallow. We willsee that these are the only \universally shallow" groups (see Theorem 9).Remark 4. (a) The subset f� 2 G0p j dim(V �) > dim(V G0p)g � G0p generates a normalsubgroup of G0p. Hence if G is a p-group, then G is shallow if and only if it is notgenerated by the elements � 2 G with dim(V �) > dim(V G).(b) Suppose that G is shallow and consider the diagonal action of G on the directsum mV = �mi=1V of m copies of V . Then G is also shallow as a linear group on mV .In Section 1, we will look at some examples of shallow groups and classify those groupsfor which all representations are shallow. In Section 2 we investigate the geometryassociated to ideals in RG. In particular we describe the variety associated to the imageof the relative transfer for a normal subgroup (Theorem 12). Section 3 is devoted toconstructing regular sequences in RG. The proof of Theorem 1 is contained in Section 4and forms the heart of the paper. The results of Section 2 and Section 3 are used inthis proof. In Section 5 we discuss the Buchsbaum property and prove that, if G isa p-group, then RG is Cohen{Macaulay if and only if it is Buchsbaum (Theorem 26,Nakajima [13, Proposition 3.7]).We thank the referees for their helpful suggestions, for pointing out an error in thepresentation of Example 7, and for prompting us to think about groups for which allrepresentations are shallow.1. Examples of shallow groupsWe have seen that every abelian group with a non-trivial cyclic Sylow p-subgroup isshallow. In this section we shall prove that these are the only groups for which everyrepresentation is shallow. Before doing so, we present a few additional examples ofshallow groups.Example 5. Let G be a non-zero, �nite subgroup of the additive group of K. ThusG is elementary abelian. Let U = K2 be the two-dimensional KG-module with theG-action given by a 7! �1 a0 1� for a 2 G � K:

4 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAUThe representation of G on U is shallow by Remark 4(a), since the identity element is theonly � 2 G with dim(U�) > dim(UG). Now take integers n1; : : : ; nk with 0 < ni < p.We claim that the representation onV := Sn1(U)� � � � � Snk(U)is also shallow, where Sni(U) is the ni-th symmetric power of U . Let � 2 G be anon-identity element. The dimensions of the �xed spaces Sm(U)� are encoded in theHilbert series, which is known to beH(S(U)� ; t) = 1(1� t)(1� tp)(see, Smith [15, Example 5.6.3]). Since ni < p, we have 1 = dimK (Sni(U)�) �dimK �Sni(U)G� � 1; and equality follows. Hence the identity is the only � 2 G withdim(V �) > dim(V G), and the claim follows. Theorem 1 now yields depth(RG) = k + 2if n1 + � � � + nk � 2. This example is signi�cant since if G is the additive group ofFq � K, then G is the Sylow p-subgroup of SL2(q), and we are looking at the actionof SL2(q) on binary forms of certain degrees (to use the language of classical invarianttheory), and direct sums thereof.Example 6. (a) Assume that the �nite �eld Fq2 is contained in K and take G to bea Sylow p-subgroup of the special unitary group SU3(q). Choosing the Hermitian formon F3q2 as � ((x1; x2; x3); (y1; y2; y3)) = xq1y3 + xq2y2 + xq3y1;we can takeG = n �a;b := 241 a b0 1 �aq0 0 1 35 j a; b 2 Fq2 with aq+1 + b+ bq = 0 o(see Taylor [19, p. 121]). It is clear that if dim(V �a;b) > dim(V G), then a = 0. But the�0;b form a proper subgroup of G, hence G is shallow by Remark 4(a). By Remark 4(b)it follows that every representation of G on a direct sum of k copies of the naturalmodule V is also shallow. Thus Theorem 1 yields depth �S(kV �)G� = k + 2:(b) A further class of non-abelian shallow groups are the Sylow 2-subgroups of theSuzuki groups. Indeed, assume that with q := 22m+1 the �nite �eld Fq is contained inK (and in particular p = 2). Then as a Sylow 2-subgroup of the Suzuki group Sz(q) wecan choose G := n �a;b := 26641 a b a2�(a) + ab+ �(b)0 1 �(a) a�(a) + b0 0 1 a0 0 0 1 3775 j a; b 2 Fq o ;where � is an automorphism of Fq such that �2(x) = x2 for all x 2 Fq (see Huppert andBlackburn [8, XI, Lemma 3.1 and Theorem 3.3]). Clearly if dimK(V �a;b) > dimK(V G)

DEPTH OF MODULAR INVARIANT RINGS 5then a = 0. But the �0;b form a proper subgroup of G, hence again G is shallow. Asabove we obtain depth �S(kV �)G� = k + 2 for the invariants of k copies of the naturalmodule. Even for k = 1, the invariant ring is not Cohen{Macaulay. It is not clearwhether the invariant ring of the full Suzuki group with the natural representation isCohen{Macaulay. By using MAGMA (see Kemper and Steel [11] and Bosma et al. [2]),we were able to compute depth �S(V �)Sz(2)� = 3. The other Suzuki groups are too largefor the computation.Example 7. Landweber and Stong [12, Appendix] considered linear groups G withthe property that every non-identity element is a transvection. By Theorem 8.2.13 inSmith [15], G is an elementary abelian p-group and either V G or (V �)G has codimensionone. If V G has codimension one, then dim(V �) = dim(V G) for all non-identity � 2 Gand hence G is shallow. If (V �)G has codimension one, then G is not necessarily shallow.Suppose that V G has codimension one. Then there exists a basis for V such that, withrespect to this basis, all group elements are represented by matrices of the form26666641 0 � � � 0 �10 1 0 �2... . . . ...1 �n�10 � � � 0 13777775 with �1; : : : ; �n�1 2 K:By Remark 4(b), the action of G on the direct sum of m copies of V is still shallow. Form � 2, Theorem 1 yields the equation depth(S((mV )�)G) = m(n � 1) + 2: Note thatfor m � 2, the ring of invariants is Cohen{Macaulay. In fact for m = 1 the invariantsare a polynomial ring (see Smith [15, Theorem 8.2.13 ]). However for m > 2, thedepth is less than the dimension of the representation, so the ring of invariants is notCohen{Macaulay (compare with Corollary 21).The next goal is to classify the (abstract) groups for which all representations areshallow. We need a small lemma, which is also worth noting in its own right.Lemma 8. If G � GL(V ) is a shallow group, then for any Sylow p-subgroup P ,V G0p = V P :Proof. Since G is shallow, there is a normal subgroup N of index p such that f� 2 G0p jdim(V �) > dim(V G0p)g � N: Since P � G0p, we certainly have V G0p � V P . Assume, byway of contradiction, that V G0p $ V P . Then for every � 2 P we have a strict inclusionV G0p $ V P � V � : Thus P � N and [G : N ] is relatively prime to p, giving the requiredcontradiction.Theorem 9. Let G be a �nite group and p a prime. Then the following statements areequivalent:(a) Every faithful linear representation of G over a �eld of characteristic p is shallow.(b) The regular representation of G over a �eld of characteristic p is shallow.(c) G is abelian with a non-trivial cyclic Sylow p-subgroup.

6 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAUProof. We have already seen the implication \(c) ) (a)" in the Introduction. \(a) )(b)" is trivial, hence we only have to show that (b) implies (c). So assume the regularrepresentation V = Vreg is shallow. For the regular representation, it is easy to see thatfor a subgroup H � G, we have dim(V H) = [G : H ]: (1)Indeed, it is true for any permutation representation that the dimension of the �xedspace equals the number of orbits. Assume that G0p is not cyclic. Then, by (1), forevery � 2G0p, dim(V �) = [G : h�i]> [G : G0p] = dim(V G0p): Hence by the hypothesis ofshallowness �2N . Thus G0p�N , contradicting the condition that N has index p in G.Therefore G0p has to be cyclic. Moreover, by (1) and Lemma 8 we obtain [G : G0p] =dim(V G0p) = dim(V P ) = [G : P ] for a Sylow p-subgroup P � G. Hence G0p = P . Wetherefore have a normal, cyclic Sylow p-subgroup P = h�i �= Zpm with m a positiveinteger. Therefore by the theorem of Schur{Zassenhaus G = h�i o H is a semidirectproduct with H �= G=G0P abelian and p - jH j. G has only one subgroup N of index p,which is N = h�pioH; and by hypothesis this must be normal in G. Take � 2 H . Then����1 = �k with an integer k. The element ����1 must lie in N , but ����1 = �1�k�;so k � 1 mod p. It follows that kpm�1 � 1 mod pm, so �pm�1 commutes with �. Butas an element in H , the order of � is coprime to p, and therefore � itself commutes with�. Since this is true for every � 2 H , we obtain G = h�i �H; as required.After having shed some light on the property of shallowness in this section, we shallnow embark on the proof of the main Theorem 1.2. Geometric preliminariesLet K denote the algebraic closure of K. For a vector space U over K, de�neU = U K K. We identify U with the subset U K K � U . For any subset X of R,de�ne V(X) = fv 2 V j f(v) = 0 8f 2 Xg � V :Proposition 10. If I is an ideal in RG, then V(I) = V(I).Proof. Since I � I , we have V(I) � V(I). On the other hand, any f 2 I can be writtenas a linear combination of elements from I with coe�cients from K. Thus if v 2 V(I),it follows that f(v) = 0. Therefore V(I) � V(I).Proposition 11. Suppose I and J are ideals in RG. Then V(I) = V(J) if and only ifpI = pJ .Proof. This is essentially the Hilbert Nullstellensatz. For a �nite group G, the a�nevariety associated to RG is the orbit space V =G. Corresponding to each ideal I inRG is a subvariety of V =G. The set V(I) is the preimage in V of the subvariety ofV =G associated to I . By the Hilbert Nullstellensatz, pI = pJ if and only if thecorresponding varieties are equal. We refer the reader to Zariski and Samuel [20, Ch.VII, x3] and Serre [14, Ch. III, x12] for details.Suppose H is a subgroup of G and let R be a set of left coset representatives for H inG. The relative transfer, TrGH(f) =P�2R �(f), is an RG-module homomorphism fromRH to RG. Therefore the image of the relative transfer IGH is an ideal in RG. If x is an

DEPTH OF MODULAR INVARIANT RINGS 7element of a G-set then the isotropy subgroup of x is Gx = f� 2 G j �(x) = xg and theorbit of x is the set Gx = f�(x) j � 2 Gg. The following theorem is a special case ofFleischmann [7, Theorem 5.9]. We include a proof for completeness.Theorem 12. If H is a normal subgroup of G, then V(IGH) = fv 2 V j p divides[Gv : Hv]g:Proof. Extend the action of G on V to a linear action on V . Using this action, theimage of TrGH applied to RH is IGH . Thus it follows from Proposition 10 that we mayassume that K is algebraically closed.First suppose that p divides [Gv : Hv] for a point v 2 V . Then there exists � 2 Gvsuch that �H has order p in G=H . Let hH; �i denote the subgroup generated by H and� . For any f 2 RH we have (TrhH;�iH (f))(v) = Pp�1i=0 f(��i(v)) = Pp�1i=0 f(v) = 0; and,since TrGH = TrGhH;�i �TrhH;�iH , it follows that �TrGH(f)� (v) = 0. Thus v 2 V(IGH ).Conversely, take v 2 V with p - [Gv : Hv] and choose f 2 R such that for � 2 G,f(�(v)) = 1 if �(v) lies in Hv, and f(�(v)) = 0 otherwise. Set g = Q�2H �(f) 2 RH .Then for � 2 G we have g(�(v)) = (1 if �(v) 2 Hv;0 otherwise:However �(v) 2 Hv if and only if � 2 H � Gv = Gv �H . Hence (TrGH(g))(v) = jf�H 2G=H j �H � GvHgj = jGv=Hvj 6= 0; so v 62 V(IGH), which completes the proof.We include the following example to illustrate the fact that in the preceding theoremthe hypothesis that H is a normal subgroup is necessary.Example 13. Let G be the symmetric group �3 and consider the usual three dimen-sional permutation representation. Let H be the subgroup generated by the transposi-tion (1; 2), take p = 2 and let v = (0; 1; 1). Since [G : H ] is coprime to p, the relativetransfer is surjective. In particular �2 = x1x2+x1x3+x2x3 = TrGH( 13�2) 2 IGH and, since�2(v) = 1, we conclude that v is not in V(IGH). However, v is �xed by the transposition(2; 3) and (2; 3) is not in H . Thus 2 divides [Gv : Hv ].Corollary 14. If H is normal and [G : H ] = p, then V(IGH ) = [�2GnHV � .Proof. If [G : H ] = p, then for any v 2 V , [Gv : Hv ] is either p or 1. If v 2 V(IGH ),then [Gv : Hv] = p and v 2 V � for some � 2 G n H . Conversely, if v 2 V � for some� 2 G nH , then [Gv : Hv ] 6= 1. Therefore [Gv : Hv ] = p and v 2 V(IGH ).If U is a subspace of V , then we de�ne the annihilator U? = f' 2 V � j '(v) = 0 8v 2Ug: Let (U?) denote the ideal in R generated by U? and de�ne IRG(U) = (U?)\RG.For f 2 R we write NG(f) =Q�2G �(f) 2 RG.Proposition 15. If U is a KG-submodule of V , then V(IRG(U)) = U .Proof. Using Proposition 10, we see that extending the �eld doesn't change the varietyand so we may assume that K is algebraically closed. It is easy to see that U �V(IRG(U)). To complete the proof we need only show that any vector in V n U is alsoin V n V(IRG(U)).

8 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAUSuppose v 2 V n U . The orbit of v is a �nite set and, since U is a KG-module, thisset is disjoint from U . Therefore, since K is in�nite, there exists a linear functional` 2 V � such that ` is zero on U and non-zero on each point in the orbit of v. Letf = NG(`). Clearly f 2 IRG(U) and it is not hard to show that f(v) is non-zero.Therefore v 2 V n V(IRG(U)) as required.We note that GU � V(IRG(U)) and so, if U is not aKG-module, V(IRG(U)) properlycontains U .Corollary 16. If H is a normal subgroup of G, then V(IRG(V H)) = V H .3. Constructing regular sequencesSuppose that U is a subspace of V . We write S(U?) for the symmetric algebra ofthe annihilator of U . If U is a KG-module, then so is U? and S(U?)G is a subalgebraof RG. As in Section 2, for f 2 R we write NG(f) =Q�2G �(f) 2 RG.Proposition 17. Suppose U � V is a KG-submodule of dimension 1 and choose x 2V � nU?. Let f1; : : : ; fm be a regular sequence on RG whose elements lie in S(U?)G+ �RG+. Then f1; : : : ; fm; NG(x) is also a regular sequence on RG.Proof. Let A denote the G-stable algebra S(U?). Then R = A[x] is the polynomialring over A in the variable x. Write f = NG(x) and assume thatg1f1 + � � �+ gmfm + gf = 0 (2)with g; gi 2 RG. We must show that g lies in the ideal generated by f1; : : : ; fm in RG.View f and gi as polynomials in x. Since the leading coe�cient of f is an element ofK, we may apply the division algorithm to get gi = qif + ri with qi; ri 2 A[x] anddegx(ri) < degx(f). Suppose � 2 G. Observe that applying � does not change thex-degree of a polynomial. Hence degx((� � 1)ri) < degx(f). However, 0 = (� � 1)gi =(� � 1)qi � f + (� � 1)ri; so f divides (� � 1)ri, implying that (� � 1)ri = 0. Thereforeri 2 RG and, consequently, qi 2 RG. Setting h = r1f1 + � � �+ rmfm, we rewrite (2) as(q1f1 + � � �+ qmfm + g)f + h = 0: (3)Since fi 2 A, we have degx(h) < degx(f). Thus h = 0. Now dividing (3) by f showsthat indeed g 2 (f1; : : : ; fm)RG.By iteration, we now obtain the following corollary. We remind the reader that, if Uis a subspace of V , then IRG(U) = (U?) \RG.Corollary 18. Let U = V G0p and m = dim(U). Suppose that f1; : : : ; fd is a regularsequence for RG whose elements lie in S(U?)G+. Then after a possible extension of Kthere exist g1; : : : ; gm 2 RG such that:(a) the images of g1; : : : ; gm form a homogeneous system of parameters of the quo-tient ring RG=IRG(U),(b) f1; : : : ; fd; g1; : : : ; gm is a regular sequence for RG.Moreover, the degree of the necessary extension of K is bounded above by '(jG=G0pj),where ' is the Euler totient function.

DEPTH OF MODULAR INVARIANT RINGS 9Proof. G acts on U as an abelian group of order coprime to p. Hence after addingenough roots of unity to K (which is an extension of degree � '(jG=G0pj)), we obtain abasis e1; : : : ; em of U such that the one-dimensional subspacesKei areKG-submodules.Extend the ei to a basis of V , take the dual basis x1; : : : ; xn and set gi = NG(xi) fori = 1; : : : ;m. To see that g1; : : : ; gm form a system of parameters of RG=IRG(U), itis enough to show that they form a system of parameters of R=(U?), since this is anintegral extension of RG=IRG(U). But writing �(ei) = a�;iei with a�;i 2 K, we see that(� � a�1�;i )xi 2 U?, hence gi � aixjGji mod (U?) with ai 2 K n f0g. But, together withU?, the xi generate V �.We show by induction on i that f1; : : : ; fd; g1; : : : ; gi is regular on RG. In fact, byassumption, f1; : : : ; fd 2 S((Kei)?), and for j < i and � 2 G we have (�(xj ))(ei) = 0,hence also gj 2 S((Kei)?). So it follows from Proposition 17 that f1; : : : ; fd; g1; : : : ; giis regular on RG. 4. The proof of Theorems 1 and 2Suppose that N is a normal subgroup of G with index p, and choose an element� 2 G nN . Consider the idealJ = (� � 1)RN \ RG = f�(g)� g j g 2 RNg \ RG � RG:Lemma 19. The ideal J is non-zero and independent of the choice of �.Proof. To see that the ideal J is independent of the choice of �, we �rst note thatelements in the same coset of N produce the same ideal, and thus it su�ces to considerthe non-trivial coset representatives �i for 1 � i � p � 1. Also, since N is normal, �induces an automorphism on RN . Next notice that �i � 1 = (�� 1)(1+�+ � � �+ �i�1)and thus (�i�1)RN � (��1)RN . Conversely, choosing j such that ij � 1 (mod p), wehave (� � 1)RN = ((�i)j � 1)RN = (�i � 1)(1 + �i + �2i + � � �+ �i(j�1)) � (�i � 1)RN :Thus (�i � 1)RN = (� � 1)RN as required.We now show that J is non-zero. First observe that the kernel of ��1 acting on RNis RG. Since RG is not equal to RN , the kernel is a proper subset and we may chooseg 2 RN such that (� � 1)g 6= 0. However (� � 1)pg = (�p � 1)g = 0. Choose i so that(� � 1)ig 6= 0 and (� � 1)i+1g = 0. Then (� � 1)ig is a non-zero element of J .Proposition 20. depthJ (RG) = minf2; ht(J )g:Proof. Assume, by way of contradiction, that there are f1; f2; f3 2 J which form aregular sequence for RG. By de�nition of J , there exist gi 2 RN such that �(gi)�gi = fifor i = 1; 2; 3, and we have0 = ������f1 f2 f3f1 f2 f3g1 g2 g3������ = u23 � f1 + u31 � f2 + u12 � f3 with uij = ����fi fjgi gj���� :Observe that uij 2 RG, since�(uij) = ���� fi fjgi + fi gj + fj���� = uij :

10 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAUThe regularity of f1; f2; f3 implies that u12 must lie in the ideal in RG generated byf1 and f2, so g2f1�g1f2 = u12 = h1f1+h2f2 with h1; h2 2 RG, which yields (g2�h1)f1=(g1+h2)f2: However f1; f2 is a regular sequence on RG and hence also a partialhomogeneous system of parameters for RG. Since R is an integral extension of RG, anypartial homogeneous system of parameters for RG is a partial homogeneous system ofparameters for R. Moreover R is Cohen{Macaulay and so f1; f2 is a regular sequence onR. Therefore f1 and f2 are coprime as polynomials in R. Hence f1 divides g1 + h2, sog1 = q �f1�h2 with q 2 R: Since f1 and h2 are invariants, we obtain f1 = (��1)g1 =f1 � (� � 1)q; or (� � 1)q = 1; which is impossible.This shows that depthJ (RG) � 2. Also clearly depthJ (RG) � ht(J ). On the otherhand, if ht(J ) = 1, then there exists f 2 J which forms a regular sequence for RG,so depthJ (RG) = 1. If ht(J ) � 2, then there exist homogeneous f1; f2 2 J such thatht(f1; f2) = 2. Thus a relation g1f1 + g2f2 = 0 with gi 2 RG implies that f1 dividesg2 in R. However, since f1 and g2 are both invariant, we conclude that f1 divides g2 inRG. This shows that f1; f2 is regular for RG. Hence depthJ (RG) = 2 if ht(J ) � 2.As a consequence of Proposition 20 we get the following generalization of Campbellet al. [5, Theorem 1.2]. This result also follows from Kemper [10, Theorem 2.3].Corollary 21. If G contains a normal subgroup N of index p, then the ring of vectorinvariants S((mV )�)G is not Cohen{Macaulay for m � 3.Proof. By Lemma 19, there exists a homogeneous g 2 S(V �)N such that f = (��1)g 2S(V �)G nf0g. Let gi; : : : ; gm be the elements corresponding to g in S((mV )�)N . De�nefi = (�� 1)gi. The sequence f1; : : : ; fm is a partial homogeneous system of parametersfor S((mV )�)G lying inside J . From Proposition 20, depthJ (RG) � 2. Therefore, ifm � 3, we have a partial homogeneous system of parameters which fails to be a regularsequence, so RG is not Cohen{Macaulay.We remind the reader that IGN denotes the image of the relative transfer and thatIRG(U) = (U?) \ RG.Proposition 22. pJ =qIGN = T�2GnN IRG(V � ):Proof. Let J = \�2GnNIRG(V � ). Note that J is a radical ideal. Thus by Proposition 11it is su�cient to show that V(J ) = V(IGN ) = V(J).Suppose that f 2 J and v 2 V � for some � 2 G n N . Then, using Lemma 19,f = (� � 1)(h) for some h 2 RN . Thus f(v) = h(��1(v)) � h(v) = 0. Hence v 2 V(J )and V(J) � V(J ).Since [G : N ] = p, we may use Corollary 14 to getV(IGN ) = [�2GnNV � � [�2GnNV(IRG(V � )) = V(\�2GnNIRG(V � )) = V(J):Suppose f 2 IGN . Then for some h 2 RN , f = TrGN(h) = Pp�1i=0 �i(h) = (� � 1)p�1(h).Thus f 2 J and IGN � J . Therefore V(J ) � V(IGN ) � V(J).Now we consider the ideal I = IRG(V G0p) � RGand bring the hypothesis that G is shallow into play.

DEPTH OF MODULAR INVARIANT RINGS 11Proposition 23. Suppose that G is shallow with respect to N . Then pI = pJ anddepthI(RG) = minf2; ht(I)g:Proof. We �rst prove that pI = pJ . By Proposition 11 it is su�cient to show thatthe corresponding varieties are equal. From Corollary 16 we know that V(I) = V G0p .From Proposition 22 and Corollary 14 we know that V(J ) = [�2GnNV � .Let � be an arbitrary element in G nN . The order of � is kpm for some k relativelyprime to p and some m � 0. However, [G : N ] = p so m > 0 and �k 62 N . Notethat G0p contains all of the p-Sylow subgroups of G and, therefore, �k 2 G0p. HenceV G0p � V �k . Since G is shallow with respect to N and �k 62 N , we have dim(V �k ) �dim(V G0p). Therefore V �k = V G0p . However V � � V �k � [�2GnNV � . Hence V � �V G0p � [�2GnHV � . Since this is true for an arbitrary � 2 G nN , the varieties are equal.Using Proposition 20, and the fact that an ideal and its radical share a commondepth and height, gives depthI(RG) = minf2; ht(I)g.We can now give a proof of the theorems stated in the Introduction.Proof of Theorem 1 and Theorem 2. First we remark that the depth of RG does notchange if the ground �eld K is extended. This follows from the characterization ofdepth in terms of Ext-functors (see, for example, Benson [1, Theorem 4.4.3]).We �rst prove the inequalitydepth(RG) � minfdim(V G0p) + 2; dim(V )g: (4)As above, we write U = V G0p . Let d = minf2; dim(U?)g. Then we can assume that Kis large enough such that there exist x1; : : : ; xd 2 U? and such that the ideal generatedby fi := NG(xi) (i = 1; : : : ; d) has height d. In fact, the xi can be obtained by \Dade'sconstruction" (see Stanley [17, page 483]). It follows that the fi lie in S(U?), andthey form a regular sequence since d � 2 (see the proof of Proposition 20). Now aftera further extension of K, Corollary 18 yields g1; : : : ; gm with m = dim(U), such thatf1; : : : ; fd; g1; : : : ; gm is a regular sequence on RG and the images of g1; : : : ; gm form ahomogeneous system of parameters for RG=I. Hence depth(RG) � m+ d = minfm+2; dim(V )g since d = minf2; dim(V )�mg. Thus we have proved (4) without using thehypothesis that G is shallow. Applying (4) to the Sylow p-subgroup P instead of G, weobtain depth(RP ) � minfdim(V P ) + 2; dim(V )g: But by Kemper [9, Proposition 14]depth(RG) � depth(RP ). Thus we have proved Theorem 2.We now show that if G is shallow, f1; : : : ; fd; g1; : : : ; gm is a maximal regular se-quence. From this the equality depth(RG) = m + d follows. In fact, by Proposi-tion 23 we have that depthI(RG) = d. Assume that there exists f 2 RG+ such thatf1; : : : ; fd; g1; : : : ; gm; f is a regular sequence for RG. Since the images of g1; : : : ; gmform a homogeneous system of parameters for RG=I, there is a k 2 N such thatfk 2 (g1; : : : ; gm)RG + I. But f1; : : : ; fd; g1; : : : ; gm; fk is also regular, and this is notchanged if we add an element from (g1; : : : ; gm)RG to fk, so we may assume that fk 2 I.But then f1; : : : ; fd; fk 2 I is regular for RG, in contradiction to depthI(RG) = d. Thusindeed we have depth(RG) = m + d, and Theorem 1 follows since m = dim(V P ) byLemma 8.

12 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAU5. Some remarks on the Buchsbaum propertyFor graded or local rings, the Cohen{Macaulay property can be weakened to obtainthe so-called Buchsbaum property. We concentrate on the graded case, so let A =�1i=0Ai be a �nitely generated graded algebra over a �eldK = A0 and let A+ = �1i=1Ai.We begin with a few de�nitions. We refer the reader to St�uckrad and Vogel [18] for amore detailed discussion.A sequence a1; : : : ; ar in A+ is called weakly regular if: for every i � r and m 2 A,m � ai 2 (a1; : : : ai�1) implies m �A+ � (a1; : : : ai�1). A is Buchsbaum if every system ofparameters is a weakly regular sequence. A system of parameters a1; : : : ; an is said tomeasure the depth of A if a1; : : : ; ad is a regular sequence when d = depth(A).Remark 24. Suppose that K is a �nite �eld with q = ps elements. Then GLn(K) is�nite and RGLn(K) is generated by the Dickson invariants d1; : : : ; dn (with deg(di) =qn � qn�i) (see Smith [15, Section 8.1]). For any subgroup G � GLn(K), the Dicksoninvariants form a homogeneous system of parameters for RG. Landweber and Stong [12]conjectured that d1; : : : ; dn measure the depth of RG. This conjecture has recently beenproved by Bourguiba and Zarati [3].Proposition 25. If A is Buchsbaum, then every system of parameters measures thedepth.Proof. Suppose that A is Buchsbaum and assume, by way of contradiction, that thereis a system of parameters a1; : : : ; an which does not measure the depth. Let k < d :=depth(A) be maximal such that a1; : : : ; ak is regular. Every regular sequence of lengthless than d can be extended to a regular sequence of length d. Therefore there existsf 2 A+ such that a1; : : : ; ak; f is regular. Since a1; : : : ; ak+1 is not regular, thereexists g 62 (a1; : : : ; ak) such that gak+1 2 (a1; : : : ; ak). However, since A is Buchsbaum,a1; : : : ; ak+1 is weakly regular. Therefore g � A+ � (a1; : : : ; ak). In particular g � f 2(a1; : : : ; ak). By the regularity of a1; : : : ; ak; f , this implies g 2 (a1; : : : ; ak) and givesthe required contradiction.Returning to the situation of Section 4 where A = RG is a ring of invariants, wecan use Corollary 18 to give an alternate proof the following result of Nakajima [13,Proposition 3.7].Theorem 26. If G is a p-group, then RG is Cohen{Macaulay if and only if it is Buchs-baum.Proof. If RG is Cohen{Macaulay, it is clearly Buchsbaum. To prove the converse,assume that RG is Buchsbaum. Let U = V G0p and consider the subalgebra S(U?).Observe that G0p = G stabilizes U?, and that there exists a basis x1; : : : ; xd of U?such that the action of G on U? takes the form of upper triangular matrices. Settingfi = NG(xi), we obtain ht(f1; : : : ; fd) = d. Let k be maximal such that the subsequencef1; : : : ; fk is regular. By Corollary 18 there exist g1; : : : ; gm 2 RG with m = dim(U)such that f1; : : : ; fk; g1; : : : ; gm is a regular sequence, and since G is a p-group, m ispositive, so depth(RG) � k + 1. Now assume k < d. Then f1; : : : ; fk+1 is not regularand can be extended to a homogeneous system of parameters which does not measurethe depth. This contradicts Proposition 25. Hence we must have k = d, and sincem+ d = dim(V ), it follows that RG is Cohen{Macaulay.

DEPTH OF MODULAR INVARIANT RINGS 13In all examples known to us regarding rings of invariants which are not Cohen{Macaulay, there are homogeneous systems of parameters that fail to measure the depth.This prompts us to make the following conjecture.Conjecture 27. For rings of invariants of �nite linear groups acting on polynomialalgebras, the Cohen{Macaulay property and the Buchsbaum property are equivalent.References[1] D. J. Benson, Polynomial Invariants of Finite Groups, Lond. Math. Soc. Lecture NoteSer. 190, Cambridge Univ. Press, Cambridge, 1993.[2] W. Bosma, J. J. Cannon, C. Playoust, The Magma algebra system I: The user language,J. Symbolic Computation 24 (1997), 235{265.[3] D. Bourguiba, S. Zarati, Depth and the Steenrod algebra, Invent. Math. 128 (1997), 589{602.[4] W. Bruns, J. Herzog, Cohen{Macaulay Rings, Cambridge University Press, Cambridge,1993.[5] H. E. A. Campbell, A. V. Geramita, I. P. Hughes, R. J. Shank, D. L. Wehlau, Non-Cohen{Macaulay vector invariants and a Noether bound for a Gorenstein ring of invariants,Canad. Math. Bull. 42 (1999), 155{161.[6] G. Ellingsrud, T. Skjelbred, Profondeur d'anneaux d'invariants en caract�eristique p, Com-pos. Math. 41 (1980), 233{244.[7] P. Fleischmann, Relative trace ideals and Cohen{Macaulay quotients of modular invariantrings, in: P. Dr�axler, G.O. Michler, C. M. Ringel, eds., Computational Methods for Rep-resentations of Groups and Algebras, Euroconference in Essen, April 1-5 1997, Progressin Mathematics 173, Birkh�auser, Basel 1999.[8] B. Huppert, N. Blackburn, Finite Groups III, Springer-Verlag, Berlin, Heidelberg, NewYork, 1982.[9] G. Kemper, Calculating invariant rings of �nite groups over arbitrary �elds, J. SymbolicComputation 21 (1996), 351{366.[10] G. Kemper, On the Cohen{Macaulay property of modular invariant rings, J. of Algebra215 (1999), 330{351.[11] G. Kemper, A. Steel, Some algorithms in invariant theory of �nite groups, in: P. Dr�axler,G.O. Michler, C. M. Ringel, eds., Computational Methods for Representations of Groupsand Algebras, Euroconference in Essen, April 1-5 1997, Progress in Mathematics 173,Birkh�auser, Basel, 1999.[12] P. S. Landweber, R. E. Stong, The Depth of rings of invariants over �nite �elds, in: Proc.New York Number Theory Seminar, 1984, Lecture Notes in Math. 1240, Springer{Verlag,New York, 1987.[13] H. Nakajima, Invariants of �nite groups generated by pseudo-re ections in positive cha-racteristic, Tsukuba J. Math. 3 (1979), 109{122.[14] Jean{Pierre Serre, Algebraic Groups and Class Fields, Springer-Verlag, Heidelberg 1988.Russian translation: �. Serr, Algebraiqeskie gruppy i pol� klassov,M., Mir, 1968.[15] Larry Smith, Polynomial Invariants of Finite Groups, A. K. Peters, Wellesley, Mass.,1995.

14 H. E. A. CAMPBELL, I.P. HUGHES, G. KEMPER, R.J. SHANK, D.L. WEHLAU[16] L. Smith, Homological codimension of modular rings of invariants and the Koszul complex,J. Math. Kyoto Univ. 38 (1998), 727{747.[17] R. P. Stanley, Invariants of �nite groups and their applications to combinatorics, Bull.Amer. Math. Soc. 1(3) (1979), 475{511.[18] J. St�uckrad, W. Vogel, Buchsbaum Rings and Applications, Springer{Verlag, Heidelberg,1986.[19] D. E. Taylor, The Geometry of the Classical Groups, Heldermann, Berlin, 1992.[20] O. Zariski, P. Samuel, Commutative Algebra, vol. 2, Springer{Verlag, Heidelberg, 1960.Russian translation: O. Zarisski�, P. Sam��l~, Kommutativna� algebra, tom 2,M., Inostr. lit., 1963.