GENERALIZED RESOLVENTS OF ISOMETRIC LINEAR RELATIONS INPONTRYAGIN SPACES, II: KREIN-LANGER FORMULA

OLEG NITZ

This is the second of three papers, devoted to a generalization of the Kreın-Langerformula for the generalized resolvents of an isometric relation in a Pontryaginspace and to its application to the Nehari-Takagi problem. In this paper thegeneralized Kreın-Langer formula is obtained.

Introduction

The parametric description of the generalized resolvents of a symmetric and an isometricoperator in a Pontryagin space by Kreın and Langer ([KL]) have been further developed,also for symmetric linear relations, by many authors (see, for example, Dijksma, Langer andde Snoo [DLS], Bruinsma [B], Constantinescu and Geondea [CG], Derkach [D], Kaltenbackand Woracek [KW]).

The problem of describing the generalized resolvents of an isometric linear relationin a Pontryagin space arose in a natural way during our investigation of the Nehari-Takagiproblem. We used an operator approach, which was developed by Adamyan, Arov andKreın for the Nehari problem in [AAK], [A]. That the operator approach might also beeffective in the Nehari-Takagi problem was indicated by Adamyan, Arov, Kreın in [AAK2].An isometric operator in a Pontryagin space can be build from the data of the problem,such that the unitary extensions of this operator correspond to the solutions of the Nehari-Takagi problem. But there arise some difficulties. A solution of the Nehari-Takagi problemmay have a pole at 0, and in this case it does not correspond to any operator, it naturallycorresponds to a linear relation. One cannot a priori choose a transformation of the dataof the problem, which guarantees the holomorphy of all the solutions at 0. That is whythe investigation of isometric and unitary linear relation play an essential role in this seriesof papers.

1991 Mathematics Subject Classification. Primary 47A20, Secondary: 47A48, 47B50.Keywords and phrases. Unitary extension, generalized resolvent, linear relation, Pontryagin space.

The main result of the first paper [N] is Theorem 4.6 about the structure of unitaryrelations in Pontryagin spaces. There is a close connection between this result and the paper[AD] by T.Ya.Azizov and A.Dijksma.

Although this paper is the second one in the series, it may be read independently,because we repeat all definitions and results of the first paper, which are needed here.Section 1 of this paper contains general definitions and results related to Kreın spaces, tolinear relations and to reproducing kernel spaces. In Section 2 we reformulate and supplementsome results on open and closed systems from the first paper. In Section 3 we build a one-to-one correspondence between simple open systems and operator-functions of the generalizedSchur class. In Section 4 we repeat the definition of the embedding of an open system intoa closed system from the first paper and we bring some new results. In Section 5 we formulatethe problem of describing the generalized resolvents of an isometric relation in a Pontryaginspace and solve it using the system approach. In the third paper we shall apply the resultsof the previous papers and we shall obtain a description of the set of all the solutions of theNehari-Takagi problem.

1 Definitions and Preliminaries

1.1 Kreın spaces

We start with some basic definitions related to Kreın spaces. All topological linear spacesare assumed separable. The proofs of all statements in this subsection can be found in [AI].

In this article by a scalar product we mean a Hermitian sesquilinear form on a com-plex linear space. A Kreın space K is a topological complex linear space K endowed witha scalar product [·, ·], such that for some continuous linear operator J in the space K withthe property J2 = I, the new scalar product (x, y)J := [Jx, y] turns K into a Hilbert space.It follows, that the topology of the Hilbert space is equivalent to the topology of the Kreınspace. The operator J is called a fundamental symmetry. In general, a fundamentalsymmetry is not unique. It is easy to see, that J is selfadjoint with respect to both scalarproducts (·, ·)J and [·, ·]. Sometimes we write [·, ·]K to indicate the space explicitly.

A vector h ∈ K is called positive (neutral, negative) if [h, h] > 0 (= 0, < 0).A subspace of a Kreın space K is a nonempty linear set L ⊂ K (not necessarily closed).A subspace L is called positive (neutral, negative) if every nonzero vector h ∈ L is positive(neutral, negative). Below some standard notation is given:

x [⊥] y :⇔ [x, y] = 0 L1uL2 := L1 + L2 if L1 ∩ L2 = 0L [⊥] := x ∈ K : ∀y ∈ L x [⊥] y L1 [+]L2 := L1 + L2 if L1 [⊥]L2

L2 [−]L1 := L2 ∩ L[⊥]1 if L1 ⊂ L2 L1 [+]L2 := L1 + L2 if L1 ∩ L2 = 0 ,L1 [⊥]L2

L1 ∨ L2 := L1 + L2, i.e. the closure of L1 + L2

A subspace L is called hypermaximal neutral if L = L [⊥] . By a regular subspace ofa Kreın space we mean a closed subspace L ⊂ K which is a Kreın space in the scalar productof K. A subspace L ⊂ K is regular if and only if L [+]L [⊥] = K.

2

Every closed subspace L of a Kreın space K admits a decomposition of the formL = L+ [+]L− [+]L0, where L+, L−, L0 are positive, negative and neutral closed subspacesrespectively. The subspace L0 is uniquely determined and can be found by the formulaL0 = L ∩ L [⊥] . It is called the isotropic part of L. In general, the subspaces L± are notunique but their dimensions do not depend on the choice and they are called the signaturesof L. We set κ0[L] = dimL0, κ

±[L] = dimL±. The whole Kreın space K has no isotropicpart, i.e. κ0[K] = 0, so it has a decomposition K = K+ [+]K−, which is called a fundamentaldecomposition. The number κ−[K] is called the number of negative squares of a Kreınspace K. A Kreın space with the finite number of negative squares is called a Pontryaginspace. A closed subspace L of a Pontryagin space is regular if and only if κ0[L] = 0.

Let K1 and K2 be Kreın spaces. We define the direct sum K1 ⊕ K2 of Kreınspaces as the direct sum of the topological linear spaces endowed with the scalar product

[

(x1

x2

),

(y1y2

)] = [x1, y1]K1 + [x2, y2]K2 . We denote by L(K1,K2) the set of all continuous and

everywhere defined linear operators from the Kreın space K1 to the Kreın space K2; we writeL(K1) instead of L(K1,K1). Let A ∈ L(K1,K2) and L ⊂ K1 be a regular subspace, then therestriction of A to L is denoted by A|L and is considered as an element of L(L,K2). Let L bea regular subspace of a Kreın space K, then the orthogonal projection onto L (with respectto the indefinite scalar product) is denoted by PL and PL ∈ L(K).

Two neutral finite-dimensional subspaces L1,L2 are called skewly linked if L1+L2

is a regular subspace. In this case L1 ∩L2 = 0, dimL1 = dimL2 = κ−[L1uL2] = κ+[L1uL2]and ∀ x ∈ L1, x = 0 ∃y ∈ L2 : [x, y] = 0. For any nontrivial finite-dimensional neutralsubspace L1 one can find infinitely many subspaces L2 such that L1, L2 are skewly linked.

1.2 Linear Relations

Here we repeat some definitions and results from Sections 3, 4 of [N].Let K1 and K2 be Kreın spaces. We define the graph-sum K1K2 of Kreın

spaces as the direct sum of the topological linear spaces endowed with the scalar product

[

[x1

x2

],

[y1y2

]] = [x1, y1]K1 − [x2, y2]K2 . We use square brackets for vectors from K1K2 to em-

phasize the disparity of the two components of the vector with respect to the scalar product.When writing block-matrices for the graph-sums of spaces we shall also use square brackets.The rules of the multiplication of a block-matrix by another one or by a block-vector are asusual. However, note that the rules for taking the adjoint of such block-matrices differ fromthe usual ones. Let K, K1, K2, K

′1, K

′2 be Kreın spaces, A ∈ L(K,K1), B ∈ L(K,K2) and

Aij ∈ L(Ki,K′j) for i, j = 1, 2. Then[

AB

]⋆=

[A⋆ −B⋆

]and

[A11 A12

A21 A22

]⋆=

[A⋆

11 −A⋆21

−A⋆12 A⋆

22

]The graph of a linear operator A ∈ L(K1,K2) is a closed subspace of K1K2,

defined by grA =

[xAx

]: x ∈ K1

. Assume, that A is invertible. It is easy to see that

3

grA⋆ = (grA−1) [⊥] . For future purposes we define the inverting operator

jK1,K2 :=

[0 II 0

]∈ L(K1K2,K2 K1)

It has the properties grA−1 = jK1,K2(grA), jK2,K1 jK1,K2 = IK1 K2 and j⋆K1,K2= −jK2,K1 .

By a multi-valued mapping from a set X to a set Y we mean a mapping fromX to the set of all subsets of Y . If A is a multi-valued mapping, then we denote by Ax

the set which is a value of A on the element x ∈ X. We also define an action of A on theset L ⊂ X by formula AL :=

∪x∈LAx. We introduce notation domA = x ∈ X : Ax = ∅,

ranA = AX.A linear relation A (in Kreın spaces) is a multi-valued mapping from a Kreın space

K1 to a Kreın space K2 such that the set grA :=

[xy

]∈ K1 K2 : y ∈ Ax

is a subspace of

K1 K2. It is easy to see that A maps subspaces to subspaces, i.e. for any subspace L ⊂ K1

the set AL is a subspace.For subsets X and Y of a linear space we write λX instead of λx : x ∈ X and

X + Y instead of x+ y : x ∈ X, y ∈ Y . Let A, A1, A2 be linear relations from K1 to K2,B be a linear relation from K2 to K3. Then we define the linear relations λA, A1 +A2, A−1,B A and A⋆ by formulae

∀x ∈K1 (λA)x = λ(Ax), λ ∈ C∀x ∈K1 (A1 +A2)x = A1x+A2x

grA−1 = jK1,K2(grA)

∀x ∈K1 (B A)x = B(Ax)

grA⋆ = (grA−1) [⊥]

We also introduce, as usual, kerA = A−1(0). It is easy to verify that

for a subspace L A⋆L [⊥] ⊂ (A−1L) [⊥] (1.1)

∀λ ∈ C λ(A1 +A2) = λA1 + λA2 (1.2)

∀λ ∈ C \ 0 (λA)−1 = λ−1A−1 (1.3)

A⋆−1 = (A−1)⋆ (1.4)

Notice that in general 0 · A = 0 since dom(0 · A) = domA.If A(0) = 0, then we identify the relation A with the operator A, uniquely defined

by the condition grA = grA (or the equivalent condition: Ax equals Ax for x ∈ domAand is the empty set for all other x).

We say that a closed subspace X in a Kreın space is the completely closed sub-space if for all closed subspaces Y the subspace X+Y is closed. It is not difficult to see thatthe closed subspace X is completely closed iff either dimX < ∞ or dimX [⊥] < ∞. It followsthat the set of completely closed subspaces of a given space is closed under the operationsof sum, intersection and taking orthogonal complements.

Let K1 and K2 be Kreın spaces. We say that a linear relation A from K1 to K2 isthe completely closed relation if it has a closed graph and the subspaces domA, ranA,

4

domA⋆ and ranA⋆ are completely closed. We denote by R(K1,K2) the set of all completelyclosed linear relations from K1 to K2; we write R(K) instead of R(K,K). If a completelyclosed linear relation is an operator, then it is continuous. Continuously invertible linearoperators from L(K1,K2) and projections on completely closed subspaces are completelyclosed relations. The following statements were proved in [N], Proposition 3.1. A completelyclosed linear relation maps closed subspaces to closed subspaces. Let A ∈ R(K1,K2) andB ∈ R(K2,K3) be completely closed linear relations, L be a subspace and λ ∈ C. Then (λA),A⋆,A−1, B A are also completely closed relations and

(B A)⋆ = A⋆ B⋆ (1.5)

A⋆L [⊥] = (A−1L) [⊥] (1.6)

We say, that z ∈ C is a regular point of linear relation A ∈ R(K) if (A− zI)−1 ∈ L(K) isa continuous operator. We say, that z = ∞ is a regular point if A ∈ L(K). The set of allregular points of A we call a regular set and denote by ρ(A). We define the spectrumof A as σ(A) = C \ ρ(A), where C means C ∪ ∞. In this paper we give more traditionaldefinition of spectrum, then in [N], because here we don’t consider a spectrum of relationswith neutral domain.

A linear relation V is called isometric if grV−1 ⊂ grV⋆, or equivalently, grV isa neutral subspace: grV ⊂ (grV) [⊥] . A linear relation U is called unitary if U−1 = U⋆, orequivalently, grU is a hypermaximal neutral subspace: grU = (grU) [⊥] .

If V is an isometric relation in a Pontryagin space subspaces, then V(0) and kerVare neutral and finite-dimensional subspaces. A unitary relation in a Pontryagin space iscompletely closed (see [N], Proposition 4.3). The class of unitary relations in a Pontryaginspace is closed under the operation of composition (see [N], Proposition 4.1).

1.3 Generalized Schur Class and Reproducing Kernel Spaces

Details and proofs on this topic one can find in [DR] and in [ADRS].Let C be a Pontryagin space andK(w, z) be an operator-function with values in L(C)

defined on Ω×Ω for some open set Ω in the complex plane. Assume thatK(w, z)∗ = K(z, w),w, z ∈ Ω and that K(w, z) is holomorphic in z for each fixed w and holomorphic in w foreach fixed z. We shall call such a function a hermitian holomorphic kernel, or justa kernel for a short term. The set Ω will be called a domain of the kernel K(w, z). We saythat K(w, z) has κ negative squares on Ω if for any finite set w1, . . . , wN ∈ Ω and vectorsc1, . . . , cN ∈ C the matrix (

[K(wj, wi)cj, ci])Ni,j=1

(1.7)

has at most κ negative squares, and at least one such matrix has exactly κ negative squares.Write sq− K = κ if the condition holds for some κ and sq− K = ∞ otherwise.

Let Ω be an open subset in the complex plane and let θ(z) be an operator-functiondefined on Ω, then we denote

Ω⋆ := z ∈ C : z ∈ Ω (1.8)

θ(z) := θ(z)⋆ (1.9)

5

Operator-function θ(z) is defined on Ω⋆.Let M, N be Pontryagin spaces. Generalized Schur class Sκ(M,N) is a set of

operator-functions θ(z) with values in L(M,N) which are meromorphic in the unit disk andsuch that the kernel

Dθ(w, z) :=

I−θ(z)θ(w)⋆

1−zwθ(z)−θ(w)

z−w

θ(z)−θ(w)z−w

I−θ(z)θ(w)⋆

1−zw

(1.10)

with values in N ⊕M has κ negative squares. The domain of the kernel Dθ(w, z) is an in-tersection of the domains of holomorphy of θ(z) and θ(z) in the unit disk. In the case κ = 0we obtain an ordinary Schur class S(M,N) = S0(M,N), which consists of holomorphic andcontractive in the unit disk operator-functions.

Let H be a Pontryagin space of C-valued holomorphic functions on an open set Ωin the complex plane. Assume that for any point w ∈ Ω, (af + bg)(w) = af(w) + bg(w) forall a, b ∈ C and all f, g ∈ H. A kernel K(w, z) on Ω× Ω with values in L(C) is said to be areproducing kernel for H if for each w ∈ Ω and c ∈ C, K(w, z)c belongs to H as a functionof z, and

[h(·), K(w, ·)c]H = [h(w), c] (1.11)

for every h(·) in H.

Theorem 1.1. (1) A Pontryagin space H of holomorphic C-valued functions on an openset Ω has a reproducing kernel K(w, z) iff for each w ∈ Ω, the evaluation mappingh 7→ h(w) acts continuously from H into C. A reproducing kernel is unique when itexists.

(2) Let K(w, z) be a kernel on Ω×Ω with values in L(C) having a finite number of negativesquares. Then there is a unique Pontryagin space H of holomorphic C-valued functionson Ω with reproducing kernel K(w, z), and κ−[H] = sq− K.

For θ ∈ Sκ(M,N) we denote by D(θ) the unique Pontryagin space with the repro-ducing kernel Dθ(w, z). Then κ−[D(θ)] = κ.

Let M be a Hilbert space. An operator-function with values in L(M)

I − PX + ρz − z01− z0z

PX

is called a simple Blaschke-Potapov factor, if PX is an orthogonal projector onto a one-dimensional subspace X, |z0| < 1 and |ρ| = 1. It is invertible for all z in the unit disk exceptfor z0, where it equals I−PX . By a Blaschke product of degree N we mean an operator-function which admits a decomposition into a product of N simple Blaschke-Potapov factors.Number of the factors does not depend on the choice of the decomposition.

6

Theorem 1.2 (Kreın-Langer factorization). If M, N are Hilbert spaces, every θ(z) ∈Sκ(M,N) has factorizations

θ(z) = Bℓ(z)−1θℓ(z) = θr(z)Br(z)

−1 (1.12)

where Bℓ(z) and Br(z) are Blaschke products of degree κ with values in L(N) and L(M),respectively, and θℓ(z), θr(z) ∈ S(M,N). In this case

kerBℓ(z)∗ ∩ ker θℓ(z)

∗ = kerBr(z) ∩ ker θr(z) = 0 , |z| < 1 (1.13)

Conversely, let Bℓ(z) and Br(z) be Blaschke products of degree κ with values in L(N) andL(M), respectively, and let θℓ(z), θr(z) ∈ S(M,N). If (1.13) holds, then operator-functionsBℓ(z)

−1θℓ(z) and θr(z)Br(z)−1 belong to Sκ(M,N).

This theorem is often formulated with the extra condition that θ(z) is holomorphicat 0, but this condition can be eliminated by a substitution of the form z := (1−zz0)

−1(z−z0).

2 Open and Closed Systems in Pontryagin Spaces

In this section we reformulate and supplement some results from Section 5 of [N].Assume that H is a Pontryagin space, M and N are Hilbert spaces and W ∈

R(H ⊕ M,H ⊕ N) is a unitary relation. Then the quadruple α = (H,M,N,W) is calleda linear stationary conservative dynamical scattering system with discrete timeand Pontryagin state space. Here H, M and N are called state space, incomingspace and outgoing space. The relation W is called a system relation. It describesthe evolution of inner states of the system with some sequence of outer data:(

hi+1

ni+1

)∈ W

(hi

mi

)for all integer numbers i from some interval. Here hi, mi and ni are interpreted as inner state,incoming and outgoing outer data at time i, respectively. We say that hi is a trajectory ofstates of the system with the sequence ((mi;ni+1) of outer data. We say, that the systemα is regular if for every finite trajectory hiNi=0 with zero sequence of outer data conditionsh0 = hN = 0 imply hi = 0 for 0 < i < N . If the system relation of the system α is anoperator, then α is regular. An adjoint system is defined by

α⋆ := (H,N,M,W⋆) (2.1)

Since in this paper we deal only with systems of the described type, below weuse the term “open system” instead of “linear stationary conservative dynamical scatter-ing system with discrete time and Pontryagin state space”. We say that the open systemα = (H,M,N,W) is a closed system if M = N = 0. We shall write a closed system asa pair β = (K,U) consisting of the state space K and the system relation U (we usually useletters K and U for closed systems, while H and W are used for open systems, which maynot be closed).

7

Let us fix some regular closed system β = (K,U). Then

Rβ0 (z) := (I − zU)−1

Rβ1 (z) := (U⋆ − zI)−1

are meromorphic in the open unit disk operator-functions (see [N], Corollary 4.7). Let us setΩβ := z ∈ ρ(U⋆) : 0 < |z| < 1. Then operator-functions Rβ

0 (z) and Rβ1 (z) are holomorphic

in Ωβ.

Lemma 2.1. Let K1 and K2 be Kreın spaces and A,B ∈ R(K1,K2). Then A = B iffA(0) = B(0), domA = domB and Ax ∩ Bx = 0 for all x ∈ domA.

The proof is omitted in view of its simplicity.

Lemma 2.2. Let K1 and K2 be Kreın spaces and A,B ∈ R(K1,K2). If (A⋆ + B⋆)(0) isa closed subspace, then (A+ B)⋆ = A⋆ + B⋆.

Proof. The closedness of (A⋆ + B⋆)(0) yields

(A⋆ + B⋆)(0) = A⋆(0) + B⋆(0) = (domA) [⊥] + (domB) [⊥] =

(domA ∩ domB) [⊥] = (dom(A+ B)) [⊥] = (A+ B)⋆(0)

Let us prove that for y ∈ dom(A⋆ + B⋆) it holds (A⋆ + B⋆)y ∩ (A + B)⋆y = 0. Fixy ∈ dom(A⋆ + B⋆) = domA⋆ ∩ domB⋆ and choose a∗ ∈ A⋆y and b∗ ∈ B⋆y, then a∗ + b∗ ∈(A⋆ + B⋆)y. In order to prove that a∗ + b∗ ∈ (A + B)⋆y, let us choose x ∈ dom(A + B)and c ∈ (A + B)x. Then we can find a ∈ Ax and b ∈ Bx such that c = a + b. Hence[c, y] = [a + b, y] = [a, y] + [b, y] = [x, a∗] + [x, b∗] = [x, a∗ + b∗] and a∗ + b∗ ∈ (A + B)⋆y. Itfollows that dom(A⋆ + B⋆) ⊂ dom(A+ B)⋆. But

dom(A+ B)⋆ ⊂ (A+ B)(0) [⊥] = (A(0) + B(0)) [⊥] =

A(0) [⊥] ∩ B(0) [⊥] = domA⋆ ∩ domB⋆ = dom(A⋆ + B⋆)

whence dom(A+ B)⋆ = dom(A⋆ + B⋆) and by Lemma 2.1 (A+ B)⋆ = A⋆ + B⋆.

Lemma 2.3. Let β = (K,U) be a regular closed system. Then for the adjoint system β⋆ it

holds Rβ⋆

0 (z) = Rβ0 (z), R

β⋆

1 (z) = Rβ1 (z), ρ(U)⋆ = ρ(U⋆) and Ωβ⋆

=(Ωβ

)⋆.

Proof. By (1.4), Rβ0 (z) = ((I − zU)−1)⋆ = (I − zU)⋆−1. Subspace (I − zU⋆)(0) = U⋆(0) is

closed, therefore by Lemma 2.2 (I − zU)⋆−1 = (I − zU⋆)−1 = Rβ⋆

0 (z). The proof for Rβ1 (z)

is analogous. The other statements follow from the definition of the spectrum.

Let α = (H,M,N,W) be an open system. We define the canonical closure of α

as a closed systemα = (

K,

U) with

K := H2(M)⊕ H⊕H2(N) and

U

µ(ζ)h

ν(ζ)

:=

ζ−1(µ(ζ)− µ(0))h1

n1 + ζν(ζ)

:

(h1

n1

)∈ W

(h

µ(0)

)

8

where H2(E) is the Hardy space. We introduce the isometric operators EH ∈ L(H,K),

EM ∈ L(M,K) and EN ∈ L(N,

K) by

EH h :=

0h0

EM m :=

m00

EN n :=

00n

In the following two propositions we give an improvement of Proposition 5.5 from [N] withmore detailed proof.

Proposition 2.4. Let α = (H,M,N,W) be an open system and letα = (

K,

U) be a canonical

closure of α. Then α is regular iffα is regular.

Proof. Let us take any trajectory hiNi=0 of α with zero sequence of outer data and h0 =hN = 0. This condition may be written as follows(

hi+1

0

)∈ W

(hi

0

), 0 ≤ i ≤ N − 1 and h0 = hN = 0 (2.2)

On the other hand, let us take any trajectory xiNi=0 ofα with x0 = xN = 0

xi+1 ∈Uxi, 0 ≤ i ≤ N − 1 and x0 = xN = 0 (2.3)

Using the definition ofα we obtain:

if xi =

µi(ζ)hi

νi(ζ)

then

xi+1 ∈

Uxi, 0 ≤ i ≤ N − 1

x0 = xN = 0⇔

(hi+1

νi+1(0)

)∈ W

(hi

µi(0)

)µi+1(ζ) = ζ−1(µi(ζ)− µi(0))νi−1(ζ) = ζ−1(νi(ζ)− νi(0))h0 = hN = 0µ0(ζ) = µN(ζ) ≡ 0ν0(ζ) = νN(ζ) ≡ 0

Condition µ0(ζ) ≡ 0 yields by induction µi(ζ) ≡ 0, 0 ≤ i ≤ N . Condition νN(ζ) ≡ 0 yieldsby induction in backward direction νi(ζ) ≡ 0, 0 ≤ i ≤ N . This means that

if xi =

µi(ζ)hi

νi(ζ)

then

xi+1 ∈

Uxi, 0 ≤ i ≤ N − 1

x0 = xN = 0⇔

(hi+1

0

)∈ W

(hi

0

)h0 = hN = 0µi(ζ) ≡ 0, 0 ≤ i ≤ Nνi(ζ) ≡ 0, 0 ≤ i ≤ N

(2.4)

Let α be regular. Then for any trajectory xiNi=0 ofα, satisfying (2.3), we take projection

hiNi=0 on H. By (2.4) hiNi=0 is a trajectory of α, which satisfies (2.2). By regularity of α,hi = 0, 0 ≤ i ≤ N , therefore (2.4) yields xi = 0, 0 ≤ i ≤ N . Vise versa, let

α be regular.

Then for any trajectory hiNi=0 of α, satisfying (2.2), we append zero functions as in (2.4)to build trajectory xiNi=0 of

α, which satisfies (2.3). By regularity of

α, xi = 0, 0 ≤ i ≤ N ,

therefore hi = 0, 0 ≤ i ≤ N .

9

Proposition 2.5. Let α = (H,M,N,W) be a regular open system and letα = (

K,

U) be a

canonical closure of α. Then the function

Θα(z) =

(E⋆

H

E⋆N

)(

U⋆ − zI)−1

(EH EM

)=

(E⋆

H

E⋆N

)R

α1 (z)

(EH EM

)(2.5)

is holomorphic in Ωα and(hn

)= Θα(z)

(h0

m

)⇔

(hn

)∈ W

(h0 + zh

m

), z ∈ Ω

α (2.6)

Proof. Let z ∈ Ωα and let

(hn

)∈ W

(h0 + zh

m

). We set ν(ζ) := z(1 − zζ)−1n ∈ H2(N),

then by the definition of the canonical closure 0h

n+ ζν(ζ)

∈U

mh0 + zhν(ζ)

(2.7)

Using the identity ν(ζ) = z(n+ ζν(ζ)), we obtain 0h

n+ ζν(ζ)

∈U

mh0

0

+ z

0h

n+ ζν(ζ)

mh0

0

∈ (U⋆ − zI)

0h

n+ ζν(ζ)

Since z ∈ Ω

α ⊂ ρ(

U⋆), it follows 0h

n+ ζν(ζ)

= (U⋆ − zI)−1

mh0

0

(2.8)

(hn

)=

(E⋆

H

E⋆N

)(

U⋆ − zI)−1

(EH EM

)(h0

m

)= Θα(z)

(h0

m

)(2.9)

Conversely, let z ∈ Ωα and let (2.9) holds. Then for some µ(ζ) ∈ H2(M) and ν(ζ) ∈ H2(N) µ(ζ)

hn+ ζν(ζ)

= (U⋆ − zI)−1

mh0

0

Analogously to the previous calculation, we derive formula µ(ζ)

hn+ ζν(ζ)

∈U

m+ zµ(ζ)h0 + zh

z(n+ ζν(ζ))

10

By the definition of the canonical closure µ(ζ)h

n+ ζν(ζ)

=

ζ−1((m+ zµ(ζ))− (m+ zµ(0)))h1

n1 + ζz(n+ ζν(ζ))

, where

(h1

n1

)∈ W

(h0 + zh

m+ zµ(0)

)

It follows that µ(ζ) = ζ−1z(µ(ζ) − µ(0)), h = h1 and n + ζν(ζ) = n1 + ζz(n + ζν(ζ)).The first equality yields µ(ζ) = (1−ζz−1)−1µ(0), but this vector-function belongs to H2(M)

iff µ(0) = 0. The third equality yields n = n1, and we obtain

(hn

)∈

U(h0 + zh

m

).

We say that Θα(z) is a resolvent matrix of α. One can consider Θα(z) as a

meromorphic operator-function in the open unit disk. We set Ωα := Ωα. Since for a closed

system it holdsβ = β, formula (2.5) yields

Θβ(z) = Rβ1 (z) (2.10)

The blocks of the representation of Θα(z) as a block-matrix with respect to the decomposi-tions H⊕M and H⊕N we denote as follows

Θα(z) =

(θα0 (z) θα1 (z)θα2 (z) θα(z)

)(2.11)

The block θα(z) ∈ L(M,N) is called the transfer function of the open system α. Assume

that W is an operator and

(T FG H

)is its representation as a block-matrix, then (2.5) yields

Θα(z) =

(T (I − zT )−1 (I − zT )−1FG(I − zT )−1 H + zG(I − zT )−1F

)Notice that the transfer function here looks as usual.

It is easy to see that the canonical closure of the adjoint system α⋆ coincides with(

α)⋆ up to the natural unitary mapping of H2(N) ⊕ H ⊕ H2(M) to H2(M) ⊕ H ⊕ H2(N),and therefore

Θα⋆

(z) = Θα(z) (2.12)

Decomposing into blocks, we obtain:

θα⋆

0 (z) = θα0 (z) θα⋆

1 (z) = θα2 (z) θα⋆

2 (z) = θα1 (z) θα⋆

(z) = θα(z) (2.13)

3 Simple Open Systems

In this section we introduce a notion of simple open system and other concerned notions.We also solve the problem of reconstructing a simple open system from its transfer function.

11

Definition 3.1. Let α = (H,M,N,W) be a regular open system and let L be a regularsubspace of H. Then subspaces

Hc,L := L ∨( ∨z∈Ωα

PHΘα(z)(L⊕M)

)=

∨z∈Ωα

(L+ θα0 (z)L+ ran θα1 (z)

)(3.1)

Ho,L := L ∨( ∨z∈Ωα

PHΘα(z)⋆(L⊕N)

)=

∨z∈Ωα

(L+ θα0 (z)

⋆L+ ran θα2 (z)⋆)

(3.2)

are called L-control subspace and L-observation subspace, respectively. The open sys-tem α is called L-simple (L-controllable, L-observable) if H = Hc,L ∨ Ho,L (H = Hc,L,H = Ho,L). If L = 0, then the letter L is omitted in the terms and in the notation: the opensystem α is called simple (controllable, observable) if H = Hc ∨ Ho (H = Hc, H = Ho),where

Hc :=∨

z∈Ωα

ran θα1 (z) Ho :=∨

z∈Ωα

ran θα2 (z)⋆ (3.3)

Since the operator-function Θα(z) is holomorphic in Ωα, the closed linear spans inthe definition may be taken over any open subset of Ω ⊂ Ωα. For example, assume thath [⊥]

∨z∈Ω ran θα1 (z), then θα1 (z)

⋆h = 0 for all z ∈ Ω. Hence holomorphic function θα1 (z)h isa zero function. Thus θα1 (z)

⋆h = 0 for all z ∈ Ωα and h [⊥]Hc.

Proposition 3.2. Let α = (H,M,N,W) be a simple open system and let Ω = Ωα ∩(Ωα

)⋆.

Assume that C := N⊕M, then Hα(z) :=(θα2 (z) θα1 (z)

)is a holomorphic operator-function

with values in L(C,H) defined in Ω. Moreover,∨z∈Ω

ranHα(z) = H (3.4)

and for all z, w ∈ Ω, m ∈ M, n ∈ NHα(z)

(znm

)n+ θα(z)m

∈ W

Hα(z)

(nzm

)θα(z)n+m

(3.5)

Hα(w)⋆Hα(z) = Dθα(z, w) (3.6)

The transfer function θα belongs to Sκ(M,N), where κ := κ−[H].

Proof. Formula (3.4) follows from the definition of the simple system. Substitute (2.11) to(2.6), then for any m ∈ M, h0 ∈ H(

θα0 (z)h0 + θα1 (z)mθα2 (z)h0 + θα(z)m

)∈ W

(h0 + z

(θα0 (z)h0 + θα1 (z)m

)m

)(3.7)

Substitute h0 := 0 (θα1 (z)mθα(z)m

)∈ W

(zθα1 (z)m

m

), m ∈ M (3.8)

12

Applying this formula to the adjoint system α⋆ and making obvious transformations, weobtain that (

zθα2 (z)nn

)∈ W

(θα2 (z)n

θα(z)n

), n ∈ N (3.9)

Now we add (3.8) to (3.9), and (3.5) follows. We set c :=

(nm

), then for all c ∈ C

Hα(z)

(zI 00 I

)(I θα(z)

) c ∈ W

Hα(z)

(I 00 zI

)(θα(z) I

) c (3.10)

Substitute w and c′ instead of z and c, then the neutrality of grW yields:

[Hα(z)

(zI 00 I

)c,Hα(w)

(wI 00 I

)c′] + [

(I θα(z)

)c,(I θα(w)

)c′]−

[Hα(z)

(I 00 zI

)c,Hα(w)

(I 00 wI

)c′]− [

(θα(z) I

)c,(θα(w) I

)c′] = 0

Therefore(wI 00 I

)Hα(w)⋆Hα(z)

(zI 00 I

)−(I 00 wI

)Hα(w)⋆Hα(z)

(I 00 zI

)=(

θα(w) I)⋆ (

θα(z) I)−

(I θα(w)

)⋆ (I θα(z)

)Let

(A BC D

):= Hα(w)⋆Hα(z), then

(wzA wBzC D

)−

(A zBwC wzD

)=

(θα(w)⋆

I

)(θα(z) I

)−(

Iθα(w)⋆

)(I θα(z)

)((wz − 1)A (w − z)B(z − w)C (1− wz)D

)=

(θα(w)⋆θα(z)− I θα(w)⋆ − θα(z)

θα(z)− θα(w)⋆ I − θα(w)⋆θα(z)

)(A BC D

)=

I−θα(w)θα(z)⋆

1−wzθα(w)−θα(z)

w−z

θα(w)−θα(z)w−z

I−θα(w)θα(z)⋆

1−wz

and (3.6) follows. Let us verify, that the kernel Dθα(w, z) has κ negative squares. Let ustake w1, . . . , wN ∈ Ω and c1, . . . , cN ∈ C, then(

(Dθα(wj, wi)cj, ci)C)Ni,j=1

=([Hα(wj)cj, H

α(wi)ci]H)Ni,j=1

On the right hand side there is a Gram matrix of the set of the vectors Hα(wi)ciNi=1,therefore it has at most κ = κ−[H] negative squares. By (3.4) one can find a set Hα(wi)ciκi=1

of negative vectors, for which κ−[∨κ

i=1Hα(wi)ci] = κ. Then the Gram matrix of this set has

exactly κ negative squares and we have proved the last statement of the proposition.

13

Definition 3.3. Two open systems α = (H,M,N,W) and α1 = (H1,M,N,W1) are calledunitarily equivalent, if there exists a unitary operator T : H → H1 such that the operatorT : (H⊕M) (H⊕N) → (H1 ⊕M) (H1 ⊕N),

T =

(T 00 IM

)0

0

(T 00 IN

) (3.11)

has the property T (grW) = grW1.

Clearly, the unitary equivalence of open systems is an equivalence relation.

Proposition 3.4. Let α = (H,M,N,W) be a simple open system and let Ω = Ωα ∩(Ωα

)⋆.

Then α is unitarily equivalent to a simple open system of the form α1 = (D(θα),M,N,W1),where W1 satisfies Dθα(z, ·)

(znm

)n+ θα(z)m

∈ W1

Dθα(z, ·)(

nzm

)θα(z)n+m

(3.12)

for all m ∈ M, n ∈ N and z ∈ Ω.

Proof. Let C := N⊕M and T0 be an operator on the linear span of vectors Hα(z)c for allc ∈ C, which maps Hα(z)c to Dθα(z, ·)c. Operator T0 is correctly defined and isometric,because by (3.6)

[Hα(z)c,Hα(w)c′]H = [Hα(w)⋆Hα(z)c, c′]C = [Dθα(z, w)c, c′]C = [Dθα(z, ·)c,Dθα(w, ·)c′]D(θα)

for all z, w ∈ Ω and c, c′ ∈ C. By (3.4) it is densely defined in H. Since H is a Pontryaginspace, there exists an isometric operator T ∈ L(H,D(θα)), which is an extension of T0 bycontinuity. The range of T is a closed linear span of the elements of the form Dθα(z, ·)cfor all z ∈ Ω and c ∈ C, so ranT = D(θα) and T is unitary. Let operator T be definedby (3.11). Then T is unitary, and therefore T (grW) is a hypermaximal neutral subspace,which coincides with the graph of some unitary relation W1 ∈ R(D(θα)⊕M,D(θα)⊕N).Formula (3.12) follows from (3.5).

Proposition 3.5. Let M and H be Hilbert spaces and let θ ∈ Sκ(M,N). Let Ω be a domainof the kernel Dθ(w, z). We define the linear relation W2 ∈ R(D(θ)⊕M,D(θ)⊕N) by

grW2 =∨z ∈ Ωm ∈ Mn ∈ N

Dθ(z, ·)(

nzm

)θ(z)n+m

Dθ(z, ·)(znm

)n+ θ(z)m

(3.13)

Then W2 is a unitary linear relation, α2 = (D(θ),M,N,W2) is a simple open system, and

θα2(z) = θ(z) θα21 (z) = Dθ(z, ·)

(0IM

)θα22 (z) = Dθ(z, ·)

(IN0

)(3.14)

14

Later we shall see, that if we take θ := θα, then α2 coincides with α1 from Proposi-tion 3.4. At first we shall prove several lemmas.

Lemma 3.6. Let V be an isometric linear relation in a Pontryagin space, which has a closedgraph. Linear relation V is unitary iff the following conditions hold:

(domV) [⊥] = kerV (3.15)

(ranV) [⊥] = V(0) (3.16)

Proof. By Proposition 4.3 of [N], V is a completely closed relation, whence subspaces domVand ranV are closed. If V is a unitary relation, then (3.15), (3.16) follow from (1.6). Back-

wards, let (3.15), (3.16) hold. We have to prove that (grV) [⊥] ⊂ grV. Let

[xy

][⊥] grV .

Since kerV 0 ⊂ grV, then x [⊥] kerV. Using (3.15) and the closedness of domV , we ob-

tain x ∈ domV . Then choose y1 ∈ Vx. The neutrality of grV implies

[xy1

][⊥] grV , so[

0y − y1

][⊥] grV . Therefore, y−y1 [⊥] ranV , and (3.16) yields y−y1 ∈ V(0). Then y ∈ Vx,

which was to be proved.

Lemma 3.7. Let M, N be Hilbert spaces. If θ(z) ∈ Sκ(M,N), and

θ(z) = Bℓ(z)−1θℓ(z) = θr(z)Br(z)

−1

is a Kreın-Langer factorization, then the operator-function(Bℓ(z) 0

0 Br(z)

)h(z)

is holomorphic in the unit disk for all h(z) ∈ D(θ) .

Proof. Since Bℓ(z)θ(z) = θℓ(z) ∈ S(M,N) and Br(z)θ(z) = θr(z) ∈ S(N,M) the vector-function (

Bℓ(z) 0

0 Br(z)

) I−θ(z)θ(w)∗

1−zwθ(z)−θ(w)

z−w

θ(z)−θ(w)z−w

I−θ(z)θ(w)⋆

1−zw

c

is holomorphic in the unit disk for all c ∈ N ⊕ M and w ∈ Ω, where Ω is the domainof the kernel Dθ(w, z). One can find a sequence Dθ(wj, z)cjj which converges to anygiven h(z) ∈ D(θ) in the topology of D(θ). By Theorem 1.1 the convergence in each pointz ∈ Ω follows. Therefore, for all z ∈ Ω the sequence of holomorphic in the unit disk vector-

functions fj(z) :=

(Bℓ(z) 0

0 Br(z)

)Dθ(wj, z)cj converges to f(z) :=

(Bℓ(z) 0

0 Br(z)

)h(z),

which is meromorphic in the unit disk. Let ε > 0 be so small, that 1− ε ≤ |z| < 1 ⊂ Ω.Then for all z0 < 1− ε

fj(z0) =1

2πi

∫|z|=1−ε

fj(z)

z − z0dz → 1

2πi

∫|z|=1−ε

f(z)

z − z0dz =: f(z0), j → ∞

15

Here f(z) is a holomorphic vector-function, and it coincides with f(z) on Ω∩ |z| < 1− ε,therefore f(z) is holomorphic.

Lemma 3.8. Let K be a Hilbert space and A,B ∈ L(K). If the operators A, B, A∗, B∗ hasfinite-dimensional kernels and closed ranges, then

dimkerAB − dimkerA− dimkerB = dimkerB∗A∗ − dimkerA∗ − dimkerB∗

Proof. It is not difficult to see that

dimkerAB = dim(kerA ∩ ranB) + dimkerB

dimkerB∗A∗ = dim(kerB∗ ∩ ranA∗) + dimkerA∗

For closed subspaces X, Y ⊂ K one can verify the following statements:

dimX = dim(X ∩ Y ⊥) + dimPXY

dimPXY = dimPYX

Taking into account that ranB = (kerB∗)⊥ and ranA∗ = (kerA)⊥, we conclude

dimkerAB = dimkerA− dimPkerA kerB∗ + dimkerB

dimkerB∗A∗ = dimkerB∗ − dimPkerB∗ kerA+ dimkerA∗

The statement of the lemma follows.

Lemma 3.9. Let B(z) be a Blaschke product of degree N in a Hilbert space, then

dimkerB(z) = dimkerB(z)∗

Proof. Each simple Blaschke-Potapov factor is either invertible or has a one-dimensionalkernel. Moreover, the dimension of the kernel of the simple Blaschke-Potapov factor (thatis 0 or 1) equals the dimension of the kernel of its adjoint operator. Each of the factors isa completely closed relation, therefore the Blaschke product is a completely closed relationand has a closed range. So we can perform an induction by N , using Lemma 3.8 on eachstep.

Proof of Proposition 3.5. Let θ(z) = Bℓ(z)−1θℓ(z) = θr(z)Br(z)

−1 be a Kreın-Langer factor-ization, then

θℓ(z)Br(z) = Bℓ(z)θr(z)

θr(z)∗Bℓ(z)

∗ = Br(z)∗θℓ(z)

∗

Therefore,

θr(z) kerBr(z) ⊂ kerBℓ(z)

θℓ(z)∗ kerBℓ(z)

∗ ⊂ kerBr(z)∗

16

By Theorem 1.2, kerBr(z) ∩ ker θr(z) = kerBℓ(z)⋆ ∩ ker θℓ(z)

⋆ = 0, then we find thatdimkerBr(z) ≤ dimkerBℓ(z) and dimkerBℓ(z)

∗ ≤ dimkerBr(z)∗. Lemma 3.9 yields

dimkerBr(z) = dimkerBℓ(z), therefore

θr(z) kerBr(z) = kerBℓ(z) (3.17)

θℓ(z)∗ kerBℓ(z)

∗ = kerBr(z)∗ (3.18)

We shall use Lemma 3.6 to prove that W2 is unitary. Using a short calculation, one canverify, that W2 is an isometric relation. Since

[

(h0(·)m0

),

Dθ(z, ·)(

nzm

)θ(z)n+m

] = [h0(·), Dθ(z, ·)(

nzm

)] + [m0, θ(z)n+m] =

[h0(z),

(nzm

)] + [m0,

(θ(z) z−1I

)( nzm

)] = [h0(z) +

(θ(z)z−1I

)m0,

(nzm

)]

we find

(domW2)[⊥] =

(h0(·)m0

): h0(z) = −

(θ(z)z−1I

)m0 ∈ D(θ)

(3.19)

Clearly, kerW2 ⊂ (domW2)[⊥] . It remains to prove that (domW2)

[⊥] ⊂ kerW2. Assume

that

(h0(·)m0

)∈ (domW2)

[⊥] . Then h0(·) ∈ D(θ). By Lemma 3.7, if the operator-function

h0(z) = −(θ(z)z−1I

)m0 belongs to D(θ), then Br(z)(−z−1m0) is holomorphic in the unit disk.

Therefore m0 ∈ ker Br(0). By (3.18) ker Br(0) = θℓ(0) ker Bℓ(0). Choose n0 ∈ ker Bℓ(0) suchthat m0 = θℓ(0)n0. By definition of W2,

F (w, ·) :=

Dθ(w, ·)(Bℓ(w)n0

0

)θ(w)Bℓ(w)n0

Dθ(w, ·)

(wBℓ(w)n0

0

)Bℓ(w)n0

∈ grW2

Using the equality θ(z)Bℓ(z) = θℓ(z), we calculate F (w, z)

Dθ(w, z)

(Bℓ(w)n0

0

)θ(w)Bℓ(w)n0

Dθ(w, z)

(wBℓ(w)n0

0

)Bℓ(w)n0

=

((1− zw)−1(Bℓ(w)n0 − θ(z)θℓ(w)n0)

(z − w)−1(θ(z)Bℓ(w)n0 − θℓ(w)n0)

)θℓ(w)n0

w

((1− zw)−1(Bℓ(w)n0 − θ(z)θℓ(w)n0)

(z − w)−1(θ(z)Bℓ(w)n0 − θℓ(w)n0

)Bℓ(w)n0

17

The vector-function F (w, z) is holomorphic on w in the unit disk. Since n0 ∈ ker Bℓ(0),

F (0, z) =

(

−θ(z)θℓ(0)n0

−z−1θℓ(0)n0

)θℓ(0)n0

(00

) =

−

(θ(z)z−1I

)m0

m0

(00

) =

(h0(z)m0

)(00

)

By assumption, h0(·) ∈ D(θ). Thus, for any w in some neighborhood of zero F (w, ·) ∈(D(θ) ⊕ M) (D(θ) ⊕ N) and F (0, z) = limw→0 F (w, z) for z ∈ Ω. It follows from (1.11)that F (0, ·) is a weak limit of F (w, ·) in the space (D(θ)⊕M) (D(θ)⊕N). Since F (w, ·) ∈grW2, and a closed subspace is also weakly closed, it follows F (0, ·) ∈ grW2. Therefore(h0(·)m0

)∈ kerW2 and we conclude that (domW2)

[⊥] ⊂ kerW2. The proof of the inclusion

(ranW2)[⊥] ⊂ W2(0) is analogous.Setting n = 0 in the generating element of grW2 (see (3.13)), we find thatDθ(z, ·)

(0m

)θ(z)m

∈ W2

Dθ(z, ·)(

0zm

)m

= W2

zDθ(z, ·)(0m

)m

By definition of the resolvent matrix it followsDθ(z, ·)

(0m

)θ(z)m

= Θα2(z)

(0m

)=

(θα21 (z)mθα2(z)m

)Analogously, setting m = 0 in the generating element of grW2 and using properties of theadjoint system, we obtain Dθ(z, ·)

(n0

)θ(z)n

=

(θα22 (z)n

θα2(z)n

)

Thus, (3.14) is proved. Now computation(∨z∈Ω

ran θα21 (z)

)∨(∨z∈Ω

ran θα22 (z)

)=

(∨z∈Ω

ranDθ(z, ·)(

0IM

))∨(∨z∈Ω

ranDθ(z, ·)(IN0

))=

∨z∈Ω

ranDθ(z, ·) = D(θ)

shows that the system α2 is simple.

Theorem 3.10. Let M and N be Hilbert spaces and let κ be a nonnegative integer. The map-ping of an open system to its transfer operator-function establishes a one-to-one corre-spondence between all classes of unitary equivalence of simple open systems of the formα = (H,M,N,W) with κ−[H] = κ and all operator-functions of the class Sκ(M,N).

18

Proof. Let an operator-function θ ∈ Sκ(M,N) is given. By Proposition 3.5 a simple opensystem α2 = (D(θ),M,N,W2) can be built, such that κ−[D(θ)] = κ and its transfer operator-function equals θ. Moreover, W2 is an operator iff θ(z) is holomorphic at 0. It remains toprove that a simple open system α = (H,M,N,W), which has the same transfer operator-function θ, is unitarily equivalent to α2. By Proposition 3.4 the system α is unitarily equiv-alent to some system α1 = (D(θ),M,N,W1), such that W1 satisfies (3.12). Note, that α2 isdefined in the same spaces as α1, and it follows from (3.13), that grW2 ⊂ grW1. But bothW2 and W1 are unitary, therefore, they coincides. Thus, the system α is unitarily equivalentto the system α2 = α1. The statement of the theorem easily follows.

In the case when the system relation is an operator, our definitions of the simpleopen systems, the transfer functions and the unitary equivalence of the open systems areequivalent to the traditional ones. And the well-known result about transfer functions ofunitary colligations in Pontryagin spaces can be formulated in our terms as follows: the sys-tem relation of the simple open system is an operator iff its transfer function is holomorphicat 0.

4 Embedding of Open System into Closed System

In this section we reformulate and supplement some results from Section 5 of [N].The construction of the canonical closure in Section 2 is analogous to the con-

struction of a unitary dilation of contraction considered in [SF] and to the construction ofLax-Phillips scattering scheme from the unitary colligation (e.g., see [Ar]). The followingdefinition generalize such constructions.

Definition 4.1. Embedding τ of an open system α = (H,M,N,W) into a closed sys-tem β = (K,U) is a pair of linear operators (Eτ1, Eτ2), where Eτ1 ∈ L(H,K), Eτ2 ∈

L(MN,KK) such that if

(h1

n1

)∈ W

(h0

m0

), then

[Eτ1h0

Eτ1h1

]+ Eτ2

[m0

n1

]∈ grU .

It is easy to see, that for any open system α we can build a canonical embeddingof α into its canonical closure

α by means of the pair of embedding operators Eτ1 := EH and

Eτ2 :=

[EM 00 EN

]. In this case as well as in all cases of embedding which we meet in the

next section operator Eτ1 is isometric and operator Eτ2 is either isometric or anti-isometric.

Proposition 4.2. Let τ be an embedding of an open system α into a regular closed systemβ and kerEτ1 = 0. Then α is regular.

Proof. Assume that α = (H,M,N,W) is not regular and β = (K,U) is regular. Fix hi ∈ H,

0 ≤ i ≤ N , such that

(hi+1

0

)∈ W

(hi

0

), h0 = hN = 0 and hi = 0, 0 < i < N . Since

kerEτ1 = 0, we obtain the trajectory Eτ1hiN0 of β such that Eτ1hi+1 ∈ UEτ1hi, Eτ1h0 =Eτ1hN = 0 and Eτ1hi = 0, 0 < i < N , in contradiction with the regularity of β.

19

Let τ be an embedding of α into β. We introduce a total embedding operator

Eτ

(h0

m0

)(h1

n1

) :=

[Eτ1h0

Eτ1h1

]+ Eτ2

[m0

n1

](4.1)

Then the condition of Definition 4.1 may be written as Eτ (grW) ⊂ grU or, equivalently,grW ⊂ E−1

τ (grU). Applying formula (1.1), we obtain

E⋆τ (grU) = E⋆

τ (grU) [⊥] ⊂ (E−1τ (grU)) [⊥] ⊂ (grW) [⊥] = grW (4.2)

Assume that the closed system β is regular. We define the holomorphic in Ωβ operator-functions M τ (z) ∈ L(K,M) and N τ (z) ∈ L(K,N) by[

M τ (z)N τ (z)

]:= E⋆

τ2

[Rβ

0 (z)

Rβ1 (z)

](4.3)

It was proved in Section 5 of [N] that if in addition the open system α is regular, then insome ring r < |z| < 1 (

E⋆τ1R

β1 (z)

N τ (z)

)= Θα(z)

(E⋆

τ1

M τ (z)

)(4.4)

By Proposition 2.5, Θα(z) is holomorphic in Ωα, therefore this equality remains in force forz ∈ Ωα ∩ Ωβ. This formula establishes a relationship between the resolvent matrix Θα(z) ofthe open system α and the resolvent matrix Rβ

1 (z) = Θβ(z) of the closed system β. Takingthe adjoint we obtain (

Rβ1 (z)

⋆Eτ1 N τ (z)⋆)=

(Eτ1 M τ (z)⋆

)Θα(z)⋆ (4.5)

Let τ be an embedding of an open system α = (H,M,N,W) into a closed system β = (K,U).Then we can construct in a natural way an adjoint embedding τ ⋆ = (Eτ⋆1, Eτ⋆2) of α⋆

into β⋆:

Eτ⋆1 := Eτ1 Eτ⋆2 := jK,K Eτ2 jN,M

Proposition 4.3. Let τ be an embedding of a regular open system α = (H,M,N,W) intoa regular closed system β = (K,U). Then

Eτ1θα1 (z) =

[Rβ

1 (z) −Rβ0 (z)

]Eτ2

[IM

θα(z)

], z ∈ Ωα ∩ Ωβ

Proof. Take formula (4.5) for the adjoint embedding τ ⋆ and multiply both sides of the formula

by

(0IM

)from the right hand side, then for z ∈ Ωα⋆ ∩ Ωβ⋆

=(Ωα ∩ Ωβ

)⋆(Rβ⋆

1 (z)⋆Eτ⋆1 N τ⋆(z)⋆)( 0

IM

)=

(Eτ⋆1 M τ⋆(z)⋆

)Θα⋆

(z)⋆(

0IM

)N τ⋆(z)⋆ = Eτ⋆1θ

α⋆

2 (z)⋆ +M τ⋆(z)⋆θα⋆

(z)⋆

Eτ⋆1θα⋆

2 (z)⋆ = −[M τ⋆(z)⋆ −N τ⋆(z)⋆

] [θα⋆(z)⋆

IM

]

20

but[M τ⋆(z)⋆ −N τ⋆(z)⋆

]=

[M τ⋆(z)N τ⋆(z)

]⋆=

[Rβ⋆

0 (z)

Rβ⋆

1 (z)

]⋆Eτ⋆2 =

[Rβ⋆

0 (z)⋆ −Rβ⋆

1 (z)⋆]Eτ⋆2, so

Eτ1θα1 (z) = −

[Rβ⋆

0 (z)⋆ −Rβ⋆

1 (z)⋆]jK,K Eτ2 jN,M

[θα(z)IM

]Eτ1θ

α1 (z) =

[Rβ

1 (z) −Rβ0 (z)

]Eτ2

[IM

θα(z)

]It remains to substitute z instead of z.

5 Unitary Extensions of Isometric Relations

Notation and assumptions of the following definition remain in force during this section.

Definition 5.1. Let D be a Pontryagin space with κ negative squares and let V be anisometric relation in D. A unitary extension of V is a unitary relation U ∈ R(K) suchthat grV ⊂ grU , where K = D⊕He and He is a Pontryagin space. The number κe of negativesquares of He is called a growth of number of negative squares of the extension U .

One can find a regular subspace X ⊂ DD, such that (grV) [⊥] = grV [+]X. Thisstatement follows from the description of orthogonal complements of neutral subspaces,which is given in [AI]. In the latter formula the orthogonal complement is taken withrespect to the space DD, but taking the orthogonal complement with respect to the spaceKK we obtain (grV) [⊥] = grV [+]X [+] (He He). Let X = X+ [+]X− be a decompositioninto the sum of a positive subspace X+ ⊂ DD and a negative subspace X− ⊂ DD.Let us introduce a Hilbert space N as X+ in the scalar product [·, ·]DD and M as X− inthe scalar product −[·, ·]DD (that is anti-space of X−). We say, that (N,M) is the pair ofthe generalized defect subspaces of the isometric relation V . IfD is a Hilbert space, thenits defect subspaces in traditional sense are an example of the generalized defect subspaces(up to some natural identification). If V is not unitary, then one can choose the pair ofthe generalized defect subspaces in infinitely many ways, but the dimensions of the defectsubspaces do not depend on the choice.

Let us fix some unitary extension U . In order to simplify further computations weshall identify D and He with the subspaces D⊕0 and 0⊕He of D⊕He = K. We are going tobuild an open system α on the base of the isometric relation V and to build an embedding ofthe system α into the closed system β = (K,U). The analogous construction for an isometricoperator in a Hilbert space appeared in [AG]. We define the state space of the open systemα as D and operator Eτ1 as inclusion IK|D of D into K. Let operator Eτ2 ∈ L(MN,KK)

maps

[mn

]to the sum of m and n, considered as elements of the subspaces X− and X+. It is

easy to see, that the operator Eτ2 is anti-isometric (that is E⋆τ2Eτ2 = −I) and ranEτ2 = X.

21

Lemma 5.2. Let Eτ be defined by

Eτ

(d0m0

)(d1n1

) :=

[Eτ1d0Eτ1d1

]+ Eτ2

[m0

n1

]=

[d0d1

]+ Eτ2

[m0

n1

](5.1)

and let grW := E−1τ (grV), then W is a unitary relation and τ = (Eτ1, Eτ2) is an embedding

of the open system α = (D,M,N,W) into the closed system β = (K,U).

Proof. Let w =

(d0m0

)(d1n1

), w′ =

(d′0m′

0

)(d′1n′1

), w,w′ ∈ (D⊕M) (D⊕N), and let w′ [⊥]w for

all w ∈ grW . We are going to prove that w′ ∈ grW .

0 = [w,w′] = [d0, d′0]− [d1, d

′1] + [

[m0

n1

],

[m′

0

n′1

]] = [

[d0d1

],

[d′0d′1

]]− [Eτ2

[m0

n1

], Eτ2

[m′

0

n′1

]] =

[Eτw − Eτ2

[m0

n1

], Eτw

′ − Eτ2

[m′

0

n′1

]]− [Eτ2

[m0

n1

], Eτ2

[m′

0

n′1

]] =

[Eτw,Eτw′]− [Eτ2

[m0

n1

], Eτw

′]− [Eτw,Eτ2

[m′

0

n′1

]] = [Eτw,Eτw

′]− [Eτ2

[m0

n1

], Eτw

′]

since Eτw ∈ grV , Eτ2

[m′

0

n′1

]∈ X and grV [⊥]X. Choose d0, d1 such that

[d0d1

]= −Eτ2

[m0

n1

],

then Eτw = 0 and we find 0 = [Eτ2

[m0

n1

], Eτw

′] for all m0 ∈ M, n1 ∈ N. Since ranEτ2 = X,

we conclude that Eτw′ ∈ (DD) [−]X. Now choose d0, d1 such that

[d0d1

]∈ grV, m0 = 0

and n1 = 0. Then we find 0 = [Eτw,Eτw′], and Eτw ∈ grV yields Eτw

′ ∈ (grV) [⊥] .Hence Eτw

′ ∈ (grV) [⊥] ∩((DD) [−]X

)= grV and w′ ∈ grW . We have proved that

(grW) [⊥] = grW , i.e. W is a unitary relation.

Notice that the constructed open system α does not depend on the choice of theunitary extension U of the isometric relation V . The condition of the regularity of the systemα can be easily expressed in terms of V , and this leads to the following

Definition 5.3. An isometric relation V in a Pontryagin space D is called regular if forany sequence diNi=0 such that di+1 ∈ Vdi condition d0 = dN = 0 implies di = 0, 0 ≤ i ≤ N .

Obviously, if the relation V is an operator, then it is regular. If the relation V isregular, then the transfer operator-function of the open system α belongs to Sκ(M,N). Wecall it a characteristic operator-function of the isometric relation V with the pair of thegeneralized defect subspaces (N,M).

We define the “extern” isometric operator Ve by grVe = grU ∩ (X [+] (He He)).Since (grU) [⊥] ⊂ (grV) [⊥] = grV [+]X [+] (He He), we obtain grU = grV [+] grVe. We

22

are going to associate a second open system αe with the isometric relation Ve and to buildan embedding of αe into the closed system β = (K,U). Let operator Eτe1 be an inclusionIK|He and operator Eτe2 ∈ L(NM,KK) be defined by

Eτe2 = Eτ2 jN,M (5.2)

Then both operators Eτe1 and Eτe2 are isometric. Note, that Eτe2 does not depend onthe extension U .

Lemma 5.4. Let Eτe be defined by

Eτe

(h0

n0

)(h1

m1

) :=

[Eτe1h0

Eτe1h1

]+ Eτe2

[n0

m1

]=

[h0

h1

]+ Eτe2

[n0

m1

](5.3)

and let grWe := E−1τe (grVe), then We is a unitary relation and τe = (Eτe1, Eτe2) is an em-

bedding of the open system αe = (He,N,M,We) into the closed system β = (K,U).Thus, we obtain the mapping U 7→ αe, which establishes a one-to-one correspondence

between all unitary extensions with a growth of number of negative squares by κe and all opensystems with a state space having κe negative squares.

Proof. Clearly, operator Eτe is isometric and grVe = grU∩(X [+] (He He)) = grU∩ranEτe ,therefore grWe = E−1

τe (grVe) = E−1τe (grU). Since operator Eτe is isometric, Eτe(grWe)

[⊥] =

(grVe)[⊥] ∩ ranEτe = grVe, thus grW [⊥]

e = grWe.Conversely, let αe = (He,N,M,We) be an open system, and let Eτe2 and Eτe be

defined by (5.2) and (5.3). Obviously, grVe := Eτe(grWe) is an isometric relation andgrV ⊂ grU := grV [+] grVe. Hence (grU) [⊥] = (grV) [⊥] ∩ (grVe)

[⊥] = (grV [+] ranEτe) ∩((ranEτe)

[⊥] [+] grVe) = grV [+] grVe = grU , therefore U is a unitary extension withthe growth of number of negative squares by κ−[He].

Assume that the closed system β is regular. By Proposition 4.2 it follows thatthe systems α and αe are also regular. We denote the blocks of their resolvent matrices(with respect to the decompositions D ⊕M, D ⊕N and He ⊕N, He ⊕M, respectively) asfollows

Θα(z) =

(χ0(z) χ1(z)χ2(z) χ(z)

)Θαe(z) =

(ε0(z) ε1(z)ε2(z) ε(z)

)(5.4)

We set Ω := Ωα ∩ Ωαe ∩ Ωβ. Then for any z ∈ Ω formula (4.4) yields for the system α

E⋆τ1R

β1 (z) = χ0(z)E

⋆τ1 + χ1(z)M

τ (z) (5.5)

N τ (z) = χ2(z)E⋆τ1 + χ(z)M τ (z) (5.6)

and for the system αe

E⋆τe1R

β1 (z) = ε0(z)E

⋆τe1 + ε1(z)M

τe(z) (5.7)

N τe(z) = ε2(z)E⋆τe1 + ε(z)M τe(z) (5.8)

23

It follows from the definition of Eτe2, that M τe(z) = −N τ (z) and N τe(z) = −M τ (z). Re-stricting formulae (5.5)–(5.8) to D and taking into account that E⋆

τ1 = PD, E⋆τe1 = PHe and

PHe|D = 0, we obtain

PDRβ1 (z)|D = χ0(z) + χ1(z)M

τ (z)|D (5.9)

N τ (z)|D = χ2(z) + χ(z)M τ (z)|D (5.10)

P ⋆HeRβ

1 (z)|D = −ε1(z)Nτ (z)|D (5.11)

−M τ (z)|D = −ε(z)N τ (z)|D (5.12)

We are going to eliminate M τ (z)|D and N τ (z)|D. Substituting (5.12) into (5.10) we find

χ2(z) = (I − χ(z)ε(z))N τ (z)|D (5.13)

Lemma 5.5. Let β be a regular system, then ker(I − χ(z)ε(z)) = 0 for all z ∈ Ω.

Proof. Let n ∈ ker(I − χ(z)ε(z)). Then χ(z)ε(z)n = n and by Proposition 4.3

Eτ1χ1(z)ε(z)n =[Rβ

1 (z) −Rβ0 (z)

]Eτ2

[IMχ(z)

]ε(z)n =

[Rβ

1 (z) −Rβ0 (z)

]Eτ2

[ε(z)nn

]Eτe1ε1(z)n =

[Rβ

1 (z) −Rβ0 (z)

]Eτe2

[INε(z)

]n =

[Rβ

1 (z) −Rβ0 (z)

]Eτ2

[ε(z)nn

]Thus, χ1(z)ε(z)n ∈ D equals ε1(z)n ∈ He, whence both equals 0. Equality χ1(z)ε(z)n = 0and the definition of the resolvent matrix yield

Θα(z)

(0

ε(z)n

)=

(χ0(z) χ1(z)χ2(z) χ(z)

)(0

ε(z)n

)=

(0n

)(

0ε(z)n

)∈ W

(0n

)

Then

[00

]+ Eτ2

[ε(z)nn

]∈ grV by definition of the system α, therefore n equals 0.

This lemma allows to consider an operator (I−χ(z)ε(z))−1 with the domain ran(I−χ(z)ε(z)). The domain in general does not equal the whole space N. To emphasize thedifference between this operator and a usual inverse operator, we shall write (I−χ(z)ε(z))[−1].

Formula (5.13) yields

N τ (z)|D = (I − χ(z)ε(z))[−1]χ2(z) (5.14)

Substituting (5.14) into (5.12), and the result into (5.9), we obtain

PDRβ1 (z)|D = χ0(z) + χ1(z)ε(z)(I − χ(z)ε(z))[−1]χ2(z) (5.15)

Substituting (5.14) into (5.11), we obtain

PHeRβ1 (z)|D = −ε1(z)(I − χ(z)ε(z))[−1]χ2(z) (5.16)

24

Notice that U is a unitary extension of V if and only if U⋆ is a unitary extension of V−1. Soall the reasoning, which leads to formulae (5.15) and (5.16), may be repeated for extensionU⋆ of V−1. Using the properties of adjoint systems, which was stated before, it is not difficultto find that

PDRβ1 (z)

⋆|D = χ0(z)⋆ + χ2(z)

⋆ε(z)⋆(I − χ(z)⋆ε(z)⋆)[−1]χ1(z)⋆ (5.17)

PHeRβ1 (z)

⋆|D = −ε2(z)⋆(I − χ(z)⋆ε(z)⋆)[−1]χ1(z)

⋆ (5.18)

Definition 5.6. A unitary extension U ∈ R(K) of an isometric relation V is called a simpleextension if β = (K,U) is a D-simple system:

K =( ∨z∈Ωβ

(D+Rβ1 (z)D)

)∨( ∨z∈Ωβ

(D+Rβ1 (z)

⋆D))

(5.19)

Notice that aD-simple system is regular by definition. We do not use the traditionalterm “minimal extension” to avoid the following possibility of misunderstanding. Let U bean extension which is an operator, then condition (5.19) may be written as K =

∨∞−∞ U jD.

If K is a Hilbert space, the latter is equivalent to the following minimum condition: if K1 ⊂ K

and U1 ∈ L(K1) is a unitary extension such that U |K1 = U1, then K1 = K and U1 = U . Butif K is not a Hilbert space, a minimal extension in the latter sense can be not simple, see

Example 5.7. Assume that K = C ⊕ (−C) ⊕ (−C) ⊕ C, D = C ⊕ 0 ⊕ 0 ⊕ 0 and He =0⊕(−C)⊕(−C)⊕C, where (−C) is the anti-space of C, i.e. [z, w] = −wz. Let V be a trivialisometric operator in D with domV = 0. Consider the following unitary operator in K

U :=

1 0 1 −10 1 −1 11 1 1 01 1 0 1

It is not difficult to verify that

U j =

1 0 j −j0 1 −j jj j 1 0j j 0 1

,∞∨−∞

U jD = D⊕ Ch0 = K, where h0 :=

0011

Thus, U is an extension of V , which is not simple. Notice that h0 is a neutral eigenvector of U .Moreover, h0 is the only eigenvector of U which belongs to He. It follows that the extensionU is minimal.

Proposition 5.8. Let β be a regular system. Then the extension U is simple iff the opensystem αe is simple.

Proof. Assume that U is a simple extension, then the system β is D-simple. We set Ω :=Ωα ∩ Ωαe ∩ Ωβ. It is easy to verify that (5.19) is equivalent to

He =(∨z∈Ω

PHeRβ1 (z)D

)∨(∨z∈Ω

PHeRβ1 (z)

⋆D)

(5.20)

25

Using (5.16) and (5.18), we obtain

PHeRβ1 (z)D = ε1(z)(I − χ(z)ε(z))[−1]χ2(z)D ⊂ ran ε1(z)

PHeRβ1 (z)

⋆D = ε2(z)⋆(I − χ(z)⋆ε(z)⋆)[−1]χ1(z)

⋆D ⊂ ran ε2(z)⋆

Let us substitute these formulae into (5.20)

He ⊂(∨z∈Ω

ran ε1(z))∨(∨z∈Ω

ran ε2(z)⋆)⊂ He (5.21)

It follows, that the open system αe is simple:

He =(∨z∈Ω

ran ε1(z))∨(∨z∈Ω

ran ε2(z)⋆)

(5.22)

Conversely, assume that αe is simple. Proposition 4.3 and equality Rβ0 (z) = I+zRβ

1 (z) yield(see [N], formula (5.4))

ran ε1(z) =[Rβ

1 (z) −Rβ0 (z)

]Eτe2

[INε(z)

]N ⊂

[Rβ

1 (z) −Rβ0 (z)

]D ⊂ D+Rβ

1 (z)D

Let us substitute this formula and the analogous formula for the adjoint system into (5.22):

He ⊂(∨z∈Ω

(D+Rβ1 (z)D)

)∨(∨z∈Ω

(D+Rβ1 (z)

⋆D))

Formula (5.19) immediately follows, therefore U is a simple extension.

Proposition 5.9. Assume that α and αe are regular systems. If the system β is regular,then ker(I − χ(z)ε(z)) = 0, z ∈ Ωα ∩ Ωαe ∩ Ωβ. Conversely, if ker(I − χ(z)ε(z)) = 0,z ∈ Ω, where Ω ⊂ Ωα ∩ Ωαe is an open set, then the system β is regular.

Proof. The first assertion was stated in Lemma 5.5, so it remains to prove the second one.Let us take a trajectory xiN0 of system β, such that x0 = xN = 0. We have to prove that

xi = 0, 0 < i < N . We set di := PDxi, hi := PHexi, 0 ≤ i ≤ N , and

[mi

ni

]:= E⋆

τ2

[xi

xi+1

],

0 ≤ i < N . Then

[−ni

−mi

]:= E⋆

τe2

[xi

xi+1

]and (4.2) yields:(

di+1

ni

)∈ W

(dimi

) (hi+1

−mi

)∈ We

(hi

−ni

)for 0 ≤ i < N (5.23)

Multiply each formula by zi, sum for 0 ≤ i < N and denote d(z) :=∑N−1

i=0 di+1zi, h(z) :=∑N−1

i=0 hi+1zi, m(z) :=

∑N−1i=0 miz

i and n(z) :=∑N−1

i=0 nizi, then(

d(z)n(z)

)∈ W

(zd(z)m(z)

) (h(z)

−m(z)

)∈ We

(zh(z)−n(z)

)By definition of the resolvent matrix, n(z) = χ(z)m(z) and −m(z) = ε(z)(−n(z)), z ∈ Ω.Therefore, (1 − χ(z)ε(z))n(z) = 0 and, by the conditions of the Proposition, n(z) = 0 andm(z) = ε(z)n(z) = 0, z ∈ Ω. Hence ni = 0, mi = 0 for all 0 ≤ i < N . Substituting this into(5.23) and using the condition of the regularity of α and αe, we conclude that di = 0 andhi = 0, 0 < i < N . So xi = di + hi = 0 and the regularity of β is stated.

26

Definition 5.10. Let αe be a regular open system and let operator-function χ(z) be mero-morphic in the unit disk. We say, that αe is χ-regular if ker(I − χ(z)ε(z)) = 0 for all zin some open set, where ε(z) is a transfer operator-function of αe.

Proposition 5.11. The mapping from Lemma 5.4 establishes a one-to-one correspondencebetween all simple extensions with a growth of number of negative squares by κe and allsimple χ-regular open systems with a state space having κe negative squares.

Proof. Assume that U is a simple extension, then β is regular by the definition of the simpleextensions. Then the system αe is simple by Proposition 5.8. By Proposition 5.9, if β isregular, then αe is χ-regular.

Conversely, assume that αe is a simple χ-regular open system. By Proposition 5.9,if αe is χ-regular, then the system β is regular. By Proposition 5.8, if β is regular and αe issimple, then the extension U is simple.

There is an another criterion of the regularity of the system β:

Proposition 5.12. If He is a Hilbert space and either κ = 0 or ranVκ ∨ domVκ = D, thenthe system β is regular.

Proof. If κ = κ−[D] = 0, then K is a Hilbert space, therefore U is an operator and β isregular. Let κ > 0. Assume that xiN0 is a trajectory of system β and x0 = xN = 0.Since xi ∈ U ix0 = U i(0) ⊂ Uκ(0), xi [⊥] domUκ. Formula U i(0) ⊂ Uκ(0), i > 0, wasstated in [N], Section 4. Analogously, xi [⊥] ranUκ. It follows that xi [⊥] ranVκ∨domVκ ⊂ranUκ ∨ domUκ, so xi ∈ D [⊥] = He. All xi are neutral vectors from the Hilbert space He,therefore they are equal to 0 for all 0 < i < N .

Definition 5.13. Two unitary extensions U ∈ R(D⊕ He) and U1 ∈ R(D⊕ He1) are calledunitarily equivalent, if there exists a unitary operator T : He → He1 such that the operatorT : (D⊕ He) (D⊕ He) → (D⊕ He1) (D⊕ He1),

T =

(ID 00 T

)0

0

(ID 00 T

) (5.24)

has the property T (grU) = grU1.

Proposition 5.14. Two unitary extensions U ∈ R(D⊕ He) and U1 ∈ R(D⊕ He1) areunitarily equivalent iff the correspondent open systems αe = (He,N,M,We) and αe1 =(He1,N,M,We1) are unitarily equivalent.

Proof. Assume that T : He → He1 is a unitary operator, T is defined by (5.24) and T isdefined by (3.11). We have to prove, that T (grU) = grU1 iff T (grWe) = grWe1. At first let

27

us prove that TEτe = Eτe1T :

TEτe

(h0

n0

)(h1

m1

) = T (

[h0

h1

]+ Eτe2

[n0

m1

]) =

[Th0

Th1

]+ Eτe2

[n0

m1

]=

[Th0

Th1

]+ Eτe12

[n0

m1

]= Eτe1

(Th0

n0

)(Th1

m1

) = Eτe1T

(h0

n0

)(h1

m1

)

Assume that T (grU) = grU1, then

grV [+] grVe1 = grU1 = T (grU) = T (grV [+] grVe) = grV [+] T (grVe)

Hence, grVe1 = T (grVe), so

T (grWe) = E−1τe1

Eτe1T (grWe) = E−1τe1

TEτe(grWe) = E−1τe1

T grVe = E−1τe1

grVe1 = grWe1

The inverse implication is proved in a similar way.

Definition 5.15. Let χ(z) be a meromorphic in the unit disk operator-function with valuesin L(M,N). We define the χ-regular generalized Schur class as follows

Sκ(N,M;χ) := ε ∈ Sκ(N,M) : ker(I − χ(z)ε(z)) = 0 for all z in some open set

Theorem 5.16. Let V be a regular isometric relation in a Pontryagin space D. Let χ(z) bea characteristic operator-function of V with the pair of generalized defect subspaces (N,M).A superposition of the mapping from Lemma 5.4 and the mapping from Theorem 3.10 estab-lishes a one-to-one correspondence between all classes of unitary equivalence of simple unitaryextensions with a growth of number of negative squares by κe and all operator-functions ofthe class Sκe(N,M;χ). Let U be a simple unitary extension and let ε ∈ Sκe(N,M;χ) bea correspondent operator-function, then

PD(U⋆ − zI)−1|D = χ0(z) + χ1(z)ε(z)(I − χ(z)ε(z))[−1]χ2(z), z ∈ Ω (5.25)

where Ω is the open unit disk without a finite set of points, depending on ε(z), and χ0, χ1,χ2 are holomorphic in Ω operator-functions that do not depend on the extension U .

Let κ be a number of negative squares of D. If either κ = 0 or ranVκ∨domVκ = D,then S(N,M;χ) = S(N,M).

Proof. By Proposition 5.11 all simple extensions with a growth of number of negative squaresby κe are in a one-to-one correspondence with all simple χ-regular open systems with a statespace having κe negative squares. By Proposition 5.14 this correspondence is carried over tothe classes of unitary equivalence. It follows from Theorem 3.10, that all classes of unitaryequivalence of simple χ-regular open systems with a state space having κe negative squaresare in a one-to-one correspondence with all operator-functions of the class Sκe(N,M;χ).Formula (5.25) follows from (5.15), and the last statement of the theorem follows fromProposition 5.12.

28

Acknowledgment

I would like to express deep gratitude to my supervisor Damir Zyamovich Arov for his advisesand valuable criticism. The research described in this publication was made possible in partby Grant No. UCZ000 from International Science Foundation and by Grant No. UM1-298from American CRDF and Ukrainian Government.

References

[A] V.M.Adamyan. Non-degenerate Unitary Coupling of Semiunitary Operators, Funk.Anal. i Prilozhen. 7 (1973) 1–16.

[AAK] V.M.Adamyan, D.Z.Arov, M.G.Krein. Infinite Hankel Matrices and GeneralizedProblem of Caratheodory-Fejer and I.Schur, Funk. Anal. i Prilozhen. 2:4 (1968), 1–17. English Transl.: Funct. Anal. Appl. 2 (1968), 269–281.

[AAK2] V.M.Adamyan, D.Z.Arov, M.G.Krein. Analytic properties of the Schmidt pairs ofa Hankel operator and the generalized Schur-Takagi problem, Math. Sb. (N.S.) 86(128)(1971), 34–75; English Transl.: Math. USSR-Sb. 15 (1971), no. 1, 31–73.

[ADRS] D.Alpay, A.Dijksma, J.Rovnyak, H.S.V. de Snoo. Schur functions, operator colli-gations, and reproducing kernel Pontryagin spaces, Operator Theory: Adv. Appl., 96(1997).

[Ar] D.Z.Arov. Stable dissipative linear stationary dynamical scattering systems, J. OperatorTheory 2 (1979), 95–126.

[AG] D.Z.Arov, L.Z.Grossman. Scattering matrices in the theory of unitary extensions ofisometric operators, Math. Nachr. 157 (1992), 105–123.

[AD] T.Ya.Azizov, A.Dijksma. Contractive Linear Relations in Pontryagin Spaces, OperatorTheory: Adv. Appl., vol 103, Birkhauser Verlag, Basel, 1998, 19–51.

[AI] T.Ya.Azizov, I.S.Iohvidov. Foundations of the Theory of Linear Operators in Spaceswith an Indefinite Metric, “Nauka”, Moscow, 1986; English Transl.: Wiley, New York,1989

[B] P.Bruinsma. Interpolation problems for Schur and Nevanlinna pairs, Doctoral disserta-tion, University of Groningen, 1991.

[CG] T.Constantinescu, A.Geondea. Minimal signature in lifting of operators, II, J. Func-tional Analysis, 103 (1992), 317–351.

[D] V.Derkach. On Generalized Resolvents of Hermitian Relations in Kreın spaces, to ap-pear.

29

[DLS] A.Dijksma, H.Langer, H.S.V. de Snoo. Generalized Coresolvents of Standard Isomet-ric Operators and Generalized Resolvents of Standard Symmetric Relations in KreınSpaces, Operator Theory: Adv. Appl. 48 (1990), 261–274.

[DR] M.A.Dritschel, J.Rovnyak. Operators on Indefinite Inner Product Spaces, Waterloo,1994.

[IKL] I.S.Iohvidov, M.G.Krein, H.Langer. Introduction to the Spectral Theory of Operatorsin Spaces with an Indefinite Metric, volume 9 of Mathematical Research. AkademieVerlag, Berlin, 1982

[KW] M.Kaltenback, H.Woracek. The Kreın Formula for Generalized Resolvents in Degen-erated Inner Product Spaces, Mh. Math. 127, 119-140 (1999).

[KL] M.G.Kreın, H.Langer. On Defect Subspaces and Generalized Resolvents of a HermitianOperator in Space Πκ, Funk. Anal. i Prilozhen., 5(2) (1971), 59–71 (Russian); 5(3)(1971), 54–69. English transl.: Funct. Anal. Appl., 5 (1971/1972), 136–146, 217–228).

[N] O.Nitz. Generalized Resolvents of Isometric Linear Relations in Pontryagin Spaces, I:Foundations , to appear.

[SF] B.Sz.-Nagy, C. Foias. Harmonic Analysis of Operators on Hilbert Space , North Holland,Amsterdam, 1970.

Oleg Nitz,South-Ukrainian State Pedagogical University,Odessa, Ukraine.E-mail: on@ibis.odessa.ua

30