Optimization of Stochastic Uncertain

Systems with Variational Norm

Constraints

†,∗Farzad Rezaei, †Charalambos D. Charalambousand

‡Nasir U. Ahmed

†,∗School of Information Technology and EngineeringUniversity of Ottawa

†Department of Electrical and Computer EngineeringUniversity of Cyprus, CYPRUS

‡Also with Department of MathematicsUniversity of Ottawa

46th IEEE Conference on Decision and ControlDecember 12-14, 2007

New Orleans, LA, U.S.A.

0

Introduction

1. Introduction and Motivations

2. Problem Formulation

3. Solution to the Maximization Problem

4. Fully Observed Uncertain Control Sys-

tems

5. Conclusion and Future Work

1

Introduction and Motivations

Optimization of uncertain stochastic systems.

Pay-off is defined as a linear functional of the

uncertain measure.

Modeling of unceratainty:

Many papers have considered relative entropy

constraint model.[meneghini96],[petersen99], and

also [petersen2000],[Hansen2001],[ahmed2005].

2

Introduction and Motivations

Shortcomings of Relative Entropy Model

• It is not a true metric on the space of prob-

ability measures.

• Relative entropy between two probability

measures is only defined when the mea-

sures are absolutely continuous.

• In the context of SDEs, relative entropy

allows uncertainty in the drift term, but it is

unable to model uncertainty in the diffusion

coefficient.

3

Introduction and Motivations

Proposed model Uncertainty model based on

the variational norm on the space of probability

measures.

• The uncertainty is described by a ball in

the space of probability measures, centered

at the nominal measure and with a non-

negative radius.

• From Pinsker’s inequality[Pinsker64], it fol-

lows that an uncertainty set described by

relative entropy is a subset of the much

larger variational norm uncertainty set.

4

Problem Formulation

• Let (Σ, d) denote a complete separable met-

ric space (a Polish space).

• Let (Σ,B(Σ)) be the corresponding mea-

surable space, in which B(Σ) is the σ-algebra

generated by open sets in Σ.

• Let M1(Σ) denote the set of probability

measures on (Σ,B(Σ)) and Uad the set of

admissible controls.

• By choosing a control policy u for the nom-

inal system, the nominal system induces a

probability measure µu ∈M1(Σ).

5

Problem Formulation

• For a given u ∈ Uad, let M(u) ⊂ M1(Σ)

denote the set of probability measures in-

duced by the perturbed system while con-

trol u is applied.

• Let `u : Σ → < be a real-valued bounded

non-negative measurable function; and

||ν −µ|| denote the variational distance be-

tween measures ν and nominal µ both from

M(u).

• The uncertainty tries to maximize the av-

erage cost functional denoted by∫Σ `udν

subject to some constraints on the varia-

tional distance between ν and µ.

6

Problem Formulation

• Let Mc(u) ⊂ M(u) denote such a constraint.

The effect of uncertainty leads to the fol-

lowing maximization problem:

supν∈Mc(u)

∫Σ

`udν,

for every control u ∈ Uad.

• The designer on the other hand, tries to

choose a control policy to minimize the

worst case average cost. This gives rise

to the min-max problem

infu∈Uad

supν∈Mc(u)

∫Σ

`udν. (1)

• The constaint set is defined as follows.

Mc(u)4= Bu

R(µ) = ν ∈ M(u) : ||ν − µ|| ≤ R

7

Solution to the Maximization Problem

• Let BC(Σ) be the space of bounded con-tinuous functions defined on Σ. and sup-pose ` is a non negative element in BC(Σ).

• Let M(Σ) = (BC(Σ))∗ be the space ofbounded regular finitely additive measureson (Σ,B(Σ)), and R > 0 a given constant.

• Let M1(Σ) ⊂M(Σ) be the space of prob-ability measures defined on the above mea-surable space. Define the uncertainty setby

BR(µ) = ν ∈M1(Σ); ||ν − µ|| ≤ R

• The average cost is defined as a linear func-tional acting on ` ∈ BC(Σ). Hence thefollowing problem

J(`, R) = supν∈BR(µ)

∫Σ

`(x)ν(dx) (2)

8

Solution to the Maximization Problem

• The set BR(µ) is weak∗-compact, and also

the cost functional is weak∗ continuous,

hence there exists a maximizing measure

in BR(µ).

• The optimization in (2) can be solved by

using Hahn-Banach theorem[Rudin91]. Since

` ∈ BC(Σ) is fixed, then there exists an

η ∈ (BC(Σ))∗ such that

η(`) =∫Σ

`dη = ||`||∞, and ||η|| = 1 (3)

Using the above we get

supν∈BR(µ)

∫Σ

`(x)ν(dx) = R||`||∞ + Eµ(`) (4)

The supremum is attained by ν∗ ∈ (BC(Σ))∗.

9

Solution to the Maximization Problem

Lemma 1 The maximizing measure ν∗ in (4)is non-negative.

The maximizing measure ν∗ is not unique. As-sume that an appropriate metric such as d(·, ·)is defined on the space Σ.

Lemma 2 Suppose ` : Σ → < is a boundednon-negative measurable function, and η is afinitely additive non-negative finite measure de-fined on (Σ,B(Σ)). Assume• the total weight of the measure η is not con-centrated on any bounded measurable subsetof Σ, and• the support of η contains the point at which` attains its maximum.

Then

sups>0

∫Σ `(x)es`(x)η(dx)∫

Σ es`(x)η(dx)= ||`||∞

10

Solution to the Maximization Problem

Theorem 1 Under the same assumptions of

Lemma 2, there exists a family of probability

measures which attain the supremum in (2).

11

Fully Observed Uncertain Control Systems

• Let x(t)t≥0 denote the state process sub-

ject to control and u(t)t≥0 be the control

process, both defined for a fixed and finite

time interval [0, T ].

• For each u ∈ Uad the nominal state and the

observation process are governed by the

following system of stochastic differential

equations:

(Ω,F , P ) :dx(t) = b(t, x(t), u(t))dt + σ(t, x(t), u(t))dw(t),

x(0) = ξ

Where P is the nominal probability measure

P ∈ M(u).

12

Fully Observed Uncertain Control Systems

Definition 1 The set of admissible controls

denoted by Uw[s, T ] is the set of all 5−tuples

(Ω,F , P, w(·), u(·)) which satisfy the following

conditions.

1) (Ω,F , P ) is a complete probability space.

2) w(t)t≥s as an <m-dimensional standard

Brownian motion defined on (Ω,F , P ) over the

time interval [s, T ], with w(s) = 0, P −a.s., and

Fs,t is generated by σw(r); s ≤ r ≤ t aug-

mented by the P−null sets in F.

3) u : [s, T ] × Ω → U is an Fs,tt≥s-adapted

process on (Ω,F , P ).

4) For each u(·), and any y ∈ <n, SDE admits a

unique weak solution x(·) on (Ω,F , Fs,tt≥s, P )

such that x(s) = y.

5) f(·, x(·), u(·)) ∈ L1F(0, T ;<).

13

Fully Observed Uncertain Control Systems

The space Ω is defined as follows: Ω = Ωx ×Ωu ×Ωw, where

Ωx = C([0, T ];<n) ,Ωu = L2([0, T ];U)

Ωw = C([0, T ];<m)

Assumptions 1 The nominal system satisfiesthe following assumptions:1) (U, d) is Polish space, where U ⊂ <k. Thecontrol u(t); t ∈ [0, T ] is non anticipative andtakes values in U which is compact and convex.

2) The maps b : [0, T ]×<n×U → <n, σ : [0, T ]×<n×U → <n×m, f : [0, T ]×<n×U → < and h :<n → < are uniformly continuous. Also σ andb are assumed to be bounded. There exists aconstant L such that φ(t, x, u) = b(t, x, u), f(t, x, u), h(x), σ(t, x, u) satisfy

|φ(t, x, u)− φ(t, x, u)| ≤ L|x− x|, ∀x, x ∈ <n,

u ∈ Uad, ∀t ∈ [0, T ]

|φ(t,0, u)| ≤ L, ∀(t, u) ∈ [0, T ]× U

14

Fully Observed Uncertain Control Systems

Under the previous two assumptions, for any(s, y) ∈ [0, T ) × <n and u(·) ∈ Uw[s, T ], theSDE admits a unique weak solution x(·) ≡x(·; s, y, u(·)).

The precise problem statement should thus, beas follows.

Problem 1 Given the nominal measure P ∈M(u), find a u∗ ∈ Uad[s, T ] and a probabilitymeasure Qu∗,∗ ∈M(Ω) which solve the follow-ing constrained optimization problem.

J(u∗, Qu∗,∗) = infu(·)∈Uw[s,T ]

supQ∈M(u)

(EQ

∫ T

sf(t, x(t), u(t))dt + h(x(T ))

)subject to fidelity

||Q− P || ≤ R, R ∈ (0,2] (5)

15

Fully Observed Uncertain Control Systems

Using Lemma 2, this problem can be re-writtenin an exponential form as follows.

J(u∗, Qu∗,∗)= inf

u(·)∈Uw[s,T ]

(R. ‖

∫ T

sf(t, x(t), u(t))dt + h(x(T )) ‖∞

+EP∫ T

sf(t, x(t), u(t))dt + h(x(T ))

)(6)

where above ‖ · ‖∞ is defined over the spaceΩ, and Qu,∗ is given by the following measure.

Qu,∗(E)

= γ.eα

∫ Ts f(t,x(t),u(t))+h(x(T ))

Eη(eα

∫ Ts f(t,x(t),u(t))+h(x(T )))

η(E)

+(1− γ))P (E) (7)

where E ∈ F and α depends on the choiceof control u(·). Notice that η ∈ M1(Ω) is anarbitrary measure which is not unique.

16

Conclusion and Future Work

• Optimal control for a class of uncertain

stochastic systems has been considered.

• The model for uncertainty allows a wide

range possibilities, including ambiguities in

the diffusion term.

• The control problem has been formulated

as a min-max game and this leads to a non-

linear control problem with an exponential

tilted cost functional.

• Due to the highly nonlinear nature of this

cost function, the issues of principle of

optimality and dynamic programming has

to be further studied.

17

References

[Pinsker64] M.S. Pinsker, Information and In-

formation Stability of Random Variables and

Processes, Holden-Day, San Francisco, 1964.

[Meneghini96] P. Dai Pra, L. Meneghini, W.

J. Runggaldier, Connections between Stochas-

tic control and Dynamic Games, Matehmatics

of Control, Signals and Systems, vol.9, No.4,

1996, pp.303-326.

[Petersen99] V. A. Ugrinovskii, I.R. Petersen,

Finite Horizon Minimax Optimal Control of Stochas-

tic Partially Observed Time Varying Uncertain

Systems, Mathematics of Control, Signal and

Systems, vol.12, 1999, pp 1-23.

18

References

[Petersen2000] I.R. Petersen, M.R. James, P.Dupuis, Minimax Optimal Control of Stochas-tic Uncertain Systems with Relative EntropyConstraints, IEEE Transactions on AutomaticControl, Vol. 45, No. 3, March 2000, pp 398-412.

[Hansen2001] L.P. Hansen, T. J. Sargent, Ro-bust Control and Model Uncertainty, The Amer-ican Economic Review, Vol.91, No.2, pp.60-66, 2001.

[Ahmed2005] N.U. Ahmed, C.D. Charalambous,Minimax Games for Stochastic Systems Sub-ject to Relative Entropy Constraints: Applica-tions to SDE’s on Hilbert Space, Mathematicsof Control, Signals, and Systems , 2007 (toappear).

[Rudin91] W. Rudin, Functional Analysis, McGraw-Hill, New York, 1991.

19