Post on 31-Mar-2023
Optimization of Stochastic Uncertain
Systems with Variational Norm
Constraints
†,∗Farzad Rezaei, †Charalambos D. Charalambousand
‡Nasir U. Ahmed
†,∗School of Information Technology and EngineeringUniversity of Ottawa
†Department of Electrical and Computer EngineeringUniversity of Cyprus, CYPRUS
‡Also with Department of MathematicsUniversity of Ottawa
46th IEEE Conference on Decision and ControlDecember 12-14, 2007
New Orleans, LA, U.S.A.
0
Introduction
1. Introduction and Motivations
2. Problem Formulation
3. Solution to the Maximization Problem
4. Fully Observed Uncertain Control Sys-
tems
5. Conclusion and Future Work
1
Introduction and Motivations
Optimization of uncertain stochastic systems.
Pay-off is defined as a linear functional of the
uncertain measure.
Modeling of unceratainty:
Many papers have considered relative entropy
constraint model.[meneghini96],[petersen99], and
also [petersen2000],[Hansen2001],[ahmed2005].
2
Introduction and Motivations
Shortcomings of Relative Entropy Model
• It is not a true metric on the space of prob-
ability measures.
• Relative entropy between two probability
measures is only defined when the mea-
sures are absolutely continuous.
• In the context of SDEs, relative entropy
allows uncertainty in the drift term, but it is
unable to model uncertainty in the diffusion
coefficient.
3
Introduction and Motivations
Proposed model Uncertainty model based on
the variational norm on the space of probability
measures.
• The uncertainty is described by a ball in
the space of probability measures, centered
at the nominal measure and with a non-
negative radius.
• From Pinsker’s inequality[Pinsker64], it fol-
lows that an uncertainty set described by
relative entropy is a subset of the much
larger variational norm uncertainty set.
4
Problem Formulation
• Let (Σ, d) denote a complete separable met-
ric space (a Polish space).
• Let (Σ,B(Σ)) be the corresponding mea-
surable space, in which B(Σ) is the σ-algebra
generated by open sets in Σ.
• Let M1(Σ) denote the set of probability
measures on (Σ,B(Σ)) and Uad the set of
admissible controls.
• By choosing a control policy u for the nom-
inal system, the nominal system induces a
probability measure µu ∈M1(Σ).
5
Problem Formulation
• For a given u ∈ Uad, let M(u) ⊂ M1(Σ)
denote the set of probability measures in-
duced by the perturbed system while con-
trol u is applied.
• Let `u : Σ → < be a real-valued bounded
non-negative measurable function; and
||ν −µ|| denote the variational distance be-
tween measures ν and nominal µ both from
M(u).
• The uncertainty tries to maximize the av-
erage cost functional denoted by∫Σ `udν
subject to some constraints on the varia-
tional distance between ν and µ.
6
Problem Formulation
• Let Mc(u) ⊂ M(u) denote such a constraint.
The effect of uncertainty leads to the fol-
lowing maximization problem:
supν∈Mc(u)
∫Σ
`udν,
for every control u ∈ Uad.
• The designer on the other hand, tries to
choose a control policy to minimize the
worst case average cost. This gives rise
to the min-max problem
infu∈Uad
supν∈Mc(u)
∫Σ
`udν. (1)
• The constaint set is defined as follows.
Mc(u)4= Bu
R(µ) = ν ∈ M(u) : ||ν − µ|| ≤ R
7
Solution to the Maximization Problem
• Let BC(Σ) be the space of bounded con-tinuous functions defined on Σ. and sup-pose ` is a non negative element in BC(Σ).
• Let M(Σ) = (BC(Σ))∗ be the space ofbounded regular finitely additive measureson (Σ,B(Σ)), and R > 0 a given constant.
• Let M1(Σ) ⊂M(Σ) be the space of prob-ability measures defined on the above mea-surable space. Define the uncertainty setby
BR(µ) = ν ∈M1(Σ); ||ν − µ|| ≤ R
• The average cost is defined as a linear func-tional acting on ` ∈ BC(Σ). Hence thefollowing problem
J(`, R) = supν∈BR(µ)
∫Σ
`(x)ν(dx) (2)
8
Solution to the Maximization Problem
• The set BR(µ) is weak∗-compact, and also
the cost functional is weak∗ continuous,
hence there exists a maximizing measure
in BR(µ).
• The optimization in (2) can be solved by
using Hahn-Banach theorem[Rudin91]. Since
` ∈ BC(Σ) is fixed, then there exists an
η ∈ (BC(Σ))∗ such that
η(`) =∫Σ
`dη = ||`||∞, and ||η|| = 1 (3)
Using the above we get
supν∈BR(µ)
∫Σ
`(x)ν(dx) = R||`||∞ + Eµ(`) (4)
The supremum is attained by ν∗ ∈ (BC(Σ))∗.
9
Solution to the Maximization Problem
Lemma 1 The maximizing measure ν∗ in (4)is non-negative.
The maximizing measure ν∗ is not unique. As-sume that an appropriate metric such as d(·, ·)is defined on the space Σ.
Lemma 2 Suppose ` : Σ → < is a boundednon-negative measurable function, and η is afinitely additive non-negative finite measure de-fined on (Σ,B(Σ)). Assume• the total weight of the measure η is not con-centrated on any bounded measurable subsetof Σ, and• the support of η contains the point at which` attains its maximum.
Then
sups>0
∫Σ `(x)es`(x)η(dx)∫
Σ es`(x)η(dx)= ||`||∞
10
Solution to the Maximization Problem
Theorem 1 Under the same assumptions of
Lemma 2, there exists a family of probability
measures which attain the supremum in (2).
11
Fully Observed Uncertain Control Systems
• Let x(t)t≥0 denote the state process sub-
ject to control and u(t)t≥0 be the control
process, both defined for a fixed and finite
time interval [0, T ].
• For each u ∈ Uad the nominal state and the
observation process are governed by the
following system of stochastic differential
equations:
(Ω,F , P ) :dx(t) = b(t, x(t), u(t))dt + σ(t, x(t), u(t))dw(t),
x(0) = ξ
Where P is the nominal probability measure
P ∈ M(u).
12
Fully Observed Uncertain Control Systems
Definition 1 The set of admissible controls
denoted by Uw[s, T ] is the set of all 5−tuples
(Ω,F , P, w(·), u(·)) which satisfy the following
conditions.
1) (Ω,F , P ) is a complete probability space.
2) w(t)t≥s as an <m-dimensional standard
Brownian motion defined on (Ω,F , P ) over the
time interval [s, T ], with w(s) = 0, P −a.s., and
Fs,t is generated by σw(r); s ≤ r ≤ t aug-
mented by the P−null sets in F.
3) u : [s, T ] × Ω → U is an Fs,tt≥s-adapted
process on (Ω,F , P ).
4) For each u(·), and any y ∈ <n, SDE admits a
unique weak solution x(·) on (Ω,F , Fs,tt≥s, P )
such that x(s) = y.
5) f(·, x(·), u(·)) ∈ L1F(0, T ;<).
13
Fully Observed Uncertain Control Systems
The space Ω is defined as follows: Ω = Ωx ×Ωu ×Ωw, where
Ωx = C([0, T ];<n) ,Ωu = L2([0, T ];U)
Ωw = C([0, T ];<m)
Assumptions 1 The nominal system satisfiesthe following assumptions:1) (U, d) is Polish space, where U ⊂ <k. Thecontrol u(t); t ∈ [0, T ] is non anticipative andtakes values in U which is compact and convex.
2) The maps b : [0, T ]×<n×U → <n, σ : [0, T ]×<n×U → <n×m, f : [0, T ]×<n×U → < and h :<n → < are uniformly continuous. Also σ andb are assumed to be bounded. There exists aconstant L such that φ(t, x, u) = b(t, x, u), f(t, x, u), h(x), σ(t, x, u) satisfy
|φ(t, x, u)− φ(t, x, u)| ≤ L|x− x|, ∀x, x ∈ <n,
u ∈ Uad, ∀t ∈ [0, T ]
|φ(t,0, u)| ≤ L, ∀(t, u) ∈ [0, T ]× U
14
Fully Observed Uncertain Control Systems
Under the previous two assumptions, for any(s, y) ∈ [0, T ) × <n and u(·) ∈ Uw[s, T ], theSDE admits a unique weak solution x(·) ≡x(·; s, y, u(·)).
The precise problem statement should thus, beas follows.
Problem 1 Given the nominal measure P ∈M(u), find a u∗ ∈ Uad[s, T ] and a probabilitymeasure Qu∗,∗ ∈M(Ω) which solve the follow-ing constrained optimization problem.
J(u∗, Qu∗,∗) = infu(·)∈Uw[s,T ]
supQ∈M(u)
(EQ
∫ T
sf(t, x(t), u(t))dt + h(x(T ))
)subject to fidelity
||Q− P || ≤ R, R ∈ (0,2] (5)
15
Fully Observed Uncertain Control Systems
Using Lemma 2, this problem can be re-writtenin an exponential form as follows.
J(u∗, Qu∗,∗)= inf
u(·)∈Uw[s,T ]
(R. ‖
∫ T
sf(t, x(t), u(t))dt + h(x(T )) ‖∞
+EP∫ T
sf(t, x(t), u(t))dt + h(x(T ))
)(6)
where above ‖ · ‖∞ is defined over the spaceΩ, and Qu,∗ is given by the following measure.
Qu,∗(E)
= γ.eα
∫ Ts f(t,x(t),u(t))+h(x(T ))
Eη(eα
∫ Ts f(t,x(t),u(t))+h(x(T )))
η(E)
+(1− γ))P (E) (7)
where E ∈ F and α depends on the choiceof control u(·). Notice that η ∈ M1(Ω) is anarbitrary measure which is not unique.
16
Conclusion and Future Work
• Optimal control for a class of uncertain
stochastic systems has been considered.
• The model for uncertainty allows a wide
range possibilities, including ambiguities in
the diffusion term.
• The control problem has been formulated
as a min-max game and this leads to a non-
linear control problem with an exponential
tilted cost functional.
• Due to the highly nonlinear nature of this
cost function, the issues of principle of
optimality and dynamic programming has
to be further studied.
17
References
[Pinsker64] M.S. Pinsker, Information and In-
formation Stability of Random Variables and
Processes, Holden-Day, San Francisco, 1964.
[Meneghini96] P. Dai Pra, L. Meneghini, W.
J. Runggaldier, Connections between Stochas-
tic control and Dynamic Games, Matehmatics
of Control, Signals and Systems, vol.9, No.4,
1996, pp.303-326.
[Petersen99] V. A. Ugrinovskii, I.R. Petersen,
Finite Horizon Minimax Optimal Control of Stochas-
tic Partially Observed Time Varying Uncertain
Systems, Mathematics of Control, Signal and
Systems, vol.12, 1999, pp 1-23.
18
References
[Petersen2000] I.R. Petersen, M.R. James, P.Dupuis, Minimax Optimal Control of Stochas-tic Uncertain Systems with Relative EntropyConstraints, IEEE Transactions on AutomaticControl, Vol. 45, No. 3, March 2000, pp 398-412.
[Hansen2001] L.P. Hansen, T. J. Sargent, Ro-bust Control and Model Uncertainty, The Amer-ican Economic Review, Vol.91, No.2, pp.60-66, 2001.
[Ahmed2005] N.U. Ahmed, C.D. Charalambous,Minimax Games for Stochastic Systems Sub-ject to Relative Entropy Constraints: Applica-tions to SDE’s on Hilbert Space, Mathematicsof Control, Signals, and Systems , 2007 (toappear).
[Rudin91] W. Rudin, Functional Analysis, McGraw-Hill, New York, 1991.
19