Parameter Estimation with Expected andResidual-at-Risk Criteria

Giuseppe Calafiore!, Ufuk Topcu†, and Laurent El Ghaoui‡

Abstract

In this paper we study a class of uncertain linear estimation problems in whichthe data are a!ected by random uncertainty. In this setting, we consider two esti-mation criteria, one based on minimization of the expected !1 or !2 norm residualand one based on minimization of the level within which the !1 or !2 norm residualis guaranteed to lie with an a-priori fixed probability (residual at risk). The randomuncertainty a!ecting the data is characterized by means of its first two statistical mo-ments, and the above criteria are intended in a worst-case probabilistic sense, thatis worst-case expectations and probabilities over all possible distribution having thespecified moments are considered. The ensuing estimation problems can be solvede"ciently via convex programming, yielding exact solutions in the !2 norm case andupper-bounds on the optimal solutions in the !1 case.

Keywords: Uncertain least-squares, random uncertainty, robust convex optimization,value at risk, !1 norm approximation.

1 Introduction

To introduce the problem treated in this paper, let us first consider a standard parameterestimation problem where an unknown parameter ! " Rn is to be determined so to minimizea norm residual of the form #A!$b#p, where A " Rm,n is a given regression matrix, b " Rm

is a measurement vector, and # ·# p denotes the "p norm. In this setting, the most relevantand widely studied case arise of course for p = 2, where the problem reduces to classicalleast-squares. The case of p = 1 also has important applications due to its resilience tooutliers and to the property of producing “sparse” solutions, see for instance [5, 7]. Forp = 1, the solution to the norm minimization problem can be e!ciently computed vialinear programming, [3, §6.2].

In this paper we are concerned with an extension of this basic setup that arises inrealistic cases where the problem data A, b are imprecisely known. Specifically, we considerthe situation where the entries of A, b depend a!nely on a vector # of random uncertainparameters, that is A

.= A(#) and b

.= b(#). Due its practical significance, the parameter

estimation problem in the presence of uncertainty in the data has attracted much attention

!G. Calafiore is with the Dipartimento di Automatica e Informatica, Politecnico di Torino, Italy. e-mail:giuseppe.calafiore@polito.it

†U. Topcu is with the Department of Mechanical Engineering, University of California, Berkeley, CA,94720 USA. e-mail: utopcu@jagger.me.berkeley.edu

‡Laurent El Ghaoui is with the Department of Electrical Engineering and Computer Science, Universityof California, Berkeley, CA, 94720, USA. e-mail: elghaoui@eecs.berkeley.edu.

1

To appear in: Systems and Control Letters (Elsevier)

in the literature. When the uncertainty is modeled as unknown-but-bounded, a min-maxapproach is followed in [8], where the maximum over the uncertainty of the "2 norm ofthe residual is minimized. Relations between the min-max approach and regularizationtechniques are also discussed in [8] and in [12]. Generalizations of this approach to "1 and"! norms are proposed in [10].

In the case when the uncertainty is assumed to be random with given distribution, aclassical stochastic optimization approach is often followed, whereby a ! is sought thatminimizes the expectation of the "p norm of the residual with respect to the uncertainty.This formulation leads in general to numerically “hard” problem instances, that can besolved approximately by means of stochastic approximation methods, see, e.g., [4]. In thespecial case where the squared Euclidean norm is considered, instead, the expected valueminimization problem actually reduces to a standard least-squares problem, which has aclosed-form solution, see [4, 10].

In this paper we consider the uncertainty to be random and we develop our results ina “statistical ambiguity” setting in which the probability distribution of the uncertainty isonly known to belong to a given family of distributions. Specifically, we consider the familyof all distributions on the uncertainty having a given mean and covariance, and seek resultsthat are guaranteed irrespective of the actual distribution within this class. We addressboth the "2 and "1 cases, under two di"erent estimation criteria: the first criterion aimsat minimizing the worst-case expected residual, whereas the second one is directly tailoredto control residual tail probabilities. That is, for given risk $ " (0, 1), we minimize theresidual level such that the probability of residual falling above this level is no larger than $.

The rest of the paper is organized as follows: Section 2 sets up the problem and givessome preliminary results. Section 3 is devoted to the estimation scheme with worst-caseexpected residual criterion, while Section 4 contains the results for the residual-at-riskcriterion. Numerical examples are presented in Section 5 and conclusions are drawn inSection 6. An Appendix contains some of the technical derivations.

Notation The identity matrix in Rn,n and the zero matrix in Rn,n are denoted as In and0n, respectively (subscripts may be omitted when they can easily be inferred from context).#x#p denotes the standard "p norm of vector x; #X#F denotes the Frobenius norm of matrixX, that is #X#F =

%Tr X"X, where Tr is the trace operator. The notation # & (#, D)

means that # is a random vector with expected value E {#} = # and covariance matrix

var {#} .= E

!(# $ #)(# $ #)"

"= D. The notation X ' 0 (resp. X ( 0) indicates that

matrix X is symmetric and positive definite (resp. semi-definite).

2 Problem setup and preliminaries

Let A(#) " Rm,n, b(#) " Rm be such that

[A(#) b(#)].= [A0 b0] +

q#

i=1

#i[Ai bi], (1)

where # = [#1 · · · #q]" is a vector random uncertainties, [A0 b0] represents the “nominal”data, and [Ai bi] are the matrices of coe!cients for the uncertain part of the data. Let

2

To appear in: Systems and Control Letters (Elsevier)

! " Rn be a parameter to be estimated, and consider the following norm residual (whichis a function of both ! and #):

fp(!, #).= #A(#)! $ b(#)#p (2)

= # [(A1! $ b1) · · · (Aq! $ bq)] # + (A0! $ b0)#p.= #L(!)z#p,

where we defined z.= [#" 1]", and L(!) " Rm,q+1 is partitioned as

L(!).= [L(!)(!) L(1)(!)], (3)

with

L(!)(!).= [(A1! $ b1) · · · (Aq! $ bq)] " Rm,q, L(1)(!)

.= A0! $ b0 " Rm. (4)

In the following we assume that E {#} = 0 and var {#} = Iq. This can be done without lossof generality, since data can always be pre-processed so to comply with this assumption,as detailed in the following remark.

Remark 1 (Preprocessing the data) Suppose that the uncertainty # is such that E {#} =# and var {#} = D ( 0, and let D = QQ" be a full-rank factorization of D. Then,we may write # = Q% + #, with E {%} = 0, var {%} = Iq, and redefine the problem interms of uncertainty % & (0, I), with L(!)(!) = [(A1! $ b1) · · · (Aq! $ bq)]Q, L(1)(!) =[(A1! $ b1) · · · (Aq! $ bq)]# + (A0! $ b0). &

We next state the two estimation criteria and the ensuing problems that are tackled inthis paper.

Problem 1 (Worst-case expected residual minimization) Determine ! " Rn thatminimizes sup!#(0,I) E {fp(!, #)}, that is solve

min"$Rn

sup!#(0,I)

E {#L(!)z#p}, z".= [#" 1], (5)

where p " {1, 2}, L(!) is given in (3), (4), and the supremum is taken with respect to allpossible probability distributions having the specified moments (zero mean and unit covari-ance).

In some applications, such as in financial Value-at-Risk (V@R) [6, 11], one is interested inguaranteeing that the residual remains “small” in “most” of the cases, that is one seeks !such that the corresponding residual is small with high probability. An expected residualcriterion such as the one considered in Problem 1 is not suitable for this purpose, sinceit concentrates on the average case, neglecting the tails of the residual distribution. Thesecond criterion that we consider is hence focused on controlling the risk of having residualsabove some level ' ) 0, where risk is expressed as the probability Prob {# : f(!, #) ) '}.Formally, we state the following second problem.

Problem 2 (Guaranteed residual-at-risk minimization) Fix a risk level $ " (0, 1).Determine ! " Rn such that a residual level ' is minimized while guaranteeing that Prob {# :fp(!, #) ) '} * $. That is, solve

min"$Rn,#%0

'

subject to : sup!#(0,I) Prob {# : #L(!)z#p ) '} * $, z".= [#" 1],

3

To appear in: Systems and Control Letters (Elsevier)

where p " {1, 2}, L(!) is given in (3), (4), and the supremum is taken with respect to allpossible probability distributions having the specified moments (zero mean and unit covari-ance).

A key preliminary result opening the way for the solution of Problem 1 and Problem 2 isstated in the next lemma. This lemma is a powerful consequence of convex duality, andprovides a general result for computing the supremum of expectations and probabilitiesover all distributions possessing a given mean and covariance matrix.

Lemma 1 Let S + Rn be a measurable set (not necessarily convex), and ( : Rn , R ameasurable function. Let z" = [x" 1], and define

Ewc.= sup

x#(x,!)E {((x)}

Pwc.= sup

x#(x,!)Prob {x " S}

Q.=

$# + xx" x

x" 1

%.

Then,

Ewc = infM=M!

Tr QM subject to: z"Mz ) ((x), -x " Rn (6)

and

Pwc = infM&0

Tr QM subject to: z"Mz ) 1, -x " S. (7)

&

A proof of Lemma 1 is given in the Appendix.

Remark 2 Lemma 1 provides a result for computing worst-case expectations and prob-abilities. However in many cases of interest we shall need to impose constraints on thesequantities in order to eventually optimize them with respect to some other design variables.In this respect, the following equivalence holds:

supx#(x,!) E {((x)} * ' (8)

./M = M" : Tr QM * ', z"Mz ) ((x), -x " Rn. (9)

To verify this fact, consider first the 0 direction: If (8) holds, we let M be the solution thatachieves the optimum in (6), and we have that (9) holds. On the other hand, if (9) holdsfor some M = M , then supx#(x,!) E {((x)} = inf Tr QM * Tr QM * ', which concludesproves the statement. By an analogous reasoning, we can verify that

supx#(x,!) Prob {x " S} * $

./M = M" : M ( 0, Tr QM * $, z"Mz ) 1, -x " S.

&

4

To appear in: Systems and Control Letters (Elsevier)

3 Worst-case expected residual minimization

In this section we focus on Problem 1 and provide an e!ciently computable exact solutionfor the case p = 2, and e!ciently computable upper and lower bounds on the solution forthe case p = 1. Define

)p(!).= sup

!#(0,I)E {#L(!)z#p}, with z"

.= [#" 1], (10)

r.= [0 · · · 0 1/2]" " Rq+1, (11)

where L(!) " Rm,q+1 is an a!ne function of parameter !, given in (3), (4). We have thefollowing preliminary lemma.

Lemma 2 For given ! " Rn, the worst-case residual expectation )p(!) is given by

)p(!) = infM=M!

Tr M

subject to: M $ ru"L(!)$ L(!)"ur" ( 0, -u " Rm : #u#p" * 1,

where #u#p" is the dual "p norm. &

Proof. From Lemma 1 we have that

)p(!) = infM=M!

Tr M

subject to: z"Mz ) #L(!)z#p, -# " Rq.

Since#L(!)z#p = sup

'u'p"(1u"L(!)z,

it follows that z"Mz ) #L(!)z#p holds for all # if and only if

z"Mz ) u"L(!)z, -# " Rq and -u " Rm : #u#p" * 1.

Now, since z"r = 1/2, we write u"L(!)z = z"(ru"L(!) + L(!)"ur")z, whereby the abovecondition is satisfied if and only if

M $ ru"L(!) + L(!)"ur" ( 0, -u : #u#p" * 1,

which concludes the proof. !

We are now in position to state the following key theorem.

Theorem 1 Let ! " Rn be given, and let )p(!) be defined as in (10). Then, the followingholds for the worst-case expected residuals in the "1- and "2-norm cases.

1. Case p = 1: Define

)1(!).=

m#

i=1

&&Li(!)"&&

2, (12)

where Li(!)" denotes the i-th row of L(!). Then,

2

*)1(!) * )1(!) * )1(!). (13)

5

To appear in: Systems and Control Letters (Elsevier)

2. Case p = 2:)2(!) =

'Tr L(!)"L(!) = #L(!)#F . (14)

&

Proof. (Case p = 1) The dual "1 norm is the "! norm, hence applying Lemma 2 we have

)1(!) = infM=M! Tr M (15)

subject to: M $ L(!)"ur" $ ru"L(!) ( 0, -u : #u#! * 1.

For ease of notation, we drop the dependence on ! in the following derivation. Note that

L"ur" + ru"L =m#

i=1

uiCi,

where

Ci.= rL"i + Lir

" =

(0q

12L

(!)i

12L

(!)"i L(1)

i

)" Rq+1,q+1,

where L"i is partitioned according to (4) as L"i = [L(!)"i L(1)

i ], with L(!)"i " R1,q , and

L(1)i " R. The characteristic polynomial of Ci is pi(s) = sq)1(s2$L(1)

i s$#L(!)i #2

2/4), hence

Ci has q $ 1 null eigenvalues, and two non-zero eigenvalues at +i,1 = (L(1)i + #Li#2)/2 > 0,

+i,2 = (L(1)i $ #Li#2)/2 < 0. Since Ci is rank two, the constraint in problem (15) takes the

form (23) considered in Theorem 4 in the Appendix. Consider thus the following relaxationof problem (15):

,.= inf

M=M!,Xi=X!i

Tr M (16)

subject to: $Xi + Ci 1 0, $Xi $ Ci 1 0, i = 1, . . . , m,*m

i=1 Xi $M 1 0,

where we clearly have )1 * ,. The dual of problem (16) can be written as

,D = sup"i,!i

m#

i=1

Tr (($i $ #i)Ci) (17)

subject to: $i + #i = Iq+1,

#i ( 0, $i ( 0, i = 1, . . . ,m.

Since the problem in (16) is convex and Slater conditions are satisfied, , = ,D. Next weshow that ,D equals )1 given in (12). To this end, observe that (17) is decoupled in the#i, $i variables and, for each i, the subproblem amounts to determining sup0*!i*I Tr (I $2#i)Ci. By diagonalizing Ci as Ci = Vi%iV "

i , with %i = diag(0, . . . , 0, +i,1, +i,2), eachsubproblem is reformulated as sup0*!i*I Tr Ci$2Tr %i#i, where it immediately follows that

the optimal solution is #i = diag(0, . . . , 0, 0, 1), hence the supremum is (+i,1 ++i,2)$2+i,2 =+i,1 $ +i,2 = |+i,1|+ |+i,2| = #eig(Ci)#1, where eig(·) denotes the vector of the eigenvalues ofits argument. Now, we have #eig(Ci)#1 = #L"i #2, then ,D =

*mi=1 #L"i #2, and by the first

conclusion in Theorem 4 in the Appendix, we have )1 = , = ,D and )1 * )1.For the lower bound on )1 in (13), assume that the problem in (17) is not feasible.

Then, for M ( 0, we have that+M : Tr M = ,D

, -

{M : Xi ( ±Ci,*n

i=1 Xi 1 M} = 2.

6

To appear in: Systems and Control Letters (Elsevier)

This last emptiness statement, coupled with the fact that, for i = 1, . . . , n, Ci is of ranktwo, implies, by the second conclusion in Theorem 4, that

+M : Tr M = ,D

, -

{M : M (*n

i=1 uiCi, -u : |ui| * */2} = 2

and !M : Tr M = $D

%/2

"-!

M : M (*n

i=1 uiCi, -u : |ui| * 1"

= 2

Consequently, we have )1 ) $D

%/2 = &1%/2 , which concludes the proof of the p = 1 case.

(Case p = 2) The dual "2 norm is the "2 norm itself, hence applying Lemma 2 we have

)2 = infM=M!

Tr M

subject to: M $ ru"L$ L"ur" ( 0, -u : #u#2 * 1.

Applying the LMI robustness lemma (Lemma 3.1 of [9]), we have that the previous semi-infinite problem is equivalent to the following SDP

)2(!) = infM=M!,'>0

Tr M subject to:

$M $ -rr" L"

L -Im

%( 0.

By the Schur complement rule, the latter constraint is equivalent to - > 0 and M (1' (L"L)+-rr". Thus, the infimum of Tr M is achieved for M = 1

' (L"L)+-rr" and, since

rr" = diag(0q, 1/4), the infimum of Tr M over - > 0 is achieved for - = 2%

Tr L"L. Fromthis, it follows that )2 =

%Tr L"L, thus concluding the proof. !

Starting from the results in Theorem 1, it is easy to observe that we can further minimizethe residuals over the parameter !, in order to find a solution to Problem 1. Convexityof the ensuing minimization problem is a consequence of the fact that L(!) is an a!nefunction of !. This is formalized in the following corollary, whose simple proof is omitted.

Corollary 1 (Worst-case expected residual minimization) Let

)+p.= min

"$Rnsup

!#(0,I)E {#L(!)z#p}, z"

.= [#" 1].

For p = 1, it holds that2

*)+1 * )+1 * )

+1,

where )+1 is computed by solving the following second-order-cone (SOCP) program:

)+1 = min

"$Rn

m#

i=1

#Li(!)"#2.

For p = 2, it holds that)+2 = min

"$Rn#L(!)#F ,

where a minimizer for this problem can be computed via convex quadratic programming, byminimizing Tr L"(!)L(!). &

7

To appear in: Systems and Control Letters (Elsevier)

Remark 3 Notice that in the specific case of # & (0, I) we have that )22 = Tr L"(!)L(!) =*q

i=0 #Ai! $ bi#22, hence the minimizer can in this case be determined by standard Least-

Squares solution method. Interestingly, this solution coincides with the solution of theexpected squared "2-norm minimization problem discussed for instance in [4, 10]. Thismight not be obvious, since in general E {# · #2} 3= (E {# · #})2. &

4 Guaranteed residual-at-risk minimization

4.1 The "2-norm case

Assume first ! " Rn is fixed, and consider the problem of computing

Pwc2(!) = sup!#(0,I)

Prob {# : #L(!)z#2 ) '} = sup!#(0,I)

Prob {# : #L(!)z#22 ) '2},

where z".= [#" 1]. By Lemma 1, this probability corresponds to the optimal value of the

optimization problem

Pwc2(!) = infM&0

Tr M

subject to: z"Mz ) 1, -# : #L(!)z#22 ) '2,

where the constraint can be written equivalently as

z" (M $ diag(0q, 1)) z ) 0, -# : z".L(!)"L(!)$ diag(0q, '

2)/z ) 0.

Applying the lossless S-procedure, the condition above is in turn equivalent to the existenceof - ) 0 such that (M $ diag(0q, 1)) ( -

.L(!)"L(!)$ diag(0q, '2)

/, therefore we obtain

Pwc2(!) = infM&0,'>0

Tr M

subject to: M ( -L(!)"L(!) + diag(0q, 1$ -'2),

where the latter expression can be further elaborated using the Schur complement formulainto $

M $ diag(0q, 1$ -'2) -L(!)"

-L(!) -Im

%( 0. (18)

We now notice, by the reasoning in Remark 2, that the condition Pwc2(!) * $ with $ " (0, 1)is equivalent to the conditions

/- ) 0, M ( 0 such that: Tr M * $, and (18) holds.

A parameter ! that minimizes the residual-at-risk level ' while satisfying the conditionPwc2(!) * $ can thus be computed by solving a sequence of convex semidefinite optimizationproblems parameterized in - , as formalized in the next theorem.

Theorem 2 ("2 residual-at-risk estimation) A solution of Problem 2 in the "2 casecan be found by solving the following optimization problem:

inf'>0

infM&0,"$Rn,#2$Rn

'2, (19)

subject to: Tr M * $$M $ diag(0q, 1$ -'2) -L(!)"

-L(!) -Im

%( 0.

&

8

To appear in: Systems and Control Letters (Elsevier)

Remark 4 The optimization problem in Theorem 2 is not jointly convex in all variables,due to the presence of product terms -'2, -L(!). However, the problem is convex inM, !, '2 for each fixed - > 0, whence an optimal solution can be computed via a linesearch over - > 0, where each step requires solving an SDP in M , !, and '2. &

4.2 The "1-norm case

We next consider the problem of determining ! " Rn such that the residual-at-risk level 'is minimized while guaranteeing that Pwc1(!) * $, where Pwc1(!) is the worst-case "1-normresidual tail probability

Pwc1(!) = sup!#(0,I)

Prob {# : #L(!)z#1 ) '},

and $ " (0, 1) is the a-priori fixed risk level. To this end, define

D .= {D " Rm,m : D diagonal, D ' 0}

and consider the following proposition (whose statement may be easily proven by takingthe gradient with respect to D and setting it to zero).

Proposition 1 For any v " Rm, it holds that

#v#1 =1

2inf

D$D

m#

i=1

0v2

i

di+ di

1=

1

2inf

D$D

.v"D)1v + TrD

/, (20)

where di is the i-th diagonal entry of D. &

The following key theorem holds.

Theorem 3 ("1 residual-at-risk estimation) Consider the following optimization prob-lem:

inf'>0

infM&0,D$D,"$Rn,#%0

' (21)

subject to: Tr M * $$M $ (1$ 2-' + Tr D)J -L(!)"

-L(!) D

%( 0,

with J.= diag(0q, 1). The optimal value of this program provides an upper bound for

Problem 2 in the "1 case, that is an upper bound on the minimum level ' for which thereexist ! such that Pwc1(!) * $. &

Remark 5 Similar to the problem (19) in Theorem 2, the above optimization problem isnon convex, due to product terms between - and ', !. The problem can however be easilysolved numerically via a line search over the scalar - > 0, where each step of the line searchinvolves the solution of an SDP problem in the variables M , D, !, and '. &

Proof. Define

S.= {# : #L(!)z#1 ) '}

S(D).= {# : z"L(!)"D)1L(!)z + Tr D ) 2', D " D}.

9

To appear in: Systems and Control Letters (Elsevier)

For ease of notation we drop the dependence on ! in the following derivation. Using (20)we have that, for any D " D,

2#Lz#1 * z"L"D)1Lz + Tr D,

hence # " S implies # " S(D), thus S + S(D), for any D " D. This in turn implies that

Prob {# " S} * Prob {# " S(D)} * sup!#(0,I)

Prob {# " S(D)}

for any probability measure and any D " D, and therefore

Pwc1 = sup!#(0,I)

Prob {# " S} * infD$D

sup!#(0,I)

Prob {# " S(D)} .= Pwc1.

Note that, for fixed D " D, we can compute Pwc1(D).= sup!#(0,I) Prob {# " S(D)} from

its equivalent dual:

Pwc1(D) = infM&0

Tr M : z"Mz ) 1, -# " S(D)

= infM&0

Tr M : z"Mz ) 1, -# : z"L"D)1Lz + Tr D ) 2'

[applying the lossless S-procedure]

= infM&0,'>0

Tr M : M ( -L"D)1L + (1$ 2-' + -Tr D)J,

where J = diag(0q, 1). Hence, Pwc1 is obtained by minimizing Pwc1(D) over D " D, whichresults in

Pwc1 = infM&0,'>0,D$D

Tr M : M ( -L"D)1L + (1$ 2-' + -Tr D)J

[by change of variable -D , D (whence -Tr D , Tr D, -D)1 , - 2D)1)]

= infM&0,'>0,D$D

Tr M : M ( - 2L"D)1L + (1$ 2-' + Tr D)J

= infM&0,'>0,D$D

Tr M :

$M $ (1$ 2-' + Tr D)J -L"

-L D

%( 0.

Now, from the reasoning in Remark 2, we have that (re-introducing the dependence on !in the notation) Pwc1(!) * $ if and only if there exist M ( 0, - > 0 and D " D such thatTr M * $ and $

M $ (1$ 2-' + Tr D)J -L(!)"

-L(!) D

%( 0. (22)

Notice that, since L(!) is a!ne in ! , condition (22) is an LMI in M, D, !, ', for fixed - .We can thus minimize the residual level ' subject to the condition Pwc1(!) * $ (whichimplies Pwc1(!) * $), and this results in the statement of the theorem. !

5 Numerical examples

5.1 Worst-case expected residual minimization

As a first example, we use data from a numerical test appeared in [4]. Let

A(#) = A0 +3#

i=1

#iAi, bT =2

0 2 1 33,

10

To appear in: Systems and Control Letters (Elsevier)

with A0 =

4

556

3 1 40 1 1$2 5 31 4 5.2

7

889 A1 =

4

556

0 0 00 0 00 0 00 0 1

7

889, A2 =

4

556

0 0 10 1 00 0 00 0 0

7

889, A3 =

4

556

0 0 00 0 01 0 00 0 0

7

889,

and let #i be independent random perturbations with zero mean and standard deviations.1 = 0.067, .2 = 0.1, .3 = 0.2. The standard "2 and "1 solutions (obtained neglecting theuncertainty terms, i.e. setting A(#) = A0) result to be

!nom2 =

4

6$10$9.7289.983

7

9 , !nom1 =

4

6$11.8235$11.588211.7647

7

9 ,

with nominal residuals of 1.7838 and 1.8235, respectively.Applying Theorem 1, the minimal worst-case expected "2 residual resulted to be )+2 =

2.164, whereas the minimal upper bound on worst-case expected "1 residual resulted to be)+1 = 4.097. The corresponding parameter estimates are

!ewc2 =

4

6$2.3504$2.07472.4800

7

9 , !ewc1 =

4

6$2.8337$2.52522.9047

7

9 .

We next analyzed numerically how the worst-case expected residuals increase with the levelof perturbation. To this end, we consider the previous data with standard deviations onthe perturbation depending on a parameter / ) 0: .1 = / · 0.067, .2 = / · 0.1, .3 = / · 0.2.A plot of the worst-case expected residuals as a function of / is shown in Figure 1. Weobserve that both "1 and "2 expected residuals tend to a constant value for large /.

0 0.5 1 1.5 2 2.5 31.5

2

2.5

3

3.5

4

4.5

!

"2#

"1#

Figure 1: Plot of )+2 (solid) and )+1 (dashed) as a function of perturbation level /.

11

To appear in: Systems and Control Letters (Elsevier)

5.2 Guaranteed residual at risk minimization

As a first example of guaranteed residual at risk minimization, we consider again thevariable perturbation level problem of the previous section. Here, we fix the risk level to$ = 0.1 and solve repeatedly problems (19) and (21) for increasing values of /. A plot of theresulting optimal residuals at risk as a function of / is shown in Figure 2. These residualsgrow with the covariance level /, as it might be expected since increasing the covarianceincreases the tails of the residual distribution.

0 0.5 1 1.5 2 2.5 3

2.28

2.3

2.32

2.34

2.36

2.38

0 0.5 1 1.5 2 2.5 34.18

4.2

4.22

4.24

4.26

4.28

4.3

4.32

! !

||.||

resid

ual

@ r

isk

0.1

2

||.||

resid

ual

@ r

isk

0.1

1

Figure 2: Worst-case "2 and "1 residuals at risk as a function of perturbation level /.

As a further example, we consider a system identification problem where one seeks toestimate the impulse response of a discrete-time linear FIR system from its input/outputmeasurements. Let the system be described by the convolution

yk =N#

j=1

hj · uk)'+1, k = 1, 2, . . . ,

where ui is the input, yi is the output, and h = [h1 · · · hN ] is the impulse response to beestimated. If N input/output measurements are collected in vectors u"

.= [u1 u2 · · · uN ],

y".= [y1 y2 · · · yN ], then the impulse response h can be computed by solving the system

of linear equations Uh = y, where U is a lower-triangular Toeplitz matrix having u inthe first column. In practice, however, both u and y might be a"ected by errors, that isU(#u) = U +

*Ni=1 #uiUi, y(#y) = y +

*Ni=1 #yiei where ei is the i-th column of the identity

matrix in RN , and Ui is a lower-triangular Toeplitz matrix having ei in the first column.These uncertain data are easily re-written in the form (1) by setting ! = h, q = 2N ,A0 = U , b0 = y, and, for i = 1, . . . , 2N ,

Ai =

:Ui, if i * N0N , otherwise

, bi =

:0, if i * Nei)N , otherwise

, #i =

:#ui, if i * N#y,i)N , otherwise

12

To appear in: Systems and Control Letters (Elsevier)

We considered the same data used in a similar example in [8], that is u" = [1 2 3],y" = [4 5 6], and assumed that the input/output measurements are a"ected by i.i.d.errors with zero mean and unit variance. The nominal system Uh = y has minimum-normsolution hnom = [4 $ 3 0]". First, we solved the "2 residual-at-risk problem in (19), settingrisk level $ = 0.1. The optimal solution was achieved for - = 0.013 and yielded an optimalworst-case residual at risk ' = 11.09, with corresponding parameter

h(2,rar = [0.7555 0.4293 0.1236]".

This means that, no matter what the distribution on the uncertainty is (as long has ithas the assumed mean and covariance), the probability of having a residual larger than' = 11.09 is smaller than $ = 0.1.

We next performed an a-posteriori Monte-Carlo test of this solution against the nominalone hnom, for two specific noise distributions, namely the uniform and the Normal distribu-tion. The empirical cumulative distribution functions of the resulting norm residuals areshown in Figure 3. Some relevant statistics are also shown in Table 1.

0 2 4 6 8 10 12 14 16 18 20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(a) Uniform noise distribution

!

Pro

b{||L

(")z

|| <

!}

0 5 10 15 20 25 300

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

!

(b) Normal noise distribution

Figure 3: Empirical "2-norm residual cumulative distribution for the h(2,rar solution (solid)and the nominal hnom solution (dashed). Simulation in case (a) assumes uniform noisedistribution; case (b) assumes Normal distribution.

Uniform Normalhnom h(2,rar hnom h(2,rar

min 0.19 0.49 0.19 0.15max 19.40 10.79 28.98 11.75mean 7.75 5.52 7.50 5.51median 7.50 5.53 7.01 5.49std 3.02 1.44 3.58 1.43rar@$ = 0.1 11.85 7.39 12.39 7.36

Table 1: "2-norm residual statistics from a-posteriori test on nominal and residual-at-risksolutions, with uniform and Normal distribution on the noise.

We next solved the same problem using an "1-norm residual criterion. In this case, solutionof problem (21) yielded - = 0.082 and an optimal upper bound on "1 residual at risk

13

To appear in: Systems and Control Letters (Elsevier)

0 5 10 15 20 25 30 350

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(a) Uniform noise distribution

0 5 10 15 20 25 30 35 40 45 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(b) Normal noise distribution

! !

Pro

b{||L

(")z

|| <

!}

1

Figure 4: Empirical "1-norm residual cumulative distribution for the h(1,rar solution (solid)and the nominal hnom solution (dashed). Simulation in case (a) assumes uniform noisedistribution; case (b) assumes Normal distribution.

' = 19.2, with corresponding parameter

h(1,rar = [0.7690 0.4464 0.1344]".

An a-posteriori Monte-Carlo test then produced the residual distributions shown in Fig-ure 4. Residual statistics are reported in Table 2.

Uniform Normalhnom h(1,rar hnom h(1,rar

min 0.38 0.81 0.11 0.57max 32.12 18.39 47.93 19.63mean 11.89 8.95 11.42 8.94median 11.34 8.95 10.52 8.93std 5.01 2.61 5.76 2.56rar@$ = 0.1 18.66 12.37 19.22 12.29

Table 2: "1-norm residual statistics from a-posteriori test on nominal and residual-at-risksolutions, with uniform and Normal distribution on the noise.

6 Conclusions

In this paper we discussed two criteria for linear parameter estimation in presence ofrandom uncertain data, under both "2 and "1 norm residuals. The first criterion is aworst-case residual expectation and leads to exact and e!ciently computable solutions forthe "2 norm case. For the "1 norm, we can e!ciently compute upper and lower boundson the optimal solution, by means of convex second order cone programming. The secondcriterion considered in the paper is the worst-case residual for a given risk level $. With thiscriterion, an exact solution for the "2 norm case can be computed by a solving a sequence of

14

To appear in: Systems and Control Letters (Elsevier)

semi-definite programs, and an analogous computational e"ort is required for computing anupper bound on the optimal solution in the "1 norm case. The estimation setup proposedin the paper is “distribution free,” in the sense that only information about the meanand covariance of the random uncertainty need be available to the user: the results areguaranteed irrespective of the actual shape of uncertainty distribution.

Appendix

A Proof of Lemma 1

We start by recalling a preliminary result. Let fi : Rn , R, i = 1, . . . , m, be functionswhose expectations qi are given and finite, and consider the following problem (P ) and itsdual (D):

(P ) : ZP .= supx E {((x)}

subject to: E {fi(x)} = qi, i = 0, 1, . . . ,m;

(D) : ZD .= infy E {y"f(x)} = inf y"q

subject to: y"f(x) ) ((x), -x " Rn,

where f0(x) = 1 and q0 = 1 correspond to the implied probability-mass constraint, andthe infimum and the supremum are taken with respect to all probability distributions sat-isfying the specified moment constraints. Then, strong duality holds: ZP = ZD, hencesupx E {((x)} can be computed by solving the dual problem (D); see for instance Sec-tion 16.4 of [2].

Now, the primal problem that we wish to solve in Lemma 1 for computing Ewc is

(P ) : ZP = Ewc = supx E {((x)}subject to: E {x} = x

E {xx"} = # + xx",

where # ' 0 is the covariance matrix of x, and the functions fi are xk, k = 1, . . . , n, andxkxj, 1 * k * j * n. Then, the dual problem is

(D) : ZD = infy0$R,y$Rn,Y =Y !$Rn,n y0 + y"x + Tr (# + xx")Y

subject to: y0 + y"x + Tr xx"Y ) ((x), -x " Rn,

where the dual variable y0 is associated with the implicit probability-mass constraint.Defining

M =

$Y 1

2y12y" y0

%, Q =

$# + xx" x

x" 1

%, z =

$x1

%,

this latter problem writes

(D) : ZD = infM=M!$Rn+1 Tr QM

subject to: z"Mz ) ((x), -x " Rn,

which is (6), thus concluding the first part of the proof.

15

To appear in: Systems and Control Letters (Elsevier)

The result in (7) can then be obtained by specializing (6) to the case when ((x) = IS(x),where IS is the indicator function of set S, since Prob {x " S} = E {IS(x)}. We thus havethat ZP = ZD for

(P ) : ZP = Pwc = supx Prob {x " S}subject to: E {x} = x, E {xx"} = # + xx",

(D) : ZD = infM=M!$Rn+1 Tr QM

subject to: z"Mz ) IS(x), -x " Rn.

The constraint z"Mz ) IS(x) -x " Rn can be rewritten as z"Mz ) 1 -x " S, z"Mz ) 0-x " Rn, and this latter constraint is equivalent to requiring M ( 0, which explains (7)and concludes the proof. !

B Matrix cube theorem

Theorem 4 (Matrix cube relaxation; [1]) Let B0, B1, . . . , BL " Rn,n be symmetricand B1, . . . , BL be of rank two. Let the problem P) be defined as:

P) : Is B0 +L#

i=1

uiBi ( 0, -u : #u#! * / ? (23)

and the problem Prelax be defined as:

Prelax: Do there exist symmetric matrices X1, . . . , XL " Rn,n satisfying

Xi ( ±/Bi, i = 1, . . . , L,*Li=1 Xi 1 B0?

Then, the following statements hold:

1. If Prelax is feasible, then P) is feasible.

2. If Prelax is not feasible, then P!2 ) is not feasible. "

References

[1] A. Ben-Tal and A. Nemirovski. On tractable approximations of uncertain linear ma-trix inequalities a"ected by interval uncertainty. SIAM Journal on Optimization,12(3):811–833, 2002.

[2] D. Bertsimas and J. Sethuraman. Moment problems and semidefinite optimization. InHandbook of semidefinite programming, pages 469–509. Kluwer Acad. Publ., Boston,MA, 2000.

[3] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press,New York, NY, 2004. ISBN: 0-521-83378-7.

[4] G. Calafiore and F. Dabbene. Near optimal solutions to least-squares problems withstochastic uncertainty. Systems and Control Letters, 54(12):1219–1232, December2005.

16

To appear in: Systems and Control Letters (Elsevier)

[5] C. Dossal. A necessary and su!cient condition for exact recovery by "1 minimization.http://hal.archives-ouvertes.fr/hal-00164738/fr/, 2008.

[6] D. Du!e and J. Pan. An overview of value at risk. Journal of Derivatives, pages7–49, Spring 1997.

[7] M.P. Friedlander and P. Tseng. Exact regularization of convex programs. SIAM J.on Optimization, 18(4):1326–1350, 2007.

[8] L. El Ghaoui and H. Lebret. Robust solutions to least-squares problems with uncertaindata. SIAM J. on Matrix Analysis and Applications, 18(4):1035–1064, 1997.

[9] L. El Ghaoui, F. Oustry, and H. Lebret. Robust solutions to uncertain semidefiniteprograms. SIAM J. Optimization, 9(1):33–52, 1998.

[10] H.A. Hindi and S.P. Boyd. Robust solutions to l1, l2, and l! uncertain linear ap-proximation problems using convex optimization. In Proc. American Control Conf.,volume 6, pages 3487–3491, 1998.

[11] R.T. Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distribu-tions. Journal of banking and Finance, 26:1443–1471, 2002.

[12] A.H. Sayed, V.H. Nascimento, and F.A.M. Cipparrone. A regularized robust de-sign criterion for uncertain data. SIAM J. on Matrix Analysis and Applications,23(4):1120–1142, 2002.

17

To appear in: Systems and Control Letters (Elsevier)