On quantitative stability in infinite-dimensional optimization under uncertainty

Hoffhues, M.; Römisch, W.; Surowiec, T. M. Article — Published Version On quantitative stability in infinite-dimensional optimization under uncertainty Optimization Letters Provided in Cooperation with: Springer Nature Suggested Citation: Hoffhues, M.; Römisch, W.; Surowiec, T. M. (2021) : On quantitative stability in infinite-dimensional optimization under uncertainty, Optimization Letters, ISSN 1862-4480, Springer, Berlin, Heidelberg, Vol. 15, Iss. 8, pp. 2733-2756, https://doi.org/10.1007/s11590-021-01707-2 This Version is available at: https://hdl.handle.net/10419/287351 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Optimization Letters (2021) 15:2733–2756 https://doi.org/10.1007/s11590-021-01707-2 ORIGINAL PAPER On quantitative stability in infinite-dimensional optimization under uncertainty M. Hoffhues1 ·W. Römisch2 ·T. M. Surowiec1 Received: 22 January 2020 / Accepted: 19 January 2021 / Published online: 28 February 2021 © The Author(s) 2021 Abstract The vast majority of stochastic optimization problems require the approximation of the underlying probability measure, e.g., by sampling or using observations. It is therefore crucial to understand the dependence of the optimal value and optimal solutions on these approximations as the sample size increases or more data becomes available. Due to the weak convergence properties of sequences of probability measures, there is no guarantee that these quantities will exhibit favorable asymptotic properties. We consider a class of infinite-dimensional stochastic optimization problems inspired by recent work on PDE-constrained optimization as well as functional data analysis. For this class of problems, we provide both qualitative and quantitative stability results on the optimal value and optimal solutions. In both cases, we make use of the method of probability metrics. The optimal values are shown to be Lipschitz continuous with respect to a minimal information metric and consequently, under further regularity assumptions, with respect to certain Fortet-Mourier and Wasserstein metrics. We prove that even in the most favorable setting, the solutions are at best Hölder continuous with respect to changes in the underlying measure. The theoretical results are tested in the context of Monte Carlo approximation for a numerical example involving PDEconstrained optimization under uncertainty. Keywords Stability ·Stochastic programming ·Optimization under uncertainty · Probability metrics ·PDE-constrained optimization ·Functional data analysis BT. M. Surowiec suro[email protected]g.de W. Römisch [email protected] 1FB12 Mathematik und Informatik, Philipps-Universität Marburg, Hans-Meerwein-Straße 6, 35032 Marburg, Germany 2Institute of Mathematics, Humboldt University of Berlin, Unter den Linden 6, 10099 Berlin, Germany 123 2734 M. Hoffhues et al. 1 Introduction In stochastic optimization, stability usually refers to the continuity properties of optimal values and solution sets as mappings from a set of probability measures, endowed with a suitable distance, into the extended reals and solution space, respectively, see [22]. The distance on the space of probability measures must be selected in order to allow the estimation of differences of the relevant functions, which depend on probability measures. There exists a wide variety of possible distances of probability measures based on various constructions [19,27]. In the present context, distances with ζ-structure introduced first in [27] appear as a natural choice. For a given metric space Ωsuch a distance is of the form dF(P,Q)=sup f∈FΩ f(ω) dP(ω) −Ω f(ω) dQ(ω),(1) where Fis a family of Borel measurable functions from Ωto Rand P,Qare Borel probability measures on Ω. Note that the distance dFis non-negative, symmetric and satisfies the triangle inequality. It also satisfies dF(P,P)=0 and is, thus, a probability metric in the sense of [27]. However, dF(P,Q)=0 only implies P=Q, when the family Fis rich enough. Hence, dFis a semi-metric in the usual terminology, in general. The smallest relevant family Fof Borel measurable functions in our stability studies contains only those functions which appear in the stochastic optimization problem under consideration. In this case, dFmay be called the minimal information (m.i.) distance. Stability results with respect to such m.i. distances serve as the starting point (i) to study stability with respect to the weak convergence of probability measures and (ii) to enlarge the family Fproperly by functions sharing essential analytical properties with the original ones. The latter strategy may lead to probability metrics that enjoy desirable properties like dual representations and convergence characterizations. This method of probability metrics provides quantitative statements on the stability of solutions and optimal values of stochastic programming problems. Nevertheless, the existing theory has not been developed for optimization problems in which the design or decision variables may be infinite-dimensional, as is the case in PDE-constrained optimization under uncertainty. By including infinite-dimensional feasible sets, we introduce a number of complications; in particular, the loss of norm compactness of the feasible set, even in the case of convex, closed, and bounded feasible sets. After fixing some essential notation in Sect. 2, we state the class of infinitedimensional stochastic optimization problems for which we study stability in the subsequent sections in Sect. 3. Section 4contains qualitative results by providing conditions that imply convergence of optimal values and solutions if the underlying sequence of probability distribution converges to a limit distribution in some sense. In Sect. 5we show that optimal values and solutions even allow Lipschitz or Hölder estimates in terms of the ζ-distance. In Sect. 6we argue that the stability analysis of the preceding sections applies to certain stochastic PDE-constrained optimization problems. Finally, in Sect. 7, we provide a study of the results in Sect. 6for the case when Monte Carlo approximations are used. 123 Stability in infinite-dimensional stochastic optimization 2735 2 Notation and preliminary results We assume throughout that Ωis a complete separable metric space, i.e., Polish space, and Fthe associated Borel σ-algebra. In addition, we will work exclusively with Borel probability measures P:F→[0,1]. that ensure (Ω, F,P)is a complete probability space. In particular, if Ωis finite, then Fmust be the power set of Ω. For the abstract portion of our results, we will always assume that Θis a real separable Hilbert space and Θad ⊂Θis a nonempty, closed, and convex set. Given an appropriately chosen integrand f:Θ×Ω→R, we will consider the potentially infinite dimensional stochastic optimization problems: ν(P):= inf θ∈Θad Ω f(θ, ω) dP(ω). (2) Here, we also introduce the notion of optimal value function νas a function from the space of all Borel probability measures P(Ω) into R. This potentially extended realvalued function will play a key role in our discussions. If necessary, we will denote the expectation by either Eor if it is not clear in context EPto denote the dependence on the measures P. Given a complete probability space (Ω, F,P)and a real Banach space W,we recall the definition of the Bochner space Lp(Ω, F,P;W)p∈[1,∞)as the space of (equivalence classes) of strongly measurable functions v, which map Ωinto W and satisfy Ωv(ω)p WdP(ω) < +∞,cf.[13]. If p=∞, then L∞(Ω, F,P;W) consists of essentially bounded W-valued strongly measurable functions. In both cases Lp(Ω, F,P;W)is a Banach space with the natural norm(s) vLp(Ω,F,P;W)=Evp W1/p,for p∈[1,∞), ess supω∈Ωv(ω)W,for p=∞. In the special case when W=R, we simply write Lp(Ω, F,P). As usual norm convergence will be typically denote by →, whereas signifies weak convergence and ∗ weak-star convergence. In our stability analysis, we make use of distances with ζ-structure on P(Ω) having the form (1). We will refer to these objects as ζ-distances for brevity. Given a family F of Borel measurable functions from Ωinto R,theζ-distance dFon (Ω, F)is a highly flexible structure that allows us to define so-called minimal information distances and Fortet-Mourier metrics; each defined in the text below. Properties of ζ-distances like a characterization of its maximal generator and its relation to weak convergence of probability measures can be found in [18,23]. Recall that a sequence of probability measures {PN}on (Ω, F)is said to narrowly/weakly converge to the probability measure Pprovided EPN[f]→EP[f]∀f∈C0 b(Ω), 123 2736 M. Hoffhues et al. where C0 b(Ω) is the space of all bounded continuous functions on Ω. A family Fof Borel measurable functions is called a P-uniformity class if lim N→∞dF(P,PN)=0 holds for each sequence {PN}of probability measures converging weakly to P.For example, it is known that Fis a P-uniformity class if Fis uniformly bounded and P({ω∈Ω:Fis not equicontinuous at ω})=0[23]. Finally, we recall that given a σ-algebra Falong with a nominal σ-finite σ-additive positive measure Pon Ω, e.g., a Borel probability measureP∈P(Ω), the dual space of L∞(Ω, F,P)can be identified with the space of all finitely additive signed measures ba(Ω) on Fabsolutely continuous with respect to P, see e.g., [9]. 3 The optimization problem In order to carry out the stability analysis, we restrict the class of allowable integrands f(θ, ω). These restrictions will henceforth be taken as standing assumptions. The particular class considered in this paper is inspired by applications in PDE-constrained optimization under uncertainty in which the PDE is given by a linear elliptic partial differential equation with random coefficients, right-hand side, and/or boundary conditions. We refer the reader to [17] for an overview of the state-of-the-art theory including more general objective functions and risk measures. In addition, many problems in functional data analysis exhibit practically the same form used below, see e.g., [21]. Let Vand Hbe real Hilbert spaces such that Vembeds continuously into H, and θd∈H.Forθ∈Θand ω∈Ω,letΣ(ω)θ =S(ω)θ −s(ω), where S(ω) :Θ→V is bounded and linear in θindependently of ωand s(ω) ∈H. We then define f(θ, ω) := 1 2Σ(ω)θ −θd2 H=1 2S(ω)θ −(θd+s(ω))2 H. Furthermore, we assume that for every θ∈Θ(or θ∈Θad) and any P∈P(Ω) f(θ, ·)∈L1(Ω, F,P). This implicitly adds mild regularity assumptions on Sand sthat are typically fulfilled when Sis related to the solution of a parametric elliptic PDE, e.g., S(·)θ, s(·)∈ L2(Ω, F,P;V). Then for α>0, we consider the optimization problems inf θ∈Θad F(θ) := EP[f(θ)]+α 2θ2 Θ.(3) Theorem 1 Problem (3)admits a unique solution θP∈Θad for every P∈P(Ω). Proof For existence, it suffices to prove Fis proper, convex, lower-semicontinuous and coercive, cf. e.g., [3, Sec. 3.3]. Since f(θ, ·)∈L1(Ω, F,P)for any θ∈Θad 123 Stability in infinite-dimensional stochastic optimization 2737 and f≥0, Fis proper. Convexity follows directly from the P-a.e. convexity of θ→ f(θ, ω)+α 2θ2 Θand the monotonicity of the expectation EP. Lower semicontinuity is a result of Fatou’s lemma: Let θk→θin Θ. Then since f≥0 and f(θk,·)→f(θ, ·) P-a.s. (by the assumptions on S)wehave lim inf kEP[f(θk)]+α 2θk2 Θ≥EPlim inf kf(θk)+α 2θ2 Θ=EP[f(θ)]+α 2θ2 Θ. Since EP[f(θ, ·)]≥0 for all θ∈Θad,Fis coercive. Given Fis proper, convex, and lower semicontinuous, Fis weakly lower semicontinuous, as well. Since Fis coercive, the level set {θ∈Θad |F(θ) ≤α0},where θ0∈Θad and α0:= F(θ0),is weakly sequentially compact. It then follows from the direct method that (3) admits a solution θP.Givenα>0, Fis strictly convex. Hence, θPis unique.  4 Qualitative stability In this section, we provide stability results that ensure the approximating optimization problems obtained by replacing Pby another probability measure Q∈P(Ω) will converge in some sense to the original problem. In particular, we show that the solutions θQwill strongly converge to θPprovided Qconverges to Pwith respect to a properly chosen ζ-distance. This basic result serves as the foundation needed to prove continuity of the solutions with respect to narrow convergence of probability measures. However, in order to do the latter, additional regularity properties will be required on the integrands with respect to ω. These stability results are in some sense more versatile than the quantitative results below. Nevertheless, they do not provide us with a rate of convergence. Theorem 2 In the context of Theorem 1, suppose we are given a sequence {PN}with PN∈P(Ω) and a probability measure P∈P(Ω) such that dF(PN,P)→0, where Fis any class of measurable functions from Ωinto Rlarge enough to contain f (θ, ·) for any θ∈{θN:N∈N}∪{θP}with θN:= θPN. Then θN→θPstrongly in Θas N→+∞. Remark 1 The obvious candidate for the set Fwould be to choose the collection of all possible integrands f(θ, ·):Ω→Rindexed by θ∈Θad. In terms of the associated ζ-distance, this would result in what is referred to in [19,20,22]asthe minimal information metric. Proof We first show {θN}is uniformly bounded in Θ. Indeed, we have α 2θN2 Θ≤EPN[f(θN)]+α 2θN2 Θ≤EPN[f(θ)]+α 2θ2 Θθ∈Θad.(4) For any fixed θ∈Θad, it follows from the hypotheses that EPN[f(θ)]=EPN[f(θ)]−EP[f(θ)]+EP[f(θ)]≤dF(PN,P)+EP[f(θ)].(5) 123 2738 M. Hoffhues et al. Substituting this into (4) we obtain the bound α 2θN2 Θ≤dF(PN,P)+F(θ) θ ∈Θad. Since dF(PN,P)→0, {θN}is bounded in Θ. Therefore, there exists a  θ∈Θad and a weakly convergent subsequence θNsuch that θN θas →+∞. For fixed P, it follows from the proof of Theorem 1that EP[f(·)]:Θ→Ris weakly lower semicontinuous. Therefore, EPf θ+α 2 θ2 Θ≤lim inf EPfθN+α 2θN2 Θ ≤lim inf EPNfθN+α 2θN2 Θ+EPfθN−EPNfθN ≤lim inf EPNfθN+α 2θN2 Θ+dFPN,P =lim inf EPNfθN+α 2θN2 Θ. (6) It then follows from (6), the optimality of θN, and (5) that for any θ∈Θad we have: EP[f( θ)]+α 2 θ2 Θ≤lim inf EPNfθN+α 2θN2 Θ ≤lim inf EPN[f(θ)]+α 2θ2 Θ ≤lim inf dF(PN,P)+EP[f(θ)]+α 2θ2 Θ =EP[f(θ)]+α 2θ2 Θ. (7) Hence, θP= θ. Since θPis unique and the previous arguments hold for all weakly convergent subsequences of {θN},wehaveθNθP= θas N→+∞. It remains to prove θN−θPΘ→0. Clearly we have the inequality lim inf NθNΘ≥  θΘ(8) by weak lower semicontinuity of the norm ·Θ. On the other hand, by rearranging terms, the definition of θNand feasibility of  θyield α 2θN2 Θ≤α 2 θ2 Θ+EPN[f( θ)]−EPN[f(θN)] =α 2 θ2 Θ+EPN[f( θ)]−EPN[f(θN)]+EP[f(θN)]−EP[f(θN)] ≤α 2 θ2 Θ+EPN[f( θ)]−EP[f(θN)]+dF(PN,P) =α 2 θ2 Θ+EPN[f( θ)]−EP[f( θ)]+EP[f( θ)]−EP[f(θN)]+dF(PN,P) ≤α 2 θ2 Θ+2dF(PN,P)+EP[f( θ)]−EP[f(θN)]. 123 Stability in infinite-dimensional stochastic optimization 2739 Therefore, we again appeal to the weak lower semicontinuity of EP[f(·)]on Θto obtain lim sup N α 2θN2 Θ≤α 2 θ2 Θ+lim sup N2dF(PN,P)+EP[f( θ)]−EP[f(θN)] =α 2 θ2 Θ+EP[f( θ)]−lim inf N[f(θN)]] ≤α 2 θ2 Θ+EP[f( θ)]−EP[f( θ)] =α 2 θ2 Θ. (9) Combining (8) and (9), we have θNΘ→  θΘ. Then since Θis a Hilbert space and θN θ, the assertion follows.  An alternative perspective on qualitative stability is offered by our next result. Here, we will prove convergence of the sequence of minimizers under different data assumptions on the integrands and a different form of weak convergence of measures. We note that in PDE-constrained optimization under uncertainty these assumptions are less restrictive than they may appear. In particular, we do not require f(θ, ·):Ω→R to be continuous as is needed below for the Fortet-Mourier metric. The caveat here is the requirement that PNis absolutely continuous with respect to P. Theorem 3 In addition to the standing assumptions, fix some P∈P(Ω) and suppose that for all θ∈Θad f(θ, ·)∈L∞(Ω, F,P). Assume furthermore that the superposition operator Φ:Θ→L∞(Ω, F,P)defined by Φ(θ)(ω) := f(θ, ω) is completely continuous. Let {PN}⊂P(Ω) such that 1. for all N ∈NP N<< P(PNis absolutely continuous with respect to P)and 2. PN→Pwith respect to the weak-star topology on (L∞(Ω, F,P))∗. Then θN→θP. Proof As noted in Sect. 2, each Q∈P(Ω) is an element of (L∞(Ω, F,P))∗provided Q<< P. The rest of the proof mirrors that of Theorem 2. Given the sequence of minimizers {θN}we immediately obtain a uniform bound on θNfrom (4) since for any θ∈Θad f(θ, ·)∈L∞(Ω, F,P)and PN∗ P. As before, we let θN∞ =1denote the weakly convergent subsequence and  θthe associated weak limit. Turning now to the estimate derived in (6), we see that EPf θ+α 2 θ2 Θ≤lim inf EPfθN+α 2θN2 Θ ≤lim inf EPNfθN+α 2θN2 Θ+EPfθN−EPNfθN =lim inf EPNfθN+α 2θN2 Θ. (10) 123 2740 M. Hoffhues et al. Here, Φ(θNl)→Φ( θ) strongly in L∞(Ω, F,P)due the assumption of complete continuity. Therefore, both EP[f(θN)]and EPN[f(θN)]converge to EP[f( θ)]. As in the proof of Theorem 2, we obtain optimality of  θby adapting the inequality (7), i.e., for every θ∈Θad we have EPf θ+α 2 θ2 Θ≤lim inf EPNfθN+α 2θN2 Θ ≤lim inf EPN[f(θ)]+α 2θ2 Θ =EP[f(θ)]+α 2θ2 Θ. (11) Here, the regularity of the integrand ensures that EPN[f(θ)]converges to EP[f(θ)]; from which it follows that  θ=θP. As in the proof of Theorem 2, we can again argue that the entire sequence θPNweakly converges to θP. In order to prove norm convergence, we note that α 2θN2 Θ≤α 2 θ2 Θ+EPN[f( θ)]−EPN[f(θN)]. Then by the complete continuity and regularity assumptions, we have lim sup N α 2θN2 Θ≤lim sup N α 2 θ2 Θ+EPN[f( θ)]−EPN[f(θN)]=lim sup N α 2 θ2 Θ. This completes the proof.  Next, we return to the setting using probability metrics to obtain some important implications of the Theorem 2under further regularity assumptions on the integrands. In our setting, we recall that the space of all (Borel) probability measures with finite p-th moments is defined by Pp(Ω) := P∈P(Ω) Ω d(ω0,ω)pdP(ω) < +∞, for some arbitrary ω0∈Ω. We recall that a sequence {PN}⊂Pp(Ω) converges weakly (narrowly) provided for all ϕ∈C0 b(Ω) EPN[ϕ]→EP[ϕ]and EPNd(ω0,·)p→EPd(ω0,·)p as N→∞. This type of weak convergence shares an intimate link with a certain class of ζ-distances known as Fortet-Mourier metrics. To start, for p∈[1,∞),we define the sets Fp(Ω) of locally Lipschitz functions with a certain p-related growth condition by 123 Stability in infinite-dimensional stochastic optimization 2747 Next we consider the mapping Kt(ω)u=u−tJ−1(A(ω)u−g)for some t∈(0,2γ L2), ω∈Ω,g∈Vand any u∈V. Then it follows from Proposition 2that Kt(ω)u−Kt(ω)uV≤κ(t)u−uV for any u,u∈Vand κ(t)=1−2tγ+t2L2<1. Furthermore, the unique fixed point of Kt(ω) belongs to the ball around zero with radius r=Kt0−0V 1−κ(t)=t 1−κ(t)J−1gV=t 1−κ(t)g. For any u∈Vand ω,ω∈Ωwe have Kt(ω)u−Kt(ω)uV=tJ−1(A(ω) −A(ω))uV=t(A(ω) −A(ω))u and apply Proposition 3from the “Appendix” with P=Ω,X=B(0,r)={u∈V: uV≤r},F(p,u)=Kt(ω)uand F(p,u)=Kt(ω)u. We obtain ¯x(g,ω)−¯x(g,ω )≤ t 1−κ(t)Crρ(ω,ω) where ¯x(g,ω)=A(ω)−1gand r=t 1−κ(t)g. This completes the proof.  We now have enough results to prove that Fconstitutes a P-uniformity class, which is a direct consequence of the following theorem. Theorem 5 In addition to the hypotheses of Lemma 2, assume g ∈L∞(Ω, F,P;H) and there exists ¯ C>0such that |g(x,ω)−g(x,ω )|≤ ¯ Cρ(ω,ω)(∀ω,ω∈Ω,a. e. in D). Then Fis uniformly bounded and equi-Lipschitz continuous with respect to ρon Ω. Proof For any z∈Zad and ω∈Ωlet F(z,ω) =A(ω)−1(z+g(ω)) −ud.Our assumptions imply that F(·,·)is P-a.s. uniformly bounded in Hby some constant ˆ C (see the proof of Lemma 1). Furthermore, we obtain for any z∈Zad and ω,ω∈Ω: |f(z,ω)−f(z,ω )|=1 2(F(z,ω)H+F(z,ω )H)|F(z,ω)H−F(z,ω )H| ≤ˆ CF(z,ω)−F(z,ω )H ≤ˆ C((A(ω)−1−A(ω)−1)zV+A(ω)−1g(ω) −A(ω)−1g(ω)V), where we used the uniform boundedness and the Poincaré-Friedrichs’ inequality. For the first term on the right-hand side we argue as in Lemma 2. The second term is estimated by 123 2748 M. Hoffhues et al. A(ω)−1g(ω) −A(ω)−1g(ω)V≤(A(ω)−1−A(ω)−1)g(ω)V +A(ω)−1(g(ω) −g(ω))V. Now, we use again Lemma 2 for the first term and both Lemma 1 and the assumption on gfor the second. Combining these observations, we obtain the assertion.  Theorem 5establishes the P-uniformity of Funder relatively mild assumptions. In particular, we do not require the terms bij and gto be smooth in any way with respect to ω. Of course in many interesting applications, see e.g. [4,5], one can demonstrate much higher regularity of F(z,ω) =A(ω)−1(z+g(ω)) −udin ωfor each z∈Zad if some smoothness of bij and gis in fact available. And though the presence of · 2 Hin f(z,ω) rules out 1-Lipschitz continuity, as required by estimates using the Wasserstein distances, the quantitative estimates using the Fortet-Mourier metric, e.g., ζ2, which are incidentally strictly sharper than the Wasserstein estimates, are still applicable provided the local growth conditions for functions in Fp(Ω) are fulfilled. On the other hand, as a general point of critique, the minimal information metric along with Theorems 4and 5preclude the need to enlarge the set of integrands Fin order make use of the rougher estimates given by the Fortet-Mourier estimates. This brings us to our second goal of this section. In order to solve optimization problems of the type min EP[f(z)]+α 2z2 Hover z∈Zad (18) with f∈Fnumerically, not only Pbut the decision variables zand the underlying partial differential equation must be approximated. In order words, using a finite-sample-based approximation PNof Pand a finite-element discretization for the deterministic quantities in Hand V, we would typically consider the finitedimensional problems of the type: min 1 2 N  i=1 πi(Ah i)−1(zh)−uh d,i2 H+α 2zh2 Hover zh∈Zh ad.(19) Here, Ah iis defined from A(ω) by replacing ωwith a realization ωiand Vby a finitedimensional subspace Vhderived by a standard finite-element approximation. The term uh d,i=(Ah i)−1(gh i−ud,h)and Zh ad is an approximate of Zad using a finite-element approximation of H. For piecewise constant approximations of the control z, the original ideas date back to Falk [10], see the recent chapter [2] for a quick reference. For another more recent, comprehensive treatment see [26] (for a posteriori error estimates) as well as the monograph [15, Chap. 3] and the many references therein. In many of these works, as well as in our concrete example in the next section, the set Zad has the concrete form: Zad := z∈L2(D)a(x)≤z(x)≤a(x)a.e. x∈D, 123 Stability in infinite-dimensional stochastic optimization 2749 where a,aare sufficiently regular; often L∞(D). Therefore, for the sake of argument, we may assume that if D,B,g,a,a, and udare sufficiently regular, then there exists a (random variable) CN≥0 along with a real q∈(0,3/2]such that zh PN−zPNH≤CNhqω∈Ω. (20) Here, the dependence on Nresults from the fact that the typical estimates, see e.g., [2, Thm. 10], depend on constants related to the coefficients of the PDE, the right-hand side g, and ud, which is stochastic. Therefore, in the estimate (20), CNis related to a realization of a random sample of length N.Using(20), we can apply Theorem 4and the triangle inequality to obtain the estimate zP−zh PNH≤2√2α−1/2dF(P,PN)1/2+CNhq.(21) It should also be noted that for piecewise constant approximations of Zh ad, we can only expected q=1. The case for q=3/2 requires a significant amount of regularity, and q∈(1,3/2)depends on both the regularity as well as the type of discretization. In light of Theorem 5,(21) guarantees the convergence of the fully discrete solutions zh PN to the original infinite dimensional solution, provided h↓0, and PN→Pweakly. 7 Monte Carlo approximation and numerical illustration for PDE-constrained optimization In this final section, we provide additional discussions on the behavior of dF(P,PN) for Monte Carlo approximations PNof Pin order to give the reader a better impression of the potential rate of convergence; in particular, for the setting in the previous section. 7.1 Monte Carlo approximation For the sake of argument, we assume that the integrands fcan be written f(z,ω)=f(z,ξ(ω)) +α 2z2 H, where ξ:Ω→Ξ⊂Rdis a random vector. This is often the case for PDE-models and is used in the example in Sect. 7.2 below. Let ξ1,ξ2,...,ξN,... be independent identically distributed Ξ-valued random vectors on some probability space (Ω, F,P)having the common probability law P, i.e., P=P◦(ξ1)−1. In this context, we define the empirical measures PN(·)=1 N N  i=1 δξi(·)(n∈N)(22) 123 2750 M. Hoffhues et al. and the empirical or Monte Carlo approximation of the stochastic program (18) with sample size N, i.e., min 1 N N  i=1 f(z,ξi(·)) :z∈Zad .(23) The optimal value ν(PN(·)) of (23) is a real random variable and the solution zPN(·)an H-valued random element. It is well known that the sequence {PN(·)}of empirical measures converges weakly to PP-almost surely (see, e.g., [8, Theorem 11.4.1]). Theorem 5implies that the class Fis a P-uniformity class and, hence, the sequence dF(PN(·), P)converges to zero P-almost surely. According to Theorem 4, the sequences {v(PN(·))}and zPN(·)of empirical optimal values and solutions converge P-almost surely to their true optimal values and solutions v(P)and zP, respectively. In order to obtain rates of convergence for the sequences of empirical optimal values and solutions, we consider their mean or mean square distance and conclude from Theorem 4and Corollary 1the estimates E[|v(PN)−v(P)|]≤EdF(PN,P)≤LE[ζ1(PN,P)] Ez(PN)−z(P)2 H1 2≤EdF(PN,P)1 2≤ˆ LE[ζ1(PN,P)]1 2, where Lis the constant appearing in Corollary 1and ˆ Lis given by ˆ L=2L!2 α. Unfortunately, convergence rates of the mean convergence of dF(PN(·), P)are not known, but for Fcontaining uniformly bounded and equi-Lipschitz continuous functions, the rate coincides essentially with that of E[ζ1(PN,P)]. For the latter it follows from [6, Theorem 1] that the estimate E[ζ1(PN,P)]≤κsMs(P)N−1 d(24) is valid if d≥3, s>d d−1,κsis a constant only depending on sand the sth absolute moment Ms(P)=Rdξsdξ1 s is finite. It is argued in [11] that the rate (24) is sharp if Pis the uniform distribution on the unit cube [−1,1]d. For the numerical experiments in the next subsection, we observe better rates than guaranteed by the probability metrics. This is a limitation of the probability metrics themselves, not their application to the problem at hand. 7.2 Numerical illustration The previous discussion provides useful upper bounds for the case of the empirical measure PN. However, these bounds are based on estimates of the Wasserstein metrics, 123 Stability in infinite-dimensional stochastic optimization 2751 not the minimal information metric. Therefore, in order to obtain a better impression of the possible rate of convergence associated with dF(P,PN)in practice, we provide here a concrete numerical example. We propose a model problem, which is based on [16, Ex. 6.1, Ex. 6.2]. Let α=10−3, D=(0,1),u(x)=sin(50.0∗x/π), and consider the optimal control problem minimizez∈L2(D) 1 2EPu−ud2 H+α 2z2 Hover z∈L2(D)(25) where z∈Zad with Zad := w∈L2(D)|−0.75 ≤w(x)≤0.75 a.e. x∈D and u=u(z)∈L∞(Ω, F,P;H1(D)) solves the weak form of −ν(ω)∂xxu(ω, x)=g(ω, x)+z(x)(ω,x)∈Ω×D,(26a) u(ω, 0)=d0(ω), u(ω, 1)=d1(ω) ω ∈Ω. (26b) Furthermore, we suppose that ν(ω) := 10ξ1(ω)−2,g(ω, x):= ξ2(ω) 100 d0(ω) := 1+ξ3(ω) 1000 d1(ω) := ξ4(ω) 1000 , with random variables ξi:Ω→R,i=1,2,3,4, such that the supports ξi,i=1,2,3, are [−1,1]and the support of ξ4is [1,3]. For the sake of illustration, we assume that each of these random variables is uniformly distributed and after the usual change of variables, we consider instead −ν(ξ)∂xxu(ξ, x)=g(ξ, x)+z(x)(ξ,x)∈Ξ×D,(27a) u(ξ, 0)=d0(ξ), u(ξ, 1)=d1(ξ) ξ ∈Ξ. (27b) with Ξ=[−1,1]×[−1,1]×[−1,1]×[1,3], endowed with the associated uniform density. We define ξ:= (ξ1,...,ξ 4)∈Ξ. Finally, we note that the linearity of the differential equation allows us to use the superposition principle to separate the unique, z-dependent solution u(z)into the sum of random fields as u(z)=S(z)+u, where S(z)maps zfrom Hinto the solution space and uis a fixed random field. With the aim of showing that the necessary continuity properties used in the previous section are fulfilled, let u:D×Ξ→Rdenote a function such that u(·,ξ) belongs to H1(D)and usatisfies the inhomogeneous random boundary condition in (27b). If we then recast (27) as the random elliptic PDE with random boundary conditions A(ξ)u=z+g(ξ), u(x)=b(x,ξ) (x∈∂D,ξ∈Ξ) (28) 123 2752 M. Hoffhues et al. Fig. 1 (l) Optimal solution zPof (25)forM=100, h=1/(210 −1). (r) Empirical mean value of u(zP) out of sample using 1000 iid random realizations of (ξ1,...,ξ 4) and determine u∈H1 0(D)such that A(ξ)u=z+g(ξ) +A(ξ)u(ξ) (ξ ∈Ξ). Then u+usolves the random boundary value problem (28) of the random elliptic PDE. Based on the model assumptions, we see that both Aand the altered right-hand side g+Ausatisfy the necessary conditions for our theory. In order to illustrate the sensitivities for this model problem, we generate a sample on Ξof size Mand replace Pby the corresponding empirical measure PM. To avoid confusion, we leave off the Msubscript in the following discussion. The underlying spaces for the control, state, and adjoint variables are discretized using standard piecewise linear finite elements on a uniform mesh with parameter h=1/(28−1). The resulting finite-dimensional deterministic optimization problems are solved by a semismooth Newton method as proposed in [14,24,25], which in the current context is mesh-independent. The forward and adjoint equations as well as the linear equations for the Hessian-vector products are solved directly and the reduced system for the semismooth Newton step is calculated using conjugate gradients as some of the associated operators are only implicitly given. The algorithm terminates once the discrete 2-norm of the residual of the optimality system reaches a tolerance of 1e-8. On average, the semismooth Newton algorithm required between three to four iterations with approximately 13 to 15 inner iterations for the CG solver. The solution zPof this problem is treated as the “true” solution. See Fig. 1for the solution zPwith M=100 and its behavior out of sample on the mean value of the state u. As expected, the control behaves well on average. We now investigate dF(PN,P), where PNis an empirical probability measure based on a random sample of size N(1 ≤N≤M)oftheMrealizations ξ1,...,ξM. Since a direct calculation of dF(PN,P)for even simple examples can be quite challenging, we calculate instead v(PN)and zPNand compute the associated errors |ν(PN)−ν(P)|and zPN−zPH. The experiment is repeated for each N=1,...,M100 times. The plots of these errors can be see in Fig. 2. 123 Stability in infinite-dimensional stochastic optimization 2753 Fig. 2 Scatter plots of the computed errors (l) |ν(PN)−ν(P)|(r) zPN−zPHfor M=100, N∈ {1,...,100}repeated 100 times The choice of setting M=100 was made for computationally expediency. Though the semismooth Newton method implemented for this example works exceptionally well, it still requires hundreds if not thousands of PDE solves for each N. Nevertheless, due to the relatively small choice of Mwe cannot use all the data points in Fig. 2in order to estimate a rate of convergence. As a compromise, we only use the data points for N=1,...,25. This provides us with the estimates zPN−zPH=O(N−0.4995)and |ν(PN)−ν(P)|=ON−0.5591 The estimates were generated by solving a linear least-squares regression problem with ridge (2) regularization term. Whereas the convergence of the optimal values would indicate a rate of roughly 1/2, the rate of convergence for the optimal solutions would be a marked improvement over the theoretical estimates based on probability metrics. One explanation for this would be that the integrands for this specific model problem have much better smoothness properties than assumed. We leave a rigorous analysis of this question for future research. 8 Conclusion We have shown that a number known results for stability of stochastic programs with finite-dimensional decision spaces can be carried over to infinite dimensions, provided certain convexity conditions are satisfied. As perhaps expected the best possible growth rates for the convergence of solutions using probability metrics are of Hölder-type. Our analysis gives rise to a number of possible future directions and open questions. For example, we consider a setting that is primarily related to risk-neutral problems, whereas problems using typically non-smooth risk measures in the objective are of significant interest for robust engineering design. In addition, the assumptions on the objective and linear operator Σare essential for the qualitative stability analysis as the infinite-dimensional setting often requires us to make use of the weak topology. Without such regularity properties, it is unclear how to proceed in general. Finally, in order to develop adaptive numerical optimization methods based on estimates of the type (21), we need to more closely investigate the convergence of dF(PN,P)for the 123 2754 M. Hoffhues et al. class of functions Fused in Sect. 6. and specific approximations PN. Section 7provides some deeper insight into this question, however the picture is far from complete. Acknowledgements TMS’s research was sponsored by the DFG Grants no. SU 963/1-1 and SU 963/2-1. The authors wish to express their gratitude to the two anonymous referees for their helpful and constructive comments. Funding Open Access funding enabled and organized by Projekt DEAL. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. A Results from fixed point theory The following is a result can be found in the monograph [12]. We provide a translation here along with a short proof for the reader’s convenience Proposition 2 (Lemma 3.1 in [12])Let V be a Hilbert space with inner product (·,·), dual V and dual pairing ·,·,b∈V,A:V→Va strongly monotone (with constant γ>0) and Lipschitz continuous (with modulus L >0) operator, and J:V→Vthe duality mapping, i.e., Ju,v=(u,v),∀u,v ∈V. Then the mapping Kt:V→V given by Ktx=x−tJ−1(Ax −b) is a contraction with constant 0<κ(t)<1, where κ(t)=!1−2γt+L2t2 and t ∈(0,2γ L2). Moreover, the unique fixed point of Ktis the unique solution of Ax =b and belongs to the ball around zero with radius r =(1−κ(t))−1Kt0−0= t(1−κ(t))−1A0−b. Remark 2 Note that min κ(t)=L−1L2−γ2and, hence, κ(t)is typically close to 1. Proof Let x,x∈H. Then Ktx−Ktx2=x−x2−2tJ−1Ax −Ax,x−x+t2J−1(Ax −Ax)2 =x−x2−2tAx −Ax,x−x+t2Ax −Ax2  123 Stability in infinite-dimensional stochastic optimization 2755 ≤1−2tγ+t2L2x−x2=κ2(t)x−x2. Clearly, 0 <κ(t)<1ifft∈(0,2γ L2). Furthermore, the unique solution ¯x(b)of Ax =bsatisfies ¯x(b)=Kt¯x(b)≤Kt¯x(b)−Kt0+Kt0−0≤κ(t)¯x(b)+Kt0−0, from which immediately obtain the estimate ¯x(b)≤Kt0−0 1−κ(t)=t 1−κ(t)A0−b. This finishes the proof.  The following result can be found, e.g., in [7]. Proposition 3 (Theorem 1A.4 in [7])Let P be a metric space with metric ρand X a complete metric space with metric d. Let F :P×X→X and assume that there exist α∈(0,1)and λ>0such that dF(p,x), F(p,x)≤αd(x,x)∀x,x∈X,p∈P dF(p,x), F(p,x)≤λρ ( p,p)∀p,p∈P,x∈X. Then, for each p ∈P, there exists a unique fixed point x(p)of F(p,·)in X and we have the estimate d(x(p), x(p)) ≤λ 1−αρ(p,p)∀p,p∈P. References 1. Alt, H.W.: Linear functional analysis. Universitext: an application-oriented introduction. Springer, London Ltd, London. Translated from the German edition by Robert Nürnberg (2016). https://doi.org/ 10.1007/978-1-4471-7280-2 2. Antil, H., Leykekhman, D.: A brief introduction to PDE-constrained optimization. Frontiers in PDEconstrained optimization. IMA Vol. Math. Appl., vol. 163, pp. 3–40. Springer, New York (2018) 3. Attouch, H., Buttazzo, G., Michaille, G.: Variational analysis in Sobolev and BV spaces. MPS/SIAM Series on Optimization, vol. 6. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA (2006) 4. Bachmayr, M., Cohen, A., Migliorati, G.: Sparse polynomial approximation of parametric elliptic PDES. PART 1: Affine coefficients. ESAIM: M2AN 51(1), 321–339 (2017). https://doi.org/10.1051/ m2an/2016045 5. Cohen, A., Devore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s. Anal. Appl. (Singapore) 9(1), 11–47 (2011). https://doi.org/10.1142/ S0219530511001728 6. Dereich, S., Scheutzow, M., Schottstedt, R.: Constructive quantization: approximation by empirical measures. Ann. Inst. Henri Poincaré Probab. Stat. 49(4), 1183–1203 (2013). https://doi.org/10.1214/ 12-AIHP489 123 2756 M. Hoffhues et al. 7. Dontchev, A.L., Rockafellar, R.T.: Implicit functions and solution mappings: a view from variational analysis. Springer Series in Operations Research and Financial Engineering, 2nd edn. Springer, New York (2014) 8. Dudley, R.M.: Real analysis and probability. The Wadsworth & Brooks/Cole Mathematics Series. Wadsworth & Brooks/Cole Advanced Books & Software, Pacific Grove, CA (1989) 9. Dunford, N., Schwartz, J.T.: Linear operators. Part I. Wiley Classics Library. General theory, With the assistance of William G. Bade and Robert G. Bartle, Reprint of the 1958 original, A Wiley-Interscience Publication. Wiley, New York (1988) 10. Falk, R.S.: Approximation of a class of optimal control problems with order of convergence estimates. J. Math. Anal. Appl. 44, 28–47 (1973). https://doi.org/10.1016/0022-247X(73)90022-X 11. Fournier, N., Guillin, A.: On the rate of convergence in Wasserstein distance of the empirical measure. Probab. Theory Related Fields 162(3–4), 707–738 (2015). https://doi.org/10.1007/s00440-014-05837 12. Gajewski, H., Gröger, K., Zacharias, K.: Nichtlineare Operatorgleichungen und Operatordifferentialgleichungen. Mathematische Lehrbücher und Monographien. II, Abteilung, Mathematische Monographien, Band 38. Akademie-Verlag, Berlin (1974) 13. Hille, E., Phillips, R.S.: Functional analysis and semi-groups. vol. 31, Rev. ed. American Mathematical Society Colloquium Publications, American Mathematical Society, Providence (1957) 14. Hintermüller, M., Ito, K., Kunisch, K.: The primal-dual active set strategy as a semismooth Newton method. SIAM J. Optim. 13(3), 865–888 (2002) 15. Hinze, M., Pinnau, R., Ulbrich, M., Ulbrich, S.: Optimization with PDE constraints. Mathematical Modelling: Theory and Applications, vol. 23. Springer, New York (2009) 16. Kouri, D.P., Surowiec, T.M.: Risk-averse PDE-constrained optimization using the conditional valueat-risk. SIAM J. Optim. 26(1), 365–396 (2016). https://doi.org/10.1137/140954556 17. Kouri, D.P., Surowiec, T.M.: Existence and optimality conditions for risk-averse PDE-constrained optimization. SIAM/ASA J. Uncertain. Quantif. 6(2), 787–815 (2018). https://doi.org/10.1137/ 16M1086613 18. Müller, A.: Integral probability metrics and their generating classes of functions. Adv. Appl. Probab. 29(2), 429–443 (1997) 19. Rachev, S.T.: Probability metrics and the stability of stochastic models. Wiley Series in Probability and Mathematical Statistics: Applied Probability and Statistics. Wiley, Chichester (1991) 20. Rachev, S.T., Römisch, W.: Quantitative stability in stochastic programming: the method of probability metrics. Math. Oper. Res. 27(4), 792–818 (2002). https://doi.org/10.1287/moor.27.4.792.304 21. Ramsay, J.O., Silverman, B.W.: Functional data analysis. Springer Series in Statistics, 2nd edn. Springer, New York (2005) 22. Römisch, W.: Stability of stochastic programming problems. In: Stochastic programming, Handbooks Oper. Res. Management Sci., vol. 10, pp. 483–554. Elsevier, Amsterdam (2003). https://doi.org/10. 1016/S0927-0507(03)10008-4 23. Topsøe, F.: On the connection between P-continuity and P-uniformity in weak convergence. Probab. Theory Appl. 12, 281–290 (1967) 24. Ulbrich, M.: Semismooth Newton methods for operator equations in function spaces. SIAM J. Optim. 13(3), 805–841 (2002) 25. Ulbrich, M.: Semismooth Newton methods for variational inequalities and constrained optimization problems in function spaces. MOS-SIAM Series on Optimization, vol. 11. SIAM, MOS, Philadelphia, PA (2011). https://doi.org/10.1137/1.9781611970692 26. Vexler, B., Wollner, W.: Adaptive finite elements for elliptic optimization problems with control constraints. SIAM J. Control Optim. 47(1), 509–534 (2008). https://doi.org/10.1137/070683416 27. Zolotarev, V.M.: Probability metrics. Theory Probab. Theory Appl. 28(2), 278–302 (1983) Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. 123