A Risk-Aversion Approach for the Multiobjective Stochastic Programming Problem

mathematics Article A Risk-Aversion Approach for the Multiobjective Stochastic Programming Problem Javier León 1,∗, Justo Puerto 2and Begoña Vitoriano 1 1HUM-LOG Research Group, Instituto de Matemática Interdisciplinar (IMI), Facultad de Ciencias Matemáticas, Universidad Complutense de Madrid, Plaza de las Ciencias 3, 28040 Madrid, Spain; [email protected] 2Mathematical Research Institute (IMUS), University of Seville, 41004 Sevilla, Spain; p[email protected] *Correspondence: [email protected] Received: 10 October 2020; Accepted: 8 November 2020; Published: 13 November 2020   Abstract: Multiobjective stochastic programming is a field that is well suited to tackling problems that arise in many fields: energy, financial, emergencies, among others; given that uncertainty and multiple objectives are usually present in such problems. A new concept of solution is proposed in this work, which is especially designed for risk-averse solutions. The proposed concept combines the notions of conditional value-at-risk and ordered weighted averaging operator to find solutions protected against risks due to uncertainty and under-achievement of criteria. A small example is presented in order to illustrate the concept in small discrete feasible spaces. A linear programming model is also introduced to obtain the solution in continuous spaces. Finally, computational experiments are performed by applying the obtained linear programming model to the multiobjective stochastic knapsack problem, gaining insight into the behaviour of the new solution concept. Keywords: multiobjective stochastic programming; linear programming; risk-aversion 1. Introduction Decision making is never easy, yet we often have to make decisions. Emergencies and disaster management are fields in which many difficulties often arise, such as high uncertainty and multiple conflicting objectives. Risk-averse decisions are usually sought to overcome such difficulties. Risk-aversion is the attitude for which we prefer to lower uncertainty rather than gambling extreme outcomes (positive or negative). Risk-aversion, although typically studied in problems with uncertainty, can as well be considered when making decisions with multiple criteria. For instance, in the field of disaster management, solutions that are sufficiently good for all criteria are usually preferred to others that perform exceptionally good for some criteria, but inadequately for the others. Multicriteria decision making (MCDM) is a field worth of consideration when studying real-world problems. This situation, in which multiple conflicting objectives have to be optimized, has led to the definition of different solution concepts and methodologies. A specific methodology should be applied depending on the problem and the type of solution considered. A key concept in MCDM is the notion of efficiency, which reflects the intuition that, for a solution to be acceptable, another cannot exist improving that one in every objective. Uncertainty is another feature that is present in the studied problems, in which risk-averse decisions will be preferred. The most common ways for dealing with the uncertainty are stochastic programming and robust optimization, in which fuzzy optimization is also included [ 1 ]. Stochastic programming is the widest used technique when there are historical data or information to infer a probability distribution. Moreover, usually discrete distributions are used, calling scenarios the Mathematics 2020,8, 2026; doi:10.3390/math8112026 www.mdpi.com/journal/mathematics Mathematics 2020,8, 2026 2 of 26 different values. The concepts of value-at-risk (VaR) and conditional value-at-risk (CVaR) are widely used for quantifying risk. They are typically defined for losses distributions in finance, where the right tail of the distributions are of interest. Consider now the following problem, in which multiple objectives to be minimized and uncertainty are included simultaneously: min x∈X(f1(x,ω), . . . , fK(x,ω)) The above problem is typically called multiobjective stochastic programming problem (MSP), especially if ω, the uncertainty source, has a known probability distribution. In this paper, we introduce a new solution concept in multiobjective stochastic programming based on risk-averse preferences. Such a concept is complemented with a mathematical programming model in order to compute it efficiently, and computational experiments are performed to assess its strengths. The remaining of this paper is organized as follows. Section 2presents a literature review of multicriteria decision-making and uncertain optimization. Section 3includes the definition of a novel concept of solution for MSP problems and studies its properties. In Section 3.2, such a solution concept is illustrated with a basic example when the decision space is finite and small. Section 4shows how to obtain such a solution with a linear programming model. An application to the multicriteria knapsack problem is developed in Section 5, and general conclusions of the research are drawn in Section 6. 2. Literature Review 2.1. Multicriteria Decision-Making and Optimization Under Uncertainty MCDM techniques have recently been used for solving real world problems as varied as: disaster management [2,3] , engineering [ 4 ], finance [ 5 , 6 ], forest planning [ 7 ], healthcare [ 8 ], location of waste facilities [ 9 ], police districting [ 10 ], route planning [ 11 ], train scheduling [ 12 ], or urban planning [13,14]. An important concept used throughout this paper is the one of efficiency. The notation used is the given in [15]: Definition 1 (Efficiency, [ 15 ]) . Let f1(x) , . . . , fK(x) be objective functions to be minimized, and let X be the feasible set. A feasible solution ˆ x∈X is called: • Weakly efficient if there is no x∈X , x6=ˆ x , such that f(x)<f(ˆ x) i.e. fk(x)<fk(ˆ x) for all k=1, . . . , K. • Efficient or Pareto optimal if there is no x∈X such that fk(x)≤fk(ˆ x) for all k= 1, . . . , K and fi(x)<fi(ˆ x)for some i ∈ {1, . . . , K}. •Strictly efficient if there is no x ∈X, x 6=ˆ x, such that f (x)≤f(ˆ x). The different approaches for dealing with uncertainty do not respond to the desires of the modeller; instead, they reflect the nature of the uncertainty. If the uncertainty comes with an underlying known or estimated probability distribution, then stochastic programming is used. For an introduction to stochastic programming, the reader is referred to [ 16 ]. On the other hand, if uncertainty comes from a lack of precision or semantic uncertainty, then robust optimization is used. Robust optimization does not assume a known (or existing) distribution [ 17 – 19 ]. A recent review of robust optimization is written in [20]. Stochastic programming seeks the optimization of a characteristic value of a random variable, usually its average. However, in risk-averse contexts the usage of value-at-risk and conditional value-at-risk is common for quantifying risk (see, for instance, [21–25]). Mathematics 2020,8, 2026 3 of 26 Definition 2 (CVaR, [ 26 ]) . Given FX(x) distribution function, and β∈[ 0,1 ] , the β -CVaR is the conditional expected value over {x:FX(x)≥β}. 2.2. Multiobjective Stochastic Programming Multiobjective stochastic programming refers to models in which there are several criteria and stochastic uncertainty simultaneously. Reference [ 27 ] develops the PROTRADE method, where utility functions are defined to aggregate objectives into a single objective stochastic problem. The resulting problem is solved with an interactive method, where the decision-maker defines an expected solution and a feasibility probability. Reference [ 28 ] reduces the stochasticity by adding some good measures to the list of objectives, such as the mean, variance, or probability of being over/below a threshold. The resulting multiobjective deterministic problem is solved aggregating the objectives, but it could be solved via other techniques. Reference [ 29 ] compares the stochastic approach with the multiobjective approach when using different techniques. The stochastic approach transforms the MSP on a single-objective stochastic problem, while the multiobjective approach first reduces the stochasticity transforming the MSP on a deterministic multiobjective problem. They highlight that “the multiobjective approach forgets the possible existence of stochastic dependencies between objectives”. Reference [ 30 ] studies stochastic goal programming, where the deviation of the objective functions to some goals set beforehand to stochastic values is minimized. In [ 31 ], a chance-constrained compromise approach is proposed, with an example presented in [ 32 ]. In [ 33 ], the INTEREST method is proposed. It is an interactive reference point method. The decision-maker gives reference levels ui and probabilities βi , hoping to achieve a solution x∗ such that P(fi(x∗)≤ui)≥βi . If this is infeasible, then the decision-maker should either increase the reference levels or decrease the probabilities of achievement. [ 34 ] reviews different solutions methods for the MSP problem, categorizing them as stochastic approach or multiobjective approach. Reference [ 35 ] surveys methods for MSP problems that do not reduce the multiple objectives before the analysis of the problem, acknowledging the difficulty of risk-averse decision-making. More recently, in [ 36 ], different ordering relations for multicriteria problems with uncertainty are presented, building upon existing notions of robustness. Some fields where MSP models have been developed are: forest management [ 37 ], multiple response optimization [ 38 ], energy generation [ 39 , 40 ], energy exchange [ 41 ], capacity investment [ 42 ], disaster management [43,44], portfolio optimization [45], and cash management [46], among others. 3. Methodology The concept of CVaR allows aggregating several scenarios by just looking at what happens in the worst ones. The ordered weighted averaging (OWA) operators are defined in [ 47 ], and independently in the field of locational analysis [ 48 , 49 ] under the name of ordered median function. These concepts will allow for us to aggregate different criteria by looking at the least desirable ones, as a risk-aversion measure. Definition 3 (OWA, [ 47 ]) . Given a1 , . . . , an∈R , the ordered weighted averaging (OWA) operator with weights λ1, . . . , λnis defined as: OWA(a1, . . . , an) = ∑ i λia(i) where a(1), . . . , a(n)is the ordered vector from largest to smallest (a1, . . . , an). Remark 1. For certain weights, the OWA represents a known quantity: •If λi=1 n, the resulting OWA is the average of a. •If λ1=1, and λj=0for j >1, the OWA is the maximum of a. •If λn=1, and λj=0for j <n, the OWA is the minimum of a. Mathematics 2020,8, 2026 4 of 26 Reference [ 50 ] later studies how to assign weights for an OWA when criteria have different importances. Definition 4 (OWA with importances, [ 50 ]) . Given a1 , . . . , an∈R with importances u1 , . . . , un such that ∑iui= 1the weights λj for the OWA can be calculated with f , the weight generating function in the following manner: 1. Sort vector a such that a(1)≥a(2)≥. . . ≥a(n). 2. With (×)as the order induced by a, define Tj=∑j k=1u(k). 3. Let f be a function, such that f:[ 0,1 ]→[ 0,1 ] and f( 0 ) = 0, f( 1 ) = 1. This function is called weight generating function. 4. Obtain the weights as λj=f(Tj)−f(Tj−1). Example 1 (of Definition 4).Consider the following weight generating function, for a given r ∈(0,1]: f(x) = (x rif x <r 1if x ≥r(1) Let (×) be the order, such that a(1)≥ · · · ≥ a(n) , u(j) the weight associated to a(j) , and also let Tj=∑j k=1u(k). We shall now see how the weights are obtained from f. Let j∗be such that Tj∗−1<r≤Tj∗. •λ1=f(T1) = f(u(1)) = u(1) r, assuming u(1)<r •λ2=f(T2)−f(T1) = f(u(1)+u(2))−f(u(1)) = u(1)+u(2) r−u(1) r=u(2) r, assuming u(1)+u(2)<r •... •λj∗=f(Tj∗)−f(Tj∗−1) = 1−u(1)+u(2)+···+u(j∗−1) r, since Tj∗≥r •λj∗+1=f(Tj∗+1)−f(Tj∗) = 1−1=0 •... •λn=f(Tn)−f(Tn−1) = 1−1=0 Consequently the OWA of a1, . . . , anwith importances u1, . . . , unis: OWA =u(1) ra(1)+u(2) ra(2)+· · · +1−u(1)+u(2)+· · · +u(j∗−1) ra(j∗) =u(1)a(1)+u(2)a(2)+· · · +r−u(1)−u(2)−. . . a(j∗) r That is, the OWA that is characterized by the weight generating function given in (1) is the average of the worst aj , weighted by their importances, with the total importance adding up to r . The values of λ reflect the preferences of the decision-maker. The parameter r leads to incorporating the different attributes, from worst to best, until a threshold is reached. The starting point of this paper is the recurrent idea of representing ordered weighted or ordered median operators while using k -sums. k -sums (or k -centra in the location analysis literature) are sums of the k -largest terms of a vector [ 51 ]. One can trace back, at least to [ 52 ], the use of k -sums to represent ordered median objectives. More recent references are, for instance, [ 53 – 56 ]. This last reference introduces a normalized version of k -centrum, named β -average, which will be used in our paper. Through the remaining of the paper, consider that fj k(x) are functions to be minimized within a feasible set X , with k= 1, . . . , K representing K different objectives with importances wk and j= 1, . . . , Jencoding Jdifferent scenarios with probabilities πj. Mathematics 2020,8, 2026 5 of 26 Definition 5 ( β -average, gβ k(x) , [ 56 ]) . Given β∈( 0,1 ] , for each criterion k it can be defined gβ k(x) which measures the average of f on the worst scenarios f1 k(x), . . . , fJ k(x) , with accumulated probability equal to β . Remark 2 ([ 56 ]) . Given a value β , if the sum of the probabilities of the worst scenarios is exactly β , then the β-average is exactly (1−β)-CVaR. Example 2. Consider a point x , a fixed criterion k , and five different scenarios with probabilities πj and values of fj k given. Table 1shows the β -averages for different values of β , in which the scenarios have been ordered from largest value of f to smallest. • For β= 0.2, the scenario j= 1is the only one needed to obtain the worst scenario with probability 0.2, and hence gβ k(x) = 0.2×10 0.2 =10. •When βequals 0.3 it is necessary to include scenario 2, obtaining a β-average of 0.2×10+0.1×7 0.3 =9. • Finally, if β= 0.5 scenario 3 needs to be added as well, but only with the probability needed until reaching 0.5: gβ k(x) = 0.2×10+0.1×7+0.2×4 0.5 =7. Table 1. Small example of β-average for different values of β. Scenario β 1 2 3 4 5 0.2 0.3 0.5 πj0.2 0.1 0.3 0.25 0.15 10 9 7 fj k(x)10 7 4 3 2 When using the β -average the functions fj k(x) were transformed into gβ k(x) , a collection of K functions not depending on the scenario. An OWA will be defined now, via its weight generating function, which will reduce the Kβ-averages into a scalar function. Definition 6 ( r -OWA, Or(~ x) ) . Given x1 , . . . , xK∈R with importances w1 , . . . , wK , such that ∑K k=1wk= 1 and r ∈(0,1], the function Or(~ x)is defined as the OWA with the following weight generating function: f(x) = (x rif x <r 1if x ≥r Remark 3. The definition of Or(~ x) is made in a similar manner that the one given of the β -average (Definition 5), but it is done on a context with importances rather than probabilities. Example 3shows the similarities between both of the approaches. Example 3. Consider a point x and let gk(x) be the evaluation of x under five different criteria with importances wj . Table 2shows the r -OWAs for different values of r , in which the criteria have been ordered from largest values of gk(x)to smallest. Consider the case r =0.5: 1. As gk(x)are already ordered for largest to smallest, the values of Tkare: T1=0.2, T2=0.2 +0.1 =0.3, T3=0.6, T4=0.85, T5=1 2. The values of Tkunder f : f(T1) = 0.2 0.5,f(T2) = 0.3 0.5,f(T3) = f(T4) = f(T5) = 1 3. The weights of the OWA: λ1=0.2 0.5,λ2=0.3 −0.2 0.5 =0.1 0.5,λ3=1−0.3 0.5 =0.2 0.5,λ4=λ5=0 Mathematics 2020,8, 2026 6 of 26 4. Consequently, the r-OWA is: r-OWA =0.2g(1)(x) + 0.1g(2)(x) + 0.2g(3)(x) 0.5 =0.2 ×10 +0.1 ×7+0.2 ×4 0.5 =7 Table 2. Small example of r-OWA for different values of r. Criterion r 1 2 3 4 5 0.2 0.3 0.5 wk0.2 0.1 0.3 0.25 0.15 10 9 7 gk(x)10 7 4 3 2 Remark 4. Given x1 , . . . , xK and its associated importances w1 , . . . , wK , then the λk of the r -OWA are λk=˜ λk r , with ˜ λkbeing determined as: Or(x1, . . . , xK) = max ˜ λ1,...,˜ λK˜ λ1x1+· · · +˜ λKxK r|˜ λk≤wk,∑˜ λk=r Given r , β∈( 0,1 ] and x∈X , let us introduce the function hβ r(x) as the r -OWA of the β -averages. That is: hβ r(x) = Orgβ 1(x), . . . , gβ K(x) Remark 5. If the importance of all the criteria is the same (wk=1 Kfor all k) and r =n Kwith n ∈ {1, . . . , K}, then hβ r(x)is the average of the n worst β-averages. Recall that this is called n-centra [57]. Definition 7 (Dominance) . Let x and y feasible solutions ( x , y∈X ) and r , β∈( 0,1 ] . Then x dominates y (x %y) if hβ r(x)≤hβ r(y), where hβ r(x)is the r-OWA of the β-averages. Definition 7induces a domination relationship with the following properties: Reflexivity Given x,hβ r(x)≤hβ r(x), and then x%x, so %is reflexive. Transitiveness Given x%y , y%z , we have hβ r(x)≤hβ r(y) and hβ r(y)≤hβ r(z) , and then hβ r(x)≤hβ r(z), which leads to x%z, and we conclude that %is transitive. Antisymmetry Given x%y , y%x , we have hβ r(x)≤hβ r(y) and hβ r(y)≤hβ r(x) , but, from hβ r(x) = hβ r(y) , it cannot be guaranteed that x=y , and, hence, % is not antisymmetric. 3.1. Idea of Solution and Dominance Properties Consider the multiobjective stochastic programming problem: min x∈X(f1(x,ω), . . . , fK(x,ω)) The previously defined concepts of β -average and r -OWA transform the MSP problem into a deterministic multiple objective problem, and then into a deterministic single objective problem. MSP →MOP →LP(MIP) fj k(x)β-average −−−−−→ gβ k(x)r-OWA −−−−→ hβ r(x) Mathematics 2020,8, 2026 7 of 26 1. For every x∈X there is a function fj k to be minimized which depends on the scenario j and the criterion k. 2. The problem is transformed into a deterministic one with multiple objectives (MOP) while using the β-average concept. 3. When computing the r -OWA, each x∈X is assigned a scalar. The problem consists of finding the x, which minimizes this hβ r(x). The solution procedure lies into what is usually called a scalarization approach. When obtaining a minimizer of hβ r(x) , it is also desired that the optimal solution is efficient for the associated MOP problem: min x∈Xgβ 1(x), . . . , gβ K(x)(MOP) Proposition 1. Given x∗minimum of hβ r(x)the following statements hold: 1. x∗is not necessarily efficient of the MOP problem. 2. x∗is weakly efficient of the MOP problem. 3. If x∗is the only minimum of hβ r(x), then x∗is efficient. 4. Given x∗ not efficient, an alternative y∗ can be found on a second phase, such that y∗ is efficient and hβ r(x∗) = hβ r(y∗). These properties are known when using scalarization techniques [ 15 ]. Hence, only an example of the first statement will be shown. Example 4 ( x∗ is not necessarily efficient) . Consider the example that is displayed in Table 3, in which there are only two feasible solutions, two equiprobable scenarios ( π1=π2=1 2 ), three equally important criteria (w1=w2=w3=1 3), and consider the values of β=1 2and r =2 3are taken. Table 3. Values of two alternatives for each scenario j and criterion k , together with their β -averages (β=1 2) and r-OWAs (r=2 3). (a) Alternative 1 (b) Alternative 2 k1k2k3k1k2k3 j10.80 0.40 0.30 j10.70 0.45 0.65 j20.60 0.20 0.65 j20.80 0.30 0.50 β-average 0.80 0.40 0.65 β-average 0.80 0.45 0.65 r-OWA 0.725 r-OWA 0.725 The β -averages are ( 0.8,0.4,0.65 ) for the first alternative and ( 0.8,0.45,0.65 ) for the second alternative. When computing the function hβ r , both of the alternatives have an objective value of 0.725. Consequently, even though the second alternative is an optimal solution of hβ r , it is not an efficient solution of the MOP problem as its β-averages are dominated by those of the first alternative. The transformation of the problem from a multiple objective one to a single objective one is done using weights. These weights correspond to a subjective scale that is introduced by the expert, representing the importance of the criteria considered in the problem as accurately as possible. 3.2. An Illustrative Example The proposed solution concept will now be applied, first with a discrete (and small) case. When the solution space is discrete, and all feasible solutions can be explicitly enumerated, the steps are as follows: Mathematics 2020,8, 2026 8 of 26 Step 0 Normalize all objective functions fj k(x). Step 1 Set values for β,r∈(0,1]. Step 2 For every x∈Xand every criterion define gβ k(x)as: gβ k(x) = average of worst scenarios for criterion k with probabilities adding up to β Step 3 Define hβ r(x)as: hβ r(x) = average of worst gβ k(x)values with importances adding up to r Step 4 Search for x∈Xminimizing hβ r(x). Assume a decision space with only four alternatives, evaluated under five different scenarios with six criteria. For each of those alternatives, it can be computed the value of the functions fj k(x) to be minimized. Table 4shows the values of f , evaluated on the feasible point x1 , by each of the scenarios and criteria considered. Table 4. Values of alternative 1 by scenario (j) and criteria (k). Criteria w1=0.20 w2=0.10 w3=0.20 w4=0.25 w5=0.15 w6=0.10 k1k2k3k4k5k6 scenarios π1=0.15 j10.51 0.27 0.39 0.45 0.75 0.76 π2=0.20 j20.58 0.65 0.47 0.26 0.90 0.24 π3=0.30 j30.48 0.44 0.90 0.50 0.93 0.65 π4=0.25 j40.76 0.18 0.01 0.90 0.56 0.02 π5=0.10 j50.86 0.36 0.21 0.28 0.63 0.72 The first step is calculating the β-averages. Let us assume a value of β=0.3: 1. For the first criterion the worst scenario is j5 , which has probability 0.1. The second worst is j4 , with a probability of 0.25. As the sum of those probabilities exceeds the β fixed, for computing the β-average just a probability of 0.2 is considered: gβ 1(x1) = 0.1 ×0.86 +0.2 ×0.76 0.3 =0.793 2. gβ 2(x1) = (0.2 ×0.65 +0.1 ×0.44)/0.3 =0.580 3. gβ 3(x1) = (0.3 ×0.90)/0.3 =0.900 4. gβ 4(x1) = 0.833, gβ 5(x1) = 0.930, gβ 6(x1) = 0.728 The last step is calculating the function hβ r(x) , which is, the r -OWA of the β -averages. Table 5 calculates the r -OWA, and also shows the information of the previously calculated β -averages, when the value of r=0.17 is taken. Mathematics 2020,8, 2026 9 of 26 Table 5. Values of alternative 1 by scenario (j) and criteria (k). Criteria w1=0.20 w2=0.10 w3=0.20 w4=0.25 w5=0.15 w6=0.10 k1k2k3k4k5k6 scenarios π1=0.15 j10.51 0.27 0.39 0.45 0.75 0.76 π2=0.20 j20.58 0.65 0.47 0.26 0.90 0.24 π3=0.30 j30.48 0.44 0.90 0.50 0.93 0.65 π4=0.25 j40.76 0.18 0.01 0.90 0.56 0.02 π5=0.10 j50.86 0.36 0.21 0.28 0.63 0.72 β-average, β=0.30 0.793 0.580 0.900 0.833 0.930 0.728 r-OWA, r=0.17 0.927 The values of the functions for the other alternatives, as well as its β -averages and r -OWAs are shown in Tables A1–A3, starting on Page 21. Table 6illustrates a summary of the results, where all of the β -averages and r -OWAs are shown, whcih determines that the optimal alternative for the values of βand rgiven is Alternative 1. Table 6. β-averages and r-OWAs for each of the four feasible alternatives of the example. β-Averages r-OWA gβ 1(x)gβ 2(x)gβ 3(x)gβ 4(x)gβ 5(x)gβ 6(x)hβ r(x) Alternative 1 0.793 0.580 0.900 0.833 0.930 0.728 0.927 Alternative 2 0.930 0.832 0.703 0.820 0.660 0.770 0.930 Alternative 3 0.765 0.775 0.468 0.643 0.950 0.883 0.943 Alternative 4 0.993 0.760 0.473 0.773 0.820 0.990 0.993 Variations on βand ryield very different results. Figure 1a shows which of the four alternatives has the lowest hvalue, depending on the values of βand r. Figure 1b shows the optimal objective value when varying the parameters β and r . It can be appreciated how h decreases when β and r increase. This is due to the fact that the original fj k functions are to be minimized and, the larger the parameters β and r , the more favourable scenarios/criteria will take part on the computation of hβ r(x), hence decreasing its optimal value. (a)(b) Figure 1. The results from illustrative example. ( a ) Optimal alternative for some values of r and β , where each of the four alternatives is colour-coded. ( b ) Optimal values of function hβ r(x) for some values of rand β. Mathematics 2020,8, 2026 16 of 26 •tMSP , tMIP : solution time in seconds of models (MSP) and (MIP). With them, the following value is calculated: ∆time :=tMSP tMIP (time penalty factor) ∆time , the time penalty factor, indicates the increase of computing time when solving model (MSP) rather than model (MIP). •z∗ MSP,z∗ MIP: optimal values of the models. •fMSP (x∗ MIP),fMIP x∗ MSP: objective value of x∗ MIP in model (MSP) and vice versa. • To grasp the difference between the MSP and the naive approach, the following will be calculated: ∆avg :=100 fMIP x∗ MSP−z∗ MIP z∗ MIP (deteriorating rate) ∆tail :=100 fMSP (x∗ MIP)−z∗ MSP fMSP x∗ MIP(improvement rate) These quantities reflect what is the effect of making decision x∗ MSP instead of x∗ MIP . Large values of ∆avg indicate high penalties for making decision x∗ MSP instead of x∗ MIP in average scenarios-criteria. Similarly, the larger ∆tail , the higher benefit obtained from making decision x∗ MSP in tail events. They will be, respectively, called deteriorating rate and improvement rate. The models are solved in GAMS 26.1.0 with solver IBM ILOG CPLEX 12.8.0.0, while using a personal computer with an Intel Core i7 processor and 16GB RAM. Experiment 1 The first experiment will consist on a full factorial design, in which the values of |I| , |J| , |K| , r , β fall in these sets: •|I|∈{50,100,200} •|J|∈{5,25,100} •|K|∈{3,6,9} •r∈ {0.33, 0.5,0.67} •β∈ {0.05, 0.1,0.5} For each tuple (I , J , K) a random instance will be generated using Algorithm 1, which will then be solved for every pair (r , β) . All of the criteria and scenarios are given the same importance and probabilities. That is, wk=1 |K|,πj=1 |J|. The time limit was set in two hours by instance, in which all but three of the 35=243 configurations were solved to optimality. Experiment 2 For the next experiment, 100 random instances will be created, keeping the values of |I| , |J| , |K| , r , β constant and equal to the median value of the previous experiment. That is, |I|= 100, |J|= 25, |K|= 6, r= 0.5, β= 0.1. All of the criteria and scenarios are given the same importance and probabilities. All 100 instances were solved to optimality. 5.2. Results Experiment 1 Table A4 (in the Appendix A) shows, for each of the 243 instances, the solution times of the MSP and the MIP models, and the deteriorating and improvement rates of using the MSP solution instead of the MIP solutions (measured in deviation to MIP solution). Mathematics 2020,8, 2026 17 of 26 Table 7shows the objective values, by scenario and criterion, of the first instance of the experiment. Such an instance contains 50 objects, five scenarios, three criteria, and the parameters r and β are set to 0.33 and 0.05 respectively. Results show that the MSP solution (Table 7a) is more balanced through every scenario and criterion, with a worst value of 13.71. On the other hand, the MIP solution (Table 7b) attains larger (worse) values on some scenarios and criteria. Table 7. Objective values by scenario-criterion of solutions obtained with the multiobjective stochastic programming problem (MSP) and MIP models, for the first instance of the first experiment. (a) MSP Solution (b) MIP Solution k1k2k3k1k2k3 j111.97 11.26 11.00 j19.65 10.75 9.71 j211.96 9.92 13.71 j211.19 10.17 14.90 j313.48 13.51 10.92 j314.05 13.55 9.40 j413.62 13.13 13.71 j414.12 13.00 13.35 j512.94 11.35 13.47 j513.64 10.33 11.42 Table 8shows the correlations between the times and rates with the parameters of the instance. It can be seen how the MSP solution has a higher impact when fewer scenarios are considered. In addition to that, it can be appreciated that the MSP solution times decrease when β increase, which is, when more scenarios are included in the β-average computation. Table 8. Correlations. |I| |J| |K| rβ tMSP 0.34 0.09 −0.11 −0.05 −0.19 tMIP 0.51 0.18 −0.14 −0.03 −0.07 ∆time 0.31 0.11 −0.08 −0.02 −0.18 ∆avg −0.05 −0.57 −0.28 −0.09 −0.36 ∆tail −0.07 −0.56 −0.18 −0.21 −0.50 This observation is confirmed by Table 9, in which it can be seen that the median time penalty factor (how much longer does it take to solve the MSP model than the MIP model) is much smaller when β=0.5 than when β=0.05. Table 9. MSP runtimes and increases as compared to MIP runtimes, grouped by β. βtMSP ∆time Min Mean Median Max Std Min Mean Median Max Std 0.05 0.12 659.49 6.32 7222.95 1787.07 0.94 3188.96 32.77 50473.04 9472.55 0.10 0.12 212.47 2.23 4765.42 728.49 0.98 1002.35 11.09 20192.48 3245.85 0.50 0.13 3.49 0.67 62.14 9.05 1.06 19.14 3.75 414.29 55.51 The solution times of the MSP model are alarmingly high for some instances, due to the fact that the admissible integrality gap has been set to zero. If that is relaxed, it can be seen that all of the 243 instances reach an integrality gap smaller than 5% in about three seconds, 2% in about five seconds and 1% in about 88 seconds. Table 10 groups instances by r and β , and shows the deteriorating and improvement rates. It can be seen that the improvement rate (in the tail) is generally higher than the deteriorating rate (in the average), especially in cases with small rand β. Mathematics 2020,8, 2026 18 of 26 Table 10. Values of ∆avg and ∆tail, grouped by rand β. rβ ∆avg ∆tail Min Mean Median Max Std Min Mean Median Max Std 0.33 0.05 0.03 1.94 1.87 5.68 1.42 0.28 4.37 4.21 9.18 2.43 0.10 0.02 1.70 1.61 5.68 1.44 0.18 3.54 2.85 9.18 2.42 0.50 0.00 0.93 0.52 4.46 1.08 0.00 1.57 0.92 4.99 1.46 0.50 0.05 0.03 1.87 1.90 4.30 1.30 0.29 3.58 3.30 6.73 1.89 0.10 0.02 1.65 1.14 4.30 1.37 0.13 2.87 2.47 6.59 1.86 0.50 0.00 0.72 0.54 3.51 0.75 0.00 1.07 0.79 3.79 1.01 0.67 0.05 0.03 1.64 1.17 3.93 1.24 0.32 3.04 3.06 6.15 1.62 0.10 0.01 1.50 1.10 3.93 1.31 0.12 2.43 2.02 5.84 1.58 0.50 0.00 0.60 0.50 3.16 0.66 0.00 0.80 0.59 3.64 0.81 This claim is also supported with Figure 2, where each of the 243 instances is shown according to the values of ∆avg and ∆tail , and are grouped by the values of (r , β) . Almost all of the instances are above the imaginary line ∆avg =∆tail , which shows that considering the MSP solution improves in the tail more than it loses in the average situations. In addition to that, it can be seen that the largest improvements in the tail are on instances with β= 0.05 (one of the usual values taken for CVaR), and especially with the smallest values of r . When r and β grow, the differences between the MIP and MSP solutions are reduced. Figure 2. Values ∆avg and ∆tail for each of the 243 instances, grouped by values of (r,β). Finally, Figure 3shows the values of fj k(x) , where x=x∗ MIP in blue squares and x=x∗ MSP in orange circles, for just one of the created instances: 200 objects, 100 scenarios, three criteria, r= 0.33, β= 0.05. The values of fj k(x) are represented for each criterion, sorting the scenarios from most to least favourable. It can be appreciated how x∗ MIP performs better than x∗ MSP in average criteria-scenarios, on the central part of the images; but, x∗ MSP is better with unfavourable situations, those with higher values of fj k(x) . This can be especially appreciated for the second criterion, in which there are three scenarios with objective values of x∗ MIP out of control. Mathematics 2020,8, 2026 19 of 26 Figure 3. Single instance with 100 scenarios and three criteria. For each k , sorted values of fj k(x) , where x= x∗ MIP in blue squares and x=x∗ MSP in orange circles. Experiment 2 Table A5 (in the appendix) contains the results for each of the 100 instances, all of them with constant parameters |I|=100, |J|=25, |K|=6, r=0.5, β=0.1. Table 11 contains a summary of the results, where it is again seen that the improvements in the tail are better than the loses in the average situations. Although single instances might take a long computing time, the median MSP solution time (3.74 s) is definitely satisfactory. It is worth mentioning that the models were implemented without providing any extra bounds or known cuts that could reduce the solution times. Table 11. Summary of experiment 2. tMSP tMIP ∆time ∆avg ∆tail mean 16.98 0.20 91.31 2.03 3.09 std 46.57 0.03 254.68 1.12 1.49 min 0.53 0.14 2.81 0.16 0.86 25% 1.37 0.17 6.73 1.18 2.09 50% 3.74 0.19 19.72 1.93 2.81 75% 15.50 0.21 86.19 2.52 3.51 max 404.70 0.34 2175.82 5.67 8.57 6. Conclusions In this paper, a new concept of solution has been proposed for Multiobjective Stochastic Programming problems, exploiting risk-aversion. The proposed concept combines the notions of Mathematics 2020,8, 2026 20 of 26 conditional value-at-risk and ordered weighted averaging operator to find solutions protected against risks due to the uncertainty and under-achievement of criteria. Thus, this concept can be particularly useful in real-life situations, where there exists a great concern with respect to unfavourable situations, such as emergency management or portfolio optimization. The solution concept is supported by an efficient way to compute it by a Mathematical Programming problem. This model is linear, provided that the underlying problem can be linearly representable. Numerical experiments have been conducted for validating this approach, solving a multiobjective stochastic knapsack problem. The research has shown that the improvements in the tail (unfavourable situations) are consistently higher than loses on average situations, especially when small values of the parameters β and r are chosen. These differences, although clearly noticeable, are not as high as one could expect. This is possibly due to the randomness of the data. It is reasonable to assume that, in actual real-life problems, there are choices that are more conservative for every scenario and criterion, and thus being preferable for risk-averse attitudes. The results have shown that there is a clear increase in computational time as compared with risk neutral methods; however, this is arguably acceptable as a price to pay for being risk-averse. Furthermore, this could also be due to the random nature of the data. Nevertheless, it was also shown that allowing for even rather small integrality gaps (1%) leads to a drastic improvement in computing times. Author Contributions: Conceptualization, J.P. and B.V.; methodology, J.L., J.P. and B.V.; software, J.L.; formal analysis, J.L., J.P. and B.V.; writing—original draft preparation, J.L.; writing—review and editing, J.L., J.P. and B.V.; visualization, J.L.; project administration, J.P. and B.V.; funding acquisition, B.V. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the UCM-Santander grant CT27/16-CT28/16, the Government of Spain grants MTM2016-74983-C02-01 and PID2019-108679RB-I00 (LOG4D), H2020 grant MSCA-RISE 691161 (GEO-SAFE), and Fundación BBVA 2019 grant Complex networks meet data science. Conflicts of Interest: The authors declare no conflict of interest. Abbreviations The following abbreviations are used in this manuscript: MCDM Multicriteria decision making VaR Value-at-risk CVaR Conditional value-at-risk MSP Multiobjective stochastic programming OWA Ordered weighted averaging MOP Multiple objective problem Appendix A Table A1. Values of alternative 2 by scenario (j) and criteria (k). Criteria w1=0.20 w2=0.10 w3=0.20 w4=0.25 w5=0.15 w6=0.10 k1k2k3k4k5k6 scenarios π1=0.15 j10.40 0.58 0.39 0.45 0.54 0.18 π2=0.20 j20.68 0.74 0.70 0.15 0.54 0.72 π3=0.30 j30.93 0.52 0.23 0.82 0.21 0.03 π4=0.25 j40.37 0.85 0.07 0.42 0.52 0.22 π5=0.10 j50.92 0.13 0.71 0.39 0.90 0.87 β-average, β=0.30 0.930 0.832 0.703 0.820 0.660 0.770 r-OWA, r=0.17 0.930 Mathematics 2020,8, 2026 21 of 26 Table A2. Values of alternative 3 by scenario (j) and criteria (k). Criteria w1=0.20 w2=0.10 w3=0.20 w4=0.25 w5=0.15 w6=0.10 k1k2k3k4k5k6 scenarios π1=0.15 j10.80 0.90 0.61 0.28 0.94 0.09 π2=0.20 j20.29 0.48 0.26 0.23 0.21 0.07 π3=0.30 j30.73 0.65 0.32 0.56 0.95 0.65 π4=0.25 j40.58 0.39 0.21 0.66 0.70 0.93 π5=0.10 j50.73 0.22 0.33 0.31 0.32 0.38 β-average, β=0.30 0.765 0.775 0.468 0.643 0.950 0.883 r-OWA, r=0.17 0.943 Table A3. Values of alternative 4 by scenario (j) and criteria (k). Criteria w1=0.20 w2=0.10 w3=0.20 w4=0.25 w5=0.15 w6=0.10 k1k2k3k4k5k6 scenarios π1=0.15 j10.30 0.52 0.12 0.68 0.46 0.73 π2=0.20 j21.00 0.57 0.46 0.82 0.90 0.72 π3=0.30 j30.18 0.76 0.30 0.34 0.54 0.99 π4=0.25 j40.53 0.21 0.13 0.12 0.66 0.86 π5=0.10 j50.98 0.46 0.50 0.29 0.27 0.40 β-average, β=0.30 0.993 0.760 0.473 0.773 0.820 0.990 r-OWA, r=0.17 0.993 Mathematics 2020,8, 2026 22 of 26 Table A4. All instances of first experiment. The three instances with 200 objects, 100 scenarios, 6 criteria and β= 0.05 did not reach the optimal solution in 2 h. The integrality gaps of the solution shown are 0.31%,0.24% and 0.19% for r=0.33, 0.5 and 0.67 respectively. β→0.05 0.1 0.5 r→0.33 0.5 0.67 0.33 0.5 0.67 0.33 0.5 0.67 |I| |J| |K| tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail 50 5 3 0.12 0.13 3.75 8.01 0.15 0.12 3.75 8.01 0.18 0.14 3.51 4.54 0.14 0.14 3.75 6.59 0.12 0.12 3.75 6.59 0.20 0.17 3.51 3.79 0.12 0.12 3.75 5.84 0.12 0.12 3.75 5.84 0.13 0.12 3.16 3.64 6 0.22 0.11 3.79 5.07 0.25 0.11 3.79 5.07 0.18 0.11 1.03 3.01 0.30 0.13 3.79 4.01 0.25 0.13 3.79 4.01 0.23 0.15 1.48 2.10 0.23 0.16 3.77 3.58 0.23 0.14 3.77 3.58 0.18 0.17 1.48 1.47 9 0.28 0.14 1.76 6.67 0.36 0.14 1.76 6.67 0.20 0.15 1.58 3.26 0.22 0.14 2.02 5.66 0.21 0.14 2.02 5.66 0.22 0.14 1.24 2.10 0.20 0.15 2.02 4.39 0.22 0.13 2.02 4.39 0.25 0.15 1.20 1.37 25 3 0.76 0.18 2.47 5.37 0.66 0.17 2.47 2.85 0.26 0.17 2.00 1.55 0.64 0.18 2.47 5.08 0.62 0.16 2.47 2.51 0.25 0.15 1.00 0.97 0.60 0.17 2.47 4.93 0.51 0.16 1.79 2.52 0.26 0.14 1.00 0.69 6 1.20 0.16 1.87 5.77 0.91 0.29 1.96 3.70 0.49 0.18 0.79 1.81 1.15 0.18 1.87 4.93 0.74 0.18 1.14 3.51 0.41 0.18 0.70 1.55 0.93 0.16 1.17 4.33 0.78 0.18 1.17 3.45 0.38 0.16 0.71 1.22 9 0.69 0.16 0.61 4.21 0.78 0.16 0.43 2.78 0.57 0.16 0.67 0.92 0.52 0.15 0.61 3.90 0.96 0.16 0.44 2.14 0.61 0.16 0.67 0.65 0.69 0.15 0.61 3.25 1.02 0.18 0.39 1.75 0.47 0.18 0.51 0.68 100 3 1.15 0.15 0.07 2.43 0.78 0.15 0.07 2.15 0.44 0.14 0.07 0.16 1.07 0.14 0.07 2.02 0.83 0.19 0.07 1.74 0.34 0.14 0.07 0.16 1.14 0.15 0.07 1.81 0.85 0.16 0.07 1.53 0.36 0.14 0.07 0.14 6 2.51 0.20 0.77 2.24 4.45 0.22 0.78 1.19 3.52 0.20 0.23 0.47 2.62 0.25 0.77 1.99 5.09 0.18 0.31 1.10 5.45 0.20 0.19 0.29 2.70 0.22 0.77 1.38 3.31 0.19 0.47 0.63 5.59 0.19 0.23 0.27 9 4.06 0.18 0.03 0.41 1.47 0.17 0.03 0.30 1.07 0.16 0.00 0.00 2.75 0.17 0.03 0.29 1.12 0.16 0.03 0.30 1.11 0.16 0.00 0.00 2.06 0.19 0.03 0.32 1.10 0.16 0.03 0.18 1.16 0.17 0.00 0.00 100 5 3 1.24 0.26 3.81 7.63 1.29 0.20 3.81 7.63 0.37 0.22 2.08 4.47 0.72 0.18 3.81 5.22 0.66 0.18 3.81 5.22 0.32 0.27 1.56 2.33 0.74 0.19 3.38 4.02 1.13 0.23 3.38 4.02 0.28 0.20 0.63 1.36 6 8.68 0.22 5.68 7.12 8.69 0.19 5.68 7.12 0.43 0.17 4.46 2.92 0.65 0.20 4.30 5.64 1.06 0.19 4.30 5.64 0.35 0.17 0.81 1.28 1.18 0.22 3.93 4.20 0.64 0.18 3.93 4.20 0.28 0.18 0.53 0.88 9 3.31 0.18 2.19 3.14 3.26 0.20 2.19 3.14 0.67 0.18 0.97 2.90 1.04 0.20 2.19 2.86 0.96 0.20 2.19 2.86 0.23 0.14 0.64 1.88 1.02 0.17 0.92 2.31 0.90 0.17 0.92 2.31 0.24 0.18 0.41 1.18 25 3 10.65 0.17 2.96 6.52 3.39 0.18 2.07 4.00 0.29 0.15 0.48 1.73 7.09 0.18 2.96 5.46 1.83 0.19 2.96 3.67 0.30 0.14 0.48 0.95 3.46 0.19 2.19 4.98 1.30 0.16 2.96 3.50 0.34 0.16 0.41 0.59 6 32.12 0.20 2.78 4.47 9.18 0.19 0.78 3.53 0.44 0.18 0.52 1.31 26.53 0.18 2.59 3.06 3.64 0.22 0.61 2.47 0.32 0.15 0.26 0.79 12.77 0.17 0.60 2.62 0.90 0.17 0.50 2.00 0.41 0.17 0.26 0.65 9 8.58 0.18 0.72 5.52 1.90 0.17 0.97 3.49 0.42 0.18 0.24 0.82 6.32 0.19 0.72 4.75 1.24 0.16 0.97 3.06 0.51 0.20 0.24 0.44 1.60 0.19 1.12 3.83 0.88 0.19 1.12 2.53 0.59 0.17 0.50 0.23 100 3 51.23 0.22 2.21 1.12 1.67 0.21 0.27 1.36 0.82 0.18 0.09 0.25 22.70 0.21 2.21 1.16 1.22 0.19 0.34 1.05 0.81 0.18 0.05 0.17 18.75 0.16 2.21 1.17 0.84 0.22 0.34 0.92 0.75 0.18 0.05 0.13 6 48.25 0.18 0.76 2.56 31.87 0.17 0.62 2.05 62.14 0.15 0.42 0.70 24.73 0.18 0.71 2.48 27.08 0.18 0.62 1.81 42.18 0.19 0.28 0.55 20.26 0.17 0.60 2.10 22.09 0.20 0.75 1.48 7.79 0.20 0.17 0.50 9 2.16 0.19 0.37 1.48 3.34 0.18 0.29 0.77 1.84 0.17 0.18 0.41 1.80 0.17 0.34 1.25 2.87 0.20 0.28 0.69 2.22 0.19 0.26 0.22 1.67 0.18 0.34 1.14 2.77 0.19 0.20 0.63 3.09 0.16 0.08 0.13 200 5 3 146.24 0.23 1.61 3.81 140.12 0.20 1.61 3.81 7.71 0.23 1.30 1.61 151.22 0.21 1.61 3.30 135.09 0.24 1.61 3.30 4.60 0.21 1.30 1.58 83.44 0.22 1.10 3.06 89.20 0.21 1.10 3.06 4.21 0.22 1.30 1.55 6 88.70 0.19 1.08 2.50 89.69 0.19 1.08 2.50 5.14 0.17 0.72 0.83 96.44 0.19 1.08 1.91 91.66 0.18 1.08 1.91 2.93 0.18 0.91 0.58 39.26 0.18 0.94 1.68 32.92 0.18 0.94 1.68 0.70 0.17 0.58 0.44 9 468.37 0.15 3.73 9.18 484.89 0.14 3.73 9.18 29.46 0.16 1.74 4.99 304.04 0.16 3.69 6.24 305.90 0.16 3.69 6.24 2.71 0.16 1.74 3.54 110.03 0.15 3.38 4.92 107.34 0.14 3.38 4.92 0.91 0.17 1.20 2.28 25 3 5629.58 0.33 2.75 7.84 4765.42 0.24 2.24 5.33 4.86 0.24 0.81 1.40 5430.90 0.25 2.75 6.73 3394.56 0.24 2.75 5.05 5.32 0.28 0.81 1.22 6896.05 0.25 2.75 6.15 2546.43 0.21 2.75 4.91 5.66 0.34 0.81 1.13 6 2886.13 0.19 1.67 4.36 146.48 0.17 1.93 2.77 0.57 0.17 0.19 0.79 1651.91 0.21 1.67 4.20 15.06 0.22 1.93 2.29 0.71 0.21 0.19 0.81 93.66 0.19 1.46 3.64 19.36 0.19 1.93 2.02 0.55 0.18 0.12 0.40 9 1235.12 0.32 2.05 2.59 342.32 0.22 0.96 1.22 1.99 0.21 0.22 0.26 404.70 0.29 1.90 2.12 28.09 0.21 0.88 0.76 0.82 0.20 0.13 0.17 99.73 0.21 1.99 1.58 2.23 0.22 0.39 0.58 0.87 0.20 0.06 0.08 100 3 703.05 0.23 2.11 4.15 373.65 0.22 2.03 2.70 1.42 0.22 0.47 0.63 731.09 0.22 2.11 2.92 157.29 0.20 2.03 1.96 1.11 0.22 0.54 0.47 596.88 0.27 2.06 2.29 349.78 0.30 2.13 1.58 3.22 0.29 0.53 0.44 6 7222.95 0.22 0.60 3.43 1814.25 0.18 0.48 2.08 22.11 0.21 0.13 0.44 7217.64 0.14 0.47 2.57 916.42 0.21 0.48 1.75 7.04 0.24 0.28 0.27 7216.94 0.15 0.47 2.11 656.48 0.20 0.37 1.41 7.89 0.22 0.24 0.20 9 3321.23 0.34 0.07 0.28 16.40 0.20 0.02 0.18 2.33 0.17 0.08 0.08 198.14 0.19 0.07 0.32 14.84 0.21 0.02 0.13 2.31 0.21 0.08 0.05 47.16 0.23 0.08 0.33 9.77 0.21 0.01 0.12 2.63 0.20 0.06 0.04 Mathematics 2020,8, 2026 23 of 26 Table A5. All instances of second experiment. |I|=100, |J|=25, |K|=6,r=0.5, β=0.1. tMSP tMIP ∆avg ∆tail tMSP tMIP ∆avg ∆tail 31.15 0.23 1.53 3.20 2.15 0.16 2.24 2.09 1.92 0.21 1.66 6.17 20.09 0.16 1.80 6.14 8.75 0.24 0.52 3.07 7.18 0.16 2.13 1.93 28.06 0.23 5.08 2.86 1.02 0.16 3.03 3.61 1.36 0.30 1.00 1.80 3.58 0.24 1.81 6.12 3.67 0.20 2.27 2.50 3.64 0.19 1.19 3.07 2.00 0.20 2.51 2.03 128.69 0.23 3.27 2.98 192.11 0.16 2.61 8.23 0.89 0.18 1.45 0.93 0.94 0.20 0.43 2.23 1.62 0.23 1.85 3.56 0.80 0.18 1.64 2.55 4.19 0.22 2.10 1.97 16.40 0.19 2.23 2.45 2.16 0.19 0.16 1.46 1.21 0.18 2.82 1.50 1.46 0.24 2.48 2.00 1.79 0.20 0.72 2.77 0.69 0.20 1.79 2.54 21.78 0.21 4.50 4.61 20.73 0.20 2.26 3.50 1.35 0.19 0.69 0.86 1.86 0.24 1.77 2.63 31.11 0.19 0.98 3.21 14.92 0.17 1.99 8.57 8.44 0.19 1.82 3.81 0.78 0.20 0.85 1.92 1.75 0.21 0.88 0.92 10.48 0.23 2.50 2.29 1.94 0.21 2.18 2.65 1.63 0.24 2.08 2.29 0.98 0.20 0.87 3.27 10.78 0.18 0.34 1.80 27.72 0.22 2.03 5.20 38.80 0.20 1.96 4.69 14.72 0.15 3.34 0.99 19.74 0.24 0.65 2.20 0.67 0.24 0.81 2.69 1.37 0.30 2.82 2.92 3.54 0.20 2.64 2.75 6.28 0.19 2.02 2.08 6.37 0.21 2.79 6.35 22.27 0.34 1.91 3.13 1.86 0.23 0.93 2.09 1.69 0.20 2.21 2.42 1.54 0.20 2.00 3.45 27.77 0.19 0.76 3.28 40.16 0.17 2.06 3.44 2.00 0.21 2.57 1.93 7.23 0.21 3.17 3.17 2.61 0.18 2.14 3.33 5.77 0.17 2.84 1.98 40.93 0.18 1.53 4.61 2.10 0.19 1.39 3.00 18.84 0.16 0.89 4.37 404.70 0.19 1.50 2.82 11.26 0.16 3.98 4.76 24.26 0.18 4.81 3.42 14.41 0.18 1.82 5.87 0.76 0.20 1.28 3.88 12.14 0.16 2.75 2.75 0.64 0.20 0.87 1.39 12.58 0.17 1.42 3.46 0.97 0.23 1.77 2.19 0.84 0.18 0.41 2.15 0.53 0.18 1.95 2.04 5.20 0.19 3.80 2.02 7.24 0.22 2.21 1.68 28.16 0.15 4.68 3.56 0.87 0.25 0.71 1.42 39.10 0.16 3.47 3.59 8.51 0.20 2.48 4.06 19.22 0.16 3.78 3.13 13.06 0.20 4.44 2.80 0.56 0.20 0.63 3.22 59.78 0.20 5.67 4.91 0.68 0.17 1.92 2.44 67.50 0.19 2.96 3.02 0.70 0.17 1.86 1.40 3.80 0.17 0.79 1.20 15.20 0.17 0.78 2.66 3.25 0.20 2.24 1.57 0.88 0.17 2.31 2.13 5.23 0.16 1.14 4.91 0.59 0.15 1.78 1.68 0.71 0.17 0.89 3.01 1.08 0.20 2.21 3.14 4.19 0.17 3.09 2.32 1.14 0.17 0.76 2.48 3.53 0.18 1.37 6.33 1.58 0.19 1.45 3.14 19.99 0.14 3.48 5.05 13.48 0.18 1.71 5.41 References 1. Rommelfanger, H. The Advantages of Fuzzy Optimization Models in Practical Use. Fuzzy Optim. Decis. Mak. 2004,3, 295–309. [CrossRef] 2. Gutjahr, W.; Nolz, P. Multicriteria optimization in humanitarian aid. Eur. J. Oper. Res. 2016 ,252, 351–366. [CrossRef] Mathematics 2020,8, 2026 24 of 26 3. Ferrer, J.; Martín-Campo, F.; Ortuño, M.; Pedraza-Martínez, A.; Tirado, G.; Vitoriano, B. Multi-criteria optimization for last mile distribution of disaster relief aid: Test cases and applications. Eur. J. Oper. Res. 2018,269, 501–515. [CrossRef] 4. Sun, G.; Zhang, H.; Fang, J.; Li, G.; Li, Q. A new multi-objective discrete robust optimization algorithm for engineering design. Appl. Math. Model. 2018,53, 602–621. [CrossRef] 5. Karsu, Ö.; Morton, A. Inequity averse optimization in operational research. Eur. J. Oper. Res. 2015 , 245, 343–359. [CrossRef] 6. Angilella, S.; Mazzù, S. The financing of innovative SMEs: A multicriteria credit rating model. Eur. J. Oper. Res. 2015,244, 540–554. [CrossRef] 7. Fotakis, D. Multi-objective spatial forest planning using self-organization. Ecol. Inform. 2015 ,29, 1–5. [CrossRef] 8. Guido, R.; Conforti, D. A hybrid genetic approach for solving an integrated multi-objective operating room planning and scheduling problem. Comput. Oper. Res. 2017,87, 270–282. [CrossRef] 9. Eiselt, H.; Marianov, V. Location modeling for municipal solid waste facilities. Comput. Oper. Res. 2015 , 62, 305–315. [CrossRef] 10. Liberatore, F.; Camacho-Collados, M. A Comparison of Local Search Methods for the Multicriteria Police Districting Problem on Graph. Math. Prob. Eng. 2016,2016, 3690474. [CrossRef] 11. Bast, H.; Delling, D.; Goldberg, A.; Müller-Hannemann, M.; Pajor, T.; Sanders, P.; Wagner, D.; Werneck, R. Route planning in transportation networks. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2016; pp. 19–80. 12. Samà, M.; Meloni, C.; D’Ariano, A.; Corman, F. A multi-criteria decision support methodology for real-time train scheduling. J. Rail Transp. Plan. Manag. 2015,5, 146–162. [CrossRef] 13. Spina, L.; Scrivo, R.; Ventura, C.; Viglianisi, A. Urban renewal: Negotiation procedures and evaluation models. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Cham, Switzerland, 2015; pp. 88–103. 14. Carli, R.; Dotoli, M.; Pellegrino, R. A decision-making tool for energy efficiency optimization of street lighting. Comput. Oper. Res. 2018,96, 223–235. [CrossRef] 15. Ehrgott, M. Multicriteria Optimization, 2nd ed.; Springer: Berlin/Heidelberg, Germay, 2005. 16. Birge, J.R.; Louveaux, F. Introduction to Stochastic Programming; Springer: New York, NY, USA, 2011. 17. Ben-Tal, A.; Nemirovski, A. Robust solutions of uncertain linear programs. Oper. Res. Lett. 1999 ,25, 1–13. [CrossRef] 18. Chen, X.; Sim, M.; Sun, P. A Robust Optimization Perspective on Stochastic Programming. Oper. Res. 2007 , 55, 1058–1071. [CrossRef] 19. Klamroth, K.; Köbis, E.; Schöbel, A.; Tammer, C. A unified approach to uncertain optimization. Eur. J. Oper. Res. 2017,260, 403–420. [CrossRef] 20. Gabrel, V.; Murat, C.; Thiele, A. Recent advances in robust optimization: An overview. Eur. J. Oper. Res. 2014,235, 471–483. [CrossRef] 21. Yao, H.; Li, Z.; Lai, Y. Mean–CVaR portfolio selection: A nonparametric estimation framework. Comput. Oper. Res. 2013,40, 1014–1022. [CrossRef] 22. Mansini, R.; Ogryczak, W.; Speranza, M.G. Linear and Mixed Integer Programming for Portfolio Optimization; Springer International Publishing: Berlin/Heidelberg, Germany, 2015. 23. Liu, X.; Küçükyavuz, S.; Noyan, N. Robust multicriteria risk-averse stochastic programming models. Ann. Oper. Res. 2017,259, 259–294. [CrossRef] 24. Dixit, V.; Tiwari, M.K. Project portfolio selection and scheduling optimization based on risk measure: A conditional value at risk approach. Ann. Oper. Res. 2020, 285, 9–33. [CrossRef] 25. Fernández, E.; Hinojosa, Y.; Puerto, J.; da Gama, F.S. New algorithmic framework for conditional value at risk: Application to stochastic fixed-charge transportation. Eur. J. Oper. Res. 2019,277, 215–226. [CrossRef] 26. Rockafellar, R.; Uryasev, S. Conditional value-at-risk for general loss distributions. J. Bank. Financ. 2002 , 26, 1443–1471. [CrossRef] 27. Goicoechea, A. Deterministic Equivalents for Use in Multiobjective, Stochastic Programming. IFAC Proc. Vol. 1980,13, 31–40. [CrossRef] Mathematics 2020,8, 2026 25 of 26 28. Leclercq, J.P. Stochastic programming: An interactive multicriteria approach. Eur. J. Oper. Res. 1982 , 10, 33–41. [CrossRef] 29. Caballero, R.; Cerdá, E.; Muñoz, M.M.; Rey, L. Stochastic approach versus multiobjective approach for obtaining efficient solutions in stochastic multiobjective programming problems. Eur. J. Oper. Res. 2004 , 158, 633–648. [CrossRef] 30. Aouni, B.; Ben Abdelaziz, F.; Martel, J.M. Decision-maker’s preferences modeling in the stochastic goal programming. Eur. J. Oper. Res. 2005,162, 610–618. [CrossRef] 31. Ben Abdelaziz, F.; Masri, H. A compromise solution for the multiobjective stochastic linear programming under partial uncertainty. Eur. J. Oper. Res. 2010,202, 55–59. [CrossRef] 32. Ben Abdelaziz, F.; Aouni, B.; Fayedh, R.E. Multi-objective stochastic programming for portfolio selection. Eur. J. Oper. Res. 2007,177, 1811–1823. [CrossRef] 33. Muñoz, M.M.; Luque, M.; Ruiz, F. INTEREST: A reference-point-based interactive procedure for stochastic multiobjective programming problems. OR Spectr. 2010,32, 195–210. [CrossRef] 34. Ben Abdelaziz, F. Solution approaches for the multiobjective stochastic programming. Eur. J. Oper. Res. 2012 , 216, 1–16. [CrossRef] 35. Gutjahr, W.J.; Pichler, A. Stochastic multi-objective optimization: A survey on non-scalarizing methods. Ann. Oper. Res. 2013,236, 475–499. [CrossRef] 36. Engau, A.; Sigler, D. Pareto solutions in multicriteria optimization under uncertainty. Eur. J. Oper. Res. 2020 , 281, 357–368. [CrossRef] 37. Álvarez-Miranda, E.; Garcia-Gonzalo, J.; Ulloa-Fierro, F.; Weintraub, A.; Barreiro, S. A multicriteria optimization model for sustainable forest management under climate change uncertainty: An application in Portugal. Eur. J. Oper. Res. 2018,269, 79–98. 38. Díaz-García, J.A.; Bashiri, M. Multiple response optimisation: An approach from multiobjective stochastic programming. Appl. Math. Model. 2014,38, 2015–2027. [CrossRef] 39. Teghem, J.; Dufrane, D.; Thauvoye, M.; Kunsch, P. Strange: An interactive method for multi-objective linear programming under uncertainty. Eur. J. Oper. Res. 1986,26, 65–82. [CrossRef] 40. Bath, S.K.; Dhillon, J.S.; Kothari, D.P. Stochastic Multi-Objective Generation Dispatch. Electr. Power Compon. Syst. 2004,32, 1083–1103. [CrossRef] 41. Gazijahani, F.S.; Ravadanegh, S.N.; Salehi, J. Stochastic multi-objective model for optimal energy exchange optimization of networked microgrids with presence of renewable generation under risk-based strategies. ISA Trans. 2018,73, 100–111. [CrossRef] [PubMed] 42. Claro, J.; De Sousa, J. A multiobjective metaheuristic for a mean-risk multistage capacity investment problem. J. Heurist. 2010,16, 85–115. [CrossRef] 43. Manopiniwes, W.; Irohara, T. Stochastic optimisation model for integrated decisions on relief supply chains: preparedness for disaster response. Int. J. Prod. Res. 2016,55, 979–996. [CrossRef] 44. Bastian, N.; Griffin, P.; Spero, E.; Fulton, L. Multi-criteria logistics modeling for military humanitarian assistance and disaster relief aerial delivery operations. Optim. Lett. 2016,10, 921–953. [CrossRef] 45. ¸Sakar, C.T.; Köksalan, M. A stochastic programming approach to multicriteria portfolio optimization. J. Glob. Optim. 2012,57, 299–314. [CrossRef] 46. Salas-Molina, F.; Rodriguez-Aguilar, J.A.; Pla-Santamaria, D. A stochastic goal programming model to derive stable cash management policies. J. Glob. Optim. 2020,76, 333–346. [CrossRef] 47. Yager, R. On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans. Syst. Man Cybern. 1988,18, 183–190. [CrossRef] 48. Fernández, F.; Puerto, J. Análisis de sensibilidad de las soluciones del problema lineal múltiple ordenado. TOP 1992,7, 17–29. 49. Nickel, S.; Puerto, J. A unified approach to network location problems. Networks 1999 ,34, 283–290. [CrossRef] 50. Yager, R.R.; Alajlan, N. Some issues on the OWA aggregation with importance weighted arguments. Knowl. Based Syst. 2016,100, 89–96. [CrossRef] 51. Puerto, J.; Rodríguez-Chía, A.M.; Tamir, A. Revisiting k-sum optimization. Math. Program. 2017 ,165, 579–604. [CrossRef] 52. Kalcsics, J.; Nickel, S.; Puerto, J.; Tamir, A. Algorithmic results for ordered median problems. Oper. Res. Lett. 2002,30, 149–158. [CrossRef]