Visualizations for Decision Support in Scenario-based Multiobjective Optimization

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Visualizations for Decision Support in Scenario-based Multiobjective Optimization © 2021 The Authors. Published by Elsevier Inc. Published version Shavazipour, Babooshka; López-Ibáñez, Manuel; Miettinen, Kaisa Shavazipour, B., López-Ibáñez, M., & Miettinen, K. (2021). Visualizations for Decision Support in Scenario-based Multiobjective Optimization. Information Sciences, 578, 1-21. https://doi.org/10.1016/j.ins.2021.07.025 2021 Visualizations for decision support in scenario-based multiobjective optimization Babooshka Shavazipour a, ⇑ , Manuel López-Ibáñez b , Kaisa Miettinen a a University of Jyvaskyla, Faculty of Information Technology, P.O. Box 35 (Agora), FI-40014 University of Jyvaskyla, Finland b Alliance Manchester Business School, University of Manchester, Manchester, UK article info Article history: Received 17 September 2020 Received in revised form 30 June 2021 Accepted 3 July 2021 Available online 6 July 2021 Keywords: Multi-dimensional visualization Scenario-based multi-criteria optimization MCDM Scenario planning Uncertainty Empirical attainment function abstract We address challenges of decision problems when managers need to optimize several conflicting objectives simultaneously under uncertainty. We propose visualization tools to support the solution of such scenario-based multiobjective optimization problems. Suitable graphical visualizations are necessary to support managers in understanding, evaluating, and comparing the performances of management decisions according to all objectives in all plausible scenarios. To date, no appropriate visualization has been suggested. This paper fills this gap by proposing two visualization methods: a novel extension of empirical attainment functions for scenarios and an adapted version of heatmaps. They help a decision-maker in gaining insight into realizations of trade-offs and comparisons between objective functions in different scenarios. Some fundamental questions that a decision-maker may wish to answer with the help of visualizations are also identified. Several examples are utilized to illustrate how the proposed visualizations support a decision-maker in evaluating and comparing solutions to be able to make a robust decision by answering the questions. Finally, we validate the usefulness of the proposed visualizations in a real-world problem with a real decision-maker. We conclude with guidelines regarding which of the proposed visualizations are best suited for different problem classes. Ó2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). 1. Introduction In many real-world optimization problems, decisionor policy-makers are faced with several (typically conflicting) criteria or objectives that should be optimized simultaneously. When satisfying all the conflicting objectives is not possible, decision support tools may help in finding a satisfactory balance between them. Because of the conflicting nature of the objectives, we do not usually have a single optimal solution but several so-called Pareto optimal ones with different trade-offs. Over the years, many multiobjective optimization methods have been designed to help and support the decision-maker (DM) to find their most preferred Pareto optimal solution. Besides multiple conflicting objectives, real-life problems are characterized by uncertainty. It is desirable to make robust decisions that are not too sensitive to the consequences of uncertainty, i.e., they perform relatively well in a wide range of future states or events [25], like in long-term strategic planning problems. The term scenario is used in the literature for various purposes, but usually, it refers to a framework to capture uncertainty in the absence of reliable probability distributions. https://doi.org/10.1016/j.ins.2021.07.025 0020-0255/Ó2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). ⇑ Corresponding author. E-mail address: [email protected] (B. Shavazipour). Information Sciences 578 (2021) 1–21 Contents lists available at ScienceDirect Information Sciences journal homepage: www.elsevier.com/locate/ins For example, a case study of the 2015 Nepal earthquake response considered seven scenariosbased on different uncertain parameters such as demands and various types of transportation [2]. Scenario-based multiobjective optimization approaches can be powerful tools in supporting decision-making processes in practice. When we have a multiobjective optimization problem to be solved in the presence of uncertainty, we seek to find solutions that are sufficiently good in all scenarios from the DM’s perspective. There are various terms like satisfactory or acceptable that one may use to describe such solutions. However, for consistency henceforth, we will utilize the term satisfactory, referring to this concept. Recently, authors from different disciplines have studied multiple scenarios in multiobjective optimization (although they may have utilized the term scenario in different ways) in real applications such as design problems [11], energy [48], portfolio optimization [13], maritime disruption management [8], berth allocation scheduling [52], location-allocation models for disaster response [2], bridge management [34], and water resources planning and management [12,50]. Scenarios provide a way to consider uncertainty and prepare oneself for different futures. Besides, even if scenarios represent ‘‘different loading or operating conditions” and not uncertainty, as described in [11], solutions must have satisfactory performances in all scenarios. In scenario-based multiobjective optimization problems, the number of scenarios is finite and, possibly, small. Scenariobased multiple criteria decision analysis problems often consider 4–6 scenarios [42] or some tens of scenarios at most [38]. However, the number of scenarios in some real-world problems may increase to a few hundred (e.g., [40]). In multiobjective optimization without scenarios, suitable graphical visualizations are often used to understand and compare different Pareto optimal solutions in the objective space, thus helping a DM to observe, learn and evaluate trade-offs between objectives and eventually choose one’s most preferred solution [29,31]. In scenario-based multiobjective optimization problems, the performance of a decision should be evaluated regarding each objective under conditions of different scenarios [40,42]. This way of scenario consideration brings an additional dimension to the performance evaluation and complicates the task of the DM. Therefore, our goal is to help a DM understand and compare trade-offs between different objective functions and evaluate and analyze trade-offs between the performances of a solution in various scenarios (we refer to it as trade-offs between scenarios hereinafter). As one of this paper’s contributions, we first enumerate possible questions that a DM may ask when facing a scenariobased multiobjective optimization problem. Existing visualization methods for multiobjective optimization fail to capture trade-offs between scenarios. Thus, we propose specific visualization methods for different cases of scenario-based multiobjective optimization, which are the main contribution of our study. In particular, we adapt ideas from the visualization of empirical attainment functions and heatmaps and propose three visualizations based on adapted empirical attainment functions. We also report encouraging feedback from a real DM on the usefulness of the proposed visualizations in a real-world production problem. Finally, we provide recommendations for applying different visualizations depending on the number of objectives, scenarios, and solutions considered. This paper is organized as follows. After a brief description of scenario-based multiobjective optimization problems and some visualization approaches in Section 2, we formulate in Section 3examples of questions that a DM may wish to be answered with the help of visualizations when dealing with scenario-based multiobjective optimization problems. In Section 4, we propose appropriate visualization methods for various needs followed by different examples in Section 5. In Section 6, we collect feedback from a real DM on the usefulness of the proposed visualizations in a real-life problem. Finally, after further discussion and suggesting guidelines for applications in Section 7, we conclude and mention future research directions in Section 8. 2. Background 2.1. General concepts in multiobjective optimization Before concentrating on scenario-based problems, let us consider the following deterministic multiobjective optimization problem (following, e.g., [30]): minimize ff 1 ðxÞ;...;f k ðxÞg subject to x2S;ð1Þ where kP2 is the number of objective functions f i :S!R(i¼1;...;k), x¼ðx 1 ;...;x n Þ T is a vector of ndecision variables in the feasible region Sin the decision space R n (S#R n ) defined by constraint functions, and z¼fðxÞ=ðf 1 ðxÞ;...;f k ðxÞÞ T is called an objective vector in the objective space R k . A decision vector x  2Sis called Pareto optimal if there does not exist another x2Ssuch that for all i,f i ðxÞ6f i ðx  Þand f j ðxÞ<f j ðx  Þfor at least one index j. Furthermore, for any two objective vectors z 0 ;z00 2 R k , we say that z 0 weakly dominates z00 if and only if for all i;z 0 i 6z i 00 and dominates if also z 0 j <z j 00 for at least one index j.Ifz 0 and z00 do not dominate each other, they are called mutually nondominated. The image of the set of Pareto optimal decision vectors in the objective space is sometimes called a Pareto front. In what follows, we refer to objective vectors as solutions, as our focus is on the objective space. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 2 2.2. Scenario-based multiobjective optimization As mentioned earlier, a scenario-based multiobjective optimization problem involves multiple conflicting objectives under uncertainty, in which scenarios (representing plausible future states) are utilized to deal with the existing uncertainties. In this paper, we assume that no reliable probabilities related to scenarios are available (called deep uncertainty) [40]. Let us assume that each scenario contains the same number of objective functions with the same meaning, and they are all to be minimized. We denote the number of plausible scenarios defining the scenario set X by s(and the number of objective functions by k). Given x2S#R n , its image in the objective space with a set of sscenarios is given by Y¼ðy 1 ;...;y s Þ, where y p ¼ðf 1p ðxÞ;...;f kp ðxÞÞ 2 R k (p2f1;...;sg) is the image of xunder the conditions of scenario pand f ip ;i¼1;...;k, are objective functions in scenario p. A decision vector x  is called Pareto optimal with respect to the scenario set X , if there does not exist any x2Ssuch that for all i;p;f ip ðxÞ6f ip ðx  Þ(i.e., weakly dominated in all scenarios) and f jq ðxÞ<f jq ðx  Þfor at least one objective jin at least one scenario q(i.e., dominated in one of them). In other words, by solving scenario-based multiobjective optimization problems, we mean finding a decision vector that is feasible in all scenarios, and there does not exist another feasible decision vector with better values in one objective function in one scenario without losing in another objective function in any scenario. Note that, even in theory, it is hardly possible to find a decision vector that is Pareto optimal in every scenario (i.e., the image of this decision vector lies on the Pareto front in all scenarios), except in a particular class of problems [21]. Therefore, here we assume it is sufficient that the decision vector is Pareto optimal for at least one scenario; i.e., the image of the decision vector lies on the Pareto front in the objective space of at least one scenario. Accordingly, in this paper, a solution to a scenario-based multiobjective optimization problem refers to a set of images of a Pareto optimal decision vector in the objective spaces of all scenarios. The task of a DM is to compare trade-offs between objectives in different scenarios and choose the most preferred one based on his/her preferences. However, as our focus in this paper is on developing visualization methods, we do not concern ourselves from where the solutions come from. Different types of preferences can be found in the multiobjective optimization literature. An example of that is a reference point consisting of aspiration levels that represent objective values that are desirable for a DM. With a best-case scenario(s), we refer to the scenario(s) in which the best possible values for at least one objective function can be reached. Conversely, the scenario(s) in which the worst possible values, for at least one objective, can be achieved is (are) called the worst-case scenario(s). 2.3. Multiobjective visualizations There are many ways to visualize high-dimensional data [26]. In the context of multiobjective optimization without scenarios, there are many visualization methods proposed in the literature [29,31]. For example, glyph plots 1 and parallel coordinate plots 2 have been used for visualizing trade-offs between objectives in problems with more than three objectives. In addition, dimensionality reduction techniques map the objectives into lower dimensions for visualization purposes (see, e.g. [6,14,35,44]). However, some relevant information, including dominance relationships between solutions, will be lost in such a mapping [10]. Other methods are used for specific purposes besides supporting a DM. For example, in multiobjective evolutionary algorithms (MOEAs), visualizations can be utilized to approximate the range, location and the shape of the set of Pareto optimal vectors in the objective space, or to monitor the progress and convergence of optimization [24]. Two critical reviews of visualization in MOEAs are given in [46,47]. In principle, the above visualizations could be used for scenario-based multiobjective optimization problems by plotting each scenario separately. [50] utilized both glyph and parallel coordinate plots to show that optimizing under varying conditions of uncertainty (different scenarios) yields different trade-offs. They performed five optimizations, one for each scenario, and generated a Pareto front for each scenario. Then, they displayed all the solutions in a glyph plot to compare the trade-offs between solutions in various scenarios. These trade-offs, together with decision variables, were also shown in a parallel coordinate plot. In addition, they compared the performances of all five solution sets under conditions of two scenarios separately. However, even for few scenarios, utilizing a series of separate plots in a scenario-based multiobjective optimization problem becomes challenging for a DM since there is much information to digest and combine. In environmental modeling (e.g., water resource planning and management), multidimensional visualizations like glyphs, parallel coordinate plots, and scatter plot matrices have been used for modeling and/or scenario analysis [12,22,23,50] in which the objective values of generated solutions, obtained from different models or scenarios, are evaluated and compared to find a suitable model and/or robust solutions. However, these visualizations do not illustrate trade-offs of a solution in different scenarios in a way that is easy to comprehend since they visualize individual solutions for different scenarios. Further uses of visualizations in multiobjective optimization in a scenario-based context (although not particularly suited for simultaneous comparison of many solutions) include scatter plot matrices for visualizing a few solutions of biobjective 1 In general, any symbol utilized in the visualization of data to describe a characteristic or an attribute is called a glyph [49]. Glyphs can simultaneously represent multiple attributes of multivariate data [31]. 2 Parallel coordinate plots can show many dimensions at once [15,51]. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 3 problems with a few scenarios in theoretical studies in robust multiobjective optimization [7,21]. However, for problems with more objectives and/or more solutions, such visualizations get messy and hard to read, and then, extracting information (like trade-offs) and getting insights are extremely difficult, if even possible. Separate scatter plots were also used in [11] for different scenarios to compare the results in bi-scenario, biobjective design problems. In addition, a scatter plot matrix (also called a pairwise scatter plot) was utilized in some environmental applications [12]. However, separate scatter plots and a scatter plot matrix are only useful for problems with few scenarios and few solutions. They are not suitable for other scenario-based multiobjective optimization problems. A real DM also brings this issue up in our real-world evaluation (discussed in Section 6). In summary, scatter plot matrices, parallel coordinate plots, and glyph plots are simple to use and understand. They may be applied to each scenario separately if the number of scenarios is small, and it becomes the task of a DM to compare different plots to understand trade-offs between scenarios. Other visualization methods reduce any additional dimensions to a 2D space via some mapping, requiring some training to understand and interpret. They may be applied to a scenario-based context by considering scenarios as additional dimensions to be reduced; however, scenarios and objectives are two different types of dimensions, and the interpretation of such visualizations becomes even more complex. Although various dimensionality reduction techniques have been proposed in the literature for multiobjective optimization problems, we did not find any scenario-based examples. Given the above summary, we believe that, in order to support a DM in comparing solutions according to trade-offs between objectives and scenarios, the existing visualizations need to be adapted and/or utilized in a different way, which is not straightforward. For example, in [26], uncertainty in high-dimensional data visualization is mentioned as an emerging area deserving future research. In the next sections, we describe two approaches (heatmaps and visualizations of an empirical attainment function). They will form the basis of our proposed visualizations for scenario-based multiobjective optimization. 2.3.1. Heatmap Heatmaps were used in [36] for simultaneously visualizing both objective and decision spaces in population-based multiobjective optimization methods. A heatmap is a 2D matrix, where rows are related to different solutions, and columns express corresponding objective function values. The magnitudes of the values of different objectives are represented by different colors/shades of the cells. One of the advantages of heatmaps, in addition to simplicity and scalability, is that they are a direct representation of the original data [46] instead of summarising it, except for limitations of the human eye in identifying shades/colors. Furthermore, trade-offs between the objectives can also be explained through the heatmap visualization [36,45]. Although heatmaps are more suitable for clustering solutions, they have also been utilized for comparing solutions [29]. Heatmaps have also been utilized to compare different algorithms in multiobjective optimization problems [1,18]. Later, in this paper, we extend heatmap visualizations and describe a new way of utilizing them in scenario-based problems. 2.3.2. Empirical attainment function Given several mutually nondominated sets of objective vectors in the traditional (i.e., not scenario-based) context of multiobjective optimization, the empirical attainment function (EAF) [20] calculates how many of these sets weakly dominate (attain) each point of the objective space. If these sets are produced by some stochastic process, such as independent runs of a stochastic multiobjective metaheuristic algorithm [27], the EAF, in this traditional context, represents the estimated probabilistic distribution of the output of such algorithm in the objective space [16,17]. With two objectives, the visualization of the EAF is rather straightforward and helps to understand the expected output of such algorithms [28]. Visualizing differences between the EAFs of two different algorithms helps to analyze which regions of the objective space are more easily attained by each algorithm [27]. With three objectives, computation of an EAF is still straightforward. However, visualization must be done in three dimensions (3D) or somehow projected into 2D [45], preferably in some interactive fashion. Visualizations of an EAF in higher dimensions are rarely discussed in the literature since they are much more challenging, both computationally and cognitively. In this paper, we adapt the concepts behind EAF visualization to the context of scenario-based multiobjective optimization. 3. Perspectives to decision making in scenario-based multiobjective optimization In this section, as the first contribution of this study, we raise possible questions that a DM may ask when facing a scenario-based multiobjective optimization problem. We do not claim that these questions cover all possible cases of relevance. However, these are aspects that either are found in the relevant literature, as we will show next, or have been faced in our practical experiences. As mentioned, in scenario-based multiobjective optimization, evaluating decisions in terms of all objectives in all scenarios means that a DM has to analyze a lot of information. In general, the objective function values corresponding to a decision must be satisfactory for a DM in all scenarios. However, in some particular cases, when no feasible solution with satisfactory performances in all scenarios exists, a DM may be satisfied with a solution that performs better in sufficiently many scenarios. In this case, achievable performances in a given number (or percentage) of scenarios or, in some particular or critical B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 4 scenarios, may be in the interest of the DM. Indeed, a DM’s expectations may change during the decision-making process when (s)he gains a better understanding of the problem, the feasibility of desires, and trade-offs between objectives in different scenarios. Visualizations can help a DM in various ways. We assume that we have generated Pareto optimal solutions to different scenarios. In what follows, we discuss examples of questions that a DM may wish to answer with the help of visualizations. Question 1. ‘‘How does a solution perform in different objectives under the conditions of all scenarios?” This question means that a DM should be able to compare trade-offs among objective function values in different scenarios, both for an individual solution and for several solutions. The overall aim of a DM is to understand what kind of trade-offs there are among objectives in different scenarios. Some authors have indirectly or explicitly mentioned a similar question in various practical applications (e.g., [34,38]). Even in the case of two objectives (f 1 and f 2 ) and two scenarios (s 1 and s 2 ), choosing the preferred solution is not apparent. For example, assume there are only two feasible solutions, if a solution z 1 has satisfactory values for f 1 in s 1 , but the value is unsatisfactory in s 2 , and another solution z 2 has a satisfactory value for f 1 in s 2 but its objective value in s 1 is unsatisfactory (this describes trade-offs between scenarios s 1 and s 2 ), which of these solutions is more desirable? In problems with more objective functions, scenarios, and solutions to compare, the task of a DM is even more challenging. However, visualizations can support a DM in choosing the most preferred solution. Question 2. ‘‘How many (or what percentage of) scenarios can reach the desired objective function values?” In this case, a DM must provide objective function values of interest called aspiration levels. Here, the aspiration levels do not depend on the scenarios as we are studying achievable performances in all considered scenarios. In this question, a DM faces a situation where it is impossible to satisfy all the preferences in all scenarios. Examples of this are real-life problems facing deep uncertainty, such as environmental planning problems. Because of the higher number of scenarios (typically hundreds or thousands), it is impossible to find an optimal solution for all scenarios, even for a single objective. Therefore, in these problems, a DM is looking for solutions that perform relatively well in most scenarios [5,12,22,23,25,50]. In this case, a DM must understand how achievable the desires are by analyzing the number of scenarios where the desires can be met. Depending on the findings, visualizations can support a DM in revising the aspiration levels or finding a solution that is acceptable in a sufficient number of scenarios. If it is not possible to have satisfactory values in all scenarios, one can set the percentage and also study the effects of different percentages. This concept is also called domain criterion and has been utilized as a robustness measure in the literature [3,37]. Alternatively, a DM may have some important scenarios where the objective functions should reach the desired objective function values. In such a case, paying particular attention to these scenarios should be prioritized. Question 3. ‘‘Which objective function values can be reached in all or selected scenarios?” A DM may want to see the best and worst objective function values that can be reached in all scenarios or scenarios that (s) he has selected (as described above). With this information about the ranges of the objectives, a DM can adjust one’s expectations on a realistic level, if needed. 4. Proposed visualization methods In this section, we propose different visualizations to be used in scenario-based multiobjective optimization. First, we adapt the concept of EAF to the scenario-based context and propose three new ways of using the adapted EAF in decision-making problems with two objectives. Then, we propose a novel way of utilizing heatmaps for problems with more than two (or three) objectives. 4.1. Scenario-based EAF We can adapt the definition of the first-order EAF [19,20, Def. 5.8] to scenario-based multiobjective optimization as follows. Given a decision vector xto a problem with kobjectives and sscenarios, where y p ¼ðf 1p ðxÞ;...;f kp ðxÞÞ, is the objective vector corresponding to evaluating xon scenario p, we can define a x ðzÞ¼X s p¼1 Ify p 6zg;ð2Þ where Ifg :R k !f0;1gis an indicator function. The scenario-based EAF (SB-EAF) can be understood in the following way: - SB-EAF indicates the number of scenarios in which a given decision vector xattains (dominate or equal) each point z2R k in the objective space. If the number of scenarios is very large, it may be useful to describe the SB-EAF in terms of percentiles, that is, the value 100 s  a x ðzÞgives the percentage of scenarios that attain z. There are some conceptual differences between the classical EAF and this scenario-based counterpart. The classical EAF is computed from several random nondominated sets of objective vectors, typically sampled from some stochastic process, where the number of sets is equivalent to the sample size and the cardinality of each set is itself a random number. By contrast, the SB-EAF for a given xis constructed from exactly sobjective vectors, and each of them corresponds to a different scenario. In a sense, the SB-EAF is a special case of the EAF where the number of samples is exactly s, and each sample B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 5 has cardinality 1. However, under deep uncertainty, there is no probability associated with each scenario. Thus the SB-EAF does not represent a probability distribution, unlike the classical EAF. The region of the objective space where the SB-EAF takes a particular value, that is, the region that may be attained by a given number of scenarios 0 <t6s, is completely characterized by a finite set of objective vectors given by: L t ¼minfz2R k : a x ðzÞPtg;ð3Þ where each L t can be understood as the best possible objective vectors that are attained by at least tscenarios. When expressed as a j%percentage of scenarios, j¼ 100 s t, then L j% is called the j%-attainment surface. For example, the median attainment surface (L d s 2 e or L 50% ) describes the best possible objective vectors attained by at least 50%of the scenarios. In other words, the j%-attainment surface is the best (in terms of Pareto optimality) set of objective vectors that may be attained under the conditions of j%of the scenarios. However, each of those objective vectors may by attained by different combinations of scenarios. From the above definition, the best attainment surface (L 1 or L 100 s % ) corresponds to the best possible objective vectors attained by at least one scenario, in other words, the nondominated set that is only reachable under the conditions of the best-case scenario(s). Each point in this set may be attained by a different best-case scenario. Conversely, the worst attainment surface (L s or L 100% ) describes the best possible objective vectors attained by all scenarios, i.e., the nondominated set that is reachable in the worst-case scenario(s). The worst-case may correspond to a different scenario for each solution in this front, yet we can guarantee to attain these objective function values under all conditions, i.e., for any scenario realization if zis the solution chosen by a DM. In the case of the SB-EAF, an attainment surface separates the objective space into two regions. On one side, we have the best possible objective vectors attainable by a number of scenarios and, on the other side, we have those objective vectors that are not attainable under the conditions of some scenarios. Therefore, a DM can simply observe the best possible objective vectors attainable under selected scenarios for a given decision vector. This would answer Question 3 identified in Section 3. Moreover, by simultaneously portraying the attainment surfaces of different solutions in one graph, a DM is able to compare different solutions and choose the most preferred one. If the best possible objective vectors cannot satisfy a DM’s preferences in all scenarios, (s) he can set some aspiration levels and observe in how many scenarios these values can be reached. A DM can also choose to track the attainments of different objective vectors in selected scenarios if desired. All of these actions can be performed in the presence of more than one solution for comparison purposes and choosing the most preferred one. Accordingly, SB-EAFs are also helping a DM in answering Question 2. Trade-offs between objective functions in different scenarios are also tractable in this visualization, both for an individual solution and several solutions. This can help a DM to learn about different options and limitations in various scenarios and support in making the final decision. Thus, Question 1 is also answered. When comparing two solutions with corresponding decision vectors x 1 and x 2 , it is also possible to visualize the differences between their SB-EAFs ( a x 1 ðzÞ a x 2 ðzÞ2½s;s), that is, in how many more scenarios a given point zin the objective space is attained by a solution corresponding to decision vector x 1 than by a solution corresponding to x 2 . Such a visualization (referred to as SB-EAF differences from now on) can help a DM focus on the actual differences between the two solutions. Finally, it is also possible to visualize the combination of the SB-EAFs of multiple solutions with corresponding decision vectors X¼fx 1 ;x 2 ;...gin one single plot by computing their combined SB-EAFs as follows: a X ðzÞ¼max x i 2X a x i ðzÞ ð4Þ that is, by considering the maximum value of their individual SB-EAFs at each point. We call this visualization all-in-one SBEAF. In this manner, a DM is able to visualize at a glance the feasibility of attaining a certain vector in the objective space within the desired number of scenarios given the solutions visualized. Moreover, objective vectors that correspond to some decision vector but do not belong to an attainment surface L t of X(having (3) with Xinstead of x) are dominated by some other solution in at least tscenarios. In total, we introduced three ways of applying the proposed SB-EAFs. Although they are powerful tools, unfortunately, practical visualization is limited to biobjective optimization problems. SB-EAF visualizations of a large number (e.g., hundreds) of solutions simultaneously can be nonintuitive for a DM due to the large amount of information. However, the SB-EAF can still be computed and, smaller subsets of solutions can be visualized according to filters set interactively by the DM. Nonetheless, SB-EAFs are useful, especially when many scenarios are to be considered. In Section 5, we illustrate the use of SB-EAFs in various examples. 4.2. Heatmaps for scenarios In this section, we propose an adapted version of the heatmap visualization for scenario-based problems. The main idea is to utilize a 2D (kp) matrix representing normalized values of the kobjective functions in pscenarios. In other words, rows are related to objective functions, columns describe scenarios, and each matrix refers to a solution (see Fig. 1 for an example). B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 6 It is also possible to utilize some indices describing the values of each objective function under conditions of each scenario. For example, one may consider distances between attainable values of a relevant objective function and an ideal point consisting of best individual objective function values under conditions of different scenarios. Then, different levels of colors/ shades in cells can display the variation. For example, for the distance of zero (the best-case) and one (the (normalized) worst-case) use black and white, respectively, while different levels of gray can be utilized for intermediate values. Moreover, using different colors with various shades may increase the readability of the heatmaps, especially when there are many scenarios, although, in this paper, we only use different levels of gray as an example. A DM is able to compare heatmaps for different solutions (as long as they are not too many) and evaluate them from both dimensions (i.e., objectives and scenarios): the darker the color, the better the solution. An ideal solution would be a full black matrix, while a white matrix would describe the worst possible values. Accordingly, when comparing solutions, darker solutions would be preferred. Different levels of shades/colors in each solution can help a DM to observe how this solution performs in different objectives in various scenarios and study trade-offs. Therefore, we are able to answer the DM’s Question 1. Moreover, it is also possible to display numerical objective function values in each cell of the matrices in heatmap visualizations. This allows a DM to perceive and compare the performances of different objective functions in various scenarios. Thus, one can simply identify the highest objective values achieved in all selected scenarios (Question 3) or scenarios in which the desires are achievable (Question 2). One of the most important advantages of using heatmaps in scenario-based problems is the applicability in problems with more than two objectives and quite a large number of scenarios. However, the number of objectives and scenarios cannot be arbitrarily large. Another advantage of applying heatmaps is that they enable a DM to compare the performance of a solution for different objectives in various plausible scenarios, which is important. Although comparing arbitrarily many solutions is not convenient with this visualization method, heatmaps (in integration with a small multiple design 3 ) can help a DM in comparing the objective values of quite many solutions in different scenarios and choose the preferred one. On the other hand, regarding the cognitive limits of human judgment [33], comparing a higher number of solutions may need a different way of utilizing the heatmaps or other visualizations. In that case, for example, solutions can first be clustered, with or without the use of heatmaps, and one can interactively ask a DM to compare some selected solutions until the DM is satisfied with the final choice. In all heatmaps shown in this paper, the range of gray levels is defined according to the maximum and minimum values shown for each objective (hence, there is always the worst value in white and the best value in black). However, it is possible to configure the gray levels relative to the given maximum or minimum values known for the problem or given by a DM and/ or focus on specific regions of the objective space. Also, if many solutions are available and a DM wishes to compare a small subset of them, we can first use all solutions to derive the gray levels and then only display the solutions of interest. In this way, the gray levels reflect the objective values relative to the best/worst values possible, although the heatmap only shows a subset of solutions. Similarly, in the SB-EAF plots, the minimum and maximum ranges can be configured if there is a priori knowledge available. 5. Examples of proposed visualization methods In this section, we demonstrate the use of the proposed visualization methods through various examples. The examples have been selected to illustrate the use of visualizations in different types of scenario-based problems. Moreover, relevant data is available in this section, so the examples are fully reproducible. We classify the examples according to the number of objective functions, scenarios, and solutions to be compared. We also discuss how the visualizations in different classes can address the questions presented in Section 3. Fig. 1. Visualizing a solution with k objectives and p scenarios by proposed heatmaps. 3 Representing multiple charts/graphs side by side or in a matrix-based form is known as small multiples in multi-dimensional data and information visualization [43]. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 7 Example 5.1 (Biobjective, 5 scenarios, 1 solution). Let us consider a problem (originally introduced in [40]) minimizing two objective functions f i (i¼1;2) in five scenarios s p (p¼1;...;5). Objective function values for a given solution in different scenarios are shown in Table 1. Fig. 2 visualizes this solution using the proposed SB-EAF. As shown in the figure, the performance of the solution in different scenarios is clearly observable to answer Question 1. One can also compare the achievable values in each objective function for any scenario as well as the best and the worst attainment surfaces to answer Question 3. In this figure, the area in dark purple ( ) bounded by s 3 corresponds to the worst attainment surface, and it is guaranteed to be attained by the solution in all five scenarios, whereas the area in yellow ( ) bounded by fs 1 ;s 5 gcorresponds to the best attainment surface attained in at least one scenario. Finally, if a DM determines a reference point of desired aspiration levels, it is easily possible to recognize how many scenarios can reach the desired values to answer Question 2. For example, the point ð15;11:78Þ, which corresponds to the intersection of s 4 and s 5 , is attained in 4 scenarios, which is not obvious by looking at Table 1. Example 5.2 (Biobjective, 3 scenarios, 5 solutions). We revisit the farming example proposed in [40], considering only the first two objectives to get a biobjective problem. The objective functions f 1 (costs) and f 2 (liquidity) are to be minimized and maximized, respectively. We have three scenarios. The objective function values for five solutions obtained with the twostage scenario-based framework (see [40]) are given in Table 2. A simple visualization of the solutions, as shown in Fig. 3, can help a DM to gain insight about the robustness of the solutions across scenarios. That is, the Pareto fronts in different scenarios are closer to each other on the left side (solutions 4 and 5) than on the right side (solutions 1–3) of the objective space. Moreover, solutions 1–3 show the same trade-off across scenarios and across objectives: (1) solutions in scenario s 3 dominate the corresponding solutions in s 2 which dominate the ones in s 1 ; and (2) moving from solution 1 to solution 3 implies, in every scenario, trading off, i.e., losses in f 2 for improvements in f 1 . Table 1 Objective function values of Example 5.1 in five scenarios. Scenario (s p ) p¼1p¼2p¼3p¼4p¼5 f 1p 7.80 5.87 20.80 15.00 8.50 f 2p 5.25 9.20 14.20 11.40 11.78 Fig. 2. SB-EAF visualization of Example 5.1. Points denote the objective vectors evaluated on different scenarios. Colored areas show regions of the objective space that can be attained by a particular number of scenarios. (see online version for color figures) Table 2 Five solutions in three scenarios for Example 5.2. Solution 1 Solution 2 Solution 3 Solution 4 Solution 5 Scenario pðf 1p ;f 2p Þðf 1p ;f 2p Þðf 1p ;f 2p Þðf 1p ;f 2p Þðf 1p ;f 2p Þ p¼1 (4.3, 9.98) (3.9, 9.04) (3.45, 7.89) (2.5, 5.44) (2.4, 5.14) p¼2 (4.5, 7.262) (4.08, 6.89) (3.607, 6.411) (2.6, 5.36) (2.5, 5.216) p¼3 (4.7, 6.65) (4.33, 6.4) (3.857, 6.09) (2.85, 5.4) (2.75, 5.297) B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 8 All-in-one SB-EAF (Fig. 14): The DM was impressed by this visualization. He liked the way that all the elements (i.e., objectives, scenarios, and solutions) have been ‘‘fitted together”. The DM mentioned that ‘‘assessing solutions one at a time is a natural choice but not necessarily enough”. This supports our claim about the DM’s challenges in comparing a series of separate plots in scenario-based multiobjective optimization problems. In summary, the DM found separate plots useful to compare details of particular trade-offs; nevertheless, the DM found all-in-one SB-EAF more beneficial to obtain an overall view. In addition, the DM gained particular insights into the problem that, as he pointed out, were not available earlier without the help of this visualization. SB-EAF differences: The DM reported that he found this visualization more different than what he is used to thinking and less easy to understand how it works. He got the idea but will need more time to get used to it. He felt that it could be useful for considering certain pairs that have been first selected. For the sake of conciseness, we do not show these plots here. General: Thereafter, when asked to give his overall impression of the visualizations, the DM stated that ‘‘I would select the all-in-one SB-EAF (Fig. 14)) to present the big picture and take the key elements and understanding of the system as a whole. Then I would use heatmaps as there are many aspects that are easiest to see in heatmaps”. The DM also mentioned that ‘‘people want to see the same thing in multiple ways” and having different visualizations can complement each other. Overall, he found various benefits in all the proposed visualizations, even in the SB-EAF differences. However, it is better first to find key details from other visualizations (all-in-one SB-EAF and heatmaps) and then use SB-EAF differences for specific pairs. To conclude, heatmaps and all-in-one SB-EAF can complement each other in biobjective scenario-based multiobjective optimization problems from two perspectives: (1) observing the same facts from different views and (2) getting various insights (big picture thinking vs. detail-oriented). Besides, developing a flexible and interactive visualization platform including different visualizations with the ability to switch between visualizations, rearranging the objectives, scenarios, and solutions and/or choosing some of them for a closer look can be helpful in practice. 7. Discussion and guidelines for applications We have proposed different visualizations to be applied in scenario-based multiobjective optimization problems. However, because of the complexity of these kinds of problems and various perspectives of a DM, one method may be more suitable than another one to be utilized in different cases or classes of problems. In this section, following the examples of the previous sections, we discuss different classes of problems in more general terms and discuss how our visualizations could have supported studies reported in the literature. To this end, we borrow some other examples from the literature where the Fig. 13. Heatmaps of the real-world evaluation: the darker, the better. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 15 proposed techniques could be applied. However, as not all the details are available to reproduce the visualizations, we can only discuss them on a general level. Table 6 compares the proposed visualizations with those found in the literature based on their capability to answer the three DM’s questions raised in this paper (Q1–Q3). The tick sign (U) shows that the corresponding visualizations, applied in the particular example considered in that paper, can completely answer the relevant question, while the cross sign ( ) represents the opposite. The sign () means that the assigned visualization can answer the question, but only in some limited problems or with some difficulties in large cases. According to this table, amongst the visualizations utilized in the literature for some scenario-based context, only scatter plots could answer all three questions for biobjective problems with very few scenarios and solutions. According to this table, amongst the visualizations utilized in the literature for some kind of scenario-based context, only scatter plots could answer all three questions for biobjective problems with very few scenarios and solutions. Therefore, as discussed earlier, its application is mostly limited to theoretical studies to describe the theory in an elementary example as a proof of concept, and not in a real-life problem. Furthermore, in biobjective problems, SB-EAFs provide extra valuable information, such as dominance relations between solutions in different scenarios and differences between solutions, in addition to answering all three questions. Let us note that the papers mentioned in Table 6 did not consider all these three questions in the same way we have formulated them. Some of them (e.g., [2]) did not look at trade-offs between scenarios, even though Fig. 14. All-in-one SB-EAF of the real-world evaluation. Points that represent the same solution evaluated in different scenarios are connected with a line. (see online version for color figures) B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 16 they did plot objective values in different scenarios in a single scatter plot. We will discuss it in more detail in the following subsections. Further discussion and some suggestions on where the proposed visualizations are best suited to be used as a guideline for real applications are accomplished in the following subsections. As mentioned earlier, we classified problems based on the number of objective functions, scenarios, and solutions. Accordingly, we discuss different cases separately in this section. 7.1. Biobjective problems As discussed earlier, the proposed SB-EAF is well-suited for biobjective optimization problems. It can handle many scenarios and solutions (although visualizing many solutions in a single plot may need further attention as discussed at the end of Section 4.1) while providing a clear overview of the best and worst possible values of each objective in any or all scenarios for those solutions. In many real applications, hundreds or thousands of scenarios are simulated, and many solutions are compared to find the best one. In this case, a visualization technique should handle quite many scenarios and/or solutions. The proposed SB-EAFs can be applied to perform analysis in a wide variety of applications such as bridge management [34], location-allocation models for disaster response [2], and berth allocation scheduling [52]. For example, biobjective models for optimizing bridge adaptation strategies under multiple future climate scenarios were proposed in [34]. Columbia and Mississippi Rivers in the United States were studied in five scenarios, and Pareto optimal solutions were presented by separate scatter graphs. Applying SB-EAFs visualization could enable comparing trade-offs between objectives in all scenarios and make the decision-making process easier and more accurate. Furthermore, a biobjective case study of the 2015 Nepal earthquake response was studied in [2]. The minimization of the total logistics costs and response time were considered. Separate sensitivity analyses were conducted considering seven scenarios for various uncertain parameters such as demands, number of available ground transportation, air and ground transportation costs, the weight of delivered relief packages, number of available air transportation, and recurring and human resource costs. The results were visualized in several separate scatter plots (using lines to connect solutions within each scenario). Utilizing SB-EAFs in this work could help to visualize the results of such a sensitivity analysis more appropriately. SBEAFs could also be helpful to study the large number of scenarios resulting from all the uncertain parameters. A berth allocation problem for container terminals was studied in [52], where a biobjective optimization model was proposed for cost minimization and maximizing the robustness of schedules. Although the method was not applied in a real problem, the problem formulation is generic enough to be applied to realistic environments. Instead of presenting the performances of schedules under different scenarios in a long table or separately displaying the average costs of schedules for each scenario, the proposed SB-EAFs visualization could be applied to help a DM compare trade-offs between objectives in all scenarios. The authors had to portray only 4 or 5 out of 9 solutions (also referred to analyzed schedules in the paper) in separate scenarios to be able to visualize them with different line graphs, while our proposed SB-EAFs could visualize all the results clearly in one figure. Naturally, the applicability of the proposed methods is not limited to the above-mentioned papers. Furthermore, as discussed earlier, all the DM’s questions of Section 3can be addressed by SB-EAFs in biobjective problems. Besides, as in the previous section, if the number of scenarios and solutions is lower than 10 (or a DM wants to concentrate on some specific scenarios and/or solutions), heatmaps can also be utilized to compare details and, thus, visualizations can complement each other. Table 6 Visualizations applied for scenario-based multiobjective optimization in the literature in comparison with the proposed ones. Visualization method Authors Num. of obj. Num. of scen. Num. of sol. Q1 Q2 Q3 Multiple bar [32] 331UU [52] 134UU 3D glyph [50] 5 5 many  Parallel coordinate plot [50] 6 5 many  [12] 4 4/5 many(6–16)  [12] 4 2 many(<50)  Scatter plot(s) [34] 2 5/6 1 UUU [2] 2 7 2–4 UU [52] 2 3/4 ⁄ 4/5 [7] 222UUU [21] 2 2 3/4 UUU [39] 233UUU Heatmaps for scenarios This paper 2–10 <10 <10 UUU This paper many(finite) many(finite) <10  SB-EAFs This paper 2 <10 <10 UUU This paper 2 many(finite) <10 UUU This paper 2 <10 many UUU This paper 2 many(finite) many UUU ⁄ Each scenario is shown as a separate scatter plot. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 17 7.2. Moderate number of objectives, scenarios and solutions As discussed in Section 4.2, the proposed heatmap visualization is suitable (i.e., it addresses the DM’s questions discussed in Section 3) for problems with more than two objectives if the number of solutions is not too high. In this case, comparing the performances in different objectives and diverse scenarios is possible. Even in the case of too many scenarios and/or solutions, comparing the performances in some selected scenarios and solutions is meaningful and applicable in real-life applications. For example, a water resources problem in [50] (in the Lower Rio Grande Valley of Texas, USA) had six objective functions and five scenarios, and 3D glyph and parallel coordinate plots were used to illustrate objectives and decision values. Nevertheless, as individual solutions for each scenario were visualized, trade-offs between scenarios were not traceable. The performances of six selected solutions were also compared in five scenarios. However, the comparisons were made with separate plots for only two out of five scenarios. Thus, again, studying trade-offs between scenarios was not supported, even for only five scenarios. In this case, the proposed heatmap visualization could help and support performance evaluations and trade-off comparisons in various scenarios, as well as the analysis in scenario effects and finding more adaptable and robust solutions. Moreover, the proposed visualizations can also be utilized to compare different solutions in scenario-based multiple criteria decision analysis problems and strategic planning. For example, the proposed heatmaps could easily be used to compare the performances of five strategic options in [38] involving food security in Trinidad and Tobago. Three criteria in 12 (reduced) plausible scenarios were considered while, originally, there were 108 possible scenario combinations. No visualizations were used to compare the performances of different alternatives (strategic options) in different scenarios, and only a plot of the regret values was used. Instead of multiple tables to present results, heatmaps proposed here would help a DM track and compare the overall performances of applying each strategic option (alternative) in different scenarios as well as the best and the worst-case scenarios. Heatmaps could also help in regret analysis and in finding the most robust option. 7.3. More than two objectives with many scenarios and/or solutions Because of the complexity of problems with several objectives and many scenarios and/or solutions, getting an overview is very challenging. Usually, we need to either reduce the dimensions and visualize results in lower dimensions or perform separate comparisons (e.g., pairwise comparisons). Decreasing the dimensions can be done by either reducing the number of scenarios or limiting the number of solutions considered. As mentioned in [38], reducing the number of scenarios regarding the DM’s preferences does not breach the philosophy of scenario planning. Similarly, the number of solutions can also be decreased to reflect the DM’s preferences. Therefore, if we can reduce the number of scenarios and/or solutions in problems with more than two objective functions, the proposed heatmaps will be applicable and can address the DM’s questions raised in Section 3for the reduced problem. Nevertheless, if the number of scenarios and/or solutions to be compared is considerably higher than, heatmap visualizations may not help much in comparisons and analysis. In this case, a 3D version of the SB-EAFs may be useful in getting an overview of the attainable values of objective functions in various scenarios, although not as intuitively as in biobjective problems. Table 7 summarizes our discussion and recommendations as a quick guideline in applying the proposed visualizations in different cases of problems. We have separated cases with less or more than 10 scenarios and solutions. The numbers in parentheses refer to the number of objective functions. 8. Conclusions In this paper, we have proposed visualizations to support managers in understanding, evaluating, and comparing the performances of management decisions in scenario-based multiobjective optimization problems. First, we identified some fundamental questions that a DM may want to answer by employing a suitable visualization. Then, we proposed the concept of scenario-based empirical attainment function (SB-EAF), which summarises the number of scenarios that may attain a parTable 7 Visualization appropriate for various cases (kis the number of objectives). Scenarios 1–10 solutions >10 solutions 1–10 Scatter plot ðk¼2Þ a SB-EAFs (k¼2) SB-EAFs (k¼2) Heatmaps ðkP2Þ b Heatmaps (kP2) >10 SB-EAFs (k¼2) SB-EAFs (k¼2) Heatmaps ðkP2Þ b Heatmaps ðkP2Þ c a Only for few scenarios and solutions. b Difficulties with very large set of solutions. c For selected solutions and not with very large set of scenarios and objectives. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 18 ticular point in the objective space. Then, we proposed three visualization methods based on SB-EAFs. In addition, we proposed an adapted version of heatmaps. We illustrated via various examples how they can support a DM in evaluating and comparing solutions, making a robust decision, and answering their questions. SB-EAFs can be applied in almost all biobjective scenario-based problems without any limitation in the number of scenarios or solutions from the computational point of view. However, in the case of many solutions, interactive filtering and visualizing a smaller subset of solutions is recommended to reduce the cognitive load. It also provides a clear overview of the best and worst possible values of each objective in any or all scenario(s). The main advantage of using SB-EAFs is the ability to treat quite a large number of scenarios and/or solutions. Such a need is often faced in real applications. This capability makes SB-EAFs applicable in a wide variety of applications. However, unfortunately, their applicability is limited to biobjective problems, and a different visualization is needed in problems with more than two objectives. Therefore, we proposed an adapted version of heatmaps, which is suitable for problems with more than two objectives if the number of objectives, scenarios, and solutions is not too high (say less than 10). Even in the case of too many scenarios and/or solutions, comparing the performances in some selected scenarios and solutions can still be meaningful and applicable in real-life applications. The proposed visualizations can help a DM both understand and compare trade-offs between different objective functions and evaluate and analyze trade-offs under conditions of various scenarios. This was also clearly confirmed in an evaluation involving a real DM. He also recommended an order of showing visualizations to support decision-making in his case. Furthermore, we classified scenario-based multiobjective optimization problems into three different classes: (1) biobjective problems; (2) problems with more than two objectives and a moderate number of scenarios and/or solutions; and (3) problems with more than two objectives and many scenarios and/or solutions. The proposed SB-EAF and heatmap visualizations are useful in the first two cases, respectively. Our suggestions can be used as a guideline in applying the proposed visualizations in real applications. The third case is open for future directions of research, but the proposed heatmaps can be utilized in some kinds of reduced problems, and there are proposals in the literature for visualization of multi-dimensional EAFs [45,46] that may be adapted to the SB-EAFs proposed here. Developing these and other novel visualizations for scenario-based multiobjective optimization problems, converting the ideas proposed into interactive visualizations that DMs can conveniently tailor and getting additional feedback from real DMs are our future research interests. Another important future direction is conducting a behavioral study. Reproducibility. Source code and data to reproduce the visualizations in this research can be found at https://doi.org/10. 5281/zenodo.5040421. CRediT authorship contribution statement Babooshka Shavazipour: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing. Manuel López-Ibáñez: Conceptualization, Methodology, Software, Writing - original draft, Writing - review & editing. Kaisa Miettinen: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements This research was partly funded by the Academy of Finland (Grants Nos. 287496 and 322221). This research is also related to the thematic research area Decision Analytics utilizing Causal Models and Multiobjective Optimization (DEMO, jyu.fi/ demo) of the University of Jyvaskyla. M.López-Ibáñez is a ‘‘Beatriz Galindo” Senior Distinguished Researcher (BEAGAL 18/00053) funded by the Spanish Ministry of Science and Innovation (MICINN). References [1] A. Ahrari, S. Elsayed, R. Sarker, D. Essam, C.A. Coello Coello, Weighted pointwise prediction method for dynamic multiobjective optimization, Information Sciences 546 (2021) 349–367. [2] H. Baharmand, T. Comes, M. Lauras, Bi-objective multi-layer location–allocation model for the immediate aftermath of sudden-onset disasters, Transportation Research Part E: Logistics and Transportation Review 127 (2019) 86–110. [3] E. Bartholomew, J.H. Kwakkel, On considering robustness in the search phase of robust decision making: A comparison of many-objective robust decision making, multi-scenario many-objective robust decision making, and many objective robust optimization, Environmental Modelling & Software 127 (2020) 104699. [5] L. Berger, J. Emmerling, M. Tavoni, Managing catastrophic climate risks under model uncertainty aversion, Management Science 63 (3) (2017) 749– 765. [6] X. Blasco, J.M. Herrero, J. Sanchis, M. Martínez, A new graphical visualization of n-dimensional Pareto front for decision-making in multiobjective optimization, Information Sciences 178 (20) (2008) 3908–3924. [7] M. Botte, A. Schöbel, Dominance for multi-objective robust optimization concepts, European Journal of Operational Research 273 (2) (2019) 430–440. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 19 [8] F. Cheraghchi, I. Abualhaol, R. Falcon, R. Abielmona, B. Raahemi, E. Petriu, Modeling the speed-based vessel schedule recovery problem using evolutionary multiobjective optimization, Information Sciences 448–449 (2018) 53–74. [10] A.R.R. de Freitas, P.J. Fleming, F.G. Guimarães, Aggregation trees for visualization and dimension reduction in many-objective optimization, Information Sciences 298 (2015) 288–314. [11] K. Deb, L. Zhu, S. Kulkarni, Handling multiple scenarios in evolutionary multi-objective numerical optimization, IEEE Transactions on Evolutionary Computation 22 (6) (2018) 920–933. [12] S. Eker, J.H. Kwakkel, Including robustness considerations in the search phase of many-objective robust decision making, Environmental Modelling & Software 105 (2018) 201–216. [13] A. Engau, M.M. Wiecek, Interactive coordination of objective decompositions in multiobjective programming, Management Science 54 (7) (2008) 1350–1363. [14] J.E. Fieldsend, R.M. Everson, Visualising high-dimensional Pareto relationships in two-dimensional scatterplots, in: R.C. Purshouse, P.J. Fleming, C.M. Fonseca, S. Greco, J. Shaw (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2013, Springer, Heidelberg, 2013, pp. 558–572. [15] P.J. Fleming, R.C. Purshouse, R.J. Lygoe, Many-objective optimization: An engineering design perspective, in: C.A. Coello Coello, A.H. Aguirre, E. Zitzler (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2005, Springer, Heidelberg, 2005, pp. 14–32. [16] C.M. Fonseca, V. Grunert da Fonseca, L. Paquete, Exploring the performance of stochastic multiobjective optimisers with the second-order attainment function, in: A.H. Aguirre, E. Zitzler (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2005, Springer, Heidelberg, 2005, pp. 250– 264. [17] C.M. Fonseca, A.P. Guerreiro, M. López-Ibáñez, L. Paquete, On the computation of the empirical attainment function, in: R.H.C. Takahashi et al. (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2011, Springer, Heidelberg, 2011, pp. 106–120. [18] H. Gao, H. Nie, K. Li, Visualisation of Pareto front approximation: A short survey and empirical comparisons, in: Proceedings of the 2019 Congresson Evolutionary Computation (CEC 2019), IEEE Press, Piscataway, NJ, 2019, pp. 1750–1757.. [19] V. Grunert da Fonseca, C.M. Fonseca, The attainment-function approach to stochastic multiobjective optimizer assessment and comparison, in: T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss (Eds.), Experimental Methods for the Analysis of Optimization Algorithms, Springer, Berlin, 2010, pp. 103–130. [20] V. Grunert da Fonseca, C.M. Fonseca, A.O. Hall, Inferential performance assessment of stochastic optimisers and the attainment function, in: E. Zitzler, K. Deb, L. Thiele, C.A.. Coello Coello, D. Corne (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2001, Springer, Heidelberg, 2001, pp. 213–225. [21] J. Ide, A. Schöbel, Robustness for uncertain multi-objective optimization: a survey and analysis of different concepts, OR Spectrum 38 (1) (2016) 235– 271. [22] J.R. Kasprzyk, P.M. Reed, G.W. Characklis, B.R. Kirsch, Many-objective de Novo water supply portfolio planning under deep uncertainty, Environmental Modelling & Software 34 (2012) 87–104. [23] J.R. Kasprzyk, S. Nataraj, P.M. Reed, R.J. Lempert, Many objective robust decision making for complex environmental systems undergoing change, Environmental Modelling & Software 42 (2013) 55–71. [24] J.D. Knowles, D. Corne, K. Deb, Introduction: Problem solving, EC and EMO, in: J.D. Knowles, D. Corne, K. Deb, D.R. Chair (Eds.), Multiobjective Problem Solving from Nature, Springer, 2008, pp. 1–28. [25] R.J. Lempert, D.G. Groves, S.W. Popper, S.C. Bankes, A general analytic method for generating robust strategies and narrative scenarios, Management Science 52 (4) (2006) 514–528. [26] S. Liu, D. Maljovec, B. Wang, P.-T. Bremer, V. Pascucci, Visualizing high-dimensional data: Advances in the past decade, IEEE Transactions on Visualization and Computer Graphics 23 (3) (2017). [27] M. López-Ibáñez, L. Paquete, T. Stützle, Hybrid population-based algorithms for the bi-objective quadratic assignment problem, Journal of Mathematical Modelling and Algorithms 5 (1) (2006) 111–137. [28] M. López-Ibáñez, L. Paquete, T. Stützle, Exploratory analysis of stochastic local search algorithms in biobjective optimization, in: T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss (Eds.), Experimental Methods for the Analysis of Optimization Algorithms, Springer, Berlin, 2010, pp. 209–222. [29] A.V. Lotov, K. Miettinen, Visualizing the Pareto frontier, in: J. Branke, K. Deb, K. Miettinen, R. Słowin ´ski (Eds.), Multiobjective Optimization: Interactive and Evolutionary Approaches, Springer, Heidelberg, 2008, pp. 213–243. [30] K. Miettinen, Nonlinear Multiobjective Optimization, Kluwer Academic Publishers, Boston, 1999. [31] K. Miettinen, Survey of methods to visualize alternatives in multiple criteria decision making problems, OR Spectrum 36 (1) (2014) 3–37. [32] K. Miettinen, J. Mustajoki, T.J. Stewart, Interactive multiobjective optimization with NIMBUS for decision making under uncertainty, OR Spectrum 36 (1) (2014) 39–56. [33] G.A. Miller, The magical number seven, plus or minus two: Some limits on our capacity for processing information, Psychological Review 63 (2) (1956) 81–97. [34] A. Mondoro, D.M. Frangopol, L. Liu, Multi-criteria robust optimization framework for bridge adaptation under climate change, Structural Safety 74 (2018) 14–23. [35] S. Obayashi, D. Sasaki, Visualization and data mining of Pareto solutions using self-organizing map, in: C.M. Fonseca, P.J. Fleming, E. Zitzler, K. Deb, L. Thiele (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2003, Springer, Heidelberg, 2003, pp. 796–809. [36] A. Pryke, S. Mostaghim, A. Nazemi, Heatmap visualization of population based multi objective algorithms, in: S. Obayashi et al. (Eds.), Evolutionary Multi-Criterion Optimization, Proceedings, EMO 2007, Springer, Heidelberg, 2007, pp. 361–375. [37] J.D. Quinn, P.M. Reed, K. Keller, Direct policy search for robust multi-objective management of deeply uncertain socio-ecological tipping points, Environmental Modelling & Software 92 (2017) 125–141. [38] C. Ram, G. Montibeller, A. Morton, Extending the use of scenario planning and MCDA for the evaluation of strategic options, Journal of the Operational Research Society 62 (5) (2011) 817–829. [39] M. Schmidt, A. Schöbel, L. Thom, Min-ordering and max-ordering scalarization methods for multi-objective robust optimization, European Journal of Operational Research 275 (2) (2019) 446–459. [40] B. Shavazipour, T.J. Stewart, Multi-objective optimisation under deep uncertainty, Operational Research (2019).. [41] B. Shavazipour, J. Stray, T.J. Stewart, Sustainable planning in sugar-bioethanol supply chain under deep uncertainty: A case study of South African sugarcane industry, Computers & Chemical Engineering 143 (2020) 107091. [42] T.J. Stewart, S. French, J. Rios, Integrating multicriteria decision analysis and scenario planning: Review and extension, Omega 41 (4) (2013) 679–688. [43] E.R. Tufte, The Visual Display of Quantitative Information, second ed., Graphics Press, Cheshire, CT, 2001. [44] T. Tušar, B. Filipic ˇ, Visualizing 4D approximation sets of multiobjective optimizers with prosections, in: N. Krasnogor, P.L. Lanzi (Eds.), Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2011, ACM Press, New York, NY, 2011, pp. 737–744. [45] T. Tušar, B. Filipic ˇ, Visualizing exact and approximated 3D empirical attainment functions, Mathematical Problems in Engineering 2014 (2014) 569346. [46] T. Tušar, B. Filipic ˇ, Visualization of Pareto front approximations in evolutionary multiobjective optimization: A critical review and the prosection method, IEEE Transactions on Evolutionary Computation 19 (2) (2015) 225–245. [47] D.J. Walker, R.M. Everson, J.E. Fieldsend, Visualizing mutually nondominating solution sets in many-objective optimization, IEEE Transactions on Evolutionary Computation 17 (2) (2012) 165–184. [48] R. Wang, J. Xiong, M.-F. He, L. Gao, L. Wang, Multi-objective optimal design of hybrid renewable energy system under multiple scenarios, Renewable Energy 151 (2020) 226–237. [49] M.O. Ward, Multivariate data glyphs: Principles and practice, in: C.-H. Chen, W.K. Härdle, A. Unwin (Eds.), Handbook of Data Visualization, Springer, 2008, pp. 179–198. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 20 [50] A.A. Watson, J.R. Kasprzyk, Incorporating deeply uncertain factors into the many objective search process, Environmental Modelling & Software 89 (2017) 159–171. [51] E.J. Wegman, Hyperdimensional data analysis using parallel coordinates, Journal of the American Statistical Association 85 (411) (1990) 664–675. [52] L. Zhen, D.-F. Chang, A bi-objective model for robust berth allocation scheduling, Computers and Industrial Engineering 63 (1) (2012) 262–273. B. Shavazipour, M. López-Ibáñez and K. Miettinen Information Sciences 578 (2021) 1–21 21