Mixed-Integer Programming Model for Ranking Universities: Letting Universities Choose the Weights

MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX Mixed-Integer Programming Model for Ranking Universities: Letting Universities Choose the Weights Jakub Kudela  Institute of Automation and Computer Science, Brno University of Technology, Czech Republic [email protected] Abstract Regardless of the shortcomings and criticisms of world university rankings, these metrics are still widely used by students and parents to select universities and by universities to attract talented students and researchers, as well as funding. This paper proposes a new mixed-integer programming model for ranking universities. The new approach alleviates one of the criticisms – the issue of the “arbitrariness” of the weights used for aggregation of the individual criteria (or indicators) utilized in the contemporary rankings. Instead, the proposed model uses intervals of different sizes for the weights and lets the universities themselves “choose” the weights to optimize their position in the rankings. A numerical evaluation of the proposed ranking, based on the indicator values and weights from the Times Higher Education World University Ranking, is presented. Keywords: ranking, university ranking, mixed integer programming, multiplecriteria decision-making. Received: 10 May 2021 Accepted: 15 June 2021 Published: 21 June 2021 1 Introduction The ranking of universities has in recent years been used as an important tool for universities to publicise their prestige and international positioning [37]. Since the conception of the Academic Ranking of World Universities (ARWU) [23], both the number of university rankings and the number of universities included in them have steadily increased. The growth of these rankings as well as the increase in universities interested in them is explained by the interest of different groups [16]. Universities themselves are eager to occupy dominant positions in these rankings, since this is a way of getting the attention of a greater number of potential students, increasing the revenues from student enrolment [14], and the rankings themselves have an irreducible reputation-making role [26]. The practice of ranking universities has become widely defined by national and international organisations as an important instrument of political and economic policy [3]. This is clearly reflected when ranking positions determine policies related to the restructuring of the higher education system, as in the French case, where different universities were merged to create a university with a high ranking [6]. Despite the shortcomings and the criticism of the university rankings, they are still widely used by students and their parents to select institutions, and by educational institutions to attract talented students and researcher, as well as funding [24]. According to [4], funding explains up to 51% of the variability of the positions attained by the universities in some rankings. Apart from the AWRU ranking (or, commonly known as Shanghai ranking), the other prominent rankings are the Times Higher Education (THE) World University Ranking , and the Quacquarelli Symonds (QS) ranking. All three rankings are based on weighing a set of indicators (or dimensions) that should adequately describe the performance of a university in different areas. These indicators typically consist of the quality of education, quality of faculty, number of citations, industry income, international outlook, etc. Recently, new rankings have focused on prominent dimensions not explicitly addressed in the abovementioned rankings, such as innovation (Scimago Institutions Rankings— SIR), web visibility [22], and impact (Webometrics Ranking), or sustainability (GreenMetric World University Ranking). The authors of [10] showed how the intelligent integration of existing data about universities may lead to an open-linked data platform which permits the construction of new indicators, which combine heterogeneous sources of data to generate indicators that address a variety of user requirements without the need to design indicators on a custom basis. Several researchers have outlined challenging concerns that should be considered before assessing the performance of universities. For instance, some rankings that include survey-based information may bias the results towards those universities that are wellknown compared with lesser-known ones [35]. A factor analysis of the ARWU, QS and THE rankings was performed in [34] and revealed that there were two factors in each of the ranking systems, which did not support the assumption that the indicators were mutually supporting and additive as conceptualised by the ranking providers. The leading global university rankings 41 https://doi.org/10.13164/mendel.2021.1.041 ISSN: 1803-3814 (Printed), 2571-3701 (Online) MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX were also explored in [28] to determine the similarities and differences in terms of their ranking criteria, main indicators, modeling choices, and the effects of these on the rankings. In [30], the authors used a sample of Scandinavian universities to show that the differences between the THE and ARWU rankings may be attributed to both small variations on what they believe are not important indicators, as well as substantial variations on what they believe are important indicators. They also provide a methodology that can be used in understanding universities’ different ranks in global university rankings. In [7], the authors studied governmentalities of globalizing higher education through a discussion of the competing logics and landscapes of reputation and ranking in two leading universities in South Korea. Their analysis draws attention to the ways in which university rankings have generated a new multi-scalar geography of institutional reputation, the mismatch between quality, reputation and ranking, and the new kinds of institutional behaviors that are emerging to respond to the proliferation of ranking systems. A comparative analysis o five world university rankings was carried out in [27], where it is argued that current rankings are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values. The authors of [17] argue that compared to other ranking tools like THE and QS, the ARWU ranking is in many ways “better” – it is simple and transparent, relying on public knowledge and third-party data, not on data provided by the universities themselves or solely on performance data provided by operators like the Institute for Scientific Information. The utilization of rankings and indicators within the universities can also be used as an instrument of new managerialism [25]. Even considering the above shortcomings, there is still a wide consensus that the main weakness of the aforementioned university rankings is linked to the determination of the weights used to measure both dimensions and indicators in the computation of university performance [14]. The combination of multiple indicators of university performance in a single aggregate measure is usually carried out in a rather arbitrary way, which prevents a clear interpretation of the aggregated measure [31,35]. Also, Using unstandardized indicators for the total scores used in ranking can lead to undesirable results [33]. In the last decade, there have been several approaches proposed to alleviate the aforementioned problems. In [8] the authors proposed a robust ranking of universities, where the aggregation of the university performances is be done by the Choquet integral preference model that is able to take into account the possible negative and positive interactions between the different criteria. A ranking methodology based on ranking hesitant fuzzy sets was developed in [2]. The authors of [29] proposed to measure the importance of a given variable within existing composite indicators via Karl Pearson’s “correlation ratio”. Hybrid multi-criteria decision making was utilized for ranking 12 private universities in Taiwan in [37]. The composite I-distance indicator methodology as an alternative to weighting was proposed in [12]. In [24], the authors used a contrast pattern mining algorithm to extract a set of patterns describing the top 100 universities in the QS World University Rankings and showed, how are these top universities separable from the rest. A robustness analysis, based on a multi-modelling approach, was performed in [32] to test the validity of the inference about the rankings produced in the ARWU, Shanghai, and THE rankings. The authors conclude that while university and country level statistical inferences are unsound, the inference on macro regions is more robust, and propose an alternative ranking. In [11], the authors proposed a conditional multidimensional approach based on a robust directional distance technique for ranking European universities in teaching and research activities. A goal programming model for the ranking of universities was developed in [14]. Data envelopment analysis model for jointly evaluating the relative teaching and research efficiencies of universities was presented in [18]. In this paper, we propose a new mixed-integer programming model for ranking universities. The model is unique in that it allows each university to “choose” the weights for the aggregation of the individual criteria to get the best possible ranking. It alleviates the issue of of the “arbibtrariness” of the particular values of weights used in different rankings and allows instead for intervals of different sizes for the weights from which the universities can “choose” to optimize their position in the ranking. The rest of the paper is organised in the following manner. Section 2introduces the mixed-integer programming model framework for computing the ranking of universities. Section 3briefly describes the dataset used for the numerical evaluation of the model. Section 4presents and discusses the empirical results. The main conclusions and implications of the paper are presented in Section 5. 2 Mathematical Model Mixed-integer programming (MIP) is one of the most ubiquitous modelling methods used in optimization [9, 20,36], with applications ranging from optimal social distancing [19] to optimal plan for the construction of waste processing plants [21]. Our MIP model for ranking universities is based on the following idea: If we let the universities themselves decide on the values of the weights for the individual indicators, how would they choose? Let us consider a situation, where there are Nuniversities to be ranked according to Dcriteria (or indicators). The values of these criteria are already known and denoted by a vector pi∈ RDfor each university i. In order to obtain the ranking, we need to determine the weights w∈ RD of the criteria to get a overall score. If we let a particular university kchoose these weights, it will naturally set them in such a way that its own ranking is as good 42 MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX as possible. This can be achieved by solving the following MIP problem, where wkdenotes that the weights are chosen by university k: minimize N− N X i=1 yi(1) subject to p0 kwk≤p0 iwk+Myi,∀i6=k, (2) p0 kwk≥p0 iwk−M(1 −yi),∀i6=k, (3) D X j=1 wk j= 1,(4) yk= 0,(5) lj≤wk j≤uj,∀j, (6) yi∈ {0,1},∀i. (7) The objective function (1) describes the position of the university kwithing the ranking (which should be minimized). The binary variable yidecodes, if the university iis ranked worse than university k(yi= 1) or not (yi= 0). This relationship is enforced by the socalled “Big-M” constraints (2)-(3). The value of the parameter Mshould be large enough so the conditions hold, but not too large, as it might bring numerical difficulties. Constraint (4) is a normalizing condition on the weights. Constraint (5) forces the ranking to start from position 1 (otherwise, the best ranking university would have position 0). And, finally, constraints (6) and (7) enforce that the weights for the indicators are within pre-specified bounds, and that the variables yi are binary. After solving the MIP problem (1)-(7) we get the best possible position of the university k(the value of the objective function), and, more importantly, the optimal weights wk, which also determine the ranking of all other universities. If we solve MIP problem (1)-(7) for all Nuniversities, we effectively get Nobservations of possible rankings from which we can easily extract meaningful statistical results. This approach alleviates the issue of the “arbitrariness” of the weights used in the different rankings – instead of a single value, the individual indicators can have a range of values, and the resulting ranking is left “on the universities themselves”. 3 Data This section presents the database used to illustrate the implementation of the aforementioned university ranking. Although there is a large variety of rankings currently in use, we chose to apply our model to the data provided by the THE ranking [1]. This ranking is among those with the largest historical data, the number of universities listed in this ranking is very large, and it has the important data readily available at their website. Table 1contains a list of the indicators and the weights used in the THE ranking in years 2018-2021. The values of the indicators are standartized: the procedure is based on the distribution of data within a particular indicator, where a cumulative probability function is calculated, using a version of Z-scoring [1]. This means that all values of the indicators fit in a range between 0 (worst performing universities in the indicator) and 100 (best performing universities). The values of the five indicators indicators and their weighted average (Overall) for the top 10 universities in 2021 can be found in Table 2. The values of the indicators for all considered universities in years 2018-2021 can be found in the supplementary file “DATA.xls”. Table 1: List of indicators used in the THE ranking. [1] Indicator Definition Weight International outlook international-to-domestic-student ratio 7.5% international-to-domestic-staff ratio international collaboration Industry income knowledge transfer 2.5% Teaching reputation survey 30% staff-to-student ratio doctorate-to-bachelor’s ratio doctorates awarded-to-academic staff ratio institutional income Research reputation survey 30% research income research productivity Citations research influence 30% 4 Empirical Results We use the indicator values from the THE ranking as a ground for the empirical evaluation of the proposed ranking. The range for the weights is also based on the THE ranking (Table 1) and denote them by wT. We use a parameter αto denote a possible deviation from the base values, which results in the equation (6) having the following form: (1 −α)wT j≤wj≤(1 + α)wT j,∀j. We compute the results for three different values of α= [0.3,0.2,0.05] and four years: in 2021 there were N= 1527 universities, in 2020 there were N= 1397 universities, in 2019 there were N= 1258 universities, and in 2018 there were N= 1103 universities considered in the THE ranking. Since the values of the indicators are within 0 and 100, their weighted average will also lie within these abounds. This means that we can set to the value of the “Big M” parameter to M= 100. The optimization model was programmed in the high-performance dynamic language JULIA [5] with the JuMP package for mathematical optimization [13]. The solution was computed by the GUROBI 8.0 solver [15]. The computations were carried out on an ordinary computer (3.2 GHz i5-4460 CPU, 16 GB RAM) and took around five minutes finish for one instance 43 J. Kudela MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX Table 2: Indicator and Overall values for the top 10 universities in 2021. [1] Rank Name Overall Teaching Research Citations Industry income International outlook 1University of Oxford 95.6 91.3 99.6 98.0 68.7 96.4 2 Stanford University 94.9 92.2 96.7 99.9 90.1 79.5 3 Harvard University 94.8 94.4 98.8 99.4 46.8 77.7 4 California Institute of Technology 94.5 92.5 96.9 97.0 92.7 83.6 5 Massachusetts Institute of Technology 94.4 90.7 94.4 99.7 90.4 90.0 6 University of Cambridge 94.0 90.3 99.2 95.6 52.1 95.7 7 University of California, Berkeley 92.2 85.8 97.2 99.1 84.3 72.3 8 Yale University 91.6 91.9 93.8 97.9 56.1 68.4 9 Princeton University 91.5 88.8 92.5 98.9 58.0 80.2 10 The University of Chicago 90.3 88.9 90.5 98.6 54.9 74.0 20 40 60 80 100 120 140 160 180 200 Original ranking position 50 100 150 200 250 300 New ranking position First and Third Quartile for = 0.3 deviation First and Third Quartile for = 0.2 deviation First and Third Quartile for = 0.05 deviation Second Quartile (Median) for = 0.05 deviation Figure 1: Results of the first 200 universities for different values of α, year 2021. Table 3: Indicator and Overall values for selected universities in 2021. [1] Rank Name Overall Teaching Research Citations Industry Income International Outlook 91 University of Bristol 63.0 40.3 48.6 95.6 39.8 88.2 92 University of Basel 62.9 44.0 41.4 91.7 99.7 97.2 92 University of Glasgow 62.9 40.9 48.1 94.3 39.9 92.8 94 Purdue University West Lafayette 62.5 57.1 65.5 62.0 69.9 71.5 94 Zhejiang University 62.5 65.9 65.6 52.3 100.0 65.1 96 Korea Advanced Institute of Science and Technology 62.4 64.4 68.1 57.9 100.0 36.6 97 National Taiwan University 62.3 57.1 66.7 66.9 69.5 44.8 170 University of Leicester 56.0 30.9 33.6 96.2 37.6 90.7 170 University of Notre Dame 56.0 52.1 45.5 71.5 38.1 57.3 170 Sant’Anna School of Advanced Studies – Pisa 56.0 45.8 39.6 79.9 85.9 57.2 174 University of Exeter 55.9 32.4 38.3 89.9 35.8 91.5 174 Lomonosov Moscow State University 55.9 80.0 67.6 12.9 97.7 70.7 176 Northeastern University 55.8 36.7 29.1 98.2 36.5 76.0 176 Ulsan National Institute of Science and Technology 55.8 34.9 40.7 90.8 85.3 49.1 178 University of Aberdeen 55.7 29.5 34.2 94.4 45.1 95.6 178 Newcastle University 55.7 31.4 38.2 90.6 40.1 88.0 178 Paris-Saclay University 55.7 37.3 48.7 80.4 34.5 65.1 44 MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX (solving (1)-(7) for one range on the weights, one year, Ntimes for all universities). As the relevant statistical information about the resulting ranking of a given university, we chose the first quartile, the second quartile (the median), and the third quartile of its position (out of the Npossible rankings that were “chosen” by the Nuniversities). The detailed results of the computations can be found in the supplementary file “RESULTS.xls”. In Fig. 1are shown the statistical information about the first 200 universities for the year 2021 and different values of α. We can see that, roughly speaking, the high ranking universities retain their high ranking and have a relatively small difference between the first and the third quartile of their position, even for the largest considered deviation. As the position of the university goes down, this interval between the quartiles widens. An interesting jump in the size of the interval can be seen around universities with original THE ranking position between 91 and 97. The values of the indicators for these universities are reported in Table 3. The thing the universities with large intervals have in common is a relatively lower value of the “Citations” indicator when compared with other similarly ranked universities (which is compensated by their relatively higher values in “Teaching” and “Research”). Similar, but a bit more dramatic effect can be seen between the positions 170 and 180, where the “ Lomonosov Moscow State University” stands out with a large difference between the first and third quartile, which is caused by its poor value of the “Citations” indicator. Although the size of the interval is higher for lower ranking universities, it does not grow to unreasonable values. The sizes of the intervals for different values of αand different years can be seen in Fig. 2. Naturally, the higher the allowed deviation α, the larger the size of the intervals. The THE ranking reports a “precise” rank for the first 200 universities and groups the rest into ranges of 201-250, 251-300, 301-350, 351-400, 401-500, 501-600, 601-800, 801-1000, and 1000+ (i.e., 1001-1527 for the year 2021). The ranking proposed obtained by the proposed method results in a smaller ranges, as can be seen in Table 4. The effect of the proposed ranking on the top 100 universities in years 2018-2021 can be seen in Table 5. 5 Conclusion Despite their problems and criticisms, rankings of the world universities are here to stay, and will go on to be used by students, parents, researchers, and funding agencies, for categorizing and selecting universities. In this paper, we proposed a mixed-integer programming model for ranking universities that aims to alleviate one of the criticisms faced by contemporary university rankings – the “arbitrariness” of the weights used for aggregate scores. We have shown, that by using intervals for the weights instead and letting the universities themselves choose the particular values of the weights from these intervals to optimize their position in the rankings, we can get a reasonable ranking. We have demonstrated the properties of the model on a numerical example that was based on the THE ranking in years the 2018-2021. The top-performing universities still occupy the best positions regardless of the approach followed by the mixed-integer programming model, confirming their leadership. For the other universities, the mixed-integer model provides a meaningful range on their ranking based on the difference between the third and first quartile of their position in all possible rankings (chosen by all the universities). These ranges are comparatively smaller than the ones reported in the THE ranking, even for large values of the allowed deviation α. Although the “arbitrariness” of the weights is alleviated, it is not completely removed, as the intervals for the possible values of the weights still have to be agreed upon. Acknowledgement: This work was supported by IGA Brno University of Technology project No. FSIS-20-6538. References [1] Times higher education university ranking, 2021. https://www.timeshighereducation.com/worlduniversity-rankings/ [Accessed 26th Apr 2021]. [2] Alcantud, J., de Andr´ es Calle, R., and Torrecillas, M. Hesitant fuzzy worth: An innovative ranking methodology for hesitant fuzzy subsets. Applied Soft Computing 38 (2016), 232– 243. [3] Amsler, S., and Bolsmann, C. University ranking as social exclusion. British Journal of Sociology of Education 33 (2012), 283–301. [4] Benito, M., Gil, P., and Romera, R. Funding, is it key for standing out in the university rankings? Scientometrics 121 (2019), 771–792. [5] Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. Julia: a fresh approach to numerical computing. SIAM Review 59 (2017), 65–98. [6] Billaut, J., Bouyssou, D., and Vincke, P. Should you believe in the shanghai ranking? Scientometrics 84 (2009), 237–263. [7] Collins, F., and Park, G. Ranking and the multiplication of reputation: reflections from the frontier of globalizing higher education. Higher Education 72 (2016), 115–129. [8] Corrente, S., Greco, S., and S lowi´ nski, R. Robust ranking of universities evaluated by hierarchical and interacting criteria. In Huber S., Geiger M., de Almeida A. (eds) Multiple Criteria Decision Making and Aiding. International Series in Operations Research & Management Science (2019), pp. 145–192. [9] da Silva, R. F., Hvattum, L. M., and Glover, F. Combining solutions of the optimum satisfiability problem using evolutionary tunneling. MENDEL Journal 26 (2020), 23–29. 45 J. Kudela MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX 100 200 300 400 500 600 700 800 900 1000 1100 Original ranking position 50 100 150 200 250 300 350 Size of the interval 2018 For = 0.3 deviation For = 0.2 deviation For = 0.05 deviation 200 400 600 800 1000 1200 Original ranking position 50 100 150 200 250 300 350 Size of the interval 2019 For = 0.3 deviation For = 0.2 deviation For = 0.05 deviation 200 400 600 800 1000 1200 Original ranking position 50 100 150 200 250 300 350 Size of the interval 2020 For = 0.3 deviation For = 0.2 deviation For = 0.05 deviation 200 400 600 800 1000 1200 1400 Original ranking position 50 100 150 200 250 300 350 Size of the interval 2021 For = 0.3 deviation For = 0.2 deviation For = 0.05 deviation Figure 2: Size of the interval between the first and the third quartile for different values of αand different years. Table 4: Interval sizes (difference between the first and third quartile) for different values of αand different years. median interval size maximum interval size α= 0.3α= 0.2α= 0.05 α= 0.3α= 0.2α= 0.05 2018 28 19 5 231 155 42 2019 34 22 6 267 183 49 2020 39 25 6 274 172 41 2021 44 29 7 325 216 51 [10] Daraio, C., and Bonaccorsi, A. Beyond university rankings? generating new indicators on universities by linking data in open platforms. Journal of the Association for Information Science and Technology 68 (2017), 508–529. [11] Daraio, C., Bonaccorsi, A., and Simar, L. Rankings and university performance: A conditional multidimensional approach. European Journal of Operational Research 244 (2015), 918–930. [12] Dobrota, M., Bulajic, M., Bornmann, L., and Jeremic, V. A new approach to the qs university ranking using the composite i-distance indicator: Uncertainty and sensitivity analyses. Journal of the Association for Information Science and Technology 67 (2016), 200–211. [13] Dunning, I., Huchette, J., and Lubin, M. Jump: a modeling language for mathematical optimization. SIAM Review 59 (2017), 295–302. [14] Garc´ ıa, F., Guijarro, F., and Oliver, J. A multicriteria goal programming model for ranking universities. Mathematics 9 (2021), 459. [15] GUROBI. Gurobi optimizer 8.0, 2021. http://www.gurobi.com/ [Accessed 26th Apr 2021]. [16] Johnes, J. University rankings: What do they really show? Scientometrics 115 (2018), 585–606. [17] Kauppi, N. The global ranking game: narrowing academic excellence through numerical objectification. Studies in Higher Education 43 (2018), 1750–1762. [18] Kuah, C., and Wong, K. Efficiency assessment of universities through data envelopment analysis. Procedia Computer Science 3 (2011), 499–506. [19] Kudela, J. Social distancing as p-dispersion problem. IEEE Access 8 (2020), 149402–149411. [20] Kudela, J., and Popela, P. Two-stage stochastic facility location problem: Ga with benders decomposition. MENDEL 21 (2015), 53–58. [21] Kudela, J., Smejkalova, V., Somplak, R., and Nevrly, V. Legislation-induced planning of waste processing infrastructure: A case study of the czech republic. Renewable and Sustainable Energy Reviews 132 (2020), 110058. [22] Lages, J., Patt, A., and Shepelyansky, D. Wikipedia ranking of world universities. The European Physical Journal B 89 (2016), 69. [23] Liu, N., and Cheng, Y. The academic ranking of world universities. Higher Education in Europe 30 (2005), 127–136. [24] Loyola-Gonz´ alez, O., Medina-P´ erez, M., Valdez, R., and Choo, K.-K. A contrast pattern-based scientometric study of the qs world university ranking. IEEE Access 8 (2020), 206088–206104. 46 MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX Table 5: The values of the first, the second (in bold), and the third quartile for the top 100 universities in years 2018-2021, allowed deviation α= 0.3. 2021 2020 2019 2018 2021 2020 2019 2018 University of Oxford 1:1:1 1:1:1 1:1:1 1:1:1 UoM 51:51:54 56:55:58 55:56:58 53:54:57 Stanford University 2:2:3 2:4:4 2:3:3 3:3:3 University of Sydney 49:51:55 56:60:61 58:59:64 60:61:65 Harvard University 2:3:4 5:7:7 5:6:6 6:6:6 UoSC 52:53:57 62:62:65 62:66:68 60:66:68 CalTech 4:4:5 2:2:3 3:5:5 2:3:5 Boston University 52:54:58 59:61:64 68:74:83 64:70:77 MIT 3:5:5 5:5:5 4:4:5 4:5:5 Kyoto University 47:54:71 52:65:76 54:65:88 58:74:99 University of Cambridge 5:6:6 2:3:4 2:2:3 2:2:4 CUoHK 53:56:60 54:57:59 51:53:55 58:58:61 UoCB 7:7:8 13:13:14 14:15:15 17:18:19 THKUoSaT 54:56:60 44:47:49 40:41:46 41:44:46 Yale University 7:8:9 8:8:8 8:8:8 10:12:12 UoNCaCH 52:56:61 54:54:59 55:57:59 53:56:62 Princeton University 8:9:9 6:6:6 7:7:7 7:7:7 ANL 54:59:61 48:50:53 47:49:52 46:48:50 The University of Chicago 10:10:10 9:9:10 9:10:10 8:9:9 SNU 51:60:69 54:64:71 56:63:77 63:74:91 Imperial College London 11:11:13 9:10:11 9:9:10 8:8:9 Brown University 57:61:64 53:53:58 51:53:54 47:50:52 Johns Hopkins University 11:12:12 11:12:12 11:12:12 13:13:14 TUoQ 59:62:63 65:66:68 67:69:74 63:65:66 University of Pennsylvania 11:13:13 10:11:12 11:13:13 10:10:11 WU&R 54:62:67 55:59:63 57:59:66 59:64:68 ETH Zurich 14:14:15 13:13:14 11:11:13 10:10:12 UoCD 58:64:67 55:55:59 61:59:62 53:54:55 UoCLA 15:15:17 15:17:17 17:17:17 14:15:16 Monash University 62:64:65 75:75:82 82:84:89 80:80:82 University College London 15:16:17 15:15:17 14:14:15 15:16:17 University of Amsterdam 60:66:66 61:62:65 59:62:65 56:59:64 Columbia University 15:17:18 15:16:17 15:16:16 13:14:15 UNSW Sydney 66:67:68 69:71:74 90:96:99 81:85:90 University of Toronto 18:18:20 18:18:19 20:21:22 22:22:23 UoCSB 62:68:72 53:57:61 47:52:56 49:53:59 Cornell University 19:19:21 19:19:19 18:19:19 18:19:19 McMaster University 65:69:73 67:72:83 70:77:90 76:78:85 Duke University 20:20:23 20:20:23 18:18:19 16:17:17 Fudan University 66:70:80 89:109:125 87:104:126 94:116:146 Tsinghua University 18:20:23 20:23:27 18:22:28 25:30:33 Leiden University 69:70:75 66:67:70 66:68:71 67:67:72 UoMAA 22:22:23 21:21:22 20:20:21 21:21:23 EUR 67:72:85 64:69:79 63:70:80 66:72:78 Peking University 19:23:28 20:24:29 25:31:34 25:27:33 University of Montreal 75:73:79 82:85:93 88:90:95 104:108:113 Northwestern University 21:24:25 21:22:24 23:25:26 20:20:22 University of Zurich 73:73:82 86:90:100 87:92:99 124:136:145 NUoS 22:25:25 24:25:28 23:23:27 20:22:24 CUB 71:75:91 73:80:95 83:90:99 117:126:136 New York University 24:26:26 26:29:28 25:27:28 27:27:29 Utrecht University 74:75:84 74:75:83 69:74:81 68:68:72 LSoEaPS 26:27:28 25:27:27 24:26:27 26:26:29 University of Warwick 75:77:84 74:77:85 76:79:87 88:90:96 Carnegie Mellon University 26:28:29 24:27:29 22:24:26 22:24:25 DUoT 68:78:95 60:67:81 55:58:64 58:63:73 University of Washington 28:29:29 24:26:26 26:28:29 26:25:28 University of T¨ubingen 76:78:83 90:91:101 87:89:92 92:94:99 University of Edinburgh 30:30:31 29:30:31 28:29:30 26:27:30 University of Groningen 74:80:91 69:73:84 74:79:89 78:82:87 University of Melbourne 31:31:32 32:32:34 31:32:33 31:32:32 HUB 70:80:93 67:73:95 63:67:76 60:62:69 LMU Munich 32:32:33 32:32:33 32:32:33 34:34:36 OSUMC 78:80:82 68:70:72 71:71:74 69:70:73 UoCSD 32:33:34 29:31:32 29:30:31 28:31:31 University of Freiburg 80:83:88 87:85:92 76:76:80 81:82:84 UoBC 35:34:36 34:34:37 36:37:39 33:34:37 University of Copenhagen 82:84:96 91:101:105 110:116:130 103:109:124 King’s College London 34:35:39 35:36:39 37:38:40 35:36:39 Emory University 78:85:102 72:80:96 75:84:97 84:98:111 Karolinska Institute 35:36:40 37:41:44 39:40:44 36:38:40 University of Minnesota 84:85:93 78:79:85 69:71:76 55:56:60 The University of Tokyo 29:36:48 30:36:49 35:43:55 38:45:56 ´ Ecole Polytechnique 85:87:102 86:93:109 99:108:128 107:115:124 GIoT 37:38:38 37:38:39 32:34:36 31:33:34 UoSaToC 79:87:102 71:80:96 79:92:102 115:132:142 University of Hong Kong 36:39:41 35:35:40 35:36:41 34:40:42 Sorbonne University 83:87:101 78:80:91 69:73:82 N/A McGill University 38:40:40 41:42:42 43:45:46 42:42:44 UoMCP 88:90:97 89:91:97 81:83:88 66:69:72 TUoM 40:41:42 41:43:44 43:45:45 41:41:43 University of Bristol 84:91:104 77:87:101 70:77:90 69:76:79 Heidelberg University 38:42:47 40:44:49 43:47:50 43:45:48 University of Basel 86:92:105 86:94:107 98:103:109 90:95:106 EPFdL 41:43:45 37:38:39 34:35:37 36:38:41 University of Glasgow 85:92:103 87:98:109 82:92:103 76:80:87 University of Texas at Austin 39:44:45 35:38:41 36:39:41 46:49:50 PUWL 81:94:127 74:88:117 61:64:73 53:60:67 KU Leuven 43:45:46 45:45:48 48:48:50 46:47:49 Zhejiang University 74:94:144 74:107:151 67:101:143 127:177:234 Paris Sciences et Lettres 41:46:48 42:45:50 39:42:45 70:72:75 KAIST 79:96:129 83:110:137 83:102:117 84:95:111 NTUS 45:47:51 45:49:49 48:51:53 49:52:55 NTU 87:97:120 99:120:146 138:171:206 166:198:246 UoIaUC 47:48:53 44:48:49 48:50:52 36:37:39 UoCI 89:98:110 83:96:109 83:96:102 90:99:109 UoWM 50:49:54 47:51:51 41:44:46 41:43:45 University of Helsinki 93:98:104 91:96:102 93:99:100 89:90:94 WUiSL 46:50:54 47:52:54 51:55:57 47:50:53 SJTU 79:100:150 111:157:218 133:188:247 142:188:252 California Institute of Technology (CalTech), Massachusetts Institute of Technology (MIT), University of California – Berkeley (UoCB), University of California – Los Angeles (UoCLA), University of Michigan-Ann Arbor (UoMAA), National University of Singapore (NUoS), London School of Economics and Political Science (LSoEaPS), University of California – San Diego (UoCSD), University of British Columbia (UoBC), Georgia Institute of Technology (GIoT), Technical University of Munich (TUoM), ´ Ecole Polytechnique F´ed´erale de Lausanne (EPFdL), Nanyang Technological University – Singapore (NTUS), University of Illinois at Urbana-Champaign (UoIaUC), University of Wisconsin-Madison (UoWM), Washington University in St Louis (WUiSL), University of Manchester (UoM), University of Southern California (UoSC), Chinese University of Hong Kong (CUoHK), The Hong Kong University of Science and Technology (THKUoSaT), University of North Carolina at Chapel Hill (UoNCaCH), Australian National University (ANL), Seoul National University (SNU), The University of Queensland (TUoQ), Wageningen University & Research (WU&R), University of California – Davis (UoCD), University of California – Santa Barbara (UoCSB), Erasmus University Rotterdam (EUR), Charit´e – Universit¨atsmedizin Berlin (CUB), Delft University of Technology (DUoT), Humboldt University of Berlin (HUB), Ohio State University – Main campus (OSUMC), University of Science and Technology of China (UoSaToC), University of Maryland – College Park (UoMCP), Purdue University West Lafayette (PUWL), Korea Advanced Institute of Science and Technology (KAIST), National Taiwan University (NTU), University of California – Irvine (UoCI), Shanghai Jiao Tong University (SJTU) 47 J. Kudela MENDEL — Soft Computing Journal, Volume 27, No.1, June 2021, Brno, Czech RepublicX [25] Lynch, K. Control by numbers: new managerialism and ranking in higher education. Critical Studies in Education 56 (2015), 190–207. [26] Marginson, S. University rankings and social science. European Journal of Education 49 (2014), 45–59. [27] Moed, H. A critical comparative analysis of five world university rankings. Scientometrics 110 (2017), 967–990. [28] Olcay, G., and Bulu, M. Is measuring the knowledge creation of universities possible?: A review of university rankings. Technological Forecasting and Social Change 123 (2017), 153–160. [29] Paruolo, P., Saisana, M., and Saltelli, A. Ratings and rankings: voodoo or science? Journal of the Royal Statistical Society Series A (Statistics in Society) 176 (2013), 609–634. [30] Piro, F., and Sivertsen, G. How can differences in international university rankings be explained? Scientometrics 109 (2016), 2263–2278. [31] Rahnamayan, S., Mahdavi, S., Deb, K., and Bidgoli, A. Ranking multi-metric scientific achievements using a concept of pareto optimality. Mathematics 8 (2020), 956. [32] Saisana, M., d’Hombres, B., and Saltelli, A. Rickety numbers: Volatility of university rankings and policy implications. Research Policy 40 (2011), 165–177. [33] Soh, K. Misleading university rankings: cause and cure for discrepancies between nominal and attained weights. Journal of Higher Education Policy and Management 35 (2013), 206–214. [34] Soh, K. What the overall doesn’t tell about world university rankings: examples from arwu, qswur, and thewur in 2013. Journal of Higher Education Policy and Management 37 (2015), 295–307. [35] Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R., van Eck, N., van Leeuwen, T., van Raan, A., Visser, M., and Wouters, P. The leiden ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology 63 (2012), 2419–2432. [36] Williams, H. Logic and Integer Programming. Springer, Verlag, 2009. [37] Wu, H.-Y., Chen, J.-K., Chen, I.-S., and Zhuo, H.-H. Ranking universities based on performance evaluation by a hybrid mcdm model. Measurement 45 (2012), 856–880. 48