An evolutionary algorithm for optimizing the target ordering in Ensemble of Regressor Chains

An Evolutionary Algorithm for Optimizing the Target Ordering in Ensemble of Regressor Chains Jose M. Moyano University of Cordoba Cordoba, Spain Email: [email protected] Eva L. Gibaja University of Cordoba Cordoba, Spain Email: [email protected] Sebasti´ an Ventura University of Cordoba Cordoba, Spain Email: [email protected] Abstract—In this article we present an evolutionary algorithm for the optimization of sequences of targets for the multitarget regression algorithm Ensemble of Regressor Chains. This algorithm selects several random sequences or chains of targets where to predict each target, the values of previous targets in the chain are included as features, considering in this way the relationship among them. Under the assumption that a target may be better predicted if it is highly correlated with the targets which were included as feature, our proposal, called CCOERC, looks for chains where each target is highly correlated with previous targets in the chain. Several methods for the combination of predictions in the ensemble and for the selection of the chains which forms the ensemble are also proposed. CCOERC is compared to other state-of-the-art algorithms in multitarget regression, presenting statistically better performance than them. I. INTRODUCTION Multi-Target Regression (MTR), also known as multi-output regression, aims to predict multiple continuous output or target variables using a common set of input variables [1]. Several applications for multi-target regression have been studied, including ecological modeling [2], [3], economics [4] and realtime predictions [5], among others. The most important challenges of these real-world problems are the multivariate nature as well as the compound dependencies between the multiple feature and target variables. In this way, multi-target regression methods tend to get a better performance than single-target methods, in which each output is treated separately [6]. These methods considers not only the relationship between features but also the relationship between targets, producing simpler models with a better computational efficiency [2]. Let be Da dataset of Ninstances including a value for each variable X1, . . . , Xm, Y1, . . . , Yd. Each instance is characterized by an input vector of mfeature variables x(l)= (x(l) 1, . . . , x(l) m)and an output vector of dcontinuous target variables y(l)= (y(l) 1, . . . , y(l) d), with l∈ {1, . . . , N}. The goal is to learn a model from Dconsisting of finding a function hthat for each new instance with unknown output, given the feature vector x, assigns a vector ˆ yof dtarget values [1]. The Ensemble of Regressor Chains (ERC) [1] algorithm selects nrandom sequences or chains of targets. In each chain, for the prediction of each target, the values of previous targets in the chain are added as features to the dataset. Hence, it takes into account the relationship among targets. The selection of several random chains for ERC is a good way to deal with the problem of the chain ordering, which has reported good results [7]. However, the fact that a given target may be more related with some targets than with others could be a relevant information to consider in the selection of chains. Our hypothesis is that if each target is highly correlated with the target values that are included as features, the performance of the regressor might be improved. Thereby, our proposal is to develop an evolutionary algorithm to select chains for ERC considering the correlation between each target and previous targets in the chain. Our goal is twofold: optimize the chains and optimize the way that they form the ensemble, including the selection of the chains that will form the ensemble and the combination of predictions of all base regressors. The rest of the paper is organized as follows: Section II presents the related work in MTR, describing state-of-the-art algorithms, Section III presents the evolutionary algorithm, Section IV shows the experimental setup including methods, metrics and datasets to use, Section V shows the results and discussion for the proposed experiments and Section VI presents the conclusions and lines of future work. II. RELATED WORK In this section, we are going to describe some of the best known algorithms for multi-target regression. The baseline method in MTR is the Single-Target (ST) method [8]. ST decomposes the regression problem in dsingle-output regression problems, each one trained to predict the value of a single target variable. Thus, despite being a fast model, the target variables are predicted independently and possible relationship among them cannot be exploited. The Multi- Target Regressor Stacking (MTRS) method [8] includes a twostage training process. First, dsingle-target models are learned as in ST. However, instead of directly using these models for prediction, it includes an additional training stage where dnew meta-models are learned, one for each target, where input vectors are augmented with estimates of the values of the targets in previous models. The Random Linear Combinations (RLC) method [9] creates new output variables as random linear combinations of koriginal output variables. Then, a multi-target algorithm is applied to predict the new variables and finally, the original targets are recovered by inverting the random linear transformation. The Regressor Chain (RC) 978-1-5090-4601-0/17 2015 method [8] is based on the idea of chaining single-target models. The training of RC consists of selecting a random chain of the set of targets, building regression models for each target linked in such way that for each model, the dataset is augmented with the values of previous targets in the chain. As RC is sensitive to the selected chain order, Ensemble of Regressor Chains (ERC) was also proposed in [8]. To solve the chain ordering problem, ERC selects 10 random chains, being the final output for each target the mean among all base regressors. III. CCO-ERC ALGORITHM In this Section the Correlation-based Chain Optimization for Ensemble of Regressor Chains algorithm (CCO-ERC) is presented. First, the main features of the evolutionary algorithm as individuals, fitness function and genetic operators are presented. Then, the evolutionary algorithm is proposed, and finally, the methods for the combination of base regressors in the ensemble are described. A. Individuals The individuals of the population are integer vectors of size d, representing each one a chain of targets for RC. This chain represents the order in which targets are included as feature in following regressors. The example of the Figure 1 represents the chain of targets c={3,5,1,4,6,2}. For the first target of the chain, y3, a single-target regressor is built. Then, for each remaining target in the chain, the feature space is augmented with the predicted values of the previous targets in the chain. The initial population is generated with random chains, and they are optimized during the evolutionary process. xy3 xy5 y 3 y1 y2 y 3 y 5 x y 3 y 5 y 1 y 4 y 6 x 153 264 ... Fig. 1. Individual representation and RC regressor operation B. Fitness function We assume that a model to predict a target yiwill perform better if targets included as features in this model are highly correlated with yi. Thereby, the fitness is calculated based on the correlation coefficients of each pair of targets. The correlation coefficient ρbetween two variables Yand Zis defined as ρY,Z =σY Z σY·σZ[10], where σY Z is the covariance, and σYand σZare the standard deviation of the variables Y and Zrespectively. It is in the range [-1, 1], where ρY,Z = 1 indicates a perfect positive correlation between Yand Z, ρY,Z =−1indicates perfect negative correlation and ρY,Z = 0 indicates no lineal correlation between these two targets. The fitness function averages the absolute value of ρbetween each target of the chain from position 2to dwith previous targets in the chain. The first target is not taken into account because it has not other targets included as features. Therefore, chains where each target is better correlated with previous ones in the chain are better scored. Thus, fitness is defined as: fitness =Pd i=2 Pi−1 j=1 |ρyc[i],yc[j]| i−1 d−1(1) where yc[i]represents the target at position iin the chain c. The Figure 2 shows in an example how the fitness is calculated. xy 3 x x y 3 y 5 x y 3 y 5 y 3 y 5 y 1 y 4 y 6 y 1 y 2 153 264 ... ... f 1 = - f 2 = |ρ 5,3 | f 3 = (|ρ 1,3 | + |ρ 1,5 |) / 2 f 6 = (|ρ 2,3 | + |ρ 2,5 | + ... + |ρ 2,6 |) / 5 itness = (f 2 + f 3 + ... f 6 ) / 5 Fig. 2. Example of fitness calculation C. Genetic operators In this Subsection both crossover and mutation operators are described. Each individual is considered to form the set of parents by tournament selection. Then, each of these individuals are crossed or mutated based on crossover and mutation probabilities respectively. 1) Partially Mapped Crossover: The Partially Mapped Crossover (PMX) [11] has been selected as crossover operator. It is an operator designed for individuals representing integer permutations. Given two parents, a portion of string of one parent is exchanged with a portion of the other parent creating the mappings, while the remaining information is copied using the mappings. Considering two example parents p1and p2: p1={163425} p2={652413} the PMX operator creates the offspring in the following way. First, it selects two random points along the strings. Suppose that cut points are selected at points a1= 2 and a2= 5: p1={1 6 |342|5} p2={6 5 |241|3} 2016 The substrings between the cut points are called the mapping sections. In our example, these sections define the mappings 2↔3,4↔4and 1↔2. Then, the mapping section from p1is copied to p2, and vice versa, generating the offspring s1and s2: s1={− − | 241| −} s2={− − | 342| −} Then, each individual of the offspring is filled copying the elements of the parent. In case a number is already present in the offspring, it is replaced according to the mappings. For example, for the first bit in s1a1(first bit of p1) must be copied, but as it is present in the individual, following the mappings 1↔2and 2↔3, a 3must be placed in this position. As seen, mappings can be concatenated if necessary Hence, offspring remains as: s1={3 6 |241|5} s2={6 5 |342|1} Therefore, PMX operator preserve information of both parents in offspring, generating feasible individuals where no index are repeated in the individual. 2) Random Mutator: Given an individual, the random mutator swaps two bits of the chain, which allows to find new unexplored chains. For example, given an individual p1={1 6 3 4 2 5}and two random points a1= 1 and a2= 3, bits at these positions are swapped, generating the new individual s1={361425}. It will always generate feasible individuals. D. Evolutionary algorithm Algorithm 1 shows the main steps of the evolutionary algorithm. First, a random population of popSize individuals is generated. Then, for each generation, popSize individuals from pare selected by tournament selection and they are applied crossover and mutation operators. Once offspring s is generated, duplicated individuals are removed and the rest are evaluated. The best(set, n)and worst(set, n)methods return the nbest or worst individuals from set respectively, being nthe desired number of regressors in the ensemble. In order to have always included the best nindividuals in the current population, the best nindividuals from pare added to s(including only those who are not already in s) and removed from p. If size of sis greater than popSize, the surplus individuals from sare removed. If not, the best individuals from pwhich are not in sare added to suntil size of sis equal to popSize. Finally, new population is assigned to p. E. Combination of regressors At the end of the generations, the algorithm stores a set of popSize chains from which nmust be selected to form the ensemble. We propose two methods to obtain the chains: get the best nchains (nBest selection) or randomly select nchains with weights (randomWeight selection). The nBest method selects the nbest individuals based on its fitness, ensuring that Algorithm 1 Evolutionary algorithm steps 1: p←randomPopulation(popSize) 2: for Ggenerations do 3: s←select popSize parents from pby tournament 4: Apply crossover and mutation to individuals in s 5: removeDuplicated(s) 6: for each individual ind from sdo 7: evaluate(ind) 8: end for 9: s←s+{best(p, n)} 10: p←p− {best(p, n)} 11: if size(s)> popSize then 12: s←s− {worst(s, size(s)−popSize)} 13: else 14: s←s+{best(p, popSize −size(s))} 15: end if 16: p←s 17: end for the chains selected for the ensemble includes the ntop correlated chains based on the fitness function previously defined. On the other hand, based on the fact that the final population includes a set of good chains, randomWeight selection selects nrandom individuals without repetition, where each individual has a weight to be selected. The weight of each individual is its own fitness value, so, the better an individual is, the more likely it is to be selected. This method provides the ensemble with a greater diversity among base regressors, since the best chains usually do not differ in many positions. Once the chains have been selected, for each chain a RC regressor is built and the final prediction for each target is calculated as the average value of the nregressors for this target (average method). A novel second variant to obtain the final prediction is proposed in this paper. Given the vector of predictions vof the nregressors for a given target, first outliers of this vector are removed and then the average of remaining values is calculated. Outliers are defined as the values which are not in the range [¯ v−2·σv,¯ v+2·σv][12], being ¯ vand σvthe average value and standard deviation of vrespectively. This method, called noOutliers removes values which could distort the final prediction, so the performance might be improved using this method. IV. EXPERIMENTAL SETUP This Section describes our experimental setup. First, the methods used in the experimentation and their parameters are presented. Next, both the evaluation measures and the datasets used in the experimentation are presented. Finally, the experiments to be carried out are described. A. Methods and parameters The experimental evaluation includes all the methods described in Section II, namely, ERC, ST, RC, RLC and MTRS. All these methods are available in the extended mulan library for MTR [13]. The default parameters for each method have 2017 been used. ERC uses k= 10 RC models in the ensemble and uses no sampling method. The method to calculate the estimates for ERC was also studied in [7], concluding that it is better to use predicted values in the train set instead of true values of targets. On the other hand, the estimates for MTRS are not calculated over the train file but are calculated using an internal f-fold cross-validation, which allows obtaining out-of-sample estimates of the target variables for all the training examples [7]. The RLC method generates r= 100 new target variables by combining k= 2 original target variables (after bringing them to the [0,1] interval). All methods use regression trees (Weka’s REPTree [14]) as singletarget regressor, as recommended in [7]. Finally, our proposal uses 100 individuals, 1000 generations for small datasets (8 or less targets) and 10000 generations for bigger datasets, and the rest of parameters as the proposed for ERC, in order to perform a fair comparison. B. Evaluation measures Let Ntest be the size of the unseen test data; y(l)and ˆ y(l)the vectors of the actual and predicted outputs for x(l), respectively; and ¯ ythe vector of averages of the actual outputs. Following, the selected evaluation measures are defined, indicating with ↓that they are metrics to minimize: •The Mean Absolute Error (MAE) computes the mean error between the predicted and the actual target for each instance as 1 Ntest PNtest l=1 |y(l)−ˆy(l)|. The average Relative Mean Absolute Error (aRMAE) [15] of a model that has been induced from a train set Dtrain is estimated based on a test set Dtest according to the following equation: ↓aRMAE =1 d d X i=1 PNtest l=1 |y(l) i−ˆy(l) i| PNtest l=1 |y(l) i−¯y(l) i|!(2) Definitely, the RMAE for a target is equal to the RMAE for that target divided by the RMAE of predicting the average value of that target in the training set. The aRMAE is calculated averaging the RMAE values of each target. •The Root Mean Squared Error (RMSE) computes the squared root of the average sum of squares of the error for each instance as q1 Ntest PNtest l=1 (y(l)−ˆy(l))2. The average Relative Root Mean Squared Error (aRRMSE) [1] of a regression model is calculated as: ↓aRRMSE =1 d d X i=1 v u u tPNtest l=1 (y(l) i−ˆy(l) i)2 PNtest l=1 (y(l) i−¯y(l) i)2(3) The described metrics are computed as the average of the errors of each target, allowing to calculate the model performance across multiple targets, which may potentially have distinct ranges. Relative measures (as aRMAE and aRRMSE) automatically re-scale the error contributions of each target, and hence, there might be no need of using any normalization operator for each target [1]. C. Datasets For all the experiments, a wide set of 11 datasets for MTR have been selected from Mulan website 1. Only datasets with more than 3 targets have been included, since for 3 or less targets the chain optimization is trivial. Table I shows the datasets’ characteristics, including name, number of examples (N), number of input variables (m), number of target variables (d) and source for each dataset. TABLE I DATASETS Dataset N m d Source andro 49 30 6 [16] atp7d 296 411 6 [7] atp1d 337 411 6 [7] rf1 9125 64 8 [7] rf2 9125 576 8 [7] osales 639 413 12 [17] wq 1060 16 14 [3] oes97 334 263 16 [7] oes10 403 298 16 [7] scm20d 8966 61 16 [7] scm1d 9803 280 16 [7] D. Experiments In this Subsection we describe the experiments that have been carried out. First the configuration of CCO-ERC is adjusted, and then, it is compared to other state-of-the-art algorithms in MTR. 1) Adjust CCO-ERC: As a previous phase, the CCO-ERC algorithm has been executed with the two methods proposed in Subsection III-E to combine the predictions of the base regressors in a final ensemble prediction: average and noOutliers. Both methods have been compared, and the one with the best performance has been selected as default configuration for the following experiments. Additionally the two methods for the selection of regressors that form the ensemble: nBest and randomWeight, which were also described in Subsection III-E, have been executed. They have been compared, selecting the best as default configuration for the following experiments. 2) Comparing CCO-ERC to other state-of-the-art algorithms: Once our proposal has been adjusted, it has been compared to other state-of-the-art algorithms. First, it has been compared to ERC, in order to know if the optimization that CCO-ERC does makes it outperform ERC. Then it has been compared, in addition to ERC, to other well known MTR algorithms as RC, RLC, MTRS and ST. V. EXPERIMENTAL ANALYSIS A. Adjust CCO-ERC The results of the two methods to combine the predictions over all datasets and the two selected metrics are shown in Table II. The noOutliers method performs always better for aRMAE metric. On the other hand, for aRRMSE, noOutliers 2018 TABLE II RESULTS FOR METHODS OF COMBINATION OF PREDICTIONS aRMAE aRRMSE average noOutliers average noOutliers andro 0.584 0.566 0.588 0.574 atp7d 0.485 0.461 0.562 0.558 atp1d 0.310 0.301 0.367 0.368 rf1 0.052 0.043 0.096 0.079 rf2 0.067 0.050 0.110 0.088 osales 0.677 0.640 0.753 0.754 wq 0.868 0.834 0.926 0.929 oes97 0.572 0.561 0.544 0.543 oes10 0.426 0.419 0.447 0.454 scm20d 0.393 0.370 0.465 0.452 scm1d 0.264 0.257 0.338 0.336 has a better performance than average in seven datasets, while the second is the best in four cases. In this case, and in the following cases involving a comparison between two methods, the Wilcoxon signed-rank test is performed [18]. It is a non-parametric test which compares two related samples. The null hypothesis is that both algorithms perform equal, and if it is rejected, there are significant differences in performance between them. For each pair of values xiand yi,ziis calculated as: zi=yi−xi. Next, all absolute values of ziare ordered, and each one is assigned a ranking Ri. Then, R+and R−are calculated as R+=Pzi>0Riand R−=Pzi<0Ri, being R+the sum of rankings where the first algorithm is better and R−the sum of rankings where the second algorithm is better. If the adjusted p-value <0.05, there are significant differences at 95% confidence, and at 99% confidence if adjusted p-value <0.01. The Table III shows the results of the Wilcoxon test for this comparison. In both metrics, the sum of ranks for the noOutliers method is greater than the sum of ranks for the average method. Also, for aRMAE metric, significant differences exist between both methods at 99% confidence. As expected, removing the outliers before calculating the mean improves the performance over only averaging the prediction values of all base regressors, since it allows to remove bad predictions in some concrete regressor. Therefore, the noOutliers method is selected as default configuration instead of average. TABLE III WILCOXON TEST FOR METHODS OF COMBINATION OF PREDICTIONS aRMAE aRRMSE Raverage 0 16 RnoOutliers 66 50 adjusted p-value 0.0033 0.1307 Once the method for the combination of predictions in the ensemble has been set, the two methods for the selection of the chains which will form the ensemble have been also executed. The results of both variants over all datasets and the two selected metrics are shown in Table IV. As could be seen, for aRMAE, nBest method is better in 5 datasets, while randomWeight method is the best in other 3 and there are 3 1http://mulan.sourceforge.net/datasets-mtr.html ties. For aRRMSE, nBest method is better in 5 datasets, while randomWeight method is the best 4 and there are other 2 ties. TABLE IV RESULTS FOR METHODS OF SELECTION OF BASE REGRESSORS aRMAE aRRMSE nBest randomWeight nBest randomWeight andro 0.566 0.576 0.574 0.589 atp7d 0.461 0.465 0.558 0.560 atp1d 0.301 0.304 0.368 0.372 rf1 0.043 0.043 0.079 0.079 rf2 0.050 0.050 0.088 0.088 osales 0.640 0.639 0.754 0.752 wq 0.834 0.834 0.929 0.928 oes97 0.561 0.562 0.543 0.546 oes10 0.419 0.420 0.454 0.456 scm20d 0.370 0.367 0.452 0.449 scm1d 0.257 0.256 0.336 0.335 The Table V shows the results of the Wilcoxon test for the chains’ selection methods. In both metrics, the sum of ranks for the nBest method is greater than the sum of ranks for the randomWeight method. Although there are no significant differences between both methods, the nBest method shows a better performance, so it is used in further experiments. The fact that the randomWeight method offers greater diversity of base regressors does not yield a better performance than if the nmost correlated chains are selected. TABLE V WILCOXON TEST FOR METHODS OF SELECTION OF BASE REGRESSORS aRMAE aRRMSE RnBest 25.5 31.5 RrandomWeight 10.5 13.5 adjusted p-value ≥0.2 ≥0.2 B. Comparing CCO-ERC to other state-of-the-art algorithms The results for the comparison between CCO-ERC and ERC are shown in Table VI. For aRMAE, CCO-ERC is better than ERC in all the datasets, while for aRRMSE CCO-ERC is the best in 5 cases and ERC in 6. TABLE VI RESULTS OF THE COMPARISON BETWEEN CCO-ERC AND ERC aRMAE aRRMSE CCO-ERC ERC CCO-ERC ERC andro 0.566 0.605 0.574 0.614 atp7d 0.461 0.486 0.558 0.554 atp1d 0.301 0.322 0.368 0.379 rf1 0.043 0.053 0.079 0.100 rf2 0.050 0.058 0.088 0.113 osales 0.640 0.677 0.754 0.751 wq 0.834 0.871 0.929 0.927 oes97 0.561 0.582 0.543 0.565 oes10 0.419 0.428 0.454 0.452 scm20d 0.370 0.376 0.452 0.445 scm1d 0.257 0.260 0.336 0.334 The Table VII shows the results of the Wilcoxon test for the comparison between CCO-ERC and ERC. For aRMAE, the Wilcoxon test affirms that there are significant differences 2019 at 99% confidence. On the other hand, for aRRMSE, while ERC was better at more datasets, the sum of ranks where CCO-ERC is better is greater than for ERC, so although there are no significant differences for this metric, the performance of CCO-ERC is slightly better. Given these results, CCO-ERC performs significantly better than ERC, at least for one metric. Therefore, selecting chains taking into account the correlation between targets performs significantly better than selecting them randomly. TABLE VII WILCOXON TEST FOR THE COMPARISON BETWEEN CCO-ERC AND ERC aRMAE aRRMSE RCCO-ERC 66 45 RERC 0 21 adjusted p-value 0.0033 ≥0.2 Finally, our proposal is compared to other well known state- of-the-art algorithms in MTR. For this comparison, in addition to ERC have also been executed RC, RLC, MTRS and ST. The results for aRMAE and aRRMSE are shown in Table VIII and Table IX respectively. As seen, for aRMAE, CCO-ERC is the best algorithm for ten datasets, while RLC is the best only one. On the other hand, for aRRMSE the best results are more spread: CCO-ERC is the best in four datasets, ERC in six and RLC in the remaining one. In order to compare the performance of the algorithms, first a Friedman test is performed for each metric as recommended by Demˇ sar [19]. The Friedman test is a non-parametric statistical test to compare several algorithms over several samples. It compares the average ranking of the algorithms for each sample. For each metric and dataset, a ranking is assigned to each algorithm, where the best algorithm has a ranking of 1, the next with better performance has a ranking of 2, and so on. Then, for each metric the average ranking is calculated averaging the values of ranking of all datasets. The null hypothesis is that all algorithms perform equally, and the alternative hypothesis is that there are significant differences among algorithms. Then, for metrics where significant differences were found, a posthoc test has to be performed. The Holm’s step-down test [20] compares the control algorithm (algorithm with the best performance) with the rest. It compares each adjusted p-value piwith α/(k−i), being kthe number of datasets and α the significance level. It orders all the adjusted p-values as p1≤p2≤. . . ≤pk−1, and starts with the most significant adjusted p-value. If p1is below α/(k−1), the null hypothesis is rejected and it continues comparing p2with α/(k−2), and so on. If a certain null hypothesis is not rejected, all the remaining hypothesis are neither rejected. If each adjusted pvalue pi<0.05 the null hypothesis that the performance of the control algorithm is equal than the ith-algorithm is rejected at 95% confidence, and if pi<0.01, it is rejected at 99% confidence. The average rankings of each algorithm for both metrics are shown in Table X, where the algorithm with best ranking is marked in bold. Also, the results of the Friedman test are shown in Table XI. CCO-ERC is the control algorithm for aRMAE metric, while for aRRMSE, ERC is the best algorithm but followed very closely by CCO-ERC. For both metrics, the Friedman test shows significant differences among algorithms’ performance. Therefore, the Holm’s post-hoc test is performed, comparing all algorithms with the control algorithm for each metric. The results are shown in Table XII. Values in bold shows that there exists significant differences with the control algorithm (marked with “-”) at 99% confidence. TABLE X AVERAGE RANKINGS CCO ERC ERC RC RLC ST MTRS aRMAE 1.09 2.09 5.68 3.45 4 4.68 aRRMSE 1.64 1.56 5.95 3.18 4.23 4.45 TABLE XI FRIEDMAN TEST RESULTS Metric Friedman p-value Conclusion aRMAE 0.0000 Rejected null hypothesis aRRMSE 0.0000 Rejected null hypothesis TABLE XII HOLM’S TEST RESULTS CCO ERC ERC RC RLC ST MTRS aRMAE - 0.2100 0.0000 0.0061 0.0008 0.0001 aRRMSE 0.9093 - 0.0000 0.0805 0.0023 0.0011 According to the results, CCO-ERC and ERC are the control algorithms each one for one metric, showing significant differences with RC, MTRS and ST for both metrics and with RLC for aRMAE at 99% confidence. Therefore, CCO-ERC and ERC are significantly better than the other state-of-the-art algorithms. Also, since in the previous pairwise comparison CCO-ERC was significantly better than ERC, CCO-ERC outperforms the rest of the state-of-the-art algorithms, confirming the hypothesis that a target will be better predicted if it is highly correlated with targets included as feature. VI. CONCLUSION In this manuscript we have proposed an evolutionary algorithm for the optimization of chains for Ensemble of Regressor Chains based on the Correlation Coefficient called CCO-ERC. Several methods for the combination of predictions of base regressors and for the selection of chains to form the ensemble have been proposed and compared. Statistical tests shows that CCO-ERC outperforms ERC and other state-of-the-art algorithms in MTR. This proves that our hypothesis that a target will be better predicted if it is highly correlated with the targets included as feature is correct. As future work, we pretend to try other combination and selection methods for the ensemble, and explore other ways to measure the relationship among targets. 2020 TABLE VIII RESULTS FOR ARMAE METRIC CCO ERC ERC RC RLC ST MTRS andro 0.566 0.605 0.714 0.562 0.668 0.634 atp7d 0.461 0.486 0.623 0.526 0.598 0.583 atp1d 0.301 0.322 0.386 0.375 0.379 0.381 rf1 0.043 0.053 0.082 0.107 0.070 0.068 rf2 0.050 0.058 0.102 0.112 0.090 0.083 osales 0.640 0.677 0.924 0.801 0.847 0.791 wq 0.834 0.871 0.905 0.884 0.893 0.900 oes97 0.561 0.582 0.706 0.662 0.708 0.706 oes10 0.419 0.428 0.544 0.475 0.543 0.543 scm20d 0.370 0.376 0.537 0.407 0.460 0.437 scm1d 0.257 0.260 0.355 0.289 0.332 0.330 TABLE IX RESULTS FOR ARRMSE METRIC CCO ERC ERC RC RLC ST MTRS andro 0.574 0.614 0.746 0.572 0.710 0.693 atp7d 0.558 0.554 0.737 0.584 0.684 0.686 atp1d 0.368 0.379 0.465 0.440 0.460 0.460 rf1 0.079 0.100 0.165 0.164 0.144 0.136 rf2 0.088 0.113 0.187 0.162 0.155 0.151 osales 0.754 0.751 0.956 0.848 0.902 0.907 wq 0.929 0.927 0.971 0.932 0.961 0.971 oes97 0.543 0.565 0.743 0.655 0.742 0.736 oes10 0.454 0.452 0.617 0.508 0.615 0.615 scm20d 0.452 0.445 0.637 0.482 0.565 0.533 scm1d 0.336 0.334 0.442 0.362 0.420 0.414 ACKNOWLEDGMENT This research was supported by the Spanish Ministry of Economy and Competitiveness, Project TIN-2014-55252-P, and by FEDER funds. This research was also supported by the Spanish Ministry of Education under FPU Grant FPU15/02948. REFERENCES [1] H. Borchani, G. Varando, C. Bielza, and P. Larra˜ naga, “A survey on multi-output regression,” Wiley Int. Rev. Data Min. and Knowl. Disc., vol. 5, no. 5, pp. 216–233, 2015. [2] D. Kocev, S. Deroski, M. D. White, G. R. Newell, and P. Griffioen, “Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition,” Ecological Modelling, vol. 220, no. 8, pp. 1159 – 1168, 2009. [3] S. Dˇ zeroski, D. Demˇ sar, and J. Grbovi´ c, “Predicting chemical parameters of river water quality from bioindicator data,” Applied Intelligence, vol. 13, no. 1, pp. 7–17, 2000. [4] J. Ghosn and Y. Bengio, “Multi-task learning for stock selection,” in Advances in Neural Information Processing Systems, 1996, pp. 946– 952. [5] Z. Han, Y. Liu, J. Zhao, and W. Wang, “Real time prediction for converter gas tank levels based on multi-output least square support vector regressor,” Control Engineering Practice, vol. 20, no. 12, pp. 1400 – 1409, 2012. [6] A. J. Burnham, J. F. MacGregor, and R. Viveros, “Latent variable multivariate regression modeling,” Chemometrics and Intelligent Laboratory Systems, vol. 48, no. 2, pp. 167 – 180, 1999. [7] E. Spyromitros-Xioufis, G. Tsoumakas, W. Groves, and I. Vlahavas, “Multi-target regression via input space expansion: treating targets as inputs,” Machine Learning, vol. 104, no. 1, pp. 55–98, 2016. [8] E. Spyromitros-Xious, W. Groves, G. Tsoumakas, and I. Vlahavas, “Multi-label classification methods for multi-target regression,” pp. 1159–1168, 2012, arXiv preprint arXiv:1211.6581, Cornell University Library. [9] G. Tsoumakas, E. Spyromitros-Xioufis, A. Vrekou, and I. Vlahavas, Multi-target Regression via Random Linear Target Combinations, 2014, pp. 225–240. [10] J. Benesty, J. C. Y. Huang, and I. Cohen, Pearson Correlation Coefficient. Springer Berlin Heidelberg, 2009, pp. 1–4. [11] P. Larra˜ naga, C. Kuijpers, R. Murga, I. Inza, and S. Dizdarevic, “Genetic algorithms for the travelling salesman problem: A review of representations and operators,” Artificial Intelligence Review, vol. 13, no. 2, pp. 129–170, 1999. [12] J. Miller, “Short report: Reaction time analysis with outlier exclusion: Bias varies with sample size,” The Quarterly Journal of Experimental Psychology Section A, vol. 43, no. 4, pp. 907–912, 1991. [13] “Mulan extended,” https://github.com/lefman/mulan-extended, last access: 21-12-2016. [14] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: An update,” SIGKDD Explor. Newsl., vol. 11, no. 1, pp. 10–18, 2009. [15] A. Osojnik, P. Panov, and S. Dˇ zeroski, “Comparison of tree-based methods for multi-target regression on data streams,” in Revised Selected Papers of the 4th International Workshop on New Frontiers in Mining Complex Patterns - Volume 9607, New York, NY, USA, 2016, pp. 17–31. [16] E. Hatzikosa, G. Tsoumakas, G. Tzanis, N. Bassiliades, and I. Vlahavas, “An empirical study on sea water quality prediction,” Knowl-Based Syst, no. 6, pp. 471–478, 2008. [17] “Kaggle competition: Online product sales,” https://www.kaggle.com/c/online-sales. [18] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945. [19] J. Demˇ sar, “Statistical comparisons of classifiers over multiple data sets,” Journal of Machine Learning Research, no. 7, pp. 1–30, 2006. [20] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979. 2021