Toward an optimal and structured feature subset selection for multi-target regression using genetic algorithm

Received 8 October 2023, accepted 22 October 2023, date of publication 26 October 2023, date of current version 6 November 2023. Digital Object Identifier 10.1109/ACCESS.2023.3327870 Toward an Optimal and Structured Feature Subset Selection for Multi-Target Regression Using Genetic Algorithm FARRUKH HASAN SYED1, MUHAMMAD ATIF TAHIR 1, JAROSLAV FRNDA 2,3, (Senior Member, IEEE), MUHAMMAD RAFI 1, MUHAMMAD SHAHID ANWAR 4, AND JAN NEDOMA 3, (Senior Member, IEEE) 1School of Computing, Department of Computer Science, National University of Computer and Emerging Sciences, Islamabad 44000, Pakistan 2Department of Quantitative Methods and Economic Informatics, Faculty of Operation and Economics of Transport and Communications, University of Žilina, 01026 Žlina, Slovakia 3Department of Telecommunications, Faculty of Electrical Engineering and Computer Science, VSB—Technical University of Ostrava, 70800 Ostrava, Czech Republic 4Department of AI and Software, Gachon University, Seongnam-si 13120, South Korea Corresponding authors: Muhammad Rafi ([email protected]) and Muhammad Shahid Anwar ([email protected]) This work was supported in part by the Ministry of Education, Youth and Sports of the Czech Republic conducted by the VSB—Technical University of Ostrava, Czechia, under Grant 2023/39 and Grant 2023/42; and in part by the European Union within the REFRESH Project—Research Excellence For Region Sustainability and High-Tech Industries, of the European Just Transition Fund’’ under Grant ‘‘CZ.10.03.01/00/22003/0000048.’’ ABSTRACT Multi Target Regression (MTR) is a machine learning method that simultaneously predicts multiple real-valued outputs using a set of input variables. A lot of emerging applications that can be mapped to this class of problem. In MTR method one of the critical aspect is to handle structural information like instance and target correlation. MTR algorithms attempt to exploit these interdependences when building a model. This results in increased model complexities, which in turn, reduce the interpretability of the model through manual analysis of the result. However, data driven real-world applications often require models that can be used to analyze and improve real-world workflows. Leveraging dimensionality reduction techniques can reduce model complexity while retaining the performance and boost interpretability. This research proposes multiple feature subset alternatives for MTR using genetic algorithm, and provides a comparison of the different feature subset selection alternatives in conjunction with MTR algorithms. We proposed a genetic algorithm based feature subset selection with all targets and with individual target keeping the structural information intact in the selection process. Experiments are performed on real world benchmarked MTR data sets and the results indicate that a significant improvement in performance can be obtained with comparatively simple MTR models by utilizing optimal and structured feature selection. INDEX TERMS Multi-target regression, feature selection, genetic algorithm, single target, multiple objectives. I. INTRODUCTION Multi-target Regression (MTR) has been receiving increasing attention in the research community [1],[2],[3]. With the exponential increase in automation and data collection, data-driven decision-making is becoming a norm in almost The associate editor coordinating the review of this manuscript and approving it for publication was Wentao Fan . all areas of life. Increasing real-world situations are being identified where the objective is to make inference about several target variables at the same time. Regression tasks in general and multi-target regression tasks in particular have a vast number of applications ranging from economics, business and finance to health, engineering and space research, etc. [4],[5],[6]. Since MTR problems deal with multiple target variables, large systems with complex 121966 2023 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ VOLUME 11, 2023 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR FIGURE 1. Classification of solar flare intensities ([7]). interrelated components can be modeled using multi-target regression algorithms. For instance, Figure 1(a) shows solar flares of different intensities while Figure 1(b) shows their irradiance intensities. The intensities of the emissions are used to categorize the solar flares. A solar event can result in emissions of multiple intensities, and is therefore a classical multi-target problem. The already challenging problem of multi-target prediction and analysis is further aggravated by (i) complex interdependencies between the target variables, and (ii) issues arising from high dimensionality of the feature space, popularly referred to as the ‘‘Curse of Dimensionality’’. To the best of our knowledge, most of the research until now has focused on the target interdependency modeling problem, but very little work has been done on addressing the dimensionality issues to improve model performance. This research aims to investigate feature selection as a viable performance improvement approach to multi-target regression problems. This research is related to its semi-supervised counterpart proposed in [8]. The problem of curse of dimensionality is inherent in all machine learning tasks. It collectively refers to the problems which arise in modeling of high dimension data [9],[10]. As a result, models become complex [11]. This complexity leads to low predictive accuracy and interpretability, and an increased run time. This problem is compounded when the number of available examples for learning is few, which is usually the case in practical cases [12]. To solve these problems, a number of techniques exist, which generally do so by reducing the dimensions of the data. Feature selection techniques [13] help rule out features which are irrelevant or redundant. The original features are not transformed or modified in any way; thus, the understandability and the context is preserved. The issue with feature selection is to decide which features to keep and which ones to drop from the set of available features. Given the large state space usually presented by the features, an exhaustive search is almost always out of question. One method that has been used extensively for the purpose of feature selection is the use of Evolutionary Algorithms which are metaheuristic algorithms inspired by the evolutionary processes in nature [14],[15]. One commonly used evolutionary algorithm is the Genetic Algorithm (GA) [16]. GAs with their operators of crossover and mutation, allow a faster exploration of different feature combinations in an effective manner. They are most commonly used algorithms for optimization problems, given their flexible nature, ease of adaptation and variability of parameter tuning possibilities [17],[18]. The process of selecting features for multi-variable learning faces added complications since there are multiple dependent variables. For example, does there exist a common feature subset which can be used for all target variables? Or, is using a different subset of features for each target the way to go? If a common feature subset is used, decreasing error for one target may increase the error for some other target and vice versa. There are a huge number of real-world analytics and optimization problems that face this issue [19],[20]. In such a case, even if average error is used as an evaluation metric, the error in some targets might reduce while for some targets, the error may increase to an unacceptable level. Keeping the above questions in mind, this paper investigates the effect of feature selection on multi-target regression tasks using genetic algorithms. The objective of this research is as follows: OI: To show that by reducing features using feature selection, at par or better results can be obtained compared to existing MTR methods. OII: To alleviate the issues collectively referred to as the curse of dimensionality issues for high dimension MTR problems. OIII: To improve interpretability and performance for existing MTR models. For any multi-target task, there are two basic approaches available. 1) To approach the problem as independent modeling problems for each of the targets. 2) To use multi-target regression algorithms that can also model the interactions between the targets in order to explore intertarget dependencies. Based on the above mentioned approaches, we propose two variations as part of this research. For the first approach, we will be using the baseline approach called Single Target (ST) [21], while for the second approach, we will be using two methods multi-target regression algorithms, Stacked Single Target (SST) and Ensemble of Regressor Chains (ERC) [22]. Although a number of other multi-target regression algorithms exist, we have chosen these three algorithms due to the fact that the same base regressor can be used for all three algorithms. Thus only the effect of feature selection can be isolated and investigated. VOLUME 11, 2023 121967 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR Based on the above mentioned parameters, the following steps shown in figure 2to identify and propose relevant solution: FIGURE 2. Study process flowchart. The rest of the paper is organized as follows: Section II presents a discussion of the existing reasearch in the area of multi-target regression and feature selection. It also give a brief overview of using GA as a tool for feature selection. Section III describes the details of the proposed method, along with a description of the algorithms and techniques used in the experiments. It also lists the datasets used in the experiments. Section IV presents comprehensive results of the experiments and a discussion on the extent to which the objectives of the research have been achieved. Section V ends the paper with a discussion on the advantages and disadvantages of the proposed approach and some future directions for the research. II. RELATED WORK Multi-target regression [23],[24] has gained focus recently with many practical applications and problems being uncovered which require prediction of values for multiple target variables having continuous outputs. Some interesting examples include predicting customer prepaid top-ups [25], prediction of drug efficacy [26], prediction of flour quality [27], human brain activity prediction [28] where the activity of adjacent areas of the brain need to be mapped simultaneously for a better understanding of brain activity, ecosystem study such as in [29] where readings for various attributes of vegetation at a certain location need to be studied together, in epidemiological studies, by studying multiple variables of a disease or diseases, and their interdependence [30], and many more. Up to a certain extent, the techniques developed for Multilabel Classification (MLC) can be adapted for MTR problems as well [21], while some methods, for example, Multi-Objective Decision Tree [31], Multi-Objective Random Forest (MORF) [32], Ensembles of Multi-Objective Decision Trees [33] have been developed specifically for Multi-target Regression (MTR). Although a lot of work has been done on feature selection for traditional machine learning algorithms, not much work has been done to improve MTR algorithm performance using feature selection and most work has focused on model development. Table 1presents a summary demonstrating this concept. TABLE 1. MTR model development. As can be from table 1, most ground-breaking MTR papers discuss algorithm development and adapting MTC algorithms for MTR. For all machine learning tasks, the quality of features plays the most important role. Even the performance of the most powerful algorithms suffers if a suitable set of features is not used. In this regard, feature selection methods play a very important role. Feature selection can help in not only improving the results by reducing the problems associated with curse of dimensionality and overfitting, but also in decreasing the run time of the model [34]. Feature selection approaches are generally divided into three categories: Filter approach, Wrapper approach and Embedded approach. Evolutionary algorithms, and especially GA and its variations have found their way into a number of feature selection applications and have been used since long. They are used as wrapper methods, which means they use the underlying machine learning algorithm to evaluate subset performance. GA with its crossover and mutation features offers a good balance of exploitation and exploration in order to search through a large space of feature combinations that make up the search space. III. METHODOLOGY A. DATASETS For experiments and evaluation, experiments were performed using benchmark multi-target regression datasets.1 1http://mulan.sourceforge.net/datasets-mtr.html 121968 VOLUME 11, 2023 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR TABLE 2. Multi-target Regression Datasets. ddenotes the number of dimensions or features, while mdenotes the number of targets for each data set. The datasets are available for download publicly. A summary of the datasets is given in Table 2. B. EVALUATION MEASURE For multi-target regression, Relative Root Mean Squared Error (RRMSE) [22] is used as the evaluation measure. RRMSE is defined as: RRMSE =v u u tP(x,y)ϵDtest (ˆyj−yj)2 P(x,y)ϵDtest (¯ Yj−yj)2(1) In the above given equation (1),yjis the ground truth value, ˆyjis the predicted value and ¯ Yjis the mean value of the target variable Yjobtained from the training set. C. BENCHMARK MTR METHODS A brief description of the algorithms is given below: •Single Target: Single Target method is the baseline method for multi-target regression and ignores target interdependencies. It is inspired by its multilabel counterpart called Binary Relevance [42]. The idea is to decompose the ‘m’ target problem into ‘m’ independent single target problems and then applying any suitable traditional regression algorithm. •Stacked Single Target: In SST, the training phase has two stages. In the first stage, ‘m’ independent models are trained, one for each target (where ‘m’ is the number of target variables). In the second stage, the ‘m’ models are used to predict the values of the ‘m’ targets. Then, the input features are augmented using these predictions. Now, a second set of ‘m’ models, i.e. one for each target, is learned using the augmented feature space. At test time, the models trained in the first phase are first used to output ‘m’ predictions, and based on these predictions, the feature space is augmented and input to the second phase from which the final output is obtained. •Ensemble of Regressor Chains: ERC is based on the Regressor Chains [23] concept and consists of an ensemble of Regressor Chains (RC). A chain of target variables in constructed by ordering the variables randomly. A separate regression model is developed for each target by using a chain of targets to augment the feature space. Since using only one chain can cause bias due to chaining order, in ERC an ensemble of such chains is used. Final prediction is obtained by averaging the predictions from each chain. Since at prediction time, the actual values for the targets are not known, the predicted values are obtained similar to SST. D. WRAPPER BASED FEATURE SELECTION APPROACH For any feature selection technique, the general process is relatively the same as shown in Figure 3: FIGURE 3. The process of feature selection. Since the objective of this research was to investigate feature selection in conjunction with MTR algorithms, therefore, a wrapper approach was used. The reason for including MTR algorithms was that each algorithm differs in how the targets are included in the final model, so using the algorithms themselves as the evaluation function will still allow any useful target interaction to be captured. Therefore, the power of MTR algorithms will be retained with the added bonus of a reduced feature count. E. FEATURE SELECTION ALGORITHM Genetic algorithm (GA) uses the operators of selection, crossover and mutation. Each of the operators perform their respective operations on a set of candidate solutions (feature subsets) called ‘population’. The selection operator selects the candidate solutions for crossover based on their performance. The crossover operator combines two (or sometimes more) candidate solutions to produce new candidate solutions. The intuition is that the child solutions from good parent solutions will combine the strengths of the parent solutions. Since continuous crossovers can result in convergence of the population towards one optimum result, possibly a local optimum, as similar solutions are repeatedly combined, mutation operator is used to add diversity to the population by randomly changing some individuals present in the population. Mutation allows the solutions to escape from local optima by creating solutions that are different from the current population. GA allow faster exploration of the state space along with relevant exploitation of promising solutions. Another advantage of GA is that these behaviours can be modified by tuning of the GA parameters which are generally simple, and thus, do not require extensive additional knowledge. GA is also domain independent and can be applied to problems or data set irrespective of their domains. However, numerous modifications are also possible in case VOLUME 11, 2023 121969 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR domain knowledge or heuristics need to be incorporated for certain situations. F. METHODOLOGY The proposed method uses the Single Target (ST) method to find suitable feature subsets for each target. For GA-FSTarget, a feature selection for each target is found independently by considering the m-target multi-target regression task as mseparate single target regression tasks. For GA-FSMTR, a common feature subset is found by finding a feature subset which minimizes the average error for all the targets. A step by step explanation of the whole process is given below: •Encoding and Population Initialization: Feature subsets are encoded in the form of strings of 0’s and 1’s. The 1’s denote the presence of a feature in a subset, while 0’s denote its absence. A number of candidate solutions or feature subsets in the form of strings of 0’s and 1’s are generated randomly. •Fitness Calculation: Each candidate solution is passed to the regressor and the error is obtained. In the case of GA-FS-Target, the candidate solution corresponds to one target at a time, while in the case of GA-FS-MTR, the candidate solution corresponds to all the targets together. Therefore, for the former case, Relative Root Mean Squared Error (RRMSE) of one target that is under consideration is considered as the fitness of the candidate solution, while for the latter case, the Average Relative Root Mean Squared Errors (ARRMSE) of all targets is considered as the fitness of the candidate solution. •Selection: The best candidate solutions i.e. the feature subsets with the lowest error are selected for further processing. •Crossover: Crossover operator is used to generate new candidate solutions by combining any two candidate solutions as depicted in Figure 4. FIGURE 4. Single point crossover. The bits from Parent 1 and Parent 2 and combined using the crossover operator to obtain Child 1 and Child 2. The crossover probability is dependent on the fitness of solutions. Candidate solutions with higher fitness are more likely to be selected as parents for crossover. •Mutation: Mutation operator is used to maintain a level of diversity in the solutions. In this case, this change consisted of changing a ‘0’ in the candidate solution string to a ‘1’ or vice versa as depicted in Figure 5. FIGURE 5. One bit mutation. A variable rate of mutation was used for these experiments. For the first few generations, the mutation rate was kept very high, as more exploration was required. The mutation rate was decreased after a few generations, and then decreased further after a few more. To compensate for the high rate of mutation and avoid mutation of good solutions, the mutation process was set up such that the best solutions in the population were never mutated. •Generation: This cycle continues until the stopping criteria is obtained. For these experiments, there were two stopping criteria: either if a better solution was not found for twenty consecutive generations, or a predefined number of generations was reached. Complexity Analysis: •If M is the dimension of the data set, then to generate N individuals, the time complexity would be close to O(M*N). •If F is the time taken to evaluate a fitness function, then the time complexity for population would be O(F*N) •Time complexity of crossover O(L), mutation O(MU) •If G is the number of generations, then the total time complexity of the algorithm may be denoted as: ComplexityGA =G∗O(M∗N∗F∗L∗MU) (2) Equation 2can be used to obtain a rough estimate of the running time of the proposed approach as well. Genetic algorithms often provide the most advantage for optimization problems where the population size is relatively small compared to the size of the problem. The state space for feature selection can be denoted as: S=2m, where mis the number of features. In the case of feature selection, the number of features can exceed hundreds or thousands which results in a huge state space and the number of population (N) is extremely small as compared to (S). Since one of the objectives of using GA for feature selection was to explore large state space of features quickly, therefore, apart from mutation, we have also introduced one more mechanism to increase the proportion of state space exploration. If the number of candidate solution goes below a threshold due to the selection process, new random candidate solutions are generated and added to the population. This allows for the number of solutions to remain at a certain number, while also introducing diverse candidate solutions into the population. Two variations of proposed approach have been evaluated for these experiments. Figures 6(a) and 6(b) show the 121970 VOLUME 11, 2023 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR proposed methods graphically and their description is given below: FIGURE 6. Graphical representations of the proposed methods. 1) GENETIC ALGORITHM FEATURE SELECTION MULTI-TARGET (GA-FS-MTR) The proposed method GA-FS-MTR works by finding a common feature subset which reduces the average error for all targets. For these experiments, the ST approach was used to find the common feature subset which was then evaluated using different MTR algorithms. A population of candidate solutions was initialized randomly. One by one, each feature subset was passed to the ST algorithm. Since ST evaluates the features per target, the obtained results were averaged to find the ARRMSE. The feature subsets with the lowest ARRMSE were selected for further processing. After a preconfigured number of generations was reached, the subset with the lowest ARRMSE was considered as the best one. Finally, the obtained feature subsets were used in conjunction with the MTR algorithms; Single Target (GA-FS-MTR(ST)), Stacked Single Target (GA-FS-MTR(SST)) and Ensemble of Regressor Chains (GA-FS-MTR(ERC)), to obtain the final results. The advantage of using this approach is that any target interdependencies are also incorporated in the final prediction model. 2) GENETIC ALGORITHM FEATURE SELECTION USING SEPARATE FEATURES (GA-FS-TARGET) The proposed method GA-FS-Target works by finding target specific features for each target separately. One advantage of using this technique is that no MTR algorithms are needed. Therefore, the complexity of the model is decreased. Another situation where this method may also be useful is when for any MTR problem, there are some targets that are more important than others. When using MTR algorithms, each target is considered equally. This proposed method can provide an improvement over baseline methods while allowing flexibility of handling each target differently. Similar to the first variation, a population of candidate solutions were initialized randomly. The candidate solutions were evaluated using ST, but the RRMSE of a single target was considered each time. Eventually, the feature subset which minimized the RRMSE for target ‘mi’ was considered the best feature subset for target ‘mi’. The process was then repeated for each target. In this way, for a dataset with ‘m’ targets, ‘m’ feature subsets were found. It would be pertinent to mention here that in each case, the ST method was used to find the feature subset. The reason is that since the MTR algorithms also incorporate target information when generating the models, only the feature subsets returned by using MTR algorithms would not have given an accurate picture of the feature importance. G. EXPERIMENTAL SETTINGS All experiments were carried out using Mulan and Weka [43]. The data was divided into two equal half. 15-fold cross validation was used for feature selection experiments using half of the data. The experiments were performed using ‘reptree’ as base regressor. Once the best feature subsets were identified, the reduced features were used for training and testing on the second half of the data to obtain final results which are reported in the paper. This was also carried out using 10-fold cross validation. For the genetic algorithm, a standard single point crossover was used. The mutation rate was set randomly. Initially, a higher mutation rate was set to encourage exploration. It was decreased over the course of the subsequently generations. IV. RESULTS AND DISCUSSIONS This section presents the results of the experiments and separate discussions for each compared method. The results are presented to show a comparison of the proposed methods with their respective baseline methods. In each case, the proposed variation can be seen to have improved the performance of the the baseline method. In the Per Target method, a Single Target (ST) approach is employed. Therefore, this proposed method is compared with all the baseline methods using Table 12. A. COMPARISON WITH (GA-FS-MTR) GA-FS-MTR-SO: The proposed GA-FS-MTR-SO method provides an improvement in performance compared to the benchmark methods in all instances except a few. Tables 3, 4and 5show the results. The ST method achieves an average performance improvement of around 7.42% compared to its baseline by using the proposed approach in the majority of the data sets, as well as on average. The standard deviation is also less, showing that results do not vary very largely. The SST method also achieves a performance improvement of around 14.5% compared to its baseline method by using the proposed approach in the majority of the data sets. The average performance and the standard deviation also reflect the performance improvement. VOLUME 11, 2023 121971 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR TABLE 3. Table showing the comparison between the ARRMSE obtained from baseline method ST [21] with the ARRMSE obtained using the proposed Single objective method GA-FS-MTR(ST)-SO. TABLE 4. Table comparing the ARRMSE obtained from baseline method SST [22], with the ARRMSE obtained using the proposed Single objective method GA-FS-MTR(SST)-SO. TABLE 5. Table comparing the ARRMSE obtained from baseline method ERC [22], with the ARRMSE obtained using the proposed Single objective method GA-FS-MTR(ERC)-SO. The ERC method also achieves a performance improvement of around 11% compared to its baseline method by using the proposed approach in the majority of the data sets. In each case, in addition to the decrease error, the standard deviation is also reduced, thus leading to a more confident prediction using the proposed approach. To test if the differences in performances were significant, paired samples t-test was carried out between each MTR algorithm and its proposed counterpart. Table 6presents the resulting p-values for Single Objective methods. The significance test shows that the proposed methods have significantly better performance at alpha value of 0.05 in TABLE 6. Significance test results showing p-values between Benchmark methods ST, SST and ERC versus proposed methods GA-FS-MTR(ST)-SO, GA-FS-MTR(SST)-SO and GA-FS-MTR(ERC)-SO. some cases, while at the alpha value of 0.1, the proposed methods have significantly better performance in almost all cases. Further, Friedman test was carried out to better understand the performance differences between the methods. Figure 7 shows critical distance diagram for the compared techniques at alpha=0.1. FIGURE 7. Critical distance diagram comparing Single Objective based GA-FS-MTR methods with the baseline methods for alpha=0.1. Lower ranks are better. It can be seen from Figure 7that the proposed methods generally outperform the baseline methods and therefore have lower ranks in the diagram. (Objectives I and I) GA-FS-MTR-MO: If the feature selection problem is formulated as a multi-objective (MO) problem, a multiobjective fitness function is used in the GA. The GA(MO) returns a Pareto front, from which the feature subset that reduces the error the most is selected. Tables 7,8and9present the results for these experiments. TABLE 7. Table comparing the ARRMSE obtained from baseline method ST [21] versus the ARRMSE obtained using the proposed Multi-objective method GA-FS-MTR(ST)-MO. The proposed GA-FS-MTR-Multi Objective methods also provides an overall performance improvement as shown in Tables 7,8and 9. In the average case an improvement of around 4% over ST, 15.5% over SST and 10.5% over ERC is obtained. In this case as well, the standard deviation across all data sets is also lower for the proposed method. To test if the differences in performances were significant, paired samples t-test was carried out between each MTR 121972 VOLUME 11, 2023 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR TABLE 8. Table comparing the ARRMSE obtained from baseline method SST [22] versus the ARRMSE obtained using the proposed Multi-objective method GA-FS-MTR(SST)-MO. TABLE 9. Table comparing the ARRMSE obtained from baseline method ERC [22] versus the ARRMSE obtained using the proposed Multi-objective method GA-FS-MTR(ERC)-MO. algorithm and its proposed counterpart. Table 10 presents the resulting p-values for Multi-Objective methods. TABLE 10. Significance test results showing p-values between Benchmark methods ST, SST and ERC versus proposed methods GA-FS-MTR(ST)-MO, GA-FS-MTR(SST)-MO and GA-FS-MTR(ERC)-MO. The proposed methods show significant differences at alpha value of 0.1 indicating that the proposed approach improves the performance considerably. Friedman test was also carried out between the proposed and baseline methods. Figure 8shows critical distance diagram for the compared techniques at alpha=0.1 for Multi-Objective based GA-FS-MTR methods. (Objectives I and I). FIGURE 8. Critical distance diagram comparing Multi-Objective based GA-FS-MTR methods with the baseline methods for alpha=0.1. Lower ranks are better. It is seen from the figures that the proposed methods generally outperform the baseline methods and therefore have lower ranks in the diagram. The GA-FS-MTR(ERC) method performs best among the proposed methods for both the cases. In order to analyze how much feature reduction the proposed approaches provide, the number of features obtained after feature selection are analyzed. Table 11 shows the percentage reduction in the number of features when the proposed GA-FS-MTR (GA-FS-MTR-Single Objective or GA-FS-MTR-Multi-Objective) methods are used. TABLE 11. Percentage reduction in number of features using GA-FS-MTR-Single Objective and GA-FS-MTR-Multi-Objective. The results show that the proposed methods provide a remarkable reduction in the number of features. With a minimum of 38 percent reduction, feature selection provides a substantial improvement in interpretability by identifying the features which affect the target variables the most. Another point of interest in the results is that even for the lower dimensional datasets, the proposed methods still allow for at least an at par performance with the reduced feature set. This scale of reduction allows for a greater possibility of visualizing the results using graphical techniques and thus gaining more insight into the data. In order to inspect the effect of feature selection on each target, Figures 9(a) and 9(b), show a comparison of the Relative Root Mean Squared Error of each target when features are reduced using GA-FS-MTR-Single Objective. Figures 10(a) and 10(b) show a comparison of the Relative Root Mean Squared Error of each target when features are reduced using GA-FS-MTR-Multi-Objective. As can be seen from the figures, the performance improvement is observed in almost all of the targets using the proposed approaches. Delving further into the feature selection process to find how often any feature is used when making predictions. In order to do that, Local interpretable model-agnostic explanations LIME [44] was used to evaluate the feature contributions for two data sets ‘jura’ and ‘edm’ and the predictions were made using a Decision Tree regressor and a Ridge regressor implemented using python’s sklearn library. Figures 11(a) and 11(b) show the percentage of the number of a times a feature was used by the model to make a prediction. As can be seen from the figures, although some features are used more frequently in making the predictions, it is not possible to decide the extent to which the models will be affected if one of the features is dropped. Therefore, any data driven activity will require an assessment of the features individually. Feature selection identifies the most important VOLUME 11, 2023 121973 F. H. Syed et al.: Toward an Optimal and Structured Feature Subset Selection for MTR FIGURE 9. Comparison of Relative Root Mean Squared Error of each target in the data set for data sets ‘andro’ and ‘jura’ when feature reduction is obtained using GA-FS-MTR-Single Objective. FIGURE 10. Comparison of Relative Root Mean Squared Error of each target in the data set for data sets ‘andro’ and ‘jura’ when feature reduction is obtained using GA-FS-MTR-MultiObjective. features and therefore allows increased interpretability and understanding into the working of the model. B. COMPARISON WITH (GA-FS-TARGET) The results of the proposed method GA-FS-Target are presented in the Table 12. Although this proposed technique ignores the possible target interdependencies, it still provides an average performance improvement of roughly 8.6% in comparison to all the MTR methods. FIGURE 11. Percentage of times a feature contributed to a prediction for two machine learning models. TABLE 12. Table comparing the ARRMSE obtained from baseline methods ST [21], SST and ERC [22], versus the ARRMSE obtained using the proposed Per target method. Table 13 shows the p-values for t-test between the benchmarked methods and the proposed method GA-FSTarget. The results how a significance in all cases at alpha value 0.1 and a significance in all but one case for alpha 0.05. TABLE 13. Significance test results showing p-values between Benchmark methods ST, SST and ERC versus proposed method GA-FS-Target. Figure 12 shows the critical distance diagram for GA-FS-Target method at alpha=0.1. The proposed method 121974 VOLUME 11, 2023