Feature Selection Guided by CVOA Metaheuristic for Deep Neural Networks: Application to Multivariate Time Series Forecasting

Feature Selection Guided by CVOA Metaheuristic for Deep Neural Networks: Application to Multivariate Time Series Forecasting Manuel Jes´us Jim´enez-Navarro1(B), Camilo Restrepo-Estrada2, Laura Melgar-Garc´ıa3, and David Gutierrez-Aviles1 1Department of Computer Languages and Systems, University of Seville, Seville, Spain {mjjimnav3,dgutierrez3}@us.es 2Faculty of Economic Sciences, University of Antioquia, Medell´ın, Colombia [email protected] 3Data Science and Big Data Lab, Pablo de Olavide University, Seville, Spain [email protected] Abstract. In the workflow of machine/deep learning, a common question that data scientists ask before training a model is whether to feed the model with the entire set of input dataset features or just a subset of them. In this scenario, the precise selection of features from the input data has a significant impact on the efficiency of model training and the final performance, especially in datasets with many variables. This decision is typically made before model training and in a static way (i.e., the feature selection is applied, and then the model is trained). This paper presents a new methodology based on metaheuristics to dynamically select features. This approach has been applied to train a deep neural network to solve two problems: univariate and multivariate time series forecasting. The performance of the proposed method has been compared to the random forest model (a well-known machine learning model with a highly efficient internal feature selection method). The proposed method outperformed the random forest model in terms of mean squared error, demonstrating the validity of the approach. Keywords: deep learning ·feature selection ·time series forecasting · multivariate c 1 Introduction The technique of feature selection can significantly improve the efficiency of machine and deep learning models by reducing the number of irrelevant features. This, in turn, reduces the burden of measuring, storing, managing, and processing unnecessary features, resulting in improved efficacy and interpretability of the model. Moreover, feature selection is particularly crucial in industries where collecting new features can be difficult and time-consuming [14]. https://doi.org/10.1007/978-3-031-42529-5_20 210 M. J. Jim´enez-Navarro et al. There are three primary types of feature selection techniques: Filter, Wrapper, and Embedded [6]. Filter methods rank the importance of each feature based on heuristics and eliminate irrelevant ones. However, the selected features may not be the most effective or related to the acquired knowledge of the model. Wrapper methods, on the other hand, use the model itself to select optimal features, but the training process can be a bottleneck. Embedded methods combine the best of Filter and Wrapper methods and do not require hyperparameter tuning, but an algorithm that includes feature selection is needed. Among wrapper methods, metaheuristic algorithms, such as genetic algorithms, particle swarm optimization, and simulated annealing, are commonly used to search for the optimal set of features that maximize the performance of the machine learning model. These algorithms explore the search space of possible feature subsets and iteratively refine the subset by removing or adding features until a satisfactory subset is found. One significant advantage of using metaheuristics for feature selection is their ability to handle high-dimensional feature spaces efficiently. They can search through a large number of features and find an optimal subset that provides a good trade-off between accuracy and computational cost. Additionally, metaheuristics can incorporate different evaluation metrics to measure the quality of the feature subset, such as accuracy, recall, quadratic error, and absolute error. In summary, metaheuristics are a powerful tool for feature selection that enables efficient exploration of highdimensional feature spaces and optimization of model performance. However, the quality of the fitness function used is critical to the effectiveness of the approach. In this work, we present a feature selection methodology that uses a wrapper method to find the best subset of features that yield the best model, in terms of error, guided by the CVOA metaheuristic [13] applied to time series forecasting. Feature selection has proven to improve the performance of the models on several ocassions, which motivates the combination of these techniques. The method has been applied to both univariate and multivariate time series forecasting problems and compared to random forests [2] (a well-known machine model with an internal feature selection method), showing promising results. The rest of the paper is organized as follows: Section 2analyzes a set of different approaches in the feature selection research field. Section 3describes the presented approach. Section 4shows the experimental results, and Sect. 5 sets out the future efforts to be made in this research line. 2 Related Works Feature selection is a crucial technique in machine learning that can enhance model efficiency and accuracy. There are various methods for feature selection, including filter, wrapper, and embedded methods. In [10], a feature ranking method based on the information gain score was applied to a heart disease dataset, and a threshold was used to select features before training a neural network. The authors in [12] built an ensemble of neural networks clustered in groups of three using PCA, RankSum, and t-test selection methods. In [16], Feature Selection Guided by CVOA Metaheuristic 211 a feature ranking method constrained the derivatives by cross-entropy error, and recursive feature elimination was used based on importance. In [8], the authors presented an incremental feature selection method that uses the correlation between features. In [15], the authors used a genetic algorithm to optimize the model. The proposal in [1] used the Mayfly metaheuristic algorithm for feature selection. The authors in [4] developed an embedded method that trains a mask during gradient descent to select features. This approach is similar to the method presented in this study, but additional loss terms are not included to binarize the mask. In [5], the authors proposed a new feature selection method that combines three weight-based feature selection methods to address the ranking stability problem. Overall, embedded methods that relate feature selection to model training seem to be the most promising, but they can be computationally expensive. The optimal feature selection method depends on the specific problem and dataset, and researchers should consider the trade-offs between efficiency, accuracy, and computational cost when selecting a method. Regarding metaheuristics, a review of metaheuristic optimization algorithms for feature selection was presented in 2022 [7]. This feature selection approach has been applied to different domains. For example, in [11] the metaheuristic feature selection PSO or Particle Swarm Optimization was applied with classification models for software fault prediction. In [9] a new feature selection method based on a hybrid metaheuristic combining Gorilla Troops Optimizer and Bird Swarms, called GTO-BSA, was presented for IoT intrusion detection. Authors in [13] proposed the Coronavirus Optimization Algorithm, a bioinspired metaheuristic based on the COVID-19 propagation model. This algorithm was successfully applied in [3] by hybridizing a deep learning network with a metaheuristic algorithm to forecast deformation in a hydropower dam, demonstrating the potential of combining different techniques for feature selection and model optimization. 3 Methodology This section outlines the methodology used to obtain the results from the dataset, which is divided into five subsections. Section 3.1 explains the selected dataset and the preprocessing techniques used. Section 3.2 provides details on the parameters used for the neural network, the training process, and the random forest. Section 3.3 presents the evaluation metrics used during the experimentation. Lastly, Sect. 3.4 illustrates the overall optimization process employed by the algorithm. 3.1 Dataset The dataset selected for this study is the Electricity Transformer Temperature (ETT) dataset [17]. This dataset comprises data from two electricity stations 212 M. J. Jim´enez-Navarro et al. (ETTh1 and ETTh2) in China, which were sampled every hour. It contains six features related to the transformer load and the isolation oil temperature. The goal of this study is to predict the future isolation oil temperature to prevent critical damage to the transformer and avoid unnecessary waste. In our case, we considered two scenarios: the univariate and multivariate datasets. In the univariate scenario, only the target data was used during experimentation, while in the multivariate scenario, all features were employed. The original data was initially divided into training, validation, and testing datasets, which represent 70%, 10%, and 20% of the data, respectively. The temporal order of the instances was maintained while dividing the data. Therefore, instances in the training dataset were from moments before the validation and test sets. Once the data was split, the mean and standard deviation were calculated from the training dataset. Using these parameters, all the datasets were standardized. Finally, since we are dealing with a time-series forecasting problem using a neural network, the data had to be processed using the windowing technique. This technique involves converting the original time series into the input window and the forecast horizon. The input window consists of Wpast instances for all variables Din the dataset, while the output comprises the Hfuture instances for the target variable. Note that this technique typically depends on the hyperparameter W, which must be tuned. Additionally, the Hparameter was fixed at 24, which represents a prediction of the next 24 h. 3.2 Models A FeedForward neural network was selected for experimentation due to its simplicity. The neural network comprises a single hidden layer with 32 units for all experiments. For training, we employed the Adam optimizer with a learning rate of 0.003, which is the most common optimizer with default configuration. The Swish activation function was used for the hidden layer as it has been demonstrated to be a good alternative to ReLU, with better results. The neural network was trained for 50 epochs, which is sufficient to converge, and the early stopping technique was used to stop training when there was no change in the loss function for 10 consecutive epochs. Finally, the loss function used was the well-known Mean Squarred Error. To compare the performance of the neural network with another widely used method, we employed the random forest model. This model incorporates a feature selection process in its design, making it suitable for comparison in our methodology. The random forest was configured using 50 estimators, a maximum depth of 20, and the absolute error as the splitting criterion. Feature Selection Guided by CVOA Metaheuristic 213 3.3 Evaluation Metric To evaluate the performance of trained models, the Mean Squared Error (MSE) has been employed. In addition, the training time is calculated to measure the number computational cost. MSE =1 N N  n=1 1 H H  h=1 (yt+h−ˆyt+h)2,(1) here, yt+h−ˆyt+hdenotes the the difference between the prediction and the actual value, respectively. The error is averaged over all horizons Hand instances N. 3.4 Optimization Process To find the optimal parameters for the neural network using the previously described data, models, and evaluation metric, the CoronaVirus Optimization Algorithm (CVOA) [13] is utilized. To use the CVOA algorithm, the domain and fitness function need to be defined. The domain specifies the parameters and their ranges. In our problem, the domain comprises the number of past instances (W) and the selected input features/moments (S). Note that Sis a one-dimensional vector selected from the two-dimensional matrix WxD, representing instances selected from both the time and features dimensions. The fitness function receives the parameters to be evaluated and returns a score. This function applies the windowing transformation using the Wparameter and selects the Sfeatures from all the datasets. The model is then trained and evaluated against the validation set using the training dataset. After 6 iterations, which is enough to converge, the CVOA algorithm finds the optimal parameters for the neural network in both the univariate and multivariate datasets. Using the same W, the neural network is then trained with and without feature selection (FS) to compare their performance. Finally, the random forest is applied using the same Was the neural network. In summary, this work reports results for the following scenarios: – Univariate vs multivariate data. – Neural network with selection vs neural network without selection. – Neural network vs random forest. 4 Results and Discussion In this section, we present the results obtained from applying our methodology divided into three subsections. Section 4.1 summarizes the results, while Sect. 4.2 analyzes the performance based on the number of features. Finally, Sect. 4.3 showcases the best and worst predictions obtained from the best model. 214 M. J. Jim´enez-Navarro et al. 4.1 Performance Results Table 1shows the results obtained by our methodology and the comparison in all the specified scenarios. In particular, the columns in Table 1refer to the model used (NN for neural network and RF for random forest), the dataset used (described in Sect. 3.1), whether or not feature selection (FS) was used, whether it was a univariate or multivariate problem, the MSE error (see Eq. 1), the training time, the number of past instances (W) and the number of selected features (#S). Table 1. Results obtained after the evaluation process. The data has been divided by the model and if the dataset was univariate or multivariate. The best MSE error and training time for each dataset is marked in bold. Model Dataset FS? Univariate? —MSE Time W#S NN ETTh1 Yes Yes 0.065 2.326 68 32 NN ETTh2 Yes Yes 0.289 2.310 64 23 NN ETTh1 No Yes 0.089 4.373 68 68 NN ETTh2 No Yes 0.330 1.943 64 64 NN ETTh1 Yes No 0.074 2.388 20 3 NN ETTh2 Yes No 0.394 2.371 24 1 NN ETTh1 No No 0.550 2.444 20 140 NN ETTh2 No No 0.399 2.261 24 168 RF ETTh1 Yes Yes 0.090 5.936 68 68 RF ETTh2 Yes Yes 0.412 6.236 64 64 RF ETTh1 Yes No 0.090 11.134 20 140 RF ETTh2 Yes No 0.633 13.129 24 168 Upon initial analysis, the neural network utilizing univariate data and feature selection appears to produce the best outcomes. When compared to the neural network without selection, the results display a marked improvement of 27% in ETTh1 and 12% in ETTh2 in terms of MSE. Including all features in the model does not appear to be advantageous, even with feature selection. This is logical, as the search space in the multivariate data version is larger. The random forest produces commendable outcomes compared to the neural network without feature selection in the ETTh1 dataset. However, the results are substantially inferior, indicating that the ETTh1 dataset is more difficult. 4.2 Number of Features Analysis Figure 1a shows the distribution of the error obtained for each number of feature in the ETTh1 dataset using the multivariate and univariate data. Firstly, in the multivariate distribution, we observed that the optimization algorithms explore a usually low number of features, even though the total Feature Selection Guided by CVOA Metaheuristic 215 Fig. 1. Error distribution for each dataset using univariate and multivariate data. number of features in the multivariate data is close to 140 instances. Additionally, we noted that selecting just one feature represents the worst possible selection. When considering the univariate data, the error and its variability are significantly lower than in the multivariate data, as expected. Furthermore, as previously mentioned, selecting only two to seven past instances tends to result in a considerable increase in error. Figure 1b displays the same distribution as mentioned before but for the ETTh2 dataset. Focusing on the multivariate distribution, we observe the same trend as in ETTh1, with a significant increase in error for a low number of features. The error appears to be lower when 15 features are selected, which seems to be the minimum to achieve good results. Regarding the univariate data, the distribution seems to exhibit the same behavior as in the ETTh1 dataset, with a higher error when only three or four instances are used for the prediction. Finally, Fig. 2shows the selected features for the best models in the univariate version of ETTh1 and ETTh2. 4.3 Best and Worst Predictions Analysis Figure 3represents the best and worst 24 h predictions for the ETTh1 and ETTh2 dataset by the best model using multivariate data. In ETTh1, the best prediction seems to be in the summer season in a epoch where the temperature seems to stabilize to a low temperature. The worst prediction is located also in summer but the temperature has suddenly decreased and 216 M. J. Jim´enez-Navarro et al. Fig. 2. Selected features for univariate datasets. Fig. 3. Best and worst prediction for multivariate datasets. The x-axis represents the period of time containing the input window and the forecasting. stabilized in a very low temperature. This seems to indicate that the model expected the temperature to stabilize around 10 degrees but the behaviour changed suddenly. In the ETTh2, the best prediction seems to represent the mean value of the real values as the increasing in the temperature is not correctly predicted. The worst prediction seems to have incorrectly predicted the behaviour of the temperature as the ups and downs seem to have been exchanged in the autumn season. Figure 4shows the best and worst predictions by the best model using the univariate data. For ETTh1, the best prediction occurs during the summer season when the temperature stabilizes at a low temperature. On the other hand, the worst prediction is also in the summer, but the temperature has suddenly decreased and stabilized at a much lower temperature. This suggests that the model was anticipating the temperature to stabilize around 10 degrees, but the actual behavior was different. Feature Selection Guided by CVOA Metaheuristic 217 Fig. 4. Best and worst prediction for univariate datasets. The x-axis represents the period of time containing the input window and the forecasting. Regarding ETTh2, the best prediction represents the average value of the actual values since the model was unable to accurately predict the temperature increase. In contrast, the worst prediction has incorrectly forecasted the temperature behavior, where the ups and downs seem to have been reversed during the autumn season. 5 Conclusions and Future Work In this research, a novel method for feature selection guided by metaheuristics has been presented. This methodology consists of finding the best subset of the attributes of the input datasets that yields a better model in terms of error; the search strategy used is the CVOA metaheuristic. The method has been applied to solve univariate and multivariate time series forecasting problems, which yield promising results. This methodology has been validated by comparing it with random forest models that implement a well-known and high-performance internal attribute selection method. However, the performance of the proposed method is considerably greater than Embedded feature selection methods like tree-based models. The future efforts for this research line include optimizing the architecture of the deep learning model in addition to the attribute set. In this way, the result of the method would be an optimal deep learning architecture with optimal hyperparameters and an optimal attribute set. Acknowledgements. The authors would like to thank the Spanish Ministry of Science and Innovation for the support under the projects PID2020-117954RB-C22 and TED2021-131311B and Junta de Andaluc´ıa for the project PYC20 RE 078 USE.