Anticipating Future Behavior of an Industrial Press Using LSTM Networks

applied sciences Article Anticipating Future Behavior of an Industrial Press Using LSTM Networks Balduíno César Mateus 1,2,* , Mateus Mendes 3,4,* , José Torres Farinha 3,5 and António Marques Cardoso 2   Citation: Mateus, B.C.; Mendes, M.; Farinha, J.T.; Cardoso, A.M. Anticipating Future Behavior of an Industrial Press Using LSTM Networks. Appl. Sci. 2021,11, 6101. https://doi.org/10.3390/app11136101 Academic Editors: Marlene Amorim, Yuval Cohen and João Reis Received: 30 April 2021 Accepted: 25 June 2021 Published: 30 June 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1EIGeS—Research Centre in Industrial Engineering, Management and Sustainability, Lusófona University, Campo Grande, 376, 1749-024 Lisboa, Portugal 2 CISE—Electromechatronic Systems Research Centre, University of Beira Interior, Calçada Fonte do Lameiro, P–62001-001 Covilhã, Portugal; [email protected] 3Polytechnic of Coimbra, ISEC, 3045-093 Coimbra, Portugal; [email protected] 4Institute of Systems and Robotics, University of Coimbra, 3004-531 Coimbra, Portugal 5Centre for Mechanical Engineering, Materials and Processes—CEMMPRE, 3030-788 Coimbra, Portugal *Correspondence: [email protected] (B.C.M.); [email protected] (M.M.) Abstract: Predictive maintenance is very important in industrial plants to support decisions aiming to maximize maintenance investments and equipment’s availability. This paper presents predictive models based on long short-term memory neural networks, applied to a dataset of sensor readings. The aim is to forecast future equipment statuses based on data from an industrial paper press. The datasets contain data from a three-year period. Data are pre-processed and the neural networks are optimized to minimize prediction errors. The results show that it is possible to predict future behavior up to one month in advance with reasonable confidence. Based on these results, it is possible to anticipate and optimize maintenance decisions, as well as continue research to improve the reliability of the model. Keywords: time series prediction; LSTM prediction; deep learning prediction; predictive maintenance 1. Introduction Modern processors, computers and high speed networks make it possible to acquire, transfer and store large quantities of data in real time. Acquisition and combination of data from different sensors makes it possible to gain an insightful view of the state of factories, industrial plants and other facilities. Large datasets can be constructed, stored and processed using information technologies such as Big Data, cloud computing, cuttingedge computing, and artificial intelligence tools. The Internet of Things (IoT) is a recent concept, which provides many benefits to different areas, such as maintenance and production management, because it facilitates the automation of tasks such as monitoring and maintenance. This results in the popularization of intelligent systems, which are highly dependent on Big Data [ 1 ] and are an important area of study, since they offer the tools and methods to acquire and process large volumes of data such as historical production processes, including many production and operating parameters. Modern time-series and other data analysis techniques have been used with success for different tasks, such as freeway traffic analysis [ 2 ] and additive manufacturing [ 3 ]. Different approaches have also been proposed in the field of predictive maintenance [ 4 , 5 ]. Satisfactory results were obtained using Big Data records as support for PCA models, which resulted in a warning alarm several days before a potential failure happened [6]. Life cycle optimization has been an important concern for decades. A physical asset with proper maintenance will have a longer useful life with a greater return on investment for the organization [7]. Predictive maintenance requires good quality data. The information that is extracted from the online or offline data must be reliable, and so the results must be good enough to justify the investment in data collection and analysis. The process starts from the Appl. Sci. 2021,11, 6101. https://doi.org/10.3390/app11136101 https://www.mdpi.com/journal/applsci Appl. Sci. 2021,11, 6101 2 of 16 correct calibration of the reading sensors and equipment [ 8 ]. The data are then stored and processed using different models, such as Principal Component Analysis (PCA) and Neural Networks [ 9 ]. Maintenance planning involves the use of several algorithms, the most common being time series [10]. Maintenance of equipment in the industry becomes a sensitive and important point that affects the equipment’s operating time and efficiency [ 5 ]. This makes maintenance one of the strategic points for the development and growth of competitiveness vis-à-vis competitors. Chen and Tseng studied the total expected cost of maintaining a flotation system, including the cost of lost production, the cost of repairs, and the cost of standby machines [11]. Daniyan et al. propose the integration of Artificial Intelligence (AI) systems, which will bring many benefits in diagnosing condition problems of industrial machines [ 12 ]. They highlight the viability of AI that combines the use of Artificial Neural Networks (ANNs) with a dynamic time series model, for fault diagnostics, to optimize the equipment intervention time. Hsu et al. demonstrated that neural networks can be a great technology in the support and decision making of large and small companies [ 13 ]. There is a trend to use those tools in predictive maintenance systems with the aim of making the prediction systems more intelligent [14]. According to Jimenez et al., there is a great effort in the development of predictive models for application in predictive maintenance [ 15 ]. Ayvaz and Alpay apply Long Short-Term Memory (LSTM) neural network approaches to predict real production data, obtaining satisfactory results, superior to conventional models [ 16 ]. In their study to improve maintenance planning to minimize unexpected stops, they apply a new method that consists of the combined use of decomposition in empirical mode of ensemble and long-term memory. Their results showed a performance superior to other state of the art models. LSTM networks use several ports with different functions to control neurons and to store information. The LSTM cell can retain important information for a longer period in which it is used. This property of information maintenance allows the LSTM to exhibit a good performance in the classification, processing, or forecasting of a complex dynamic sequences [17]. The present work uses different LSTM models to predict future trends of six variables, on a dataset containing three years of data samples grabbed in an industrial press, which aims to operate continuously with minimum downtime. Different data pre-processing techniques, network architectures and hyperparameters were tested in order to determine the models that best fit the data and provide the lowest prediction errors. Section 2contains a summary of related work. Section 3describes the theory of the LSTM networks. Section 4describes the methods used for the present work. Section 5 describes the results and validation of the predictive model. Section 7draws some conclusions and suggestions for future work. 2. Related Work 2.1. Predictive Maintenance In smart industries, predictive maintenance is one of the most used techniques to improve condition monitoring, as it allows one to evaluate the conditions of specific equipment in order to predict problems before failure [ 18 ]. For good performance of predictive models, it is important that the sensor data collected are of good quality. Deep neural models have been used with success to improve prediction for condition monitoring of industrial equipment. Wang et al. [ 19 ] use a model of long short-term recurrent neural networks (LSTMRNN) with the objective of predictive maintenance based on past data. The main objective of predictive maintenance is to make an accurate estimate of a system’s Remaining Useful Life (RUL). Traditional systems are only able to warn the user when it is too late and the Appl. Sci. 2021,11, 6101 3 of 16 failure occurs, causing an unpredictable offline period during which the system cannot operate properly with a consequent waste of time and resources [20]. In order to assess the condition of a system, the predictive maintenance approach employs sensors of different kinds. Some examples are temperature, vibration, velocity or noise sensors, which are attached to the main components whose failure would compromise the entire operation of the system. In this sense, predictive maintenance analyzes the history of a system in terms of the measurements collected by the sensors that are distributed among the components, with the objective of extracting a “failure pattern” that can be exploited to plan an optimal maintenance strategy and thus reducing offline periods [ 21 ]. In a case related to the steel industry, Ref. [ 22 ] used neural networks for classification of maintenance activities, so that interventions are planned according to the actual status of the machine and not in advance. Using multiple neural networks to identify status and RUL at a higher resolution can be very difficult, as the system can predict failure classifications and may not be able to recognize neighboring states. One limitation arises from the need for maintenance records to label datasets and the need for large amounts of data of adequate quality with maintenance events, such as component failures. When systems start to be very complex or the number of sensor measurements to manage is very large, it can be difficult to estimate a failure. For this reason, in recent years, machine learning techniques are used more and more to predict working conditions of a component. Mathew et al. [ 23 ] propose several approaches to machine learning such as support vector machines (SVMs), decision trees (DTs), Random Forests (RFs), and others that show which technique has the best performance in RUL forecast for turbofan engines. A major challenge in operations management is related to predicting machine speed, which can be used to dynamically adjust production processes based on different system conditions, optimize production performance and minimize energy consumption [ 24 ]. Essien and Giannetti [ 25 ] use a deep convolutional LSTM encoder–decoder architecture model on real data, obtained from a metal packaging factory. They show that it is possible to perform combinations of LSTM with other networks to significantly improve the results. 2.2. Prediction with LSTM Models LSTM neural networks achieved the best performance in a number of computational sequence labeling tasks, including speech recognition and machine translation [ 26 ]. There are a variety of engineering problems that can be solved using predictive neural models. Beshrand Zarzoura used neural network models to predict problems of suspended road bridge structures based on global navigation satellite system observations [ 27 ]. Sak et al. demonstrated that the proposed LSTM architectures exhibit better performances compared to deep neural networks (DNNs) in a large vocabulary speech recognition task with a large number of output states [ 28 ]. Chen et al. adopted LSTMs for predicting the failure of heavy truck air compressors [ 29 ]. They concluded that the use of LSTMs leads to more consistency in predictions over time compared to models that ignore history, such as random forest models. Gosh et al. [ 30 ] presented an extension that they called Contextual LSTM (CLSTM). This model was also used for the forecasting of pollutants. There is also the proposal for a genetic long short-term memory (GLSTM), which has been used in the study of wind energy forecasting [ 31 ]. Guo et al. presented a combination method based on real-time prediction errors in which the support vector regression (SVR) and LSTM outputs are combined in the final results of the model’s prediction, thus obtaining results of greater precision [32]. Ren et al. used a combination of a Convolution Neural Networks (CNNs) and LSTM in order to extract more in-depth information from data to predict the useful life of ion batteries [ 33 ]. Niu et al. used an LSTM and developed an effective speed prediction model to solve prediction problems over time [ 34 ]. Feng et al. report that the LSTM algorithm is superior and, according to them, it performs better than conventional neural network models [35]. Appl. Sci. 2021,11, 6101 4 of 16 The architecture of an LSTM network includes the number of hidden layers and the number of delay units, which is the number of previous data points that are considered for training and testing. Currently, there is no general rule for selecting the number of delays and hidden layers [ 36 ]. A deep LSTM can be built by stacking multiple LSTM layers, which generally works better than a single layer. Deep LSTM networks have been applied to solve many real-world sequence modelling problems [ 37 ]. The LSTM can also be used for planning studies [38], namely for planning the analysis of road traffic speed. To produce a prediction model with good accuracy, it is necessary to optimize neural models’ hyperparameters. While simple models can often produce good results with default hyperparemeters, the optimization process can greatly improve the results [ 39 – 41 ]. The selection of hyperparameters often makes the difference between underperformance and state-of-the-art performance. Optimization is often performed using machine learning algorithms, such as grid search, grey wolf optimization or particle swarm optimization. In the present prediction model, however, the hyperparameters were optimized manually, following a trial and error guided process, one variable at a time. This method was followed because it was the most convenient considering the limited computing power available. 2.3. LSTM with Encoder and Decoder Experiments were performed with a predictive model based on the LSTM with encoder and decoder architecture. The model consists of two LSTMs, in which the first LSTM has the function of processing an input sequence and generating an encoded state. The encoded state compresses the information in the input stream. The second LSTM, called a decoder, uses the encoded state to produce an output sequence. Those input and output sequences can be of different lengths. This technique has already been used to solve problems such as the prediction of vehicle trajectories based on deep learning [ 42 ]. This architecture [ 43 ] has shown great performance for tasks of translating from sequence to sequence. LSTM encoder–decoder models have also been proposed for learning tasks such as automatic translation [ 43 , 44 ]. There is the application of this model to solve many practical problems, such as the study of the equipment condition, applications in language translations, among others [45–47]. 3. Theoretical Background The present work uses LSTM networks, considering the referred different studies showing their usefulness for time series predictions [ 48 , 49 ]. The LSTM is a deep learning recurrent neural network architecture that is a variation of traditional recurrent neural networks (RNNs). It was introduced by Hochreiter and Schmidhuber in 1997. The most popular version is a modification refined by many works in the literature [ 50 , 51 ], which is called vanilla LSTM (hereinafter referred to as LSTM). The LSTM is excellent at handling time series data only with its network parameters. For example, weights and polarization are adjusted or optimized [ 52 ]. The primary modification of the LSTM when compared to the RNN architecture is the structure of the hidden layer [ 53 ]. The LSTM model is a powerful type of recurrent neural network (RNN), capable of learning long-term dependencies [ 54 ]. They became popular due to their power of representation and effectiveness in capturing long-term dependencies [55]. Many networks showed instability when dealing with exploding or vanishing gradient problems during learning. Those problems happen when the gradient of the error is too large or too small. If it is too large, it overflows and the errors cannot propagate properly through different layers during learning. If it is too small, it vanishes and the network does not learn. Different methods were proposed to solve those problems, known as a kind of of “door control” that is used in RNN models. For example, Gated Recurrent unit (GRU) algorithms [ 56 , 57 ], as the LSTMs [ 58 , 59 ], are to a large extent immune to the gradient problems and learn well. The LSTM network structure is based on three ports whose function is to regulate the flow. Those ports are called the entrance door, the forget gate, and the exit door. The Appl. Sci. 2021,11, 6101 5 of 16 main port of entry is to regulate the entry of new memory data; the forget gate has the function of regulating the storage time in the network memory and the output port intends to regulate how much the value retained in memory influences the activation of the output block [60]. Kong et al. demonstrate some relevant conclusions such as (1) LSTM has a good predictive capacity; (2) their use can significantly improve the profit of service providers, so there is an opportunity when it comes to exploring the forecast in real time [ 61 ]. LSTM networks are the de facto gold standard for deep learning algorithms for analyzing time series data [55]. Figure 1shows the internal architecture of an LSTM unit cell. According to [ 62 , 63 ], the internal calculation formulae of the LSTM unit are defined as follows: it=σ(xtUi+ht−1Wi+bi)(1) ft=σ(xtUf+ht−1Wf+bf)(2) ot=σ(xtUo+ht−1Wo+bo)(3) at=tan(xtUC+ht−1WC+bC)(4) where Ui , Uf , Uo and UC are the weight matrices for mapping the current input layer on three ports and the state of the current input cell. Figure 1. Detailed layout of a long short-term memory unit [63]. Wi , Wf , Wo and WC are the weight matrices for mapping the previous output layer on three ports and the current state of the input cell. bf , bi , bo , and bc are polarization vectors for calculating the state of the door and the input cell. σ is the gate activation function, which is normally a sigmoid function. tan is the hyperbolic tangent function which is the activation function for the current state of the input cell. Then, the current state of the output cell and the output layer can be calculated using the following equations. Ct=σ(ft×Ct−1+it×at)(5) ht=tanh(Ct)×ot(6) To assess the quality of the prediction model, one of the most popular metrics is the Root Mean Square Error (RMSE), which is given by Equation (7): RMSE =s1 n n ∑ t=1 (Yt−ˆ Y)2(7) where Yt represents the desired (real) value and ˆ Y is the predicted (obtained from the model) value. The difference between Y and ˆ Y is the error between the value expected Appl. Sci. 2021,11, 6101 6 of 16 to obtain and the value actually obtained from the network. n represents the number of samples used in the test set. The RMSE, however, is an absolute error. Therefore, there are also the Mean Absolute Percentage Error (MAPE) and the Mean Absolute Error (MAE). Those errors are given by the following formulae: MAE =1 n n ∑ t=1 |Yt−ˆ Yt|(8) MAPE =1 n n ∑ t=1 |Yt−ˆ Yt| |Yt|(9) where Yt represents the real value, ˆ Yt the predicted value and n represents the total number of samples. 4. Data Preparation Data are key to developing efficient modeling and planning. However, to be valuable, data need to be processed and structured before being analyzed. 4.1. The Problem The main goal of the present work is to predict potential failures in an industrial drying press before they happen. Data come from six sensors installed in the press. Those sensors monitor the operation of the press, with a sampling period of one minute. The monitored variables are: (1) electric current intensity; (2) oil level at the hydraulic unit; (3) VAT pressure; (4) rotation speed; (5) temperature in the hydraulic unit; and (6) torque. The dataset contains six time series, one for each sensor, with the values stored in the database from 2016 to August 2020. Figure 2shows a plot of the six time series, before any processing is applied. These data present some upper and lower extremes, which may be discrepant data. Those discrepant samples may be due to reading errors or periods when the equipment was off or in another atypical state. Figure 2. Plot of the original dataset values. The variables are electric current intensity, hydraulic unit oil level, VAT pressure, motor velocity, temperature at the hydraulic unit, and torque. Some of the samples, such as those when the equipment was off but the sensors were still reading, can compromise the training of the machine learning models to be developed. Table 1shows some statistical parameters such as mean, standard deviation (std), minimum, third quantiles, and maximum value. Appl. Sci. 2021,11, 6101 7 of 16 Table 1. Statistical parameters of the dataset variables, before processing: C. intensity, hydraulic unit oil level, torque, VAT pressure, velocity, and temperature. C. Intensity Hydraulic Torque VAT Velocity Temperature mean 30.26 75.90 15.28 18.25 4.59 38.22 std 1.36 4.54 0.69 2.67 0.98 1.62 min 26.34 62.93 13.59 9.67 1.27 33.19 Q1—25% 29.30 72.86 14.90 17.13 3.92 37.17 Q2—50% 30.46 75.53 15.43 18.72 4.57 38.33 Q3—75% 31.28 79.52 15.78 19.97 5.28 39.35 max 34.26 88.97 17.09 26.17 7.87 43.10 4.2. Cleaning Discrepant Data In order to facilitate the training process, discrepant samples were identified and removed using the quantiles method. Samples which are beyond the Q1− 3 ×std or Q3+ 3 ×std are replaced by the mean value. The extreme values were replaced with the average. Figure 3shows the same variables after discrepant data samples were removed. As the figure shows, the lines are now smoother and easier to read. Figure 4shows that the samples are evenly distributed after the withdrawal of discrepant data. Figure 3. Plot of the dataset values after cleaning discrepant data. The variables are current intensity, hydraulic unit oil level, VAT pressure, velocity, temperature, and torque. Figure 4. Distribution of data points of all the sensors, with lowly and highly discrepant data cleaned. The predictive models to be used are robust and tolerant to noise. However, the cleaner data are expected to show better results. As an example, a provisional experiment to train Appl. Sci. 2021,11, 6101 8 of 16 a neural network LSTM model with a historical window of 70 samples and 40 LSTM unit cells showed higher and undetermined errors. The model was not able to learn or predict some variables, as shown in Table 2. With clean data, there were better and determinable results, as shown in Table 3. The tables show the MAPE and MAE for all input variables, as determined in the test set. They also show the RMSE, as calculated in the train and test sets, globally for all variables. Table 2. Prediction results without cleaning discrepant data in the database, with a window of 70 samples and 40 LSTM units. Window 70 Days C. Intensity Hydraulic Torque VAT Velocity Temperature MAPE inf 8.46 inf 98.19 inf 11.59 MAE 3.52 6.57 24.73 10.53 14.88 4.21 Train Test RMSE 79.52 79.64 Table 3. Forecast results with treatments in the database with 40 LSTM units. Window 70 Days C. Intensity Hydraulic Torque VAT Velocity Temperature MAPE 2.52 3.02 2.44 13.10 inf 2.48 MAE 0.76 2.28 0.37 1.32 0.57 0.94 Train Test RMSE 1.71 1.97 5. Experiments and Results Experiments were performed with the aim of validating the model that has the best performance in predicting data from the industrial press. The tests are divided into two subsections, first with resampling of data to one sample per day and then with resampling for a sample each 12 h. 5.1. LSTM Models and Dataset Partition After processing the data, experiments were performed with an LSTM model. The model included an encoder and decoder, with one hidden LSTM layer in the middle and a dense layer at the output. The model was used to train and predict, with six variables that represent data coming from the paper press sensors. The goal was to forecast the value of those variables with the highest possible level of confidence so that it brings added benefits in predictive maintenance. Figure 5describes the architecture of one of the network models used. The models were implemented in Python using the TensorFlow library and Keras. The experiments were performed aiming to obtain a prediction for all variables one month in advance, from a window of a number of past samples. The LSTM models received, as an input, a sequence consisting of the composition of a number of samples for each variable. The number of samples depended on the window size and the resampling rate used. The output sequence is composed of the values predicted for each of the variables. Appl. Sci. 2021,11, 6101 9 of 16 Figure 5. Model summary of one of the LSTM networks used. The model receives a window of n samples of each variable and predicts the value of those variables as predicted 30 days ahead. To train and test the models, the dataset was divided into train and test subsets. Validation was performed using the test set, but those samples were not incorporated into the training set. The training set contained 85% of the samples and the test set the remaining 15% of samples. These values are adequate for convergence during learning. As an example, Figure 6shows a learning curve for a model with 70 units in the middle layer and a window of 30 lag samples. The figure shows that learning converges and takes fewer than 10 epochs. The remainder experiments were performed using 100 epochs. Figure 6. Example of learning curve, showing the loss measured during training of an LSTM model. 5.2. Experiments to Determine Historical Window Size and Number of LSTM Units Using One Sample per Day The first experiments performed aimed to determine the best window size to use. The smaller the window, the smaller and faster the model that can be used. However, if the window is too small, it may be insufficient to make accurate predictions. The original dataset had 1,445,760 data points, which is very large and would require a lot of memory and time to train and test. The experiments were performed after downsampling the data, so that there is only one sample per day. That sample is the average of 1004 original samples. The downsampled dataset is, therefore, less than the one thousand of the original dataset. The results are measured in the test set. The figure above shows the MAPE and MAE measured for each variable. It also shows the global RMSE measured globally for the train and test sets. As Figure 7shows, models with windows of 40 and 50 samples allow better learning and produce smaller prediction errors. Appl. Sci. 2021,11, 6101 16 of 16 42. Park, S.H.; Kim, B.; Kang, C.M.; Chung, C.C.; Choi, J.W. Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture. In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China, 26–30 June 2018; pp. 1672–1678. [CrossRef] 43. Cho, K.; van Merrienboer, B.; Gülçehre, Ç.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv 2014, arXiv:1406.1078. 44. Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to Sequence Learning with Neural Networks. arXiv 2014, arXiv:1409.3215. 45. Wang, T.; Chen, P.; Amaral, K.; Qiang, J. An Experimental Study of LSTM Encoder-Decoder Model for Text Simplification. arXiv 2016, arXiv:1609.03663. 46. Bengio, S.; Vinyals, O.; Jaitly, N.; Shazeer, N. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. arXiv 2015, arXiv:1506.03099. 47. Du, S.; Li, T.; Yang, Y.; Horng, S.J. Multivariate time series forecasting via attention-based encoder–decoder framework. Neurocomputing 2020,388, 269–279. [CrossRef] 48. Gers, F.A.; Eck, D.; Schmidhuber, J. Applying LSTM to Time Series Predictable Through Time-Window Approaches. In Neural Nets WIRN Vietri-01; Tagliaferri, R., Marinaro, M., Eds.; Perspectives in Neural Computing; Springer: Berlin/Heidelberg, Germany, 2002; pp. 193–200._20. [CrossRef] 49. Meng, Q.; Wang, H.; He, M.; Gu, J.; Qi, J.; Yang, L. Displacement prediction of water-induced landslides using a recurrent deep learning model. Eur. J. Environ. Civ. Eng. 2020. [CrossRef] 50. Sarikaya, R.; Hinton, G.E.; Deoras, A. Application of Deep Belief Networks for Natural Language Understanding. IEEE/ACM Trans. Audio Speech Lang. Process. 2014,22, 778–784. [CrossRef] 51. Sundermeyer, M.; Schlüter, R.; Ney, H. LSTM neural networks for language modeling. In Proceedings of the Thirteenth Annual Conference of the International Speech Communication Association, Portland, OR, USA, 9–13 September 2012. 52. Hu, Y.; Sun, X.; Nie, X.; Li, Y.; Liu, L. An Enhanced LSTM for Trend Following of Time Series. IEEE Access 2019 ,7, 34020–34030. [CrossRef] 53. Gers, F.A.; Schmidhuber, J.; Cummins, F. Learning to Forget: Continual Prediction with LSTM. In Proceedings of the 1999 Ninth International Conference on Artificial Neural Networks ICANN 99 (Conf. Publ. No. 470), Edinburgh, UK, 7–10 September 1999; pp. 850–855. [CrossRef] 54. Greff, K.; Srivastava, R.K.; Koutník, J.; Steunebrink, B.R.; Schmidhuber, J. LSTM: A Search Space Odyssey. IEEE Trans. Neural Netw. Learn. Syst. 2017,28, 2222–2232. [CrossRef] [PubMed] 55. Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997,9, 1735–1780. [CrossRef] [PubMed] 56. Li, C.; Jiang, P.; Zhou, A. Rigorous solution of slope stability under seismic action. Comput. Geotech. 2019 ,109, 99–107. [CrossRef] 57. Li, C.; Hu, B.C.; Hu, D.; Xu, X.F.; Zong, X.C.; Li, J.P.; Wu, M.C. Stereoselective ring-opening of styrene oxide at elevated concentration by Phaseolus vulgaris epoxide hydrolase, PvEH2, in the organic/aqueous biphasic system. Catal. Commun. 2019 , 123, 1–5. [CrossRef] 58. Qin, Y.; Li, K.; Liang, Z.; Lee, B.; Zhang, F.; Gu, Y.; Zhang, L.; Wu, F.; Rodriguez, D. Hybrid forecasting model based on long short term memory network and deep learning neural network for wind signal. Appl. Energy 2019,236, 262–272. [CrossRef] 59. Zhang, Z.; Ye, L.; Qin, H.; Liu, Y.; Wang, C.; Yu, X.; Yin, X.; Li, J. Wind speed prediction method using Shared Weight Long Short-Term Memory Network and Gaussian Process Regression. Appl. Energy 2019,247, 270–284. [CrossRef] 60. Zhao, R.; Yin, Y.; Shi, Y.; Xue, Z. Intelligent intrusion detection based on federated learning aided long short-term memory. Phys. Commun.2020,42, 101157. [CrossRef] 61. Kong, X.; Kong, D.; Bai, L.; Xiao, J. Online pricing of demand response based on long short-term memory and reinforcement learning. Appl. Energy 2020,271, 114945. [CrossRef] 62. Li, Y.; Lu, Y. LSTM-BA: DDoS Detection Approach Combining LSTM and Bayes. In Proceedings of the 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD), Suzhou, China, 21–22 September 2019; pp. 180–185. [CrossRef] 63. Alameer, Z.; Fathalla, A.; Li, K.; Ye, H.; Jianhua, Z. Multistep-ahead forecasting of coal prices using a hybrid deep learning model. Resour. Policy 2020,65, 101588. [CrossRef]