Machine Learning-Based Node Characterization for Smart Grid Demand Response Flexibility Assessment

sustainability Article Machine Learning-Based Node Characterization for Smart Grid Demand Response Flexibility Assessment Rostislav Krˇc 1, Martina Kratochvílová 1, Jan Podroužek 1,2,* , Tomáš Apeltauer 1and Václav Stupka 2 and Tomáš Pitner 2   Citation: Krˇc, R.; Kratochvílová, M.; Podroužek, J.; Apeltauer, T.; Stupka, V.; Pitner, T. Machine Learning-Based Node Characterization for Smart Grid Demand Response Flexibility Assessment. Sustainability 2021,13, 2954. https://doi.org/10.3390/ su13052954 Academic Editor: Mehdi Seyedmahmoudian, Alex Stojcevski, Ben Horan, Saad Mekhilef and Eklas Hossain Received: 26 January 2021 Accepted: 1 March 2021 Published: 9 March 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1Institute of Computer Aided Engineering and Computer Science, Faculty of Civil Engineering, Brno University of Technology, 602 00 Brno, Czech Republic; rostislav[email protected] (R.K.); [email protected] (M.K.); [email protected] (T.A.) 2Centre for Education, Research and Innovation in Information and Communication Technologies-ExecUnit, Faculty of Informatics, Masaryk University, 602 00 Brno, Czech Republic; [email protected] (V.S.); [email protected] (T.P.) *Correspondence: podr[email protected].cz Abstract: As energy distribution systems evolve from a traditional hierarchical load structure towards distributed smart grids, flexibility is increasingly investigated as both a key measure and core challenge of grid balancing. This paper contributes to the theoretical framework for quantifying network flexibility potential by introducing a machine learning based node characterization. In particular, artificial neural networks are considered for classification of historic demand data from several network substations. Performance of the resulting classifiers is evaluated with respect to clustering analysis and parameter space of the models considered, while the bootstrapping based statistical evaluation is reported in terms of mean confusion matrices. The resulting meta-models of individual nodes can be further utilized on a network level to mitigate the difficulties associated with identifying, implementing and actuating many small sources of energy flexibility, compared to the few large ones traditionally acknowledged. Keywords: smart grid; electricity network; flexibility assessment; renewable energy sources; machine learning; network simulation; artificial neural networks; convolutional neural networks 1. Introduction Safe and reliable operation of energy systems depends on maintaining a balance between consumption and production in real-time, while an increasingly large part of the production portfolio depends on the inherently variable non-stationary climate, such as renewable wind or solar sources [1]. The variability of the consumption, on the other hand, can be relatively easily controlled, when compared to the climate-dependent part of the production portfolio, by purposefully influencing the consumption behavior, i.e., targeting the reducible (epistemic) uncertainty components on the consumer side, rather than focusing only on mostly aleatory (irreducible) uncertainties related to long-term weather forecasting. The ability to purposefully influence both the production and consumption behavior of selected elements of the electricity system is therefore gradually gaining in importance (e.g., [ 1 – 4 ], as it can effectively reduce fluctuations in the overall load diagram and thus reduce the demands on available power and dynamics of Support Services, as well as the associated costs. The electricity market is currently at a crossroads. The current market model assumes that the market will ensure both short-term optimization, such as effective allocation of the necessary production among existing capacities, and long-term investment signals for the construction of new capacities. However, the significant degree of market distortions in the sector practically paralyzed this function of the market model. Such development Sustainability 2021,13, 2954. https://doi.org/10.3390/su13052954 https://www.mdpi.com/journal/sustainability Sustainability 2021,13, 2954 2 of 18 leads to a situation where investors are only looking for the construction of sources with guaranteed (subsidized) prices. Investments in resources and networks are thus driven by state incentives, instead of the market. Under such conditions, market development without adjustments by the state leads to an unbalanced resource mix with a number of strategic and systemic risks for the future [1]. An important part of any state’s critical infrastructure is its electricity network. This is traditionally based on centralized generation in large power plants, however, as the share of renewable energy production increases, the grid will need to adapt to a large number of smaller sources. Decentralized production growth is enabled by the spread of new technologies and typically benefits the local economy. However, the transformation of energy must meet the basic conditions. These are secure supplies during normal operation as well as in the event of sudden changes in external conditions, and competitive prices. At the same time, energy must be sustainable in the sense that it does not harm the environment, is able to provide raw materials for its operation and the whole sector is economically stable. The aim of this work is to contribute to the theoretical framework for quantifying network flexibility potential by introducing a machine learning-based node characterization. It is unique in the successful utilization of state-of-the-art convolutional neural network models for the classification of historic demand data from the Ausgrid distribution zone substation data. After introducing the related concepts of smart grid, grid flexibility and network modeling, demand interval data used for this study are introduced together with the clustering analysis performed. Next, machine learning-based time-series classification and surrogate resampling concepts are discussed, together with various architectures of convolutional neural network models. Finally, the statistics of resulting classifiers are discussed. 1.1. Smart Grid Decentralizing the energy system and thus at least partially replacing large-scale energy production (e.g., fossil, nuclear or hydroelectric) is an increasingly common effort. These facilities are usually far from the end consumer and therefore require an extensive and reliable high-voltage transmission network. The global tendency to achieve a sustainable economy and improve the environment leads to higher use of energy from renewable sources and thus, for example, to reduce the global temperature disruption [ 5 ]. Carbon dioxide emissions during energy production account for about two-thirds of all greenhouse gases [ 6 ]. Power plants using renewable sources are usually smaller in format and closer to the end-user. Thus, energy is not transmitted over such long distances and the transmission network has a decentralized structure. This results in fewer losses during transmission and the network is less vulnerable because it does not depend on a small number of remote large power plants. The whole system is therefore composed of smaller subsystems, which do not have to be interdependent, but still communicate with each other and can help each other. An ideal (smart) grid is a modernized electrical self-monitoring grid that can combine conventional central sources with alternative sources of electricity [ 7 ]. This includes an intelligent control system that monitors and adjusts the operation of the network in realtime, including the self-healing capabilities and supported by intelligent elements, without the need for human intervention. Smart grids communicate with the customer in real-time and help to optimize the consumption with regard to the current price of electricity and the burden on the environment, allowing better integration of renewable electricity sources and improving the efficiency, reliability, economics, and sustainability of the production and distribution of electricity. As the production of energy using renewable sources is difficult to predict in the long run (climate-dependent production), it is necessary to be able to target consumers and ensure communication between individual entities. In this way, the demands on the peak loads and the operational cost and costs of providing support services can be reduced [8]. Sustainability 2021,13, 2954 3 of 18 This can be done through a combination of technical and economic tools. Various smart metrics for measuring, communication, synchronization, forecasting and control are being developed [ 9 ]. Among the economic instruments, it is possible to name, for example, the real-time pricing [ 10 ] or the Adaptive Billing Mechanism [ 11 ], which can work, with negative energy prices and thus flatten the oscillation of the overall load diagram. A whole new market with new entrants can be expected. Information security is discussed in [ 12 ], where risk propagation model based on the Susceptible–Exposed–Infected–Recovered (SEIR) infectious disease model is proposed for a smart grid. The high volatility and uncertainty of load profiles and the tremendous communication pressure are discussed in a two-stage household electricity demand estimation study by [ 13 ]. Investigation of voltage control at consumers connection points based on smart approach has recently been carried out by [ 14 ], proposing a voltage control system for use in the Russian distribution grid. 1.2. Grid Flexibility Flexibility is considered a key enabler for the smart grid according to O’Connell et al. [15] , and is required to facilitate Demand-Side Management (DSM) programs, manage electrical consumption to reduce peaks, balance renewable generation and provide ancillary services to the grid. The ISO 50002:2014 [ 16 ] specifies the process requirements for carrying out an energy audit in relation to energy performance. It is applicable to all types of establishments and organizations, and all forms of energy and energy use. This standard can be used to assess flexibility and formulate optimization requirements [ 2 , 15 ]. According to a given scale, flexibility analysis can help to identify and quantify the available electrical load at a network or node level, i.e., substation, site or building. U.S. Energy Information Administration (EIA) [ 17 ] defines DSM programs as those including planning, implementing, and monitoring activities of electric utilities which are designed to encourage consumers to modify their level and pattern of electricity usage. In its international energy outlook or other EIA annual reports, projected and actual energy production can be compared with the global changes in manufacturing and services share, an important component in any flexibility analysis for smart grid DSM. The primary objective of most DSM programs in the past was to provide cost-effective energy and capacity resources in order to help defer the need for new sources of power, including generating facilities, power purchases, and transmission and distribution capacity additions. However, due to changes that are occurring within the industry, electric utilities are also using DSM as a way to enhance customer service. According to EIA, DSM refers to only energy and load-shape modifying activities that are undertaken in response to utility-administered programs. It does not refer to energy and load-shape changes arising from the normal operation of the marketplace or from government-mandated energyefficiency standards. Moreover, the European Commission’s (EC) 2020 targets [ 18 ] to generate 20% of Europe’s energy from renewable energy and reduce greenhouse gasses emissions by 20% have already resulted in increased climate-dependent production. In order to further increase this production to 25%, all aspects of grid flexibility have to be carefully addressed to ensure grid resilience and stability. This includes, among others, the ability to balance non-dispatchable sources and managing the power locally. In the International Energy Agency’s annex 67 [ 19 ], energy flexibility is also presented as a key asset in the smart building future, where buildings can manage itself, interact with their users and take part in demand response. 1.3. Network Modeling In the current state of discussions, energy flexibility is typically associated with “smartness” and evaluated either in a qualitative framework according to the number and type of services provided by its components, or, as presented in this paper, by quantitative and physical indicators, utilizing measured (historic) data and network-level simulations. Sustainability 2021,13, 2954 4 of 18 To better understand the functioning of Smart grids and investigate the possibilities of optimization of their functions, it is appropriate to create a mathematical meta-model of individual network nodes and simulate the operation of the whole network, where the nodes are connected according to real network topology and edge capacity. Because it is a decentralized system, decision intelligence is divided between the individual nodes of the network. Agent models are commonly used for simulations, where each agent has its own decision-making power and none of them depends on any central authority [ 20 ]. Relationships and connections between agents are usually modeled using network theory [ 21 ]. The agent decision-making process and behavior prediction can be modeled, for example, using machine learning (ML) [22]. Among recent contributions to the integrated simulation of power and communication networks for smart grid, applications can be found [ 4 ], where the smart grid discreteevent simulator is implemented in C++ using the open-source OMNeT++ simulation environment. In [ 23 ], a comprehensive real-time simulation of the smart grid is presented, including a microgrid model of a small community. A recent overview of simulation and modeling application to residential demand response can be found in [24]. In order to simulate the behavior of the entire network, and to evaluate the impact of various control strategies on the power grid, it is necessary to validate the behavior of the individual nodes first. In this paper, a historic 15 min interval demand data from Ausgrid substations [3,25] have been classified using machine learning methods. 2. Interval Demand Data Publicly available distribution zone substation 15-min interval demand data from the Australian network operator Ausgrid [ 25 ] have been used for the machine learning-based node characterization in order to support computational reproducibility of this research. In particular, historic data from the year 2019 (between May 2018 and April 2019) and from the year 2020 (between May 2019 and April 2020) from 185 substations from distribution networks around Sydney and Newcastle have been used. These substations form the boundary between the sub-transmission network and the distribution (11 kV) network. The time is in Australian Eastern Standard Time (AEST) format during the winter period and Australian Eastern Daylight Time (AEDT) during the summer period. Figure 1shows the irregular topology of the investigated distribution network, where real node positions (red dots) correspond to population density clusters, resulting in large variability of edge length (black lines). Data from individual substations were sampled at 15 min and divided into time series by days, resulting in 96 data points. The proposed classification cannot intentionally distinguish between workdays, weekends or holidays, as there is no information on the actual date attached to the individual time series, although the dataset exhibit typical daily, weekly and seasonal fluctuations in electricity demand. Nevertheless, the achieved accuracies are far from prohibitive, as discussed in the next chapter. The daily fluctuations include morning and afternoon peaks throughout all four seasons, including workdays and weekends, see Figures 2–5for an example from Harbord substation, with group averages in dark color. In summer, a lower overall demand can be observed, including less distinct morning and evening peaks. Sustainability 2021,13, 2954 5 of 18 Figure 1. Irregular topology of the investigated distribution network (map source [26]). Clustering Analysis In general, the goal of clustering is to identify structure in an unlabeled data set by objectively organizing data into homogeneous groups where the within-group-object similarity is minimized and the between-group-object dissimilarity is maximized [ 27 ]. The time-series demand data presented in this section have been clustered by the basic k-means algorithm [ 28 ] in order to split substations into several groups. K-means clustering is a renowned heuristic method for crisp partitions (i.e., each object belongs to exactly one cluster, as opposed to fuzzy if one object is allowed to be in more than one cluster to a different degree), where each cluster is represented by the mean value of the objects in the cluster. Sustainability 2021,13, 2954 6 of 18 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Power demand [MW] Weekday Weekend Figure 2. Daily fluctuations and seasonal scattering at Harbord substation, May–July 2018. 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 4 6 8 10 12 14 16 18 20 Power demand [MW] Weekday Weekend Figure 3. Daily fluctuations and seasonal scattering at Harbord substation, August–October 2018. Sustainability 2021,13, 2954 7 of 18 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 4 6 8 10 12 Power demand [MW] Weekday Weekend Figure 4. Daily fluctuations and seasonal scattering at Harbord substation, November 2018–January 2019. 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 4 6 8 10 12 14 16 Power demand [MW] Weekday Weekend Figure 5. Daily fluctuations and seasonal scattering at Harbord substation, February–April 2019. For effective optimization of the distribution grid, and for any network-level simulations in general, the behavior of nodes (i.e., stations and substations) must be understood. Daily fluctuations of power demand may depend on the type of supplied neighborhood. Three basic types of substation neighborhoods—residential, business and combined—were Sustainability 2021,13, 2954 8 of 18 considered as only 96 equally spaced data points were available per signal. K-means clustering analysis with k = 3 identified the following clusters (Figures 6–8): • C1: 48 substations, residential, morning and evening demand peak, • C2: 66 substations, combined, morning and evening peaks less distinct than in the clusters C1, • C3: 71 substations, business, high and flat distribution of energy demand during the day. Note that the qualitative evaluation of characteristics of the three identified clusters is based only on the assumption of daily peak distribution and has not been verified by any other on-site investigation as it was not the goal of this research. Note the normalization of amplitudes of daily signals to the maximum value in order to ensure classification based on demand patterns instead of absolute values of demand. 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Normalized power demand [-] Figure 6. Cluster C1 with average daily power demand for each substation (grey) and overall average demand (blue). Sustainability 2021,13, 2954 9 of 18 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Normalized power demand [-] Figure 7. Cluster C2 with average power demand for each substation (grey) and overall average demand (blue). 00:00 03:00 06:00 09:00 12:00 15:00 18:00 21:00 00:00 Time 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Normalized power demand [-] Figure 8. Cluster C3 with average power demand for each substation (grey) and overall average demand (blue). Sustainability 2021,13, 2954 16 of 18 (4) Certain classes may be mutually confused due to similarities in substations and their neighbourhoods. Some types of neighbourhood such as tourist or holiday locations may also increase demand variability in different parts of a year. The mutual confusions are similar for all investigated CNN models, and since these models have a different number of convolutional layers, the confusion is more likely to stem from substations variability rather than from the different architecture of CNN. For example, Swansea has fishing and tourism, Campsie has business and commercial areas, while Darlinghurst is a vibrant central district dependent on high season. Limitations of this study include a limited range of considered years due to computational intensity. In order to optimize the distribution grid as a whole, network topology including its inner dependencies and boundary conditions must be considered (e.g., using hierarchical neural networks). In this study, only classification of network nodes was presented. 6. Conclusions After introducing the importance of quantifying network flexibility potential for the safe integration of renewable energy sources and sustainable economy, the background on the smart grid and network modeling has been presented together with state-of-the-art machine learning techniques and their application to the classification of historical demand data. The resulting characterization of individual substations is important for future work on network-level simulations, the aim of which is to mitigate the difficulties associated with identifying, implementing and actuating many small sources of energy flexibility, compared to the few large ones traditionally acknowledged. The proposed CNN models do not require any pre-processing of the 15 min interval demand data, the only subjective choice associated with the classification is the architecture of the neural network. Three scenarios were evaluated considering data from year 2019, 2020 and both years respectively. In the first two scenarios, data were divided between training and testing set using a typical split ratio 80/20, and in the third scenario, denoted as 2019–2020, data from the year 2019 were used for training and data from the year 2020 for testing. This enabled the verification of both statistical significance of the classifier, based on bootstrapping, as well as generalization of the resulting meta-models. The sparse mean confusion matrices indicate a robust modeling approach, considering very similar structures across the investigated architectures, and relatively small standard deviation of accuracies. A more detailed (qualitative) assessment of substations and their neighbourhoods was beyond the scope of this paper, as well as the effects of boundary conditions on real network topology. Future work will continue with hierarchical neural networks modeling of the entire network. Author Contributions: Conceptualization, methodology and writing J.P. and R.K.; programming, validation R.K. and J.P.; formal analysis, M.K.; project administration, funding acquisition and review, T.P., V.S. and T.A.; All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by The SecureFlex project TK01030078 and by the TJ04000232 project, both supported by the Technology Agency of the Czech Republic. Data Availability Statement: Publicly available datasets were analyzed in this study. This data can be found here: https://www.ausgrid.com.au/Industry/Our-Research/Data-to-share/Distributionzone-substation-data (accessed on 11 January 2021). Conflicts of Interest: The authors declare no conflict of interest. References 1. Stˇrelec, M.; Mamula, O.; Pitner, T.; Kasl, F.; Hrycej, D. Technical aspects of usability aggregated flexibility for business and technical services. Energetika 2020,70, 1–20. 2. O’Connell, S.; Reynders, G.; Seri, F.; Sterling, R.; Keane, M.M. A standardised flexibility assessment methodology for demand response. International Journal of Building Pathology and Adaptation 2019,38, 20–37. [CrossRef] Sustainability 2021,13, 2954 17 of 18 3. Xenophon, A.; Hill, D. Open grid model of Australia’s National Electricity Market allowing backtesting against historic data. Sci. Data 2018,5, 180203. [CrossRef] 4. Mets, K.; Verschueren, T.; Develder, C.; Vandoorn, T.L.; Vandevelde, L. Integrated simulation of power and communication networks for smart grid applications. In Proceedings of the 2011 IEEE 16th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Kyoto, Japan, 10–11 June 2011; pp. 61–65. 5. Gielen, D. Perspectives for the Energy Transition Investment Needs for a Low-Carbon Energy System. 2017. Available online: https://iki-alliance.mx/de/download/Mexirec%20decarbonization(4).pdf (accessed on 11 January 2021). 6. Field, C.B. Climate Change 2014–Impacts, Adaptation and Vulnerability: Regional Aspects; Cambridge University Press: Cambridge, UK, 2014. 7. EUROPA–Smart Grids|SETIS–European Commission. Available online: https://setis.ec.europa.eu/publications/setismagazine/smart-grids (accessed on 5 January 2021). 8. Shayeghi, H.; Shahryari, E.; Moradzadeh, M.; Siano, P. A Survey on Microgrid Energy Management Considering Flexible Energy Sources. Energies 2019,12, 2156. [CrossRef] 9. Avancini, D.B.; Rodrigues, J.J.P.C.; Martins, S.G.B.; Rabêlo, R.A.L.; Al-Muhtadi, J.; Solic, P. Energy meters evolution in smart grids: A review. J. Clean. Prod. 2019,217, 702–715. [CrossRef] 10. Tabors, R.D.; Parker, G.; Caramanis, M.C. Development of the smart grid: Missing elements in the policy process. In Proceedings of the 2010 43rd Hawaii International Conference on System Sciences, Kauai, HI, USA, 5–8 January 2010; pp. 1–7. 11. At˘an˘asoae, P.; Pentiuc, R.D.; Hopulele, E. Considerations Regarding the Negative Prices on the Electricity Market. Proceedings 2020,63, 26. [CrossRef] 12. Zhu, B.; Deng, S.; Xu, Y.; Yuan, X.; Zhang, Z. Information security risk propagation model based on the SEIR infectious disease model for smart grid. Information 2019,10, 323. [CrossRef] 13. Liu, Y.; Sun, Y.; Li, B. A two-stage household electricity demand estimation approach based on edge deep sparse coding. Information 2019,10, 224. [CrossRef] 14. Vanin, A.; Aleshin, S.; Nasirov, R.; Novikov, D.; Tulsky, V. Investigation of voltage control at consumers connection points based on smart approach. Information 2016,7, 42. [CrossRef] 15. O’Connell, S.; Riverso, S. Flexibility analysis for smart grid demand response. arXiv 2017, arXiv:1704.01308. 16. ISO, I. 50002: 2014 Energy Audits—Requirements with Guidance for Use; ISO: Geneva, Switzerland, 2014. 17. Capuano, L. International energy outlook 2018 (IEO2018); US Energy Information Administration (EIA): Washington, DC, USA, 2018; Volume 2018, p. 21. 18. Vinois, J. DG ENER Working Paper: The Future Role and Challenges of Energy Storage. 2012. Available online: https: //ec.europa.eu/energy/sites/ener/files/energy_storage.pdf (accessed on 9 January 2021). 19. Jensen, S.Ø.; Madsen, H.; Lopes, R.; Junker, R.; Aelenei, D.; Li, R. Annex67: Energy Flexible Buildings-Energy Flexibility as a Key Asset in a Smart Building Future; IEA Energy in Buildings and Communities Program (EBC): Birmingham, UK, 2017. 20. Monroe, J.G.; Hansen, P.; Sorell, M.; Berglund, E.Z. Agent-Based Model of a Blockchain Enabled Peer-to-Peer Energy Market: Application for a Neighborhood Trial in Perth, Australia. Smart Cities 2020,3, 1072–1099. [CrossRef] 21. Nakayama, K.; Shinomiya, N. Distributed control based on tie-set graph theory for smart grid networks. In International Congress on Ultra Modern Telecommunications and Control Systems; IEEE: Moscow, Russia, 2010; pp. 957–964. [CrossRef] 22. Naz, A.; Javed, M.U.; Javaid, N.; Saba, T.; Alhussein, M.; Aurangzeb, K. Short-Term Electric Load and Price Forecasting Using Enhanced Extreme Learning Machine Optimization in Smart Grids. Energies 2019,12, 866. [CrossRef] 23. Guo, F.; Herrera, L.; Murawski, R.; Inoa, E.; Wang, C.L.; Beauchamp, P.; Ekici, E.; Wang, J. Comprehensive real-time simulation of the smart grid. IEEE Trans. Ind. Appl. 2013.49, 899–908. [CrossRef] 24. Wang, Z.; Paranjape, R.; Sadanand, A.; Chen, Z. Residential demand response: An overview of recent simulation and modeling applications. In Proceedings of the 2013 26th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Regina, SK, Canada, 5–8 May 2013; pp. 1–6. 25. Distribution Zone Substation Data—Ausgrid. Available online: https://www.ausgrid.com.au:443/Industry/Our-Research/ Data-to-share/Distribution-zone-substation-data (accessed on 30 December 2020). 26. OpenStreetMap. Available online: https://www.openstreetmap.org/copyright/en (accessed on 8 January 2021). 27. Liao, T.W. Clustering of time series data—A survey. Pattern Recognit. 2005,38, 1857–1874. [CrossRef] 28. MacQueen, J.; others. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Oakland, CA, USA, 21 June–18 July 1967; Volume 1, pp. 281–297. 29. Franke, J.; Neumann, M.H. Bootstrapping neural networks. Neural Comput. 2000,12, 1929–1949. [CrossRef] [PubMed] 30. Krˇc, R.; Podroužek, J.; Kratochvílová, M.; Vukušiˇc, I.; Plášek, O. Neural Network-Based Train Identification in Railway Switches and Crossings Using Accelerometer Data. J. Adv. Transp. 2020,2020. [CrossRef] 31. Kratochvílová, M.; Podroužek, J.; Apeltauer, J.; Vukušiˇc, I.; Plášek, O. Train Type Identification at S&C. J. Adv. Transp. 2020 ,2020. [CrossRef] 32. Yoon, H.; Hyun, Y.; Ha, K.; Lee, K.K.; Kim, G.B. A method to improve the stability and accuracy of ANN-and SVM-based time series models for long-term groundwater level predictions. Comput. Geosci. 2016,90, 144–155. [CrossRef] 33. Ghaderpour, E.; Pagiatakis, S.D. LSWAVE: a MATLAB software for the least-squares wavelet and cross-wavelet analyses. GPS Solut. 2019,23, 50–58. [CrossRef] Sustainability 2021,13, 2954 18 of 18 34. Podrouzek, J.; Bucher, C.; Deodatis, G. Identification of critical samples of stochastic processes towards feasible structural reliability applications. Struct. Saf. 2014,47, 39–47. [CrossRef] 35. Ismail Fawaz, H.; Forestier, G.; Weber, J.; Idoumghar, L.; Muller, P.A. Deep learning for time series classification: a review. Data Min. Knowl. Disc. 2019,33, 917–963. [CrossRef]