Network intrusion detection based on extended RBF neural network with offline reinforcement learning

Received October 14, 2021, accepted November 5, 2021, date of publication November 11, 2021, date of current version November 19, 2021. Digital Object Identifier 10.1109/ACCESS.2021.3127689 Network Intrusion Detection Based on Extended RBF Neural Network With Offline Reinforcement Learning MANUEL LOPEZ-MARTIN 1, (Senior Member, IEEE), ANTONIO SANCHEZ-ESGUEVILLAS 1, (Senior Member, IEEE), JUAN IGNACIO ARRIBAS 1,2, AND BELEN CARRO 1 1Department of TSyCeIT, ETSIT, University of Valladolid, 47011 Valladolid, Spain 2Castilla-Leon Neuroscience Institute, University of Salamanca, 37007 Salamanca, Spain Corresponding author: Manuel Lopez-Martin ([email protected]) This work was supported in part by the Proyectos de I+D+iRetos investigación, Programa Estatal de I+D+i Orientada a los Retos de la Sociedad, Plan Estatal de Investigación Científica, Técnica y de Innovación 2017–2020 under Grant RTI2018-098958-B-I00; in part by the Spanish Ministry for Science, Innovation and Universities; in part by the Agencia Estatal de Investigación (AEI); and in part by the Fondo Europeo de Desarrollo Regional (FEDER). ABSTRACT Network intrusion detection focuses on classifying network traffic as either normal or attack carrier. The classification is based on information extracted from the network flow packets. This is a complex classification problem with unbalanced datasets and noisy data. This work extends the classic radial basis function (RBF) neural network by including it as a policy network in an offline reinforcement learning algorithm. With this approach, all parameters of the radial basis functions (along with the network weights) are learned end-to-end by gradient descent without external optimization. We further explore how additional dense hidden-layers, and the number of radial basis kernels influence the results. This novel approach is applied to five prominent intrusion detection datasets (NSL-KDD, UNSW-NB15, AWID, CICIDS2017 and CICDDOS2019) achieving better performance metrics than alternative state-of-the-art models. Each dataset provides different restrictions and challenges allowing a better validation of results. Analysis of the results shows that the proposed architectures are excellent candidates for designing classifiers with the constraints imposed by network intrusion detection. We discuss the importance of dataset imbalance and how the proposed methods may be critically important for unbalanced datasets. INDEX TERMS Communication system security, intrusion detection, neural networks, radial basis function networks. I. INTRODUCTION Network intrusion detection (NID) focuses on classifying network traffic as either normal or attack carrier. The classification is based on information extracted from the network flow packets. It is a major problem for modern data networks with an active research community currently focused on applying machine learning and deep learning models [1]–[4]. Radial Basis Function Neural Networks (RBFNN) [5] are variants of the feed-forward Neural Network (NN) architecture. RBFNNs has been used for NID in their basic configuration. A basic RBFNN classifier is made of a single hidden-layer NN where each neuron implements a radial basis activation The associate editor coordinating the review of this manuscript and approving it for publication was Christian Esposito . function (RBF), usually a Gaussian function. In addition to the network weights, these networks have two additional types of parameters that need to be optimized: (a) the center of the RFB functions, and (b) its dispersion/scaling parameter. The optimization of these three types of parameters: centers, dispersion factors and weights, are generally carried out following specific methods for each of them [6], causing a complex training process. The optimization methods used for estimating centers and dispersion factors are usually clustering techniques (e.g., k-means) or generic optimization methods, such as: Immune Radial Basis Function (IRBF), Particle swarm optimization (PSO) and Quantum PSO (QPSO). For network weights, the optimization methods used are least squares or gradient descent. This nonintegrated and laborious training process makes it difficult to increase the complexity VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ 153153 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning of the network, resulting in the use of simple network architectures consisting of a single hidden layer network (basic RBFNN). We propose a novel extension of the RBFNN basic model. The RBFNN basic architecture is based on a single hidden-layer where each neuron implements a radial basis activation function e.g., Gaussian. These functions are also called radial kernels, and are defined by two parameters: a center and a scaling/dispersion parameter [6]. These parameters are usually optimized (learned from data) separately and with ad-hoc techniques, such as: clustering techniques (e.g., k-means) or generic optimization methods (e.g., PSO, Quantum PSO). The weights between the RBF layer and the output layer are also separately optimized using least squares or gradient descent. Therefore, the learning process for the basic RBFNN consists of optimizing three types of parameters (centers, dispersion/scaling, and weights) within three distinct and separate learning tasks [6]. In this work we address the above mentioned problems of RBFNN by extending its basic architecture as follows: (a) End-to-end learning of the entire set of parameters using gradient descent; reducing three separate optimization processes, one for each of the model parameter types (weights, centers, and dispersion parameters), to a single optimization mechanism driven by minimization of the classification error (loss function). (b) An alternative optimization scheme for the network parameters based on offline Deep Reinforcement Learning (DRL) principles by including the entire RBFNN network as the policy network of a reinforcement learning model; in this way, the loss function is replaced by a more generic reward function [7]. (c) Increase the complexity of the RBFNN network by including additional network layers with different configurations, which is made possible by the end-to-end training of the entire network. This work shows that the basic RBFNN, that has already obtained good classification results for NID, can be successfully extended to more complex architectures while maintaining its basic properties of clustering the feature space into separate groups with an indicator of the level of membership in that group. The group membership score is produced by the RBF function which provides a value that depends on the distance between the inputs and each cluster center. It is a useful representation learning technique and is even better when the parameters that define the location of the groups (centers and dispersion factors) are learned with the ultimate goal of minimizing the classification loss function, which is achievable by the simultaneous end-to-end training of all model parameters. This work explores the extension of the RBFNN model considering two separate learning frameworks: (a) A classic supervised learning framework, where the complete set of model parameters are optimized by minimizing a loss function (e.g., cross-entropy, hingeloss) related with the classification error. TABLE 1. List of abbreviations. (b) A reinforcement learning framework, with two elements: an agent (implementing a policy) and an environment. The policy produces actions based on the current state of the environment, and the environment produces new states and rewards depending on the agent’s actions. The rewards are values that indicate the adequacy of the agent’s actions to a final goal. A reinforcement learning framework attempts to optimize the 153154 VOLUME 9, 2021 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning policy by maximizing the expected sum of rewards produced by the environment under the actions generated by the policy. The iterative process between the environment and the agent (policy) is exercised with the objective of optimizing the expected total value of rewards in all successive steps. This framework is originally intended for interactive agentenvironment scenarios (online scenario) [8], but can be extended to supervised learning problems with labeled datasets (offline scenario) [7] by interpreting the actions as predicted labels, the state of the environment as predictor variables, and the environment as a sampling function that randomly selects samples from the labeled dataset. In this approach, the function that implements the environment is also responsible for generating rewards as values related to the prediction quality of the labels. The inclusion of the RBFNN as the policy network of an offline reinforcement learning framework assumes and further extends this approach. The offline reinforcement learning scenario is important for applications where rewards cannot be directly assigned by the environment without human intervention, which means that we have to rely on labeled datasets to score the actions based on closeness between predictions and ground truth values (stored in the labeled dataset). NID corresponds to this type of application. We tested the applicability of the proposed methods for various NID scenarios; however, the proposed methods are generic and perfectly applicable to implement classifiers in other fields. We have chosen NID for its complexity and strong limitations in terms of noisy and unbalanced data sets, which also appear in other fields (e.g., medicine, malware detection, agriculture. . .), and which could also benefit from these results. We present a comprehensive analysis of results based on several performance metrics (Accuracy, F1-score, Precision and Recall) where the proposed extensions to the RBFNN architecture are applied to several distinctive intrusion detection (ID) datasets: AWID, UNSW-NB15, NSL-KDD, CICIDS2017 and CICDDOS2019. These are five of the most representative ID datasets [1]. The datasets exhibits quite different behaviors, allowing a better generalization of the conclusions. The five datasets correspond to five different NID scenarios, according to the types and distribution of intrusions. The proposed new models are compared to a broad set of alternative machine learning (ML) and deep learning (DL) models for all five datasets. The results obtained show that the new RBFNN extended architectures achieve the best results in most of the performance metrics for the five proposed scenarios. Some interesting conclusions from this work, when the proposed methods are applied to NID, are: (a) The inclusion of additional dense layers in RBFNN networks has a significant impact on the classification performance. (b) Equally important is the selected loss function and learning TABLE 2. Comparative summary between proposed RBFNN-based methods and similar alternative methods. framework for these networks. In particular, the proposed offline reinforcement learning scheme provides the best results, especially in the case of highly unbalanced and/or noisy datasets, which are common in network intrusion detection. As a summary, Table 2 provides a comparison between the proposed RBFNN-based methods and alternative proposals based on similar models. The main contributions of this research are to propose novel extensions of the RBFNN architecture, with the following objectives: (a) Propose a novel method for optimization of parameters/weights within a reinforcement learning framework, where the RBFNN classifier plays the role of the policy network in a Deep Reinforcement Learning (DRL) training scheme. (b) Adapt the reinforcement learning scheme from the usual online interactive agent-environment scenario to an offline scenario working with a labeled dataset (offline reinforcement learning) [7], [9]. (c) Provide an end-to-end training process where the weights, centers, and dispersion parameters of an RBFNN are learned simultaneously by gradient descent. (d) Propose extensions to the classic RBFNN configuration from a single hidden layer network to a multilayer network. (e) Show that the different RBFNN variants resulting from these new premises produce classifiers that can be especially useful for NID, considering the results obtained when applied to five different and representative ID datasets This paper is organized as follows: Section II presents the works related to this research following different criteria. The data used for the experiments and the proposed algorithms are presented in Section III. The results of the different experiments are given in Section IV, with the final conclusions in Section V. II. RELATED WORKS There is a significant literature that presents solutions for network intrusion detection based on RBFNN. The RBFNNs introduced in the literature are usually not based on endto-end training of the network using gradient descent, but on alternative optimization methods, e.g., linear regression, genetic optimization. These related works mainly use simple networks with a single hidden layer (the RBF layer itself), and typically a two-stage model optimization is applied, training the RBF parameters and the neural network weights VOLUME 9, 2021 153155 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning separately. Meanwhile, our proposed extension to RBFNN produces multi-layer models, allowing end-to-end training of the parameters of the entire network and with the ability to be flexible in the learning framework chosen (either supervised or reinforcement learning). In particular, our proposed architecture is, to our knowledge, the first offline reinforcement learning architecture with an RBFNN and, consequently, the first application of this model to NID. The related works on RBFNN can be grouped into the following categories oriented by goals and methods used: (a) Application of regular RBFNN to NID with different network optimization methods: Authors in [10] use the NSL-KDD dataset for intrusion detection by applying an RBFNN. They optimize the network parameters without using end-to-end training and gradient descent, but an algebraic solution based on the calculation of the pseudo-inverse matrix. They report high accuracy on an ad-hoc test set of 40000 samples that does not correspond to the proposed test set of 22544 samples provided by NSL-KDD. The work in 11] proposes an RBFNN where an optimal network structure and weights are obtained with a combination of quantum-behaved particle swarm optimization (QPSO) and gradient descent applied to the QPSO algorithm. An overview of the challenges for RBFNN-based intrusion detection systems is provided in [12], identifying as the most important aspects the optimization of the location of cluster centers and width spread for the RBF neurons and the optimization of the network weights. This literature review presents the recursive least mean squares method as the main proposal to adjust the weights, and various optimization algorithms to optimize the other RBF parameters, such as: IRBF, PSO and QPSO. None of the works in this review present an end-to-end learning approach based on gradient descent across the entire network. In [13] the authors use a simple RBFNN for intrusion detection in a host-based scenario with a proprietary dataset from the University of New Mexico. The model consists of a single hidden layer RBFNN where the output weights are optimized with linear regression with no gradient descent. They claim to obtain excellent detection results for this dataset. A similar approach to [13] is adopted in [14] for the KDD99 dataset, also reporting excellent results. The work in [15] presents RBFNN as an important asset to mitigate adversarial examples, applying the results to image datasets e.g. MNIST and CIFAR10. Authors in [16] propose an RBFNN with an optimization algorithm to obtain the cluster centers for the RBF network based on an immune optimization algorithm and a least squares recursive method to adjust the network weights. An alternative solution to calculate the RBF centers different from the usual k-means approach is offered in [17], using an eigenvector based clustering method; once the centers are obtained, the weights are optimized by gradient descent. There is no subsequent end-to-end training of the entire network. The work in [18] presents a simple one-hidden layer RBFNN to detect intrusions for the KDD99 dataset. They use Levenberg-Marquardt as the optimization algorithm, with an ad-hoc method to determine the RBF centers. Authors in [19] present a particle swarm optimization algorithm (PSO) for the optimization of parameters of the RBF layer. The algorithm is applied to the KDD99 dataset. In [20] the imperialist competitive algorithm is proposed to perform network optimization in an architecture that combines RBFNN with a self-organizing map. (b) RBFNN extensions: In [21] is introduced a novel architecture for intrusion detection with kernel principal component analysis for features dimensionality reduction and an extreme learning machine model (ELM). It is applied to the KDD99 and UNSW-NB15 datasets with state-of-the-art (SOTA) results, achieving an average F1-score for the 10 labels prediction (UNSW-NB15) of 0.604. A different approach is adopted in [22] with a combination of RBF and Elman recurrent neural network which is appropriate for data with a time-series structure. The Elman layer is added as an additional layer after the RBF layer. The work in 23] proposes an iterative learning process that allows incorporating new intrusion labels by comparing the maximum value associated with the already expected labels with a threshold per label; if a sample provides output values smaller than the threshold it implies a new label and the addition of a new RBF neuron. The allocation of initial center values is done with an unsupervised self-organizing map algorithm. The method is applied to the KDD99 intrusion detection dataset. Authors in [24] use a neural network formed by an initial stage of stacked autoencoders to perform dimensionality reduction followed by an RBF layer. They apply the method to the UNSW-NB15 dataset for the 2-labels scenario using the MATLAB platform and an unclear procedure to perform RBF parameters optimization. Training the RBFNN network end-toend using gradient descent is presented theoretically in 25] under a proposal for regularization theory and regularization networks. (c) Online reinforcement learning: Applications of online reinforcement learning with RBFNN networks are mainly in the control engineering field. A Deep Q-Learning (DQN) reinforcement learning model is used in [26] with a RBF neural network as the policy network. The network weights are learned with gradient descent, but the RBF parameters are learned separately. The method is applied in the control field. A temporal difference reinforcement learning approach is applied in [27] with an RBFNN where their parameters are learned independently of the weights. The application is for a scheduling optimization problem. In [28], to solve a maze problem is used a DQN model 153156 VOLUME 9, 2021 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning with an RBFNN with their parameters learned with an evolutionary algorithm. The work in 29] applies a RBF layer at the end of the network and all parameters are adjusted by gradient descent. It extends the DQN model to continuous control. The model is applied to a continuous state space for control applications. Likewise, in [30] the authors use gradient descent and particle swarm optimization (PSO) for the learning of parameters of a neural network used with an actor-critic reinforcement learning framework. It is applied to an adaptive controller implementation. This line of work is also applied in 31]–[33] with an actor-critic DRL model applied to a continuous state-space applied to control problems. In this case the RBF parameters are adjusted as part of the training. In all these works, the DRL framework is applied to an online optimization/control problem. There are also interesting online learning works based on autoencoders [34], [35]. (d) Offline reinforcement learning: The application of deep reinforcement learning (DRL) to NID is a topic of increasing interest [36]. DRL is generally limited to online environments modeled by a Markov decision process corresponding to an interactive environment. This scheme cannot work with a dataset of recorded attacks, which is the subject of offline reinforcement learning [9]. There are currently few applications of offline reinforcement learning for NID [7], [37] even though it is currently an important area of research as an extension/alternative to supervised learning schemes [9]. Table 3 presents a summary of the related works offering a comparison between their main characteristics. The two groups of proposed solutions, with their main characteristics, are also included at the end of the table, to help compare the alternative works and the differences with the proposed methods. III. WORK DESCRIPTION This Section describes (a) the datasets used to perform the comparison between the models proposed in this work, and (b) the new proposed models in detail. The datasets applied are introduced in Section III.A. The different proposed models are described in Section III.B. A. SELECTED DATASETS To explore the different challenges imposed by an intrusion detection problem [1], which is characterized by complex behavior patterns with unbalanced datasets, we have used five ID datasets with distinctive properties to draw better conclusions on the prediction performance of the proposed models. We will use the NSL-KDD and UNSW-NB15 datasets for binary classification and the AWID, CICIDS2017 and CICDDOS2019 datasets for multiclass classification. The two-label scenario for NSL-KDD provides a reasonable balanced dataset obtained from an underlying very different TABLE 3. Categorization of related works with their main distinctive characteristics, and comparison with the characteristics of the proposed solutions. frequency distribution of basic attacks between the training and test sets. Additionally, it presents a change in the distribution of the majority class between the Normal and Attack classes for the training and test sets. This change in the distribution of the majority class is reversed for the UNSW-NB15 dataset for the two-label scenario. UNSW-NB15 is also very VOLUME 9, 2021 153157 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning FIGURE 1. Frequency of intrusions for the training and test datasets (NSL-KDD) for the 2-labels scenario. different from NSL-KDD in having the same basic attacks for the training and test sets with a very similar frequency distribution between them, which is far from being the case for the NSL-KDD dataset. The AWID dataset is the most unbalanced one with four basic attacks and has been chosen to verify the behavior of the different models for a multiclass classification. The CICIDS2017 and CICDDOS2019 datasets also have four unbalanced classes, but not as unbalanced as AWID. Each dataset (NSL-KDD, UNSW-NB15, AWID, CICIDS2017 and CICDDOS2019) with the chosen label configuration (2, 2, 4, 4 and 4, respectively) provides different characteristics, restrictions and challenges allowing a better validation of results (Section IV). 1) NSL-KDD DATASET The NSL-KDD dataset is an intrusion detection dataset evolved from the original KDD-99 [38] dataset. The NSL-KDD dataset is considered classic, but it remains a reference for intrusion detection problems as it is extremely well-known with a large literature that uses it. The NSL-KDD dataset is composed of 125,973 and 22,544 training and test samples, respectively. It contains 41 features, with 38 continuous and 3 categorical. The continuous features have been scaled to the [0–1] range. The categorical features are onehot encoded. The resulting dataset has 122 features with 38 continuous and 84 (one-hot encoded) binary features. The features are formed by aggregating the information contained in the data packets associated to the normal/attack traffic. The total number of distinct labels is 40. The number of labels presented in the training and test datasets is different with 23 in the training and 38 in the test sets. There are labels in the test set which are not in the training set and vice versa (16,6% of the test samples correspond to a class that is not present in the training set); this contributes to creating a particularly noisy dataset. The original labels can be grouped according to various hierarchies into different categories [38]. Fig. 1 presents the frequency distribution of the labels grouped into two categories: Normal and Attack, for the training and test sets. In this work we will use this grouping into two distinct categories. FIGURE 2. Frequency of intrusions for the training and test datasets (UNSW-NB15) for the 2-labels scenario. 2) UNSW-NB15 DATASET The UNSW-NB15 [39] is an intrusion detection dataset. It is a more recent and larger dataset than NSL-KDD, with 2,540,044 and 82,332 training and test samples, respectively. The dataset provides 10 labels (9 attacks plus normal traffic) with 42 features: 39 continuous and 3 categorical. Following a preprocessing similar to that presented for NSL-KDD (Section III.A.1), the total number of final features is 196. Similar to NSL-KDD, the features are formed by aggregating the information of the data packets. It is an imbalanced dataset. Fig. 2 presents the frequency distribution of the labels grouped into two categories: Normal and Attack, for the training and test sets. In this work we will use this grouping into two distinct categories. The label distribution for the training and test sets is quite similar to NSL-KDD, but with the majority class swapped in the training set (i.e., Normal is the majority class for the NSL-KDD training set while Attack is the majority class for the UNSW-NB15 training set). 3) AWID DATASET Aegean Wi-Fi Intrusion Dataset (AWID) [40] is a public intrusion detection dataset with three types of attacks for the IEEE 802.11 networks. It is also a larger and more up-todate dataset than NSL-KDD. AWID provides several dataset instances from which the AWID-CLS-R has been chosen. This dataset has four classification labels: normal, flooding, injection, and impersonation. It provides 1,795,574 and 575,642 training and test samples, respectively. The dataset has a total of 154 continuous and categorical features. The number of features can be reduced to 24 after removing features with less importance and with null and constant values [40], which after a preprocessing similar to NSL-KDD (Section III.A.1) produces a final feature number of 58. It is an imbalanced dataset with frequencies from 91% to 9% for the Normal and Attack labels. It is more unbalanced than the NSK-KDD and UNSW-NB15 datasets, but contrasting with NSL-KDD, the label frequencies for the training and test sets are extremely similar, more similar than the corresponding frequency distribution in UNSW-NB15. Fig. 3 presents the frequency distribution of all distinct labels for the training and test sets. In this work we will use the original 153158 VOLUME 9, 2021 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning FIGURE 3. Frequency of intrusions for the training and test datasets (AWID) for the 4-labels scenario. FIGURE 4. Frequency of intrusions for the training and test datasets (CICIDS2017) for the 4-labels scenario. four labels to perform multiclass classification, which will allow us to test the algorithms under a particularly unbalanced dataset. 4) CICIDS2017 AND CICDDOS2019 DATASETS CICIDS2017 [41] and CICDDOS2019 [42] datasets are two recent intrusion detection datasets from University of New Brunswick (UNB), Canada. CICIDS2017 contains three types of attacks plus normal traffic. CICDDOS2019 is dedicated to distributed denial of services (DDoS) attacks, also containing three types of DDoS attacks plus normal traffic. CICIDS2017 has 70 continuous and 7 categorical features with 562539 training and 139670 test samples. CICDDOS2019 has 67 continuous and 7 categorical features with 539579 training and 134884 test samples. The features have been processed, eliminating the features with very low variance, scaling the continuous features in the range of values [0-1], and one-hot-encoding the categorical ones. Figs. 4 and 5 present the frequency distribution of all distinct labels for the training and test sets of these datasets. CICIDS2017 and CICDDOS2019 have been selected for being modern multi-class NID data sets with different types of attacks. They are unbalanced but not as extremely unbalanced as AWID, and offer the opportunity to explore other types of prediction behaviors for the proposed classifiers. B. MODEL DESCRIPTION We propose a novel extension of the RBFNN basic model. The RBFNN basic architecture is based on a single hidden-layer where each neuron implements a radial basis FIGURE 5. Frequency of intrusions for the training and test datasets (CICDDOS2019) for the 4-labels scenario. activation function e.g., Gaussian. These functions are also called radial kernels, and are defined by two parameters: a center and a scaling/dispersion parameter [6]. These parameters are usually optimized (learned from data) separately and with ad-hoc techniques, such as: clustering techniques (e.g., k-means) or generic optimization methods (e.g., PSO, Quantum PSO). The weights between the RBF layer and the output layer are also separately optimized using least squares or gradient descent. Therefore, the learning process for the basic RBFNN consists of optimizing three types of parameters (centers, dispersion/scaling, and weights) within three distinct and separate learning tasks [6]. In this work, an end-to-end learning process is proposed in which the three types of parameters are learned simultaneously end-to-end by gradient descent. We consider the centers and dispersion parameters as additional special weights that are included in the computational graph of the network [43]. This end-to-end and simultaneous learning of the complete set of parameters allows us to consider: (a) additional loss functions (e.g., hinge-loss for maximum class separation), (b) additional layers (and layer configuration) to be incorporated after the first RBF layer, and (c) to include the entire RBFNN network as the policy function of a DRL model, where the differentiable loss function is replaced by a more generic and not necessarily differentiable reward function [7]. The extended RBFNN models described here are divided into two groups according to: (a) using a supervised learning framework based on direct minimization of the classification errors (Fig. 6), or (b) using an offline reinforcement learning framework to train the RBFNN model (Fig. 8). In both cases, all model parameters are learned end-to-end by gradient descent. 1) EXTENDED RBFNN WITHIN A SUPERVISED LEARNING FRAMEWORK Fig. 6 shows the proposed RBFNN architecture where the first hidden layer is always a layer with neurons that implement a radial basis activation function. In Fig. 6, the RBF layer is shown in detail by zooming into an RBF neuron. The RBF activation (1) is different to a sigmoid or ReLU activation. The dot product between the input and the weights, appropriate for a sigmoid or ReLU activation, is replaced in the case of an RBF neuron by the difference between the input VOLUME 9, 2021 153159 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning FIGURE 6. Generic architecture of the extended RBFNN within a supervised learning framework. and the RBF center. The RBF activation acts on this difference by computing its squared norm followed by a scaled exponential function instead of a sigmoid or ReLU function. The resulting expression is differentiable and feasible to incorporate into an end-to-end gradient descent algorithm. A radial basis function (RBF) is a real-valued function whose output depends on the distance between the input and a certain fixed point that we will call the center. The center values have similar dimension to the input values. For this study we will use a particular type of RBF: Gaussian RBF, which has the form shown in (1), where Xis the input vector, βis a scaling/dispersion parameter (scalar value), Cis the centers vector and k. . .k2is the squared vector norm, which in our case will be the squared Euclidean distance [6]: ϕ(X)=exp(−βkX−Ck2) (1) Note: In mathematical expressions a term in bold indicates a vector. Each neuron implementing the activation function in (1) will have a distinct center and dispersion values; these values are considered fixed after training, but will be adjusted during training to achieve an optimal value that minimizes the defined loss function. The first layer should always be the RBF layer since the intention is to apply the different RBF functions (one per RBF neuron) to the input, i.e., each RBF neuron will provide a value related to the distance from the input to its center (C), where the distance is scaled by the scaling/dispersion parameter (β). A graphical representation of how the RBF layer works is provided in Fig. 7, for an RBF layer with three neurons, where Xiis a particular input sample and Cjand βjare, respectively, the center and dispersion parameter for the RBF neuron j. After the RBF layer, the architecture may have several subsequent layers, which may be generic in nature, but, for this work, we have considered fully connected (FC) dense layers (Fig. 6). The final layer will provide the prediction (ˆ Y) which can be generated with: 1) a softmax activation that offers normalized probabilities, which are interpreted as the FIGURE 7. Graphical depict on how an RBF layer works. The diagram shows an RBF layer with three neurons acting on an input sample (Xi), where the output of neuron j(i.e., ϕj) is computed with an RBF function (1) with particular center (Cj) and dispersion (βj) parameters. probability of the associated class; or 2) a linear activation that produces real values, and where the selected class is associated with the highest output value. Related to the type of output produced, we can employ different types of loss functions. In this work we apply two loss functions: crossentropy (2), and hinge-loss (3): Cross Entropy Y,ˆ Y = − 1 NXN i=0XL j=1hYi,jlog ˆ Yi,j+1−Yi,j log 1−ˆ Yi,j i (2) Hinge Loss Y,ˆ Y, 1 =1 NXN i=1Xj6=zihmax(0,ˆ Yi,j−ˆ Yi,zi+1 i+γkWk2(3) In (2), the cross-entropy between two sets of Npredicted values ( ˆ Y) and their corresponding ground-truth values (Y) is an average of the log probability of the predicted class, where Lis the number of predicted classes, and Yi,jis a {0,1} value associated to the one-hot encoded ground-truth value for class jof sample i. In this case, ˆ Yi,jcorresponds to the associated predicted value for class jof sample i, which is a value in the range [0, 1] and interpreted as the probability that sample i (i.e., Xi) belongs to class j. In (3), the hinge-loss is also defined as the average of a distance between two sets of Npredicted values ( ˆ Y) and their corresponding ground-truth values (Y). In this case, the distance is a real value distance (for each sample i), between the output for each class other than the correct class (indexed by zi) if the distance is greater than a margin (1), and zero otherwise. In this case, ˆ Yi,jcorresponds to a real value (not a probability), and the predicted class for sample i(i.e., Xi) is chosen by selecting the class jthat maximizes ˆ Yi,j. The crossentropy loss requires the use of a softmax activation for the last layer and the hinge-loss a linear activation. In (3) (hingeloss) the term kWk2corresponds to the norm of the weight vector (sum of the squares of its values) and is a regularization term to avoid over-fitting, with the parameter γadjusting the effect of this regularization term. Previous to the final adjustment by gradient descent, and, instead of a random initialization, we use an initialization of the RBF centers using k-means and a fixed value for the dispersion parameter (normally a value of 2). Some more complex alternative methods for centers initialization are: (a) Learning Vector Quantization (LVQ) [44], (b) RBF 153160 VOLUME 9, 2021 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning center selection based on Fisher Ratio Class Separability measure [45], and (c) PCA based k-means, which is a novel method that could be of interest for large datasets [17]. It is important to note that this is just an initialization of parameters that will be further adjusted by gradient descent during training; therefore, in our case, it is not necessary to use complex methods, the simple and fast k-means algorithm is sufficient to provide an effective initialization that improves the convergence and the time required for the learning process. To train the RBFNN model within a supervised learning framework (Fig. 6), we have used 100 epochs and a batch size of 20 to 200 samples (depending on the dataset). We have used early-stopping to avoid overfitting with a patience period of 10 and using a validation set of 20% of the training set [43]. 2) EXTENDED RBFNN WITHIN AN OFFLINE REINFORCEMENT LEARNING FRAMEWORK An alternative proposal consists of extending the training of an RBFNN from a regular supervised scheme to an offline reinforcement learning scheme, where the RBFNN plays the role of the policy network interacting with a specifically designed ‘environment’ (Fig. 8). As presented in the Introduction, a reinforcement learning framework contains two elements: an agent (implementing a policy) and an environment. The policy produces actions based on the current state of the environment, and the environment produces new states and rewards depending on the agent’s actions. The rewards are values that indicate the adequacy of the agent’s actions to a final goal. The objective is to optimize the policy by maximizing the expected sum of rewards produced by the environment under the actions generated by the policy. The policy can be implemented through an intermediate function that gives a value for the quality (Q function) of each stateaction pair, and by choosing the action that maximizes the Q function we can select the best action for each state produced by the environment, hence implementing the policy. The Q function can be parameterized, and implemented with a neural network (NN). This NN can be trained end-to-end with a loss function that attempts to reduce the error between the output of the network and the estimation of the sum of future rewards that depends on the Q function (Bellman’s equation) [8]. Using a reinforcement learning scheme, the RBFNN parameters can be updated, during training, using one of several deep reinforcement learning approaches e.g. Deep Q-Learning (DQN), Double DQN (DDQN), Policy gradient, Actor-critic [46]. The offline reinforcement learning adopted is outlined in Fig. 8, where a specifically designed environment generates new training samples by random sampling the dataset and producing reward values based on the errors made by the policy network, which is implemented by the RBFNN (Fig. 8). This learning approach allows assimilating the loss function (supervised learning) to the reward generation process (reinforcement learning), and the space of possible actions to the FIGURE 8. High-level view of the extended RBFNN playing the role of the Policy Network in an offline reinforcement learning framework. The circled numbers indicate the process steps following the algorithm in Fig. 9. The symbol ←acts as an assignment operator. The operator = def means that their left and right operands are defined as being equal. The whole process is repeated multiple times indicated by the circular loop symbol (to the left of the DRL environment). labels to be predicted by the algorithm. The reward function is open to any function that associates a higher value with better predictions. In this work we have used the DDQN [47] as our DRL algorithm, and a simple 1/0 reward function (1 in case of positive reward and 0 otherwise), but other DRL algorithms and reward functions are possible. Fig. 8 provides a high-level view of the process involved, where 4 steps can be observed in each learning cycle. Each learning cycle is iterated multiple times until the learning process is complete (indicated by the circular loop symbol to the left of the DRL Environment). The convergence of the learning process is discussed in Section IV.G. Fig. 9 provides a more detailed view of the offline DRL training algorithm presented in Fig. 8. The training process is based on a cycle with 4 basic steps, shown as circled numbers in Fig. 8 which correspond to the numerical sequence in Fig. 9. The algorithm in Fig. 9 corresponds to an adapted Q-learning algorithm [8] that allows creating an intrusion detection classifier using a labeled dataset. The Q-learning algorithm attempts to find the best Q-function for the agent (the RBFNN model). A Q-function receives as input a state-action pair and generates a score value that indicates how good it is to take that action for that state. The Q-function can be obtained iteratively by (4) (Bellman’s equation) [8]; where Stis the current state, St+1is the next state, Atis the current action, Rtis the current reward value. The parameters αand λare two hyperparameters: learning rate and discount factor, respectively. The future Q-value (Qt+1) is formed by updating the current value (Qt) with a weighted sum of the current reward (Rt) plus the difference VOLUME 9, 2021 153161 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning FIGURE 18. Comparison of F1-score and complexity (number of parameters) of all models applied to AWID (Section IV.C). tends to slow down the training phase (e.g., reinforcement learning). However, and despite its limitations, the number of parameters can be used as a first approximation to model complexity [57], and from this point of view, Fig. 18 is appropriate to compare the different NN models and determine that the proposed models provide a good balance between complexity and prediction performance. Considering all the previous results obtained by applying the proposed RBFNN architectures to the different challenges presented by the five datasets, the main conclusions are: (a) The extended RBFNN architectures proposed in this work obtain the best performance metrics. The extended DRL+RBFNN achieves the best performance in two of the five scenarios, and the extended RBFNN architecture obtains the best results in the third scenario. The DRL learning scheme applied to RBFNN networks seems to stand out over other architectural alternatives for highly unbalanced datasets (AWID) or noisy label (NSL-KDD) scenarios. (b) The importance of the network structure after the initial RBF layer, showing that the addition of extra dense layers improves performance. The best architecture for the dense layer block appears to be a trapezoidal shape with a descending number of layers and with the first dense layer being wider than the RBF layer. (c) The great impact of the loss function and the learning approach (supervised vs. reinforcement). In particular, how using a DRL learning approach can provide better results mainly with extremely unbalanced datasets, and how the hinge-loss in its various configurations does not provide good results. (d) The limited impact of the number of RBF neurons on performance, as far as this number is greater than a certain threshold which is generally not large. (e) The best execution times required for training and prediction are obtained with linear ML models (logistic regression and linear SVM). The proposed models are in an intermediate position in terms of execution times, with the DRL +RBFNN models being the worst in this regard. Models based on CNN’s architectures offer prediction capabilities similar to the proposed models with generally higher execution times. We implemented all the neural network models (NN, RBFNN, DRL+RBFNN) in python using Tensorflow/Keras [43], and the rest of ML models in python using the scikit-learn package [54]. H. CHALLENGES AND LIMITATIONS It is important to explicitly mention the challenges and limitations of this work, which are: - The difficulties of DRL methods when increasing the dimensionality of the action space (number of labels) are known. This is an issue that should be considered and addressed in future research. - The additional prediction performance of these methods comes with additional RBF layer complexity and increased processing time, mainly for the training stage. - A good initial assignment of the RBF cluster centers is important, and introducing additional methods to optimize this initial step is challenging. - The execution times of the proposed models are high, but lower than the required times of alternative deep learning models (e.g., CNN-1D) with similar prediction performance. V. CONCLUSION Network intrusion detection is an increasingly important problem in modern data networking, and it is an active research field in which many types of machine learning and deep learning models have been applied. We propose novel extensions to the RBFNN model. These extensions are based on an end-to-end training scheme using gradient descent for all the parameters of the network: the network weights, and the centers and dispersion parameters of the radial basis functions. This end-to-end training scheme allows us to propose several alternative loss functions, different from the cross-entropy generally used for classification. It also allows a complete RBFNN network to be included as the policy network of an offline reinforcement learning model where the loss function is replaced by a reward function that is not necessarily differentiable. This approach also offers the opportunity to include additional dense hidden layers after the initial RBF layer. We show through extensive analysis of results, with five different intrusion detection datasets, that the proposed RBFNN extended architectures achieve the best results in most of the performance metrics for the five proposed datasets. We also show that the inclusion of additional dense layers and the application of the reinforcement learning framework for training are both crucial to increasing the performance metrics of the models. The reinforcement learning scheme applied to RBFNN networks stands out above other architectural alternatives, especially in the case of highly unbalanced and/or noisy datasets which are common in network intrusion detection. As promising directions for future work, we propose: (a) To continue with this line of research, particularly in exploring new loss functions and their application to RBFNN end-to-end training, e.g., contrastive loss. (b) It would also 153168 VOLUME 9, 2021 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning be of interest to analyze the impact of alternative reinforcement learning algorithms e.g., actor-critic, policygradient, adversarial [37]. (c) To apply eXplainable Artificial Intelligence (XAI) techniques to extrapolate the particular policy/classification procedure inferred by the proposed algorithms [58]. REFERENCES [1] A. Aldweesh, A. Derhab, and A. Z. Emam, ‘‘Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues,’’ Knowl.-Based Syst., vol. 189, Feb. 2020, Art. no. 105124, doi: 10.1016/j.knosys.2019.105124. [2] A. Drewek-Ossowicka, M. Pietrołaj, and J. Rumiński, ‘‘A survey of neural networks usage for intrusion detection systems,’’ J. Ambient Intell. Humanized Comput., vol. 12, no. 1, pp. 497–514, Jan. 2021, doi: 10.1007/s12652020-02014-x. [3] S. Gamage and J. Samarabandu, ‘‘Deep learning methods in network intrusion detection: A survey and an objective comparison,’’ J. Netw. Comput. Appl., vol. 169, Nov. 2020, Art. no. 102767, doi: 10.1016/j.jnca.2020.102767. [4] S. M. Tahsien, H. Karimipour, and P. Spachos, ‘‘Machine learning based solutions for security of Internet of Things (IoT): A survey,’’ J. Netw. Comput. Appl., vol. 161, Jul. 2020, Art. no. 102630, doi: 10.1016/j.jnca.2020.102630. [5] D. Broomhead and D. Lowe, ‘‘Multivariable functional interpolation and adaptive networks,’’ Complex Syst., vol. 2, no. 3, pp. 321–355, 1988. [6] F. Schwenker, H. A. Kestler, and G. Palm, ‘‘Three learning phases for radial-basis-function networks,’’ Neural Netw., vol. 14, no. 4, pp. 439–458, 2001, doi: 10.1016/S0893-6080(01)00027-2. [7] M. Lopez-Martin, B. Carro, and A. Sanchez-Esguevillas, ‘‘Application of deep reinforcement learning to intrusion detection for supervised problems,’’ Expert Syst. Appl., vol. 141, Mar. 2020, Art. no. 112963, doi: 10.1016/j.eswa.2019.112963. [8] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: Bradford Book, 2018. [9] S. Levine, A. Kumar, G. Tucker, and J. Fu, ‘‘Offline reinforcement learning: Tutorial, review, and perspectives on open problems,’’ 2020, arXiv:2005.01643. [10] A. AbuGhazleh, M. Almiani, B. Magableh, and A. Razaque, ‘‘Intelligent intrusion detection using radial basis function neural network,’’ in Proc. 6th Int. Conf. Softw. Defined Syst. (SDS), Jun. 2019, pp. 200–208, doi: 10.1109/SDS.2019.8768575. [11] R. Ma, Y. Liu, X. Lin, and Z. Wang, ‘‘Network anomaly detection using RBF neural network with hybrid QPSO,’’ in Proc. IEEE Int. Conf. Netw., Sens. Control, Apr. 2008, pp. 1284–1287, doi: 10.1109/ICNSC.2008.4525415. [12] H. Sheth, P. B. Shah, and S. Yagnik, ‘‘A survey on RBF neural network for intrusion detection system,’’ Int. J. Eng. Res. Appl., vol. 4, no. 12, pp. 17–22, 2014. [13] U. Ahmed and A. Masood, ‘‘Host based intrusion detection using RBF neural networks,’’ in Proc. Int. Conf. Emerg. Technol., Oct. 2009, pp. 48–51, doi: 10.1109/ICET.2009.5353204. [14] J. Bi, K. Zhang, and X. Cheng, ‘‘Intrusion detection based on RBF neural network,’’ in Proc. Int. Symp. Inf. Eng. Electron. Commerce, May 2009, pp. 357–360, doi: 10.1109/IEEC.2009.80. [15] J. Chenou, G. Hsieh, and T. Fields, ‘‘Radial basis function network: Its robustness and ability to mitigate adversarial examples,’’ in Proc. Int. Conf. Comput. Sci. Comput. Intell. (CSCI), Dec. 2019, pp. 102–106, doi: 10.1109/CSCI49370.2019.00024. [16] P. Yichun, N. Yi, and H. Qiwei, ‘‘Research on intrusion detection system based on IRBF,’’ in Proc. 8th Int. Conf. Comput. Intell. Secur., Nov. 2012, pp. 544–548, doi: 10.1109/CIS.2012.128. [17] Y. Hu, J. J. You, J. N. K. Liu, and T. He, ‘‘An eigenvector based center selection for fast training scheme of RBFNN,’’ Inf. Sci., vol. 428, pp. 62–75, Feb. 2018, doi: 10.1016/j.ins.2017.08.092. [18] Z. Yang, X. Wei, L. Bi, D. Shi, and H. Li, ‘‘An intrusion detection system based on RBF neural network,’’ in Proc. 9th Int. Conf. Comput. Supported Cooperat. Work Design, vol. 2, 2005, pp. 873–875, doi: 10.1109/CSCWD.2005.194301. [19] Z. Chen and P. Qian, ‘‘Application of PSO-RBF neural network in network intrusion detection,’’ in Proc. 3rd Int. Symp. Intell. Inf. Technol. Appl., vol. 1, 2009, pp. 362–364, doi: 10.1109/IITA.2009.154. [20] S. Mohammadi and F. Amiri, ‘‘An efficient hybrid self-learning intrusion detection system based on neural networks,’’ Int. J. Comput. Intell. Appl., vol. 18, no. 1, Mar. 2019, Art. no. 1950001, doi: 10.1142/S1469026819500019. [21] L. Lv, W. Wang, Z. Zhang, and X. Liu, ‘‘A novel intrusion detection system based on an optimal hybrid kernel extreme learning machine,’’ Knowl.-Based Syst., vol. 195, May 2020, Art. no. 105648, doi: 10.1016/j.knosys.2020.105648. [22] X. Tong, Z. Wang, and H. Yu, ‘‘A research using hybrid RBF/Elman neural networks for intrusion detection system secure model,’’ Comput. Phys. Commun., vol. 180, no. 10, pp. 1795–1801, 2009, doi: 10.1016/j.cpc.2009.05.004. [23] L.-Y. Tian and W.-P. Liu, ‘‘Incremental intrusion detecting method based on SOM/RBF,’’ in Proc. Int. Conf. Mach. Learn. Cybern., Jul. 2010, pp. 2849–2853, doi: 10.1109/ICMLC.2010.5580770. [24] Y. Kadhim and A. Mishra, ‘‘Radial basis function (RBF) based on multistage autoencoders for intrusion detection system (IDS),’’ in Proc. 1st Int. Informat. Softw. Eng. Conf. (UBMYK), Nov. 2019, pp. 1–4, doi: 10.1109/UBMYK48245.2019.8965627. [25] T. Poggio and F. Girosi, ‘‘Networks for approximation and learning,’’ Proc. IEEE, vol. 78, no. 9, pp. 1481–1497, Sep. 1990, doi: 10.1109/5.58326. [26] Y.-M. Du, B.-B. Yan, Y.-C. Jiang, and Q.-S. Liu, ‘‘Reinforcement learning method based on radial basis function neural network,’’ in Design, Manufacturing and Mechatronics. Singapore: World Scientific, 2016, pp. 21–27. [27] R. S. Williem and K. Setiawan, ‘‘Reinforcement learning combined with radial basis function neural network to solve job-shop scheduling problem,’’ in Proc. IEEE Int. Summer Conf. Asia Pacific Bus. Innov. Technol. Manage., Jul. 2011, pp. 29–32, doi: 10.1109/APBITM.2011.5996285. [28] S. Slušný, R. Neruda, and P. Vidnerová, ‘‘Comparison of RBF network learning and reinforcement learning on the maze exploration problem,’’ in Artificial Neural Networks—ICANN 2008. Berlin, Germany: Springer, 2008, pp. 720–729. [29] K. Asadi, N. Parikh, R. E. Parr, G. D. Konidaris, and M. L. Littman, ‘‘Deep radial-basis value functions for continuous control,’’ 2020, arXiv:2002.01883. [30] N. Shafiabady, M. A. N. Vakilian, and D. Isa, ‘‘An adaptive controller using radial basis function neural network with reinforcement learning,’’ Commun. Appl. Electron., vol. 1, no. 7, pp. 7–13, May 2015. Accessed: Mar. 14, 2021. [Online]. Available: http://www.caeaccess.org [31] J. Li, J. Yi, D. Zhao, and G. Xi, ‘‘A reinforcement learning based radial-bassis function network control system,’’ in Advances in Neural Networks—ISNN 2005. Berlin, Germany: Springer, 2005, pp. 640–645. [32] C.-G. Li, M. Wang, Z.-J. Huang, and Z.-F. Zhang, ‘‘An actor-critic reinforcement learning algorithm based on adaptive RBF network,’’ in Proc. Int. Conf. Mach. Learn. Cybern., vol. 2, Jul. 2009, pp. 984–988, doi: 10.1109/ICMLC.2009.5212431. [33] Y.-H. Cheng, J.-Q. Yi, and D.-B. Zhao, ‘‘Application of actorcritic learning to adaptive state space construction,’’ in Proc. Int. Conf. Mach. Learn. Cybern., vol. 5, 2004, pp. 2985–2990, doi: 10.1109/ICMLC.2004.1378544. [34] Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai, ‘‘Kitsune: An ensemble of autoencoders for online network intrusion detection,’’ 2018, arXiv:1802.09089. [35] G. Bovenzi, G. Aceto, D. Ciuonzo, V. Persico, and A. Pescapé, ‘‘A hierarchical hybrid intrusion detection approach in iot scenarios,’’ in Proc. IEEE Global Commun. Conf. (GLOBECOM), Dec. 2020, pp. 1–7, doi: 10.1109/GLOBECOM42002.2020.9348167. [36] T. Thi Nguyen and V. Janapa Reddi, ‘‘Deep reinforcement learning for cyber security,’’ 2019, arXiv:1906.05799. [37] G. Caminero, M. Lopez-Martin, and B. Carro, ‘‘Adversarial environment reinforcement learning algorithm for intrusion detection,’’ Comput. Netw., vol. 159, pp. 96–109, Aug. 2019, doi: 10.1016/j.comnet.2019.05.013. [38] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, ‘‘A detailed analysis of the KDD CUP 99 data set,’’ in Proc. IEEE Symp. Comput. Intell. Secur. Defense Appl., Jul. 2009, pp. 53–58. [39] N. Moustafa and J. Slay, ‘‘UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),’’ in Proc. Mil. Commun. Inf. Syst. Conf. (MilCIS), Nov. 2015, pp. 1–6, doi: 10.1109/MilCIS.2015.7348942. VOLUME 9, 2021 153169 M. Lopez-Martin et al.: NID Based on Extended RBF NN With Offline Reinforcement Learning [40] C. Kolias, G. Kambourakis, A. Stavrou, and S. Gritzalis, ‘‘Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset,’’ IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp. 184–208, 1st. Quart., 2016, doi: 10.1109/COMST.2015.2402161. [41] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, ‘‘Toward generating a new intrusion detection dataset and intrusion traffic characterization,’’ in Proc. 4th Int. Conf. Inf. Syst. Secur. Privacy, 2018, pp. 108–116, doi: 10.5220/0006639801080116. [42] I. Sharafaldin, A. H. Lashkari, S. Hakak, and A. A. Ghorbani, ‘‘Developing realistic distributed denial of service (DDoS) attack dataset and taxonomy,’’ in Proc. Int. Carnahan Conf. Secur. Technol. (ICCST), Oct. 2019, pp. 1–8, doi: 10.1109/CCST.2019.8888419. [43] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, and S. Ghemawat, ‘‘TensorFlow: Large-scale machine learning on heterogeneous distributed systems,’’ 2016, arXiv:1603.04467. [44] F. Schwenker, H. A. Kestler, G. Palm, and M. Hoher, ‘‘Similarities of LVQ and RBF learning—A survey of learning rules and the application to the classification of signals from high-resolution electrocardiography,’’ in Proc. IEEE Int. Conf. Syst., Man Cybern., vol. 1, Oct. 1994, pp. 646–651, doi: 10.1109/ICSMC.1994.399913. [45] K. Z. Mao, ‘‘RBF neural network center selection based on Fisher ratio class separability measure,’’ IEEE Trans. Neural Netw., vol. 13, no. 5, pp. 1211–1217, Sep. 2002, doi: 10.1109/TNN.2002.1031953. [46] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, ‘‘Deep reinforcement learning: A brief survey,’’ IEEE Signal Process. Mag., vol. 34, no. 6, pp. 26–38, Nov. 2017, doi: 10.1109/MSP.2017.2743240. [47] H. van Hasselt, A. Guez, and D. Silver, ‘‘Deep reinforcement learning with double Q-learning,’’ 2015, arXiv:1509.06461. [48] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’ 2014, arXiv:1412.6980. [49] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, ‘‘Playing atari with deep reinforcement learning,’’ 2013, arXiv:1312.5602. [50] R. Boutaba, M. A. Salahuddin, N. Limam, S. Ayoubi, N. Shahriar, F. Estrada-Solano, and O. M. Caicedo, ‘‘A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities,’’ J. Internet Serv. Appl., vol. 9, no. 1, p. 16, Jun. 2018, doi: 10.1186/s13174-018-0087-2. [51] K. A. P. da Costa, J. P. Papa, C. O. Lisboa, R. Munoz, and V. H. C. de Albuquerque, ‘‘Internet of Things: A survey on machine learning-based intrusion detection approaches,’’ Comput. Netw., vol. 151, pp. 147–157, Mar. 2019, doi: 10.1016/j.comnet.2019.01.023. [52] M. Azizjon, A. Jumabek, and W. Kim, ‘‘1D CNN based network intrusion detection with normalization on imbalanced data,’’ in Proc. Int. Conf. Artif. Intell. Inf. Commun. (ICAIIC), Feb. 2020, pp. 218–224, doi: 10.1109/ICAIIC48513.2020.9064976. [53] M. Lopez-Martin, B. Carro, A. Sanchez-Esguevillas, and J. Lloret, ‘‘Shallow neural network with kernel approximation for prediction problems in highly demanding data networks,’’ Expert Syst. Appl., vol. 124, pp. 196–208, Jun. 2019, doi: 10.1016/j.eswa.2019.01.063. [54] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, ‘‘Scikit-learn: Machine learning in Python,’’ J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011, Accessed: Sep. 17, 2020. [Online]. Available: http://scikit-learn.sourceforge.net [55] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, ‘‘Extreme learning machine: Theory and applications,’’ Neurocomputing, vol. 70, nos. 1–3, pp. 489–501, 2006, doi: 10.1016/j.neucom.2005.12.126. [56] J. E. B. Maia, V. R. S. Laboreiro, F. E. Chaves, F. J. A. Maia, T. G. N. Silva, and T. N. Ferreira, ‘‘Performance comparison between edited kNN and MQ-RBFN for regression and classification tasks,’’ in Proc. 11st Brazilian Congr. Comput. Intell., Mar. 2016, pp. 1–15, doi: 10.21528/CBIC2013319. [57] X. Hu, L. Chu, J. Pei, W. Liu, and J. Bian, ‘‘Model complexity of deep learning: A survey,’’ Knowl. Inf. Syst., vol. 63, no. 10, pp. 2585–2619, Oct. 2021, doi: 10.1007/s10115-021-01605-0. [58] A. Nascita, A. Montieri, G. Aceto, D. Ciuonzo, V. Persico, and A. Pescape, ‘‘XAI meets mobile traffic classification: Understanding and improving multimodal deep learning architectures,’’ IEEE Trans. Netw. Service Manage., early access, Jul. 19, 2021, doi: 10.1109/TNSM.2021.3098157. MANUEL LOPEZ-MARTIN (Senior Member, IEEE) received the Ph.D. degree in machine learning applied to network traffic analysis and prediction from the University of Valladolid, Spain, in 2019. He is currently a Research Associate with the University of Valladolid. He has worked as a Data Scientist with Telefonica and has more than 25 years of experience in the development of IT software projects. His research activities involve applying machine learning to intrusion detection in data networks and time-series forecasting. ANTONIO SANCHEZ-ESGUEVILLAS (Senior Member, IEEE) received the Ph.D. degree in QoS over IP networks from the University of Valladolid, Spain, in 2004. He has managed innovation at Telefonica and has been an Adjunct Professor and an Honorary Collaborator with the University of Valladolid, supervising several Ph.D. students. He has coordinated very large international research and development projects and has over 50 international publications and several patents. His current research interests include digital services and machine learning. JUAN IGNACIO ARRIBAS received the M.Sc. and Ph.D. degrees in electrical engineering from the University of Valladolid, Spain. He is currently a part of the Department of Electrical Engineering, University of Valladolid, as an Associate Professor. He was an FPI Postgraduate Research Fellow with the Department of Science, Spain. He has visited the University of Maryland, Baltimore, MD, USA, and the Barrow Neurological Institute, Phoenix, AZ, USA. His research interests include machine learning, neural networks, and expert systems and their applications to computer aided diagnosis and computer vision. BELEN CARRO received the Ph.D. degree in broadband access networks from the University of Valladolid, Spain, in 2001. She is currently a Professor and the Director of the Communications Systems and Networks (SRC) Laboratory, Universidad de Valladolid, working as a Research Manager in NGN communications and services, VoIP/QoS, and machine learning. She has supervised 12 Ph.D. students and has extensive research publications experience as an author, a reviewer, and an editor. 153170 VOLUME 9, 2021