Semi-supervised Multivariate Statistical Network Monitoring for Learning Security Threats

TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 1 Semi-supervised Multivariate Statistical Network Monitoring for Learning Security Threats Jos´ e Camacho, Gabriel Maci´ a-Fern´ andez, Noem´ ı Marta Fuentes-Garc´ ıa, and Edoardo Saccenti Abstract—This paper presents a semi-supervised approach for intrusion detection. The method extends the unsupervised Multivariate Statistical Network Monitoring approach based on Principal Component Analysis by introducing a supervised optimization technique to learn the optimum scaling in the input data. It inherits the advantages of the unsupervised strategy, capable of uncovering new threats, with that of supervised strategies, able of learning the pattern of a targeted threat. The supervised learning is based on an extension of the gradient descent method based on Partial Least Squares (PLS). Moreover, we enhance this method by using sparse PLS variants. The practical application of the system is demonstrated on a recently published real case study, showing relevant improvements in detection performance and in the interpretation of the attacks. Index Terms—Multivariate Statistical Network Monitoring, Anomaly Detection, Intrusion Detection, Semi-supervised learning, Partial Least Squares regression, Principal Components Analysis. I. INTRODUCTION Cibersecurity incidents are considered one of the most relevant threats for businesses in almost any market. According to the VERIZON annual Data Breach Investigation Report (DBIR) [1], tens of thousands of attacks targeted private and public companies during 2017. To effectively fight against this real menace, the security industry has identified that an essential line of defense should be based on the joint effort of all stakeholders combining their technical skills and information [2]. In this regard, the use of anomaly-based Intrusion Detection Systems (IDS) [3] is fundamental to unveil previously unknown attack strategies and thwart potential attacks to organizations. Main technical challenges in the field of IDS design are the need for handling massive and disparate sources of information, the extraction of useful knowledge for the forensic analysis of incident data and the optimization of the detection system to targeted threats. Dealing with very different sources of information and a vast amount of data has fostered the use of machine learning and data mining techniques [4]. However, it becomes essential to utilize tools and approaches that provide interpretability of results, that is, information about the features and the real cause of an attack in order to efficiently respond to it. Regretfully, the bulk of machine learning and data mining approaches does not satisfy this requirement. J. Camacho, G. Maci´ a-Fern´ andez and N. M. Fuentes-Garc´ ıa are with the Department of Signal Theory, Telematics and Communications, School of Computer Science and Telecommunications - CITIC University of Granada, Granada, Spain. E. Saccenti is with the Laboratory of Systems and Synthetic Biology, Wageningen University & Research, Wageningen, the Netherlands. Manuscript received June 18, 2018; revised XX, 2018. Regarding the optimization to targeted threats, proper system update is a relevant feature, in particular to make the most of information sharing technologies that distribute warnings among corporations when new threats arise. The adaptation to targeted threats of machine learning models is more technically challenging than that of traditional rule-based systems (e.g. antivirus, rule-based IDS like snort, correlation engines, etc.), more extended in the industry. Multivariate Analysis has been recognized as an outstanding approach for anomaly detection in several domains, including industrial monitoring [5] and networking [6]. In the field of industrial processing, a well developed strategy is Multivariate Statistical Process Control (MSPC). A main tool within MSPC is Principal Component Analysis (PCA), which was proposed by Lakhina et al. for intrusion detection in [7]. Some of the benefits of PCA are its unsupervised nature, that does not require any a-priori knowledge on the data, and its capability to provide diagnosis information for a given anomaly, a main advantage over other machine learning methodologies. This diagnosis support allows shortening the lag from detection to response in a security incident, and therefore has practical implications. In previous work [8], we introduced a methodology named Multivariate Statistical Network Monitoring (MSNM), an extension of PCA-MSPC applied to the intrusion detection problem. MSNM overcomes some reported limitations of the original PCA approach of Lakhina et al. [9]. It is based on an over-parameterization of the feature space, that is, on defining a large number of data features within the detection system. This combines with the multivariate approach based on PCA, that can handle high-dimensional data with millions of variables. However, an open problem and main limitation in MSNM is how to select the relative relevance of the features in the system. Experiments in [8] showed high sensitivity of the detection to the relative relevance (scaling) of the features in PCA. We can make the most of this sensitivity to enhance the detection ability to a set of targeted threats. In particular, defining an optimum scaling of the features to identify the pattern of a recently identified threat is useful to update the monitoring system. This would enhance MSNM with an adaptation mechanism equivalent to what it is done in a traditional rule-based detection system when adding rules with the fingerprint of recently identified threats. This paper presents a new procedure to optimize the scaling of the features in MSNM for the detection of targeted threats. This makes MSNM a semi-supervised learning approach, where the original unsupervised PCA model is enhanced to be optimum for the detection of specific anomalies. For this purpose, we employ and improve an optimization algorithm TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 2 [10][11] originally introduced in the context of process optimization. This algorithm is based on Partial Least Squares (PLS) [12][13], a multivariate regression technique. We refer to this algorithm as run-to-run PLS (R2R-PLS). The R2R-PLS has been shown to outperform state-of-the-art optimization techniques, like genetic algorithms, in large search spaces [10]. The contributions of this paper are the following: •We cast the problem of adapting an anomaly detection system to targeted threats into an optimization problem. In particular, we apply an optimization scheme in the context of anomaly detection with MSNM, overcoming one of its principal limitations: the sensitivity of detection to the features scaling. While this is specially interesting in the context of intrusion detection, this contribution also applies to different application domains [5]. •For this purpose, we extend the original R2R-PLS in [10], [11] to a general optimization problem, simplifying its implementation and improving its computational efficiency. •We further extend the R2R-PLS optimization to sparse PLS variants, very popular in biological sciences [14], leading to an improvement of the optimization performance and in the understanding of the connection between the features scale and the MSNM detection. •We demonstrate previous contributions in a recently published real case study [15]. The rest of the paper is organized as follows. Section 2 discusses related work to this paper. Section 3 introduces the MSNM technique. Section 4 reviews PLS and two sparse variants. Section 5 presents our particular implementation of the R2R-PLS algorithm. Its application in the adaptation of MSNM to new threats is presented in Section 6. Section 7 demonstrates the application of the approach to the real case study. Section 8 draws some concluding remarks. II. RELATED WORK Machine learning (ML) techniques have been widely applied to cybersecurity problems. ML refers to the combination of statistics and artificial intelligence to learn a model from data [4][16][17]. This is a global term widely used to refer to the task in which one automatically calibrates (trains) a model or algorithm to obtain a descriptive output for a given input. If the value for the output is previously known and used in the training, then the learning is called supervised and it usually applies to classification and regression problems. However, if only the input data are known and the objective is to extract patterns or common behaviour from the data, the learning is known as unsupervised [16][18]. Mixed approaches are considered to be semi-supervised learning [18]. Support vector machines, neural networks or decision trees are common examples of supervised learning, thought extensions exist in the unsupervised setting. Factorization methods like PCA and clustering algorithms such as K-means are often applied for unsupervised analyses and often combined with supervised methods. In cybersecurity, unsupervised ML methods are applied to the anomaly detection problem, a.k.a. intrusion detection problem, while supervised methods can be used to detect and classify previously observed attacks. In this context, the use of PCA was proposed more than a decade ago [19]. Due to its unsupervised nature, PCA does not require –and is not limited by– an a-priori specification of potential anomalies in the system. This means that PCA is still useful to detect new types of anomalies, something mandatory in real world anomaly detection. The most referred work for PCA anomaly detection is that of Lakhina et al. [7], form which alternative proposals have been developed [20][21][22][23]. One of them, the MSNM methodology [8], is the base of the approach of this paper. This methodology allows to combine traffic data with other security data sources, demonstrating high detection capabilities and with the advantage of providing diagnosis support [24]. The learning process in ML yields a model from a training data set. This learning process is also referred to as model calibration, and it is generally performed by optimizing the model parameters in consecutive steps until convergence [4][17]. The calibration of the parameters is performed following optimization strategies, which pursue to find, at least, local optimums for their values and, hopefully, global solutions [25]. These approaches can be classified as follows [16]: Stochastic approximation methods, expectation-maximization methods and greedy optimization. Within stochastic optimization, gradient descent methods apply the derivative of the optimization function to obtain the direction of search. In recent papers [10][11] a variation of this type of optimization based on Partial Least Squares (PLS) was presented. PLS is a particular form of regression model suitable to handle high dimensional data sets. For this reason, the PLS-based optimization is specially suited to solve optimization problems where the search space is of high dimensionality. Therefore, it is a practical choice in ML to optimize a large number of model parameters. While PLS is well suited to model high dimensional data, a recent trend of research has explored the ability of a type of methods that perform PLS regression combined with variable selection. These are generally referred to as sparse PLS (SPLS) methods. Several variants of SPLS been proposed [14][26][27][28][29] with the goal of performing variable selection during model calibration, discarding non-informative variables. Results show that SPLS variants are more stable and often yield improved performance in very high dimensional set-ups. This paper presents a semi-supervised approach for intrusion detection. The method extends the unsupervised MSNM approach by introducing a supervised optimization technique to learn the optimum scaling of the features. It inherits the advantages of the unsupervised strategy, capable of uncovering new threats, with that of supervised strategies, capable of learning the pattern of a given threat. Considering that the number of features of MSNM corresponds to the dimension of the search space in the optimization problem, and that this number can be very large, we apply the PLS-based optimization for the supervised learning. Furthermore, we extend this optimization technique using sparse variants of PLS, improving the learning ability and model interpretability. TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 3 III. MULTIVARIATE STATISTICAL NETWORK MONITORING The MSNM follows 4 main steps: 1) Parsing,2) Fusion, 3) Detection, and 4) Diagnosis. The first three steps are equivalent to what it is commonly done in other machine learning methodologies. However, step 4 is a main advantage in MSNM. This step is possible thanks to the white-box, exploratory characteristics of PCA as the core of the approach. PCA is a linear model and as such it is easy to interpret in terms of the connection between anomalies and features, something much more complicated in the non-linear machine learning variants. A. Parsing The information captured from a network is usually presented in the form of system logs or network traces, and cannot be directly used to feed a typical tool for anomaly detection. Therefore, some sort of parsing and feature engineering needs to be done in order to generate quantitative features that can be used for data modeling. Lakhina et al. [7] proposed the definition of counters obtained from Netflow records as quantitative features for anomaly detection using PCA. In [30], we generalized this definition to consider several sources of data, proposing the feature-as-a-counter approach. Each feature contains the number of times a given event (e.g. number of packets sent from public IPs or number of flows associated to destination port 80) takes place during a given time window. The parsing transforms the raw data in a stream of features, where each time interval of e.g. 1 minute is represented by a feature vector of counts. B. Fusion The feature-as-a-counter definition simplifies the fusion of different data sources in a single set of features. For each different source of data, a set of features (counters) is defined. The sampling rate for each source may be different, due to the specific dynamics of the source. Thus, to combine the features from different sources these need to be stretched/compressed to a common sampling rate, yielding a unique matrix of data of high dimensionality. In practice, when possible, all sources are parsed at the same time rate, so that the fusion operation is done by simply appending the features associated to the different sources. The combination of the feature-as-a-counter and the fusion procedure is specially suited for the subsequent multivariate analysis. It yields high dimensional feature vectors that need to be analyzed with dimension reduction techniques, like PCA. The diagnosis procedure benefits from the definition of a large number of features for a better description of the anomaly taking place. Furthermore, counters and their correlation are easy to interpret. C. Detection The core of MSNM is PCA. PCA is applied to data sets where Mvariables or features are measured on Nobservations with the aim of finding the subspace of maximum variance in the M-dimensional feature space. The original features are linearly transformed into the Principal Components (PCs), which are the eigenvectors of XX := XT·X, typically for mean centred (MC) Xand sometimes also after auto-scaling (AS, normalization to unit variance). The PCA model follows the expression: X=TA·Pt A+EA,(1) where Ais the number of PCs, TAis the N×Ascore matrix, PAis the M×Aloading matrix and EAis the N×Mmatrix of residuals. For each observation, corresponding to a feature vector collected in a given time interval, the corresponding score vector is computed as follows: tn=xn·PA(2) where xnis a 1×Mvector representing the observation and tna1×Avector with the corresponding scores, while en=xn−tn·Pt A(3) corresponds to the residuals. For the detection of anomalies in MSNM, a pair of charts are monitored: the Q-statistic (Q-st), which compresses the residuals; and the D-statistic (D-st) or Hotelling’s T2 statistic, computed from the scores. The D-st and the Q-st for an observation can be computed from the following equations: Dn=tn·(ΣT)−1·tt n(4) Qn=en·et n(5) where ΣTrepresents the covariance matrix of the scores in the calibration data. With the statistics computed from the calibration data, upper control limits (UCL), i.e. detection thresholds, can be established in the charts at a certain confidence level [31] to decide if future events are anomalous. A straightforward approach to define the UCLs is by using percentiles over the statistics computed from the calibration data X. Once the system is calibrated and control limits computed, it can be applied to incoming data/traffic. An anomaly is identified when either the D-st or the Q-st exceeds the pre-defined UCLs. D. Diagnosis Once an anomaly is detected, a diagnosis step is performed to identify the features associated with it. This information is very useful to identify and, eventually, troubleshoot the possible root causes of the anomaly. The contribution of the features to a given anomaly can be investigated with the contribution plots or similar tools, like oMEDA [32]. Thus, anomalies are detected in the D-st and/or Q-st statistics, and then the diagnosis is performed with e.g. oMEDA. The output of oMEDA is a 1×Mvector where each element contains the contribution of the corresponding feature to the anomaly under study. Those contributions with large magnitude, either positive or negative, are considered to be relevant. TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 4 While the diagnosis capabilities of MSNM are a main advantage over other ML techniques, these capabilities are not the focus of this paper. The interested reader is referred to [24]. IV. PARTIAL LEAST SQUARES REGRESSION AND SPARSE VARIANTS This section introduces the regression techniques that we will employ within the optimization of the scaling parameters of MSNM. The regression models are used to estimate the gradient in the optimization. Since the scaling parameters, and so the gradient, are high dimensional, we need regression techniques that are suitable for the analysis of high dimensional data sets. This is the case of PLS and sparse variants. A. Partial Least Squares PLS is a particular form of regression that is suitable in presence of collinearity in the predictors, common in high dimensional problems. Collinearity makes the classical multivariate least-squares (LS) regression break down due to singularity of the covariance matrix. PLS extends LS by inheriting the philosophy of PCA. Briefly, given a N×Oresponse matrix Yand a N×M set Xof predictors variables, the PLS algorithm defines a subspace of Xwhich maximizes its covariance with Y. The PLS model can written as: X=T·PT+E(6) Y=T·QT+F(7) where Tis the N×Ascore matrix, Athe number of latent variables (LVs) to be used to fit the model, Pand Qare the M×Aand O×Aloading matrices for the predictors and the response, respectively, and Eand Fare the N×Mand N×O residual matrices of Xand Y. Setting ˆ BP LS =W·(PT·W)−1·QT(8) with Wthe M×Amatrix of weights, the two models given by Equations (6) and (7) can be re-arranged in a single regression model given by Y=X·ˆ BP LS +F(9) The PLS model can be obtained, among others, using the NIPALS algorithm, and the optimal number of LVs Acan be estimated by cross-validation. B. Sparse PLS Although PLS can effectively handle noisy predictor variables, the inclusion of variables which are non-relevant for the prediction of the response usually decreases the prediction performance of the model. Several variants of the PLS algorithm have been proposed [14][26][27][28][29] with the goal of performing variable selection during model calibration. This family of algorithms is termed Sparse PLS (SPLS), and are often reported to show improved performance over PLS in high dimensional data. The most extended approach to define sparse models is based on the LASSO regularization [33], often applied using a soft-thresholding operation [34]. One popular SPLS algorithm is the variant proposed by Lˆ e Cao et al. [35] where the (sparse) PLS problem is solved using singular value decomposition [36]. Briefly, given the response (Y) and predictor (X) matrices, the R-rank matrix C=XTY(10) can be decomposed as C=GDUT(11) where the matrices G(N×R) and U(L×R) are orthonormal and Dis R×Rdiagonal containing the singular values of C. Using this formulation, the loading vectors prand qrfor Xand Yare the singular vectors grand urof Gand U, respectively. A sparse PLS solution can be obtained by penalizing (i.e. forcing to 0) the loadings [37], which are a measure of the relative importance of each variable to the PLS model, by solving the optimization problem: arg min p,q kC−pqTk2 F+λ1kpk1+λ2kqk1(12) whose solution is given by the soft-thresholding gλ(x) = sign(x) (|x| − λ)+applied to the standard PLS solution. The two parameters λ1and λ2control the sparsity for Xand Y, respectively. However, a more practical and equivalent alternative is to select the number of non zero components of the loadings Nxand Ny[35] in the soft-thresholding [34]. In the context of this paper we restrict ourselves to set Nx, so that only loadings in the x-block are sparse, since the response is univariate. Optimum values of this parameter can be found by cross-validation. C. Group-wise PLS A different approach to obtain sparse PLS solutions is the recently proposed Group-wise PLS (GPLS) [38], where the solution is found by defining groups of correlated predictor variables rather than using regularization. Briefly, the GPLS algorithm starts by defining a set of K (possibly overlapping) groups of correlated variables that are obtained from a M×Mcorrelation map Mcomputed from C=XTY. Subsequently, the weights and scores of KPLS models of 1 LV, each of them considering only the set of variables corresponding to one of the groups, are computed. From these, the one with the largest correlation with Yis retained while the others are discarded. This is repeated for a number of LVs. The GPLS approach is particularly suited for data exploration, but when data is sparse in a group-wise fashion (i.e. when there are groups of correlated variables related to the response) the algorithm outperforms PLS and SPLS in terms of goodness of prediction. The fitting of a GPLS model requires the definition of a threshold, γ, to identify the groups of variables in M, controlling simultaneously the number and size of the groups of variables to be used. Optimal values for γcan be chosen by graphically inspecting the correlation map M, by controlling TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 5 the trade-off between group size and dimension or by using cross-validation. V. RUN-TO-RUN XPLS OPTIMIZATION The optimization of the scaling parameters of MSNM is defined as a gradient descent algorithm, where the gradient is estimated with the PLS-based methods discussed in the previous section. This optimization approach has been shown to outperform state-of-the-art optimization techniques, in particular genetic algorithms, in high dimensional search spaces [10]. Let us define uas a set of inputs we can vary as desired, typically within some specific bounds or constraints, to a system we would like to optimize. Let us also define yas the set of outputs we would like to optimize (either maximize or minimize) by setting appropriate values to u. The goal of the optimization algorithm is to find those values in the input that optimize the output, see Fig.1. Without loss of generality, in the following we will assume we desire to maximize the values in yby properly setting u, which is contrary to common optimization literature but more appropriate for our specific case. The run to run (R2R) optimization algorithm [10], extended for a general optimization problem and for PLS, SPLS and GPLS, can be summarized as follows: 0. Select user defined parameters K(number of individual solutions in each iteration) and rc(level of exploration). Initialize input solution candidate uifor i= 0. 1. Repeat for k={1...K} 1.1. Generate random variant solution ˜uk i=ui+rc·rk i, for rk idrawn from a multinormal random distribution. 1.2. Apply input ˜uk ito the system in Fig.1 and measure output ˜yk i. 2. Fit a xPLS model with the Kinstances of the inputs ˜uk i arranged in the rows of ˜ Ui, and outputs ˜yk iarranged in rows of ˜ Yi: ˜ Yi=˜ Ui·ˆ Bi+F(13) 3. Compute the next input solution candidateaas: ui+1 = ui+ 3 ·ˆ Bi 4. Check for convergence in the solution and otherwise loop back to Step 1. In each iteration, the xPLS model captures the variability around uirelated to the response. This allows the estimation of the gradient in the optimization. A random signal is added to the input to ensure that there is enough trade-off between exploration and exploitation. We have intentionally overlooked the problem of metaparameter estimation within the optimization. As already discussed, we need to define the number of LVs in PLS and there is one additional meta-parameter in each of the two sparse variants. Unfortunately, the use of cross-validation or other automatic means for meta-parameter selection is computational intensive, but this problem can be overcome. Within a aFor minimization, the second addend is subtracted to the current solution. gradient based optimization, we can simplify meta-parameter selection for the sake of computational efficiency. This is done by fixing meta-parameters during the optimization. To minimize a detrimental effect on performance, we set PLS, SPLS or GPLS to be very parsimonious, with the intuition that if the model was parsimonious in excess, this would be overcome by performing more iterations in the optimization. First, we suggest to use one single LV in step 2., since any further contribution of additional LVs can be done in future iterations. Regarding parameters Nxin SPLS and γin GPLS, we suggest to set them so that models are very sparse, since again any missing variability in one iteration can be taken into account in future iterations. System R2R UY Fig. 1: Optimization scheme. VI. SEMI-SUPERVISED MULTIVARIATE STATISTICAL NETWORK MONITORING This section particularizes the R2R optimization algorithm to our specific application: the optimization of the scaling parameters in MSNM. Here, the input uwe would like to set is the scaling of the features in MSNM so as to maximize its detection performance, which is the output y. In the following we discuss in detail inputs and outputs and the complete system. A. Output of the optimizer There are several possibilities to measure the detection performance of an anomaly detection system like MSNM. Here we will use the Area Under the ROC Curve (AUROC or AUC) computed from a labelled data set, where observations are labelled as normal or attacks. The receiver operating characteristic (ROC) curve shows the evolution of the true positive rate (TPR) versus the false positive rate (FPR) for different values of the classifying threshold, discussed below. The TPR is the percentage of true attacks that are identified by the MSNM system, while the FPR is the percentage of normal observations identified as attacks. The AUC is a scalar that quantifies the quality of the anomaly detector. An anomaly detector should present an AUC as close to 1 as possible, while an AUC around 0.5 corresponds to a random classifier. The ROC curves for MSNM are obtained by varying a threshold in a specific combination of the Q-st and the D-st: MSNMn=A·Dn M·UCLD +(M−A)·Qn M·UCLQ (14) TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 6 where MSNMnis the output of the anomaly detector at a given observation n,Dnand Qnthe corresponding statistics and UCLDand UCLQthe corresponding 99% Upper Control Limits (UCL), computed as percentiles in the calibration data. Recall that Ais the number of components in PCA and M the number of features in the data. B. Input of the optimizer To optimize the AUC, we modify the values of the scaling of the features. These features were computed in the parsing step of MSNM. The scale of the features changes their relative importance in the PCA model. Since PCA maximizes variance, the higher the relative scale (weight) of a variable, the more percentage of its information is captured in the scores and the less in the residuals of the model in Equation (1). However, understanding how this scaling impacts the detection ability of a MSNM system for a given attack is a real challenge, due to the non-linear nature of the detection statistics. Given a vector xmof size N×1, corresponding to a column of Xwith the values of one given feature (counter) in the N calibration observations, which we assume to be mean centred, its scaled version is given by ˜ xi=xiwi(15) Here we propose to define a set of values of w1, w2, . . . , wM to be chosen such that when applied to the calibration data X, detection by MSNM will be optimal in terms of AUC for one type or a set of types of attacks identified in a labelled data set. C. Optimization procedure The complete semisupervised system is depicted in Fig. 2. A detailed description of the optimization procedure follows: 1) Row vector w0,1×M, is initialized such as w0= (1,1,...,1)M. The number of individual solutions in each iteration, K>> 1, and the level of exploration rc are chosen. 2) Row vector w1 0,1×M, is generated such as w1 0= w0+rc·r1 0where r1 0is a 1×Mrandom vector whose entries are ≈N(0,1). 3) The weighting vector w1 0is applied to the calibration data Xand the detection performance of the resulting MSNM system AUC1 0is recorded. 4) Step 1 and 2 are repeated Ktimes: resulting vectors w1 0,w2 0,...,wK 0are arranged in a K×Mmatrix S0and the corresponding system performances in a K×1vector y0= (AUC1 0, AUC1 0, . . . AUCK 0)t. The k-th row of S contains the k-th vector of weights wk 0and yk=AUCk 0. 5) A xPLS model is fitted regressing y0on S0obtaining a set of regression coefficient ˆ B0. 6) The current solution is updated: w1=w0+ 3 ·ˆ B0. 7) We check for convergence in w, and otherwise loop to step 2. MSNM PCA ROC R2R GPLSSPLSPLS AUC 𝝎 Calibration Data (unsupervised) Test Data (supervised) Fig. 2: Semi-supervised MSNM Optimization scheme. TABLE I: Features of the calibration and the test sets in the UGR’16 dataset. Feature Calibration Test Capture start 10:47h 03/18/2016 13:38h 07/27/2016 Capture end 18:27h 06/26/2016 09:27h 08/29/2016 Attacks start N/A 00:00h 07/28/2016 Attacks end N/A 12:00h 08/09/2016 Number of files 17 6 Size (compressed) 181GB 55GB # Connections ≈13,000M ≈3,900M VII. CASE STUDY: UGR’16 DATA SET A. Experimental Framework We evaluate our approach to optimize variables scaling in a real scenario. The dataset considered is the publicly available UGR’16 dataset [15]. These data consist of Netflow network traces taken from a real Tier 3 ISP network composed of virtualized and hosted services of many companies and clients. Netflow sensors were located in the border routers of the network, capturing all the incoming and outgoing traffic from the ISP. All the details related to the dataset are summarized in Table I and can be consulted in [15]. Two blocks of data are provided, one for training models (calibration set), and the other for testing the results obtained from those models (test set). The UGR’16 dataset is especially interesting for our experiments because the collected traffic includes controlled attack traffic against fake victims generated by 25 virtual machines that were deployed within the network. Thus, our aim is to test if our optimization algorithm is able to capture the relevant variables for every attack type and properly scale them to achieve good detection results. Although the variety of attacks in this dataset is limited, this is the only recent dataset (see TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 7 TABLE II: Variable values considered as features in our detection system. Variable #features →values Source IP 2 →public, private Destination IP 2 →public, private Source port 50 →specific services, Other Destination port 50 →specific services, Other Protocol 5 →TCP, UDP, ICMP, IGMP, Other Flags 6 →A, S, F, R, P, U ToS 3 →0, 192, Other # Packets in 5 →very low, low, medium, high, very high # Packets out 5 →very low, low, medium, high, very high # Bytes in 5 →very low, low, medium, high, very high # Bytes out 5 →very low, low, medium, high, very high survey of datasets in [15]) that includes real background traffic for a considerable amount of time (4 months), which is an essential requisite to properly evaluate the false positive rate in our detection results. The attack traffic was performed during the test set collection, in particular during its first 12,000 observations, and it presents these different patterns: •Low-rate DoS (dos): TCP SYN attack during 3 minutes by using the tool hping3. There are three different variants, where one-to-one or many-to-one are combined with different schedulings. •Port scanning (scan11): Continuous one-to-one scanning from an attacker to a single victim’s IP during 3 minutes by using the nmap tool. •Port scanning (scan44): Continuous scanning from 4 different attacker machines to four victim’s IP in parallel during 3 minutes by using the nmap tool. •Botnet traffic (nerisbotnet): The test set includes botnet traffic traces obtained from the execution of the malware known as Neris, corresponding to the capture CTUMalware-Capture-Botnet-42 available in [39]. In this malware version, infected bots send SPAM, connect to an HTTP C&C server and use HTTP to perform some ClickFraud. For the processing of the dataset, we consider time intervals of one minute. All the flows during an interval are aggregated and summarized into a M-dimensional vector, which corresponds to an observation. In particular, we define a set of M=138 network-related features, corresponding to 11 different Netflow variables, as shown in Table II. Calibration data was cleaned of outliers following the Phase 1 approach in [8], so we expect it to be mainly composed of normal observations. A main real threat identified and cleaned from the calibration and test data is a SPAM campaign driven from some of the virtual machines located in the ISP. The cleaned calibration set is used to calibrate the MSNM system. The first 12,000 observations in the cleaned test data were split in two independent parts with 6,000 observations. The first set is used within the optimization to select the optimum scaling. The second set is used to validate the results. We also used the unclean second test data set, including the SPAM traffic, to assess the performance of the methods with previously unseen attacks. B. Results To assess the performance of the semi-supervised MSNM approach, we compare it with the unsupervised MSNM and the Support Vector Machine (SVM) technique, bluethe latter being a representative of a supervised approach. Two variants of the unsupervised MSNM for two preprocessing schemes, Mean-Centering (MSNM-MC) and Auto-Scaling (MSNM-AS), are considered. Two variants of the SVM are also considered, one based on the linear kernel (SVM-L) and one on the radial basis function kernel (SVM-RBF). To calibrate the anomaly detector, unsupervised techniques only make use of the (cleaned) calibration data, described in the previous section. Supervised techniques employ only the (cleaned) first test data set, which includes both normal traffic and all the types of artificial attacks. SVM metaparameters are selected according to recommendations in the Matlab documentation of function ’fitcsvm’. The semi-supervised MSNM is initially set to the MSNM-MC, and then optimized with the R2R algorithm using the (cleaned) first test data set. In all optimization runs, Kis set to 100 and rcis set to 0.01. Methods are compared in terms of the AUC computed for the (cleaned) second test set, and thus from independent data to that used in the calibration of the anomaly detectors. We derive confidence intervals in the performance using resampling techniques without replacement. Fig. 3 shows the AUCs of the different detection approaches for the four artificial attack patterns. For the calibration, no distinction is made on the type of attack, that is, the anomaly detectors are calibrated using two types of labels: normal and attack. For the evaluation, each group of AUCs is computed comparing normal data with the specific type of attack, leaving out the observations corresponding to the other attack patterns. We expect supervised approaches to outperform the others in this experiment, since all the attacks under evaluation were used in their calibration. As expected, unsupervised methods generally yield the worst results, in particular the MSNM-MC approach. The MSNM-AS is generally outperformed by semisupervised techniques. The improvement is more notable in the case of the nerisbotnet pattern. It is remarkable that this is generated by a botnet that is equipped with mechanisms to hide its traffic and, thus, the detection of this traffic following unsupervised methodologies is a real challenge. In general we can conclude that the R2R optimization is improving the performance of the MSNM, but this improvement is limited when diverse attacks patterns are optimized at the same time, since different patterns may counteract in the optimization. Supervised approaches based on SVM yield very good results. In particular, the SVM-RBF shows a similar performance to semi-supervised MSNM, and the linear kernel is among the best anomaly detectors for all the anomalies and clearly outperforms the other methods in ’nerisbotnet’. One typical advantage that is claimed for unsupervised (and most semi-supervised) methods in comparison to supervised approaches is that they are capable of identifying new threats. This is because they are based on a model of normality, and any new threat that does not follow that model can be detected. To check whether the semi-supervised MSNM retains this capability after the scaling optimization, we compared its TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 8 dos scan11 scan44 nerisbotnet AUC 0.5 0.6 0.7 0.8 0.9 1 MSNM-MC MSNM-AS MSNM-R2R-PLS MSNM-R2R-SPLS MSNM-R2R-GPLS SVM-L SVM-RBF Fig. 3: AUC values per artificial attack type. Supervised and semi-supervised approaches calibrated without distinguishing attack types. Quartiles (25% and 75%) and median are shown in the bars. performance with the other methods in the detection of the real SPAM traffic in the (unclean) second test data set. The anomaly detectors are the same as those used before, and therefore they were calibrated using independent and cleaned data, in absence of any SPAM trace. Therefore, supervised techniques are not expected to outperform the others in this experiment. Results are shown in Fig. 4. We can see that the semi-supervised methods provide a good performance in comparison to the rest, and even outperform the unsupervised MSNM. This cannot be understood as a general result. In principle, all MSNM variants have the same apriori probability to detect new threats, since this depends very much on the relationship between the new threat and the normality model. As the new threat can be anything, this relationship will likely vary from case to case. However, this experiment shows that the optimization did not have a negative effect on the ability of MSNM to detect the unseen threat. Regarding supervised approaches, the SVM-L yielded a very poor result on the new attack type, as expected for a supervised technique, but the SVM-RBF showed the best result among the methods. blueThis result can be explained due to the special properties of SVMRBF. This method is supervised since during calibration, the classifier is optimized to distinguish between the two classes of data. However, the model identifies a support that contains one of the classes, and the rest of the feature space is assigned to the other class. If the former is the class for normal observations, and the latter models the attacks, the configuration of SVM-RBF mimics that in MSNM, or in general of an anomaly detector, and is indeed very useful to detect new anomalies. Note, however, that if the labels for normal and anomalous classes are switched, the SVM-RBF provides the worst performance, and that, according to our previous discussion, this specific result is very dependent on the relationship between the new threat and the normality model. Another experiment we can conduct is to calibrate the semiMSNM-MC MSNM-AS MSNM-R2R-PLS MSNM-R2R-SPLS MSNM-R2R-GPLS SVM-L SVM-RBF AUC 0 0.2 0.4 0.6 0.8 1 Fig. 4: AUC values for the real SPAM traffic. Quartiles (25% and 75%) and median are shown in the bars. dos scan11 scan44 nerisbotnet AUC 0.5 0.6 0.7 0.8 0.9 1 MSNM-MC MSNM-AS MSNM-R2R-PLS MSNM-R2R-SPLS MSNM-R2R-GPLS SVM-L SVM-RBF Fig. 5: AUC values per artificial attack type. Supervised and semi-supervised approaches calibrated per attack type. Quartiles (25% and 75%) and the median are shown in the bars. supervised and supervised methods specifically per attack type. This approach can be used to calibrate ensembles of detectors, which may be a suitable approach when the profiles of different attacks vary to a large extent. Mimicking rulebased detectors, where each rule corresponds to a single type of attack, we can have an anomaly detector optimized for each attack type. Performance results are shown in Fig. 5. We can see that this approach is generally beneficial for semisupervised approaches, while in supervised methods it even degrades the results. Among the semi-supervised methods, sparse PLS methods tend to outperform standard PLS, in particular for the ’nerisbotnet’ attack. blueFrom the previous experiments, we can conclude that the R2R optimization leads to a general improvement of the MSNM performance, conforming a semi-supervised approach that is competitive with state-of-the-art techniques and that retains the capability to detect new threats. Note that the un- TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, MAY 2017 9 supervised MSNM was generally outperformed by supervised techniques, but this comes at a price: supervised techniques are not generally applicable to most real cases, where the labelling of observations is not available. This is actually the commonplace in the cybersecurity industry [40]. Semi-supervised techniques can still be used when none or a partial labelling is available. Furthermore, the MSNM approach has the additional advantage over state-of-the-art supervised techniques to be an interpretable model, and thus easier to use and understand by practitioners [8]. While above we only compared in terms of detection, MSNM also provides diagnosis support, that is, information about why an attack was identified as such. This is actually a principal ability to reduce the time of response to an attack or to quickly identify a false positive. Black-box models, like the non-linear SVM, cannot be interpreted. Therefore, they do not provide the information about why an attack was identified. Finally, it should be noted that none of the SVMs generally outperformed the semi-supervised methods in all the detection experiments performed. It turns out that the result of the R2R optimization can also be interpreted. Fig. 6 compares the scaling profiles obtained from the optimization with PLS, SPLS and GPLS, and with and without distinguishing among the attack types. In general, sparse methodologies provide clearer profiles, with lower numbers of picks and easier to interpret. The picks reflect those features that are relevant for the detection of the type of attack. We can also see that the optimization using all attack types (GloOpt) is dominated by the ’nerisbotnet’ attack pattern, since in all cases the profile shows a large pick in the same feature than the profile specifically optimized for that attack. Again, this illustrates the limitation of using this semisupervised approach for a set of disparate attacks, rather than on a per-attack basis. To interpret the profiles in Fig. 6, we selected those peaks exceeding the average scaling value plus one standard deviation and listed them in Tables III, IV and V. Recall that these variables will have a higher influence on the MSNM detector, but the rest of variables will also impact the detection, to a lesser extent. In Table III (PLS) we see that the GloOpt selects three features: the number of connections with source port 1080 (sport_socks), with source port between 49152 and 65535 (sport_private) and with destination port 6667 (dport_irc). The first feature is related to the SOCKS proxy, an Internet proxy service. That port has been associated in the past to several types of attacks, mainly trojans and SPAM. The second feature is a very general one, and might have been incorrectly selected due to the low signal-to-noise ratio in R2R-PLS. The last feature is related to the nerisbotnet. It is out of question that the IRC port is also related to normal activity. However, in the traffic of the network we are monitoring, the amount of IRC is low and the counter in dport_irc can be a valid means to detect malicious activity. Looking at the selection in Table IV, for SPLS, we can see that the number of features selected is reduced and more interpretable, but we still find some potentially inconsistent results. For instance, in the list of the most relevant variables for nerisbotnet, we can see that dport oracle is present, despite TABLE III: Variables with highest weights from PLS optimization. GloOpt: sport socks, sport private dport irc SpecOpt dos: sport telnet, sport rapservice dport http SpecOpt scan11: sport syslog, sport reserved dport kpasswd tcpflags ACK npackets medium SpecOpt scan44: sport cups dport telnet, dport kpasswd SpecOpt nerisbotnet: dport irc tcpflags PSH npackets medium the fact that this botnet does not generate any traffic towards the oracle port. This potential inaccuracy is not affecting the detection results of this attack, as shown in Fig. 5, which are considerably improved in comparison to the other MSNM variants. Using GPLS (Table V) the features are subsequently reduced. While this is in general convenient, it does not necessarily imply a benefit in terms of performance. For instance, the global optimizer is mainly focus on the nerisbotnet attack. Differences between SPLS and GPLS may be associated to the degree of sparseness of the models, which is governed by the specific metaparameters used but also by the specificities of the training data. In SPLS we used Nx= 2, the most parsimonious value that results in multivariate regression vectors. In GPLS we set γ= 0.8. The same metaparameters may result in opposite sparseness levels for a different data set. However, we can generally conclude that the use of sparse methods within the R2R algorithm led to improvements in terms of AUC and parsimony, and therefore of interpretability of results. TABLE IV: Variables with highest weights from SPLS optimization. GloOpt: sport private dport kpasswd, dport irc SpecOpt dos: sport telnet SpecOpt scan11: dport kpasswd SpecOpt scan44: sport ldaps dport telnet, dport kpasswd SpecOpt nerisbotnet: dport oracle, dport irc TABLE V: Variables with highest weights from GPLS optimization. GloOpt: dport irc SpecOpt dos: sport telnet SpecOpt scan11: dport kpasswd SpecOpt scan44: dport telnet, dport kpasswd SpecOpt nerisbotnet: dport irc VIII. DISCUSSION AND CONCLUSION In this paper, we provide a solution to combine the advantages of unsupervised and supervised learning in the context of intrusion detection. For this, we use the Multivariate Statistical Network Monitoring approach, recently proposed, and we enhance an optimization algorithm based on Partial