Towards Autonomous Defense of SDN Networks Using MuZero Based Intelligent Agents

Received July 7, 2021, accepted July 23, 2021, date of publication July 28, 2021, date of current version August 6, 2021. Digital Object Identifier 10.1109/ACCESS.2021.3100706 Towards Autonomous Defense of SDN Networks Using MuZero Based Intelligent Agents JON GABIRONDO-LÓPEZ 1,2, JON EGAÑA 1, JOSE MIGUEL-ALONSO 3, (Member, IEEE), AND RAUL ORDUNA URRUTIA1 1Vicomtech Foundation, Basque Research and Technology Alliance (BRTA), 20009 Donostia-San Sebastián, Spain 2Faculty of Informatics, University of the Basque Country (UPV/EHU), 20018 Donostia-San Sebastián, Spain 3Department of Computer Architecture and Technology, University of the Basque Country (UPV/EHU), 20018 Donostia-San Sebastián, Spain Corresponding author: Jon Gabirondo-López ([email protected]) This work was supported in part by the Spanish Centre for the Development of Industrial Technology (CDTI) through the Project ÉGIDA—RED DE EXCELENCIA EN TECNOLOGIAS DE SEGURIDAD Y PRIVACIDAD under Grant CER20191012, in part by the Spanish Ministry of Science and Innovation under Grant PID2019-104966GB-I00, in part by the Basque Business Development Agency (SPRI)-Basque Country Government ELKARTEK Program through the projects TRUSTIND under Grant KK-2020/00054 and 3KIA under Grant KK-2020/00049, and in part by the Basque Country Program of Grants for Research Groups under Grant IT-1244-19. ABSTRACT The Software Defined Networking (SDN) paradigm enables the development of systems that centrally monitor and manage network traffic, providing support for the deployment of machine learningbased systems that automatically detect and mitigate network intrusions. This paper presents an intelligent system capable of deciding which countermeasures to take in order to mitigate an intrusion in a software defined network. The interaction between the intruder and the defender is posed as a Markov game and MuZero algorithm is used to train the model through self-play. Once trained, the model is integrated with an SDN controller, so that it is able to apply the countermeasures of the game in a real network. To measure the performance of the model, attackers and defenders with different training steps have been confronted and the scores obtained by each of them, the duration of the games and the ratio of games won have been collected. The results show that the defender is capable of deciding which measures minimize the impact of the intrusion, isolating the attacker and preventing it from compromising key machines in the network. INDEX TERMS Automated response, cybersecurity, intelligent agents, Markov games, MuZero, network security, OpenFlow, software defined networking. I. INTRODUCTION The number of Internet users has grown considerably in recent years and more and more services—from e-commerce to banking—are provided over the Internet. Consequently, not only the number, but also the severity of cyberattacks on organizations and businesses has been increasing, causing millions of dollars in losses [1]. Enterprise security systems have traditionally been designed and implemented manually by expert personnel and the response to attacks or intrusions has also been carried out by those technicians. The new network infrastructure paradigm introduced by Software Defined Networking (SDN) has enabled the development of automatic attack detection and mitigation systems based on machine learning techniques [2]–[6]. Compared to conventional detection The associate editor coordinating the review of this manuscript and approving it for publication was Nabil Benamar . systems or human-driven response strategies, these systems can detect attacks faster and more accurately, and even implement countermeasures autonomously and automatically, minimizing the reaction and response time to an attack and thus reducing the damage caused in the network. Additionally, the development of reinforcement learning algorithms capable of outperforming human marks in board games—such as chess or Go—has fostered the idea of approaching network intrusion as if it were a Markov game, using these algorithms to train intelligent agents that can autonomously find security strategies to mitigate intrusions [7]. This paper presents an intelligent system which can minimize the impact caused by an intrusion in an SDN network by autonomously choosing and executing adequate countermeasures. The problem is posed as a partially observed Markov game in which the attacker (the intruder) tries to compromise a critical machine and the defender (an automatic security 107184 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ VOLUME 9, 2021 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Network agent that manages the infrastructure) tries to reduce the impact of the attack. The game has been designed taking into account that it must be able to represent the state of the nodes of a real network and that the actions of the defender in the game must be implementable by an SDN controller. The model has been trained using the MuZero model-based reinforcement learning algorithm [8]. The major contributions of this paper are: •The design of a Markov game suitable to represent attack attempts to a computer network in which nodes have vulnerabilities. •The implementation of the necessary countermeasures by means of an SDN controller and OpenFlow-enabled switches. •The implementation of a virtual environment in which the countermeasures chosen by the defender are carried out autonomously in an emulated SDN network. The rest of this paper is organized as follows: Section II introduces the context of SDN and reinforcement learning and presents the state of the art in both fields. Section III presents the proposal made in this work, explaining how the game works and its integration with an SDN network. Section IV shows the information regarding the training of the model and the results obtained. Finally, Section Vevaluates these results and highlights some possible lines of future work. II. BACKGROUND AND STATE OF THE ART A. BACKGROUND 1) SOFTWARE DEFINED NETWORKING In recent years, the applications and services offered on the Internet have become increasingly complex and demanding. This has highlighted the need for a paradigm shift in the world of network infrastructures, as conventional networks lack the dynamism and adaptability required by the new platforms [9]. Conventional networks are composed of elements (such as routers and switches) usually treated as black boxes that rely on limited or manufacturer-specific control interfaces. These devices are typically configured individually, and traffic management decisions are made directly at the device level, intermingling the control plane responsible for making such decisions with the data plane composed of the network elements [10], [11]. The lack of independence between these two planes makes it extremely difficult to dynamically adapt the network to the needs of the applications deployed, or to cope with certain type of events. This problem is one of the main causes of the ‘‘ossification of the Internet’’: the development of new protocols and infrastructures has been severely hampered by the very architecture of network elements [10], [12]. Software defined networking proposes systems in which the control plane and the data plane are completely decoupled, allowing centralized configuration of the infrastructures, as shown in Fig. 1. Although there are different FIGURE 1. Structural differences between (a) a conventional network and (b) a SDN. SDN architectures, this paper only considers those based on the OpenFlow protocol [12] in the communication between the control plane and the data plane, as it is a widely used protocol endorsed by the Open Networking Foundation (ONF)—the non-profit organization dedicated to the development and standardization of SDN networks [11]. The elements that constitute the network (known as OpenFlow switches) are only responsible for forwarding traffic, whereas decisions are made by the network controller. The controller installs flow tables in the switches using the OpenFlow protocol. The tables consist of flows that determine how packets that meet certain criteria are processed and forwarded. A flow is composed of different elements: the Match fields are the conditions that the incoming packet has to meet (such as the source IP address or the incoming port, for example) to execute the instructions defined in the Instructions field. These instructions may include blocking all traffic coming from some port, forwarding the matching traffic through a specific output port or forwarding some packets to the controller, for example. When an incoming packet matches with more than one flow, the one with the highest value in the Priority field is executed. The Counters field collects the number of packets processed by that flow, the Timeout field defines how much time must pass without inputs for a flow to be dropped and the Cookie field is just an identifier set by the controller [13]. When a packet that does not match any flow is received, the instructions defined in the special tablemiss flow are executed, which may include actions such as dropping the packet or sending it to the controller for further analysis. VOLUME 9, 2021 107185 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Networks FIGURE 2. Diagram of the components of a SDN and the interfaces between them. Fig. 2shows the structure of this type of SDN networks, which consists of three main groups of components (planes) and two interfaces that allow communication between them. There are more complex structures that include several controllers and more interfaces, but in this paper, for the sake of simplicity, we only consider the case of a single controller. The controller uses the Southbound Interface (SBI) to communicate with the network devices. In this way, the controller manages all packet processing by installing flows in the switches. The controller can also collect information and statistics about the data plane. The SBI is vendorindependent and the most widely used is the aforementioned OpenFlow [14]. The controller also communicates with the application plane through a Northbound Interface (NBI). The application plane consists of several SDN applications that implement traffic control and management strategies, such as load balancing, packet filtering (firewalling), traffic monitoring, etc. Therefore, the NBI allows applications to process the data plane information received by the controller, perform actions at a high level and have the controller execute them on the infrastructure. There are no standardized NBIs, they are controller-specific. However, they are typically built using REST APIs [15]. An important property of a SDN is its ability to react to different events. The controller can perform an initial, proactive configuration of the network. However, this configuration can be dynamically modified (by adding or removing flows), adapting it to the traffic observed in the network, whether harmless or dangerous. SDN networks have therefore been used on many occasions to design systems capable of detecting and responding dynamically to cyberattacks [6], [16], [17]. 2) REINFORCEMENT LEARNING AND MARKOV GAMES Reinforcement Learning (RL) is an area of machine learning that studies how an agent learns to make decisions by following a trial-and-error strategy [18], [19]. The main elements of the RL models are shown in Fig. 3. The agent is the subject of the training and the one who has to learn to perform decisions. The environment represents the world with which the agent interacts [20]. In each interaction t, the agent receives an observation ot∈Oof the state of the environment st∈S, FIGURE 3. Summary of the main elements of reinforcement learning problems and the interactions among them. where Ois the set of observations of the ensemble of possible states S. Based on the observation it decides which action at∈A(st) to take, where A(st) is the set of possible actions in state st. Then, the environment changes in response to the agent’s action or independently. Once the action is executed, the agent receives a reward rt+1and a new observation of the environment, which allows it to evaluate the effect of the action just performed, as well as to perform a new step. It is worth noting the difference between the state and an observation of the environment. The state represents all the information of the environment and thus defines it completely. An observation, however, is a view that usually contains only a part of the information about the state. In the particular case where the observation includes all the information about the state, the environment is fully observed; otherwise, the environment is partially observed. In many cases, the term ‘‘state’’ is used to actually refer to ‘‘observation’’, and the symbol stis used instead of ot. In this paper we have chosen to use an explicit expression of the observation, for two main reasons: the first is that a main characteristic of the model we have implemented is that it is based on a partially observed environment; the second is that the authors of the algorithm used to train the model use this notation in their article [8]. A main piece of a RL model is the policy, i.e., the strategy that determines what action the agent should take given an observation. This function is updated as the model is trained. In general, the policy is the probability that the agent will take an action atbased on the observation ot. The policy πusually depends on a set of parameters θ, so its complete representation would be πθ(at|ot). A succession of states visited and actions taken, τ= (s0,a0,s1,a1, . . .), is often referred to as an episode. The agent’s main goal is to maximize the sum of all rewards obtained in an episode. This sum of rewards is known as return. A strategy called discounted return is often used, which makes rewards earned several steps back worth less than rewards earned closer in time. This return is defined as R(τ)=rt+1+γrt+2+γ2rt+3+. . . = ∞ X k=0 γkrt+k+1(1) where γis a parameter, 0 ≤γ≤1, known as discount rate. Beyond the intuition established so far, those processes studied in RL are Markov Decision Processes (MDP) which 107186 VOLUME 9, 2021 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Network are defined by a six-component tuple as shown in (2) [21]. M= hS,A,Pss0,Ra ss0, γ, ρ0i(2) being Pa ss0=Pr st+1=s0|st=s,at=a(3) Ra ss0=Ert+1|at=a,st=s,st+1=s0(4) where Pa ss0is the probability of reaching state s0by applying an action ain state s, and Ra ss0is the expected reward after that transition. The term ρ0:S7→ [0,1]is the distribution of the initial state [7]. The MDPs are so named because the transition from state st to state st+1satisfies the Markov property (5): it only depends on the mentioned states and not on the rest of the previous states [20]. Pr [st+1|st]=Pr [st+1|s1,...,st](5) As mentioned in the introduction, the model we describe in this paper is a Markov game based on a Partially Observed Markov Decision Process (POMDP), in which the agent makes decisions based on its (partial) observation of the system and not directly on the state. At a step t, the observation ot∈Ois related to the state st∈Sby the function Z, so a POMDP is defined as follows: MP= hS,A,Pss0,Ra ss0, γ, ρ0,O,Zi(6) where Ois the set of observations of the system and Z= Pr [o|s]is the probability of observing oin state s. Bringing the two fields together, the problem posed in RL can be understood as the search for the optimal policy π∗that maximizes the expected value Eof the sum of rewards of an MDP for a maximum of Tsteps: π∗=arg max π E"T X t=0 γtrt+1#(7) In RL algorithms, value functions are defined, which calculate the expected return of actionstaken from an initial state and following a certain policy. Formally, in the case of MDPs, the function Vπ(s) called state-value function for policy πis defined as Vπ(s)=Eπ[R|st=s] =Eπ"∞ X k=0 γkrt+k+1|st=s#(8) and represents the expected return when the agent follows a policy πstarting from a state s. Analogously, the value of taking an action ain a state s, the action-value function for policy π, is also defined as Qπ(s,a)=Eπ[R|st=s,at=a] =Eπ"∞ X k=0 γkrt+k+1|st=s,at=a#(9) which represents the expected return obtained by following a policy πafter performing action ain state s. The Bellman equations set out in (10) and (11) show that the value of a state or state-action pair is the expected return obtained by following the optimal policy π∗defined in (7) [22]. V∗(s)=max a Ert+1+γV∗(st+1)|st=s,at=a(10) Q∗(s,a)=Ert+1+γmax a0Q∗(st+1,a0)|st=s,at=a(11) Recall that the ultimate goal of a RL algorithm is to find the optimal policy π∗. For finite MDPs, (10) has a unique solution, so once V∗has been calculated it is relatively simple to obtain the optimal policy, since if this function is used to evaluate the short-term actions (evaluating the state sarrived at after performing a) the policy followed when taking the best option at each step is the optimal long-term policy [20]. Similarly, using Q∗, the agent no longer has to search for the new state sthat maximizes V(s), but only has to search for the action athat results in a higher Q(a). This allows the agent to make decisions without having to know all the consecutive states, that is, without having to know the dynamics of the environment with which it interacts. Estimating these value functions is a fundamental part of model-free algorithms, which seek to find the optimal policy without having a model of the environment, as the Q-learning [23] algorithm does, for example. On the other hand, model-based algorithms also use such functions to construct models that have the same value functions as the original environment and are therefore equivalent. The algorithm MuZero [8] is one of the most recent examples of model-based algorithms. Finally, it only remains to mention that decision processes involving multiple agents are studied within game theory [24], [25]. These agents interact with the environment simultaneously or in turn and each one obtains a corresponding payoff, thus refining their policies. When designing a game, the following factors must be determined, which also serve to classify them [26]: •Zero-sum: Whether the sum of all players’ rewards is 0 or not. In two-player games, rewards sum to 0 if both players are strictly competing against each other. •Information: Whether the game state is fully or partially observable by the players. •Determinism: Whether the outcome of the game depends to some extent on luck. •Sequential: Whether the agents interact sequentially or simultaneously. •Discrete: Whether actions are implemented in real time or not. Specifically, Markov games are the theoretical context of multi-agent RL algorithms [27]. A Markov game MGwith Nagents is defined by a tuple similar to the one given in (5): MG= hS,A1,...,AN,T,R1,...,RN, γ, ρ0i.(12) VOLUME 9, 2021 107187 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Networks Compared to an MDP, the Markov game defined by (12) presents a list of sets of possible actions to be performed by the Nagents in which to an agent i∈[1,N]corresponds its set Aiinstead of the unique set A. The transition function Toperates on all possible combinations between the set of states and the combined action space of all agents: T:S× A1×A2. . .×AN7→ S. Similarly, the functions Ridefine the rewards obtained by the agents: Ri:S×Ai7→ R. In the case of a game based on a partially observed process, the definition (12) also includes the lists of sets of observations O1,...,ON and of observation functions Z1,...,ZN[7]. B. STATE OF THE ART 1) INTRUSION DETECTION AND PREVENTION SYSTEMS Intrusion detection systems (IDS) attempt to identify malicious activity occurring on a network (network IDS) or on a computer (host IDS) by capturing and analyzing different sources of information. A network IDS captures and analyses the packets traversing the network (both protocol headers and content), as well as aggregate measurements on traffic: sessions, IP addresses involved, traffic volumes, etc. They are usually guided by a set of rules against which the captured data is compared. If there is a match, an alert is issued: the IDS has detected suspicious activity. Intrusion detection and prevention systems (IDPS) go one step further: they are not only capable of detecting an intrusion attempt, but also of reacting to reduce or completely prevent the malicious effects by executing some countermeasures. To this end, an IDPS is able to perform network modifications, for example by automatically adding a rule in the corporate firewall [28]. SDNs are particularly suitable for IDPS deployment. They can be deployed as control applications, capable of collecting information from the network with the help of the controller. The detection subsystem will then search the collected data for signs of intrusion. If dangerous activities are detected, the IDPS intelligence will choose the best course of action, consisting of a series of countermeasures that affect the flow tables of the network devices. The necessary changes will be implemented through requests to the controller. The work presented here is intended to be only one piece of a complete IDPS. We assume that there is a working IDS that tells our intelligent agent what type of attack has been detected. Our agent will choose the best countermeasures and ask the controller to apply them. This is the part we focus on. The design and implementation of a good network IDS goes beyond the scope of our work. The interested reader can find relevant and up-to-date information on network IDS in [29]–[32]. 2) AUTONOMOUS INTRUSION MITIGATION SYSTEMS IN SDNs As stated before, the abstraction, flexibility and programmability of infrastructures based on the SDN paradigm have enabled the development of attack detection and automatic countermeasure deployment systems [5], [33]. Focusing on the defense activities, several intelligent systems capable of deciding which countermeasures to take have been proposed. One of the most complete systems is NICE (Network Intrusion detection and Countermeasure sElection in virtual network systems) [6], which integrates the detection of infected virtual machines in cloud computing environments with the automatic deployment of optimal countermeasures, which include patching the software of an attacked machine or quarantining a suspicious node. The entire infrastructure proposed in NICE is based on SDN elements using the OpenFlow protocol. Each machine in the network has a vulnerability registered in the Common Vulnerabilities and Exposures (CVE) list [34], and the attack is represented by an attack graph where each node represents the previous state or consequence of the exploit on one of the machines. The selection of the response to an attack is made taking into account the intrusiveness and cost of the countermeasure, so as to minimize the impact of the response itself. The SnortFlow [17] system follows the guidelines set by NICE, but all proposed countermeasures are based solely on actions taken on the network itself (such as redirecting traffic or blocking a port) and are implemented via an OpenFlow controller. Apart from proposals focusing on a practical aspect of countermeasure implementation, works such as [7] propose to use multi-agent Markov games to search for defense strategies against cyberattacks. Specifically, the game is set up as a partially observed game and the authors study different scenarios in which the abilities of the attacker and defender vary. In their game, the attacker must move through a graph to reach a target machine, simulating a problem similar to those posed in the Cyber Ranges [35]. In that work, the model-free algorithms PPO [36] and REINFORCE [37] are used to train the agent. 3) REINFORCEMENT LEARNING ALGORITHMS AND THE MuZero ALGORITHM A classic way of evaluating reinforcement learning algorithms is to pit them against games such as chess or Go in an attempt to beat human scores and obtain superhuman results. One of the first milestones in this field, apart from specialized computers for winning at chess or shogi [38], was the AlphaGo algorithm developed by the company DeepMind, which managed to beat a professional Go player for the first time [39]. A modification of that algorithm called AlphaGo Zero [40] achieved superhuman results playing Go and led to its successor AlphaZero, which managed to beat world champions in chess, shogi and Go with only 24 hours of training in each case [41]. AlphaZero was trained using 5000 first-generation TPUs to generate selfplay games and 64 second-generation TPUs to train the neural networks. Model-based algorithms have repeatedly outperformed humans in classic games such as checkers, chess, Go or poker, as they are able to develop a long-term strategy. 107188 VOLUME 9, 2021 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Network FIGURE 4. Diagram of the proposal made in this work, in which the trained model, the network controller and the simulation environment are completely independent. However, those algorithms have tended to perform poorly when confronted with complex environments like the Atari 2600 computer games. Model-free algorithms obtain better results in these environments [42]–[44], but perform much worse than model-based algorithms in games such as those mentioned above, which require precise and sophisticated lookahead [8]. The MuZero algorithm introduced by DeepMind in 2020 (a model-based algorithm) completely changed the state of the art. It obtains performances comparable to those of model-free algorithms in visually complex environments, while maintaining superhuman results in precision planning tasks such as classic board games, as previous model-based algorithms [8]. III. PROPOSED ARCHITECTURE In this work we propose to use the MuZero algorithm to train a model that is able to decide which countermeasures to implement when an attacker intrudes an SDN network, with the intention of mitigating the attack and minimizing the number of compromised machines. The interaction between the intruder and the defender has been modeled as a stochastic, partially observed, zero-sum Markov game, in which both agents perform actions sequentially. Furthermore, the model has been integrated into a realistic SDN network emulation environment based on Mininet [45]. The (virtual) switches deployed within the Mininet environment support the OpenFlow specification and accept control from an external controller—we have used the Ryu controller [46]. Fig. 4summarizes the three blocks developed for this work. Below we present the programs and libraries used for the implementation of our proposal, the decisions related to the game design and the virtual network environment implemented. The hardware used to train and test the model is described in Section IV-C. A. DESIGN OF THE GAME This section explains the different elements designed to obtain a partially observed Markov game representing the interaction between the attacker and the defender. 1) AGENTS AND THE ENVIRONMENT The game is set up as a match between two players, the attacker and the defender, in which the attacker tries to compromise key targets in a network and the defender tries to mitigate the intrusion. Although the MuZero algorithm was designed to deal with board games or video games, in this case the environment to be simulated is a network of interconnected computers, so we have chosen to use a graph to represent it. For practical purposes, MuZero can be used to train agents in any environment represented by a vector or a matrix. The translation from the network graph to a matrix suitable for MuZero is explained in Section III-A2. In this game, the attacked network is composed of N vulnerable nodes, of which mare part of a honeynet. A honeynet is an isolated part of the network where machines (called honeypots) are used as traps for attackers [47]. These machines do not provide actual services, but are vulnerable to attacks, causing attackers to waste time and resources exploring while allowing defenders to obtain intruder-related information [48]. This structure is similar to the one proposed by the Science DMZ [49]. Inspired by the NICE project, each node in the network (also known as host) is assigned a Base Score (BS), an Exploitability, an Impact, and a Scope. The National Vulnerability Database (NVD) [50] uses those factors to quantitatively assess a vulnerability, which represents a weakness in software and hardware components of a system that, when exploited, negatively affects its confidentiality, integrity, or availability. In a real setup where the IDS would detect the intrusion, those factors would be computed using the equations defined by the version 3.1 of the Common Vulnerability Scoring System (CVSS) [51]. According to that standard, the value of the BS depends on the Impact subscore (ISS), Exploitability, and Impact. The impact sub-score is defined as ISS =1−[(1 −C)×(1 −I)×(1 −A)](13) where Cis the Confidentiality Impact, Ithe Integrity Impact and Athe Availability Impact. Those metrics measure the impact to the confidentiality, integrity and availability of the information resources caused by the exploitation of the vulnerability of a software component. The Exploitability of a host depends on the Attack Vector (AV ), the Attack Complexity (AC), Privileges Required (PR) and User Interaction (UI) according to (14). Those metrics represent the properties of the vulnerability that lead to a successful attack. The Attack Vector increases with the physical and logical distance between the attacker and a successfully compromised component. The Attack Complexity represents the conditions that are beyond the attacker’s control and that are necessary to exploit the component. The Privileges Required metric describes the privileges that the attacker must have before exploiting the vulnerable component and the User Interaction represents if the activity of another user other than the attacker is required to successfully compromise the vulnerability. Exploitability =8.22 ×AV ×AC ×PR ×UI.(14) The Impact of the vulnerability depends on its Scope (S). This factor determines whether the affected host is the one with the vulnerability (Unchanged) or a different one VOLUME 9, 2021 107189 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Networks (Changed). In the case where the Scope is Changed, the Impact is computed as Impact =6.42 ×ISS (15) whereas if it is Unchanged, Impact=7.52×(ISS −0.029)−3.25×(ISS−0.02)15.(16) Finally, BS is calculated differently depending on the Impact and Scope. If Impact =0, BS =0. If Impact >0, BS is computed as roundup min 1.08×(Impact+Exploitability),10 (17) if Scope is Changed, and as roundup min (Impact +Exploitability),10 (18) if Scope is Unchanged. The roundup function returns the smallest number, specified to one decimal place, that is equal to or greater than its input. In terms of the game, the Scope of a vulnerability determines what actions the attacker can take if he succeeds in exploiting it: if the Scope is Unchanged, the attacker could only read files from the target machine, whereas if the Scope is Changed, the attacker could scan and exploit the machines to which the compromised host is connected. The design decisions related with the Base Score, the Exploitability and the Impact will be discussed in Section III-A3. The initial network therefore consists of N−mhosts connected to each other forming a fully connected logical network and mhosts isolated from the main network but connected to each other (see Table 1and Fig. 6in Section IV-B). Each node is assigned a vulnerability and it can be in different states—normal (state 0), scanned (state 1) and attacked (state 2)—allowing only the transitions 0 → 1→2. Machines cannot revert to a previous state, so that it is guaranteed that there is a reduced number of possible actions. At the start of the game one of the hosts on the main network is set to flag (state -1) and the attacker’s main objective will be to reach that host; another host with Scope ‘‘Changed’’ is set to state 2, allowing the attack to start from there. In order to illustrate the explanations that follow, in the rest of this paper we will represent the state of the network using graphs, since they are also used to show the state of the game. Formally, a graph G=(V,E) is an ordered pair where Vis a set of nodes or vertices and Eis a set of edges such that E⊆ {{x,y} | x,y∈V∧x6= y}. In our representation of the system, the nodes in the graph represent the hosts, and the edges show that traffic between two hosts is not blocked (i.e. there is connectivity between them). As hosts can be in different states, a color coding has been chosen to represent them in the figures (see Fig. 5). In this coding, those machines in its initial state are represented by white background nodes, the explored ones by yellow nodes and the attacked ones by the purple nodes. A node with a dasheddotted border represents the flag. The label placed inside each FIGURE 5. Encoding used to represent the states in which the nodes can be. TABLE 1. Example of a game environment. The red colored row represents the attacked node and the green colored one is the flag. FIGURE 6. Graphical representation of the environment described by Table 1. node is just an identifier, and has no relevance from the point of view of the game. As an example, Table 1defines a possible environment consisting of N=12 nodes (with m=4 nodes forming a honeynet), and Fig. 6shows the graph representation of the network. It should be noted that these are reachability graphs that do not represent the physical topology of the network (consisting of switches, hosts and Ethernet links), since an edge between two nodes does not represent a physical link, but means that they can exchange packets. Note also that switches are not represented in these graphs. 2) THE OBSERVATIONS The evolution of the intrusion and its mitigation is modeled as a partially observed decision process, since neither the attacker nor the defender has full information of the state of the network. To represent this fact we have used three different graphs: the actual or general graph GG=(VG,EG) with the complete state of the network, the attacker’s graph GA=(VA,EA) with the partial view that the attacker has, and the defender’s graph GD=(VD,ED) that is also a partial view. 107190 VOLUME 9, 2021 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Network FIGURE 7. Calculation of the matrix Mrepresenting the observed state of the graph G. Remember that we need a matrix representation of a game board, so we need to translate the above graphs into equivalent matrices. Although the adjacency matrix representation of a graph makes it possible to reflect the existing (logical) connections in the network, it is not sufficient to describe it in its entirety, since it does not include the different states in which nodes can be. Taking advantage of the fact that the main diagonal of the adjacency matrix is null, if we represent the state (real or observed) of all nodes according to the coding described in Fig. 5as a diagonal matrix, a complete board can be represented by adding the two matrices. Therefore, the graphs GG,GAand GDare fully represented by the matrices MG,MAand MD, respectively. Fig. 7summarizes the derivation of a board matrix for a simple graph with three nodes. The three network graphs representing the state of the game are different from each other; an action taken by a player (explained in Section III-A3) only has an effect on the general graph and on the graph of that player. Consequently, each player only has a partial observation of the game. Fig. 8shows three graphs/matrices computed for the same state. The actual state can be described as follows: starting from the initial situation determined by Table 1, the attacker has attacked machine 4 and has explored machine 2. In reality, machine 1 is isolated from the rest (as seen in the graphs GGand GD) but the attacker has not yet noticed the change in connectivity between the nodes of the network. Similarly, the defender has detected that machine 1 has been attacked (and we can assume that it has taken some of the actions explained in Section III-A3) but does not notice that node 2 is in state 1 and node 4 is in state 2. 3) THE ACTIONS The following conditions have been considered in order to design the actions that each player can execute: •Only defensive actions that can be implemented using an SDN controller are considered, leaving out of the study options such as software upgrades or changes to the underlying physical topology—such as adding or removing a switch or changing the switch/port to which a host is connected. •Defensive actions can only be applied against machines that have already been attacked, but network monitoring is allowed. •Offensive actions can be only carried out against the hosts: both switches and the controller are invulnerable. FIGURE 8. Example of the general state of a game and the observations seen by the defender and the attacker. •The MuZero algorithm requires the board size to remain constant throughout the game, so the number of nodes cannot vary: new machines cannot be deployed or removed completely. It should be noted that an agent’s decision making is based on its observation of the game, not on the actual state. Therefore, legal actions that an agent can take theoretically may not be feasible in practice or may not have the expected effect due to differences between the agent’s observation and the actual state. Moreover, as mentioned above, this is a stochastic game, so there are actions that have a probabilistic factor. For example, whether or not the attacker succeeds in exploiting a vulnerability in a machine depends directly on the characteristics of that vulnerability, so the agent will not always get the same result. Actions available to the defender are: •Check status The defender verifies whether the state of a node corresponds to that of its observation. If a node has been attacked, the probability of detecting it is proportional to the Base Score of the vulnerability of the attacked machine. Pr [oD(ni)=2|s(ni)=2,a=check]=BS/(10N) (19) VOLUME 9, 2021 107191 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Networks FIGURE 9. Main network and honeynet status before redirecting all traffic from node1 to honeynet nodes (up) and after (bottom). where oD(ni) and s(ni) are the defender’s observation and the real state of the affected machine niand Nis the total number of nodes in the network. •Isolate node Deterministic action that completely isolates an exploited machine. •Send node to honeynet Deterministic action that blocks all traffic between the affected machine and the main network and allows the node to communicate with the rest of the nodes in the honeynet (see Fig. 9). In our game, the goal of this action is to make the defender gain time to check the state of the rest of the network while the attacker scans and attacks the honeypots. •Move the flag This action simulates the migration of critical services from one machine to another. The defender can try to perform this action if the objective host is not compromised according to its observation. Even though it is a deterministic action, if the machine has been attacked (the actual state of the node is 2 instead of the observed state 0), the action cannot be performed, but the defender can detect that the machine has been attacked with a probability equal to that used in the action Check status (19). Migration of services to multiple machines (flag splitting) is not considered. Each countermeasure has been assigned a value that represents the total cost of carrying out that action. The countermeasures proposed in NICE [6] and those proposed in this paper are equivalent, so each action has been assigned the cost and intrusiveness of its equivalent measure. The action Check status has no equivalent in NICE, as it is not a countermeasure but a monitoring task, so it has been assigned a low cost and zero intrusiveness. The total cost of each measure, which takes into account the effect it has on the network and the TABLE 2. Costs and intrusiveness of proposed countermeasures. resources needed to implement it, is obtained by adding its cost and intrusiveness, see Table 2. These costs have been defined in order to make the defender mitigate attacks by following an optimal strategy, thus minimizing the resources used. For the attacker, the following actions are available: •Explore the topology The attacker agent obtains the list of machines connected to those already attacked with Scope ‘‘Changed’’, thus updating its observation. •Scan for vulnerabilities This action scans the vulnerability of a machine from another machine with Scope ‘‘Changed’’. If the action is viable, the probability of detecting a vulnerability is its Exploitability/10, which is in the range of 0 to 1. Once scanned, node status changes from 0 to 1. •Attack vulnerability Attacker attempts to exploit one of the vulnerable machines from another with Scope ‘‘Changed’’. The probability of compromising an explored machine is, again, proportional to the Exploitability of the vulnerability of that host: Exploitability/10. Once attacked, node status changes from 1 to 2. Having presented all the actions, it is pertinent to return to the following idea: an attacker may decide to perform an action, but that does not mean that he succeeds in performing it. The pseudo-code presented in Algorithm 1shows how an attacker action is executed. When carrying out an exploration or an attack, the intruder takes the set of neighboring nodes of the target machine nO∈VAfrom the graph GA. From that set it randomly chooses one of the nodes nA∈VAwith Scope ‘‘Changed’’ and tries to perform the action. If the edge between nOand nAdoes not exist in GG, the attacker cannot use that connection to attack, so it removes the edge from its graph and selects another node. If the edge exists, a random number between 0 and 1 is generated and the Exploitability of the objective machine is divided by 10 to scale it to that range. Thus, the action succeeds if the random number is smaller than the scaled Exploitability, leading to a success probability of Exploitability/10. 4) THE OBJECTIVES The attacker’s main objective is to attack the flag machine and the defender’s main objective is to prevent this from happening. To do this, both players take actions sequentially until the attacker reaches the flag (the attacker wins) or cannot take any more actions (the defender wins). 107192 VOLUME 9, 2021 J. Gabirondo-lópez et al.: Towards Autonomous Defense of SDN Network [51] Common Vulnerability Scoring System Version 3.1: Specification Document, Forum of Incident Response and Security Teams, Cary, NC, USA, Jun. 2019. [52] G. Van Rossum and F. L. Drake, Python 3 Reference Manual. Scotts Valley, CA, USA: CreateSpace, 2009. [53] A. H. W. Duvaud. (2019). MuZero General: Open Reimplementation of MuZero. [Online]. Available: https://github.com/werner-duvaud/muzerogeneral [54] RYU Project Team. (Feb. 2014). RYU SDN Framework. [Online]. Available: https://book.ryu-sdn.org/en/Ryubook.pdf [55] S. Bhatt, P. K. Manadhata, and L. Zomlot, ‘‘The operational role of security information and event management systems,’’ IEEE Security Privacy, vol. 12, no. 5, pp. 35–41, Sep. 2014. [56] A. B. Asif, M. Imran, N. Shah, M. Afzal, and H. Khurshid, ‘‘ROCA: Auto-resolving overlapping and conflicts in access control list policies for software defined networking,’’ Int. J. Commun. Syst., vol. 34, no. 9, p. e4815, Jun. 2021. [57] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’ presented at the 3rd Int. Conf. Learn. Represent., San Diego, CA, USA, 2015. JON GABIRONDO-LÓPEZ received the degree in physics, the degree in electronical engineering, and the degree (advanced) in computational engineering and intelligent systems from the University of the Basque Country (UPV/EHU), in 2020 and 2021, respectively. During the 2018–2019 academic year, he obtained an IKASIKER Fellowship to work at the Physics Department, UPV/EHU, where he carried out a research project aimed at the computational study of optical properties in metals. During the 2020–2021 academic year, he was a Researcher on reinforcement learning and cybersecurity with the Department of Digital Security, Vicomtech. He is currently part of the Physics Department, UPV/EHU. He develops web applications for the Bilbao Crystallographic Server. JON EGAÑA received the Telecommunication Engineering degree and the advanced degree in telecommunication engineering from UPV/EHU, in 2014 and 2016, respectively. He is currently with the Department of Digital Security, Vicomtech, where he works as a Researcher in the field of data analytics for cybersecurity. He is an Active Member of the 5GPPP Security Work Group. JOSE MIGUEL-ALONSO (Member, IEEE) graduated in computer science from the University of the Basque Country (UPV/EHU), Spain, in 1989, and received the Ph.D. degree from UPV/EHU, in 1996. He is currently a Full Professor with the Department of Computer Architecture and Technology, UPV/EHU. He is a member of the Intelligent Systems Group, UPV/EHU. He carries out research related to networks and parallel-distributed systems, in areas such as cybersecurity, performance modeling (with focus on the interconnection network), resource management in supercomputers, cloud infrastructures and high-performance scientific, and technical applications. He has published 2 books, 33 journal articles, and 30 papers in international conferences. He is a member of the IEEE Computer Society and the HiPEAC Network of Excellence on High Performance and Embedded Architecture and Compilation. RAUL ORDUNA URRUTIA received the degree in computer engineering from the University of the Basque Country (UPV/EHU), in 1999, and the Ph.D. degree from the Public University of Navarre (UPNA), in 2010. From 2001 to 2018, he has worked in private companies as S21se, Panda Security, or Tracasa in cybersecurity and innovation positions. He is currently the Digital Security Director at Vicomtech. He has taken part or led projects related with ethical hacking, forensic analysis, malware analysis, access control, and cryptography. The current research lines are focused on anomaly detection in information systems and communication networks, automatic responses, identity management using biometrics and federated systems, and secure traceability systems. VOLUME 9, 2021 107199