Reinforcement learning applied as a game mechanic and design element in a custom boss for World of Warcraft

Final Master Thesis Reinforcement Learning Applied as a Game Mechanic and Design Element in a Custom Boss for World of Warcraft Master Degree in Artificial Intelligence Author: Carlos Alejandro Goncevatt Director: Mario Martin Mu˜noz 23rd of June, 2020 Universitat Polit` ecnica de Catalunya Universitat de Barcelona Universitat Rovira i Virgili Contents 1 Abstract 2 2 Introduction 3 3 Context and Scope of the Project 5 3.1 Project’s Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2 ScopeoftheProject .............................. 17 3.3 StateoftheArt................................. 18 3.3.1 RainbowAlgorithm........................... 18 3.3.2 Asynchronous Methods for Deep RL . . . . . . . . . . . . . . . . . 20 3.3.3 RL in Starcraft II: AlphaStar . . . . . . . . . . . . . . . . . . . . . 21 4 World of Warcraft and the MMORPG Framework 23 4.1 TheWorldofWarcraft............................. 23 4.2 MMORPG Frameworks: Emulators . . . . . . . . . . . . . . . . . . . . . . 24 4.3 ProjectEnvironment.............................. 25 4.4 FrameworkandML............................... 27 4.5 PointsofInterest ................................ 28 5 Boss Encounter: ”The Existential Will” 29 5.1 Boss, NPC and Encounter Design . . . . . . . . . . . . . . . . . . . . . . . 29 5.1.1 Aru, the Existential Will . . . . . . . . . . . . . . . . . . . . . . . . 29 5.1.2 EncounterDesign............................ 31 5.1.3 Entering the World of Warcraft . . . . . . . . . . . . . . . . . . . . 47 5.2 AIDesign .................................... 48 5.3 Boss Encounter Implementation . . . . . . . . . . . . . . . . . . . . . . . . 52 5.4 AIImplementation ............................... 58 5.5 PointsofInterest ................................ 63 6 Experiment 65 6.1 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.2 ExperimentResults............................... 67 6.3 ResultsAnalysis................................. 73 7 Conclusions 76 8 Future Work 77 1 1 Abstract Reinforcement Learning is one of the main categories of Machine Learning algorithms and has seen many applications throughout a wide variety of fields. On the other side, the industry of video games has been in a continuous growth for many years. The world of video games offers many diverse environments that can be used to develop and research new and powerful algorithms. Moreover, in these last years very interesting projects have been testing the current limits of the application of Reinforcement Learning within complex game environments. In this project, I propose the usage of Reinforcement Learning as one of the main features of the AI system of a boss fight within the massively multiplayer online game: World of Warcraft. The project is focused into taken into account this AI approach as a relevant element during the design of a boss encounter. The idea is to be able to apply Reinforcement Learning to allow the NPC to make a better use of its available mechanics, allowing it to surprise players while still being able to lose the fight. To do that, I design and implement a complex fight with various mechanics for a 15man raid, where the RL AI will have control over the choice of which mechanic to execute during the 2 main stages of the fight. The project is done using an MMORPG framework based on TrinityCore, built in C++. The ML aspect is implemented using Python scripts, adapting an implementation of the current state-of-the-art Rainbow algorithm, as well as including the capability of asynchronous learning. Both systems are put together using the CPython libraries availables within the C language. The experiments are done in an online World of Warcraft: cataclysm private server, with an average of 100 players online. The results indicate that, using the corresponding architecture and resources, is viable to apply RL to the MMORPG environment of World of Warcraft. Is also possible to design a RL AI model that respects the constraints established and can modify its behaviour in order to gain the best performance out of its actions. The application of asynchronous Rainbow worked correctly and allowed the agent to be trained based on the corresponding parameters within a parallel environment that could have many agents within their own instance of the environment. The usage of a predefined behaviour through scripting to train the network in early stages was useful to keep the early choices under control and train all the necessary actions considering the constraints. There is still margin to make more complex and larger types of researches using RL and MMORPG games, together with the application of other different approaches. 2 2 Introduction ML has various applications in a lot of broad and diverse fields. Between the different types of ML, RL is one that focuses on the learning process throughout the own experience of the agent, using the feedback from the environment to enhance its behaviour and reach its goal. The world of video games is one that arises a lot of interest in ML research as a testbed for various algorithms. Video games provide, usually, controlled environments with defined rules in which a player is able to execute a wide variety of actions in order to reach a goal or objective. Games not only provide the environment, but they can be of a lot of different genres and types, each one having environments and actions that can differ hugely from others. This project is focused on the Massively Multiplayer Online Role Playing Game (MMORPG) genre, most specifically, in the game: World of Warcraft [1], also known as “WoW”. The idea is to design and implement a balanced boss encounter that will be faced against a raid of up to 15 players. The agent, in this case the boss, is going to have a certain set of actions, with constraints, from which to choose to execute. Each action represents a boss mechanic that will affect the players directly or indirectly. The goal of the agent is to kill all the players, while the target of the players is to defeat the boss. The core aspect of this project is the AI of the agent, which is a hybrid approach that uses scripting and RL to train the boss. In this way, it can learn when some of the actions are better than others and, in consequence, improve the effectiveness of their usage. To make sure that the NPC has a fair start, the experiment will consist of 2 clearly differentiable phases. The first phase is “training”, most usually done through simulations. As there is no way to simulate or generate data in a consistent way for this case, we take advantage of the ability to define the behaviour of the agent through a script, to create the first version. Once engaged in combat with the players, this version of the agent will use hard-coded timers as basis on to when to execute the different actions, respecting the key constraints. After using each action, the data is used to train the agent. This phase will be maintained for some days to ensure that the agent has gathered a good amount of experience. Then, the second phase will disable the usage of time-based actions and enable the usage of the trained agent to further decide which actions to execute. These new actions, and their consequences, will also be used to keep training the agent.Throughout the experiment, data will be gathered with respect to the training of the agent. There will also be a questionnaire for the players that fight the boss, to try to gather data regarding their perception of the boss in each of the 2 experimental phases. Section 2 covers a general and simple introduction to the topics that will be discussed in this document. In section 3, I talk in more detail about the reasons behind the project, and how all the topics relate to each other within this context. Moreover, I make an overview of some of the state-of-the-art algorithms and projects related to reinforcement learning and video games. Section 4 is focused into describing the elements and environments of the game World of Warcraft, as well as of the MMORPG frameworks, or emulators. This section also gets into detail of the implementation and changes done to make possible the application of ML through Python into the C++ framework. Within section 5, I make 3 an in-depth description of the design process of the boss fight, the agent and how it is introduced to the game. In this part, I also specify the way in which the AI system was designed and implemented. The subsections Points of Interest (4.5, 5.5) in section 4 and section 5 summarize and emphasize the most interesting aspects of their corresponding parts. Section 6 develops over the experimental procedure, the results obtained in it, and their analysis. In section 7, I comment about what can be concluded from the analysis of the results. Finally, in section 8, I make overall comments about other possible works that may be done within this area. 4 3 Context and Scope of the Project This project makes use of technologies, methods and concepts from Machine Learning, and tries to apply them into the world of video games. This is not the first time that this is done, as there are various examples of computer games being used to test different algorithms, like the cases of [2] [4] [16]. So, to better understand how these two sectors can relate to each other, this section will make an overview over each one, starting with some general ideas and then focusing on describing the fields, concepts and methods that are going to be relevant for the project. The first main pillar involved is Machine Learning (referred as ML from here onward), which is a field of computer science. ML methods have as aim teaching computers how to learn and act, without having the behaviour explicitly programmed in the code. ML involves creating and adapting models, that allow the programs to “learn” through experience gathered in different ways [6]. The construction of algorithms that take and adapt these models to improve and enhance their ability to make predictions is also part of this field. Uses and discussions about ML date back to the 1950’s, the adoption of the term increased dramatically throughout these last years [6]. There are a variety of different methods that can be used to solve a problem with ML. These methods and algorithms can be classified in different categories based on the general idea behind them, as well as other characteristics. The first of these categories is Supervised Learning [8]. A way to define this type of algorithms of ML is that supervised learning is the algorithmic process of approximating the underlying function between sets of attributes or features and their corresponding labeled data. Supervised learning is applied in a lot of different areas, some of these applications are financial services, medical diagnosis, fraud detection, web page categorization, image and speech recognition, as well as user modeling, among many others. Is evident that supervised learning needs to have a set of labeled training examples to work, without them it is impossible to apply this kind of algorithms. To be more specific, each of the data examples (or instances), that is provided to be used in a supervised learning algorithm, comes as a pair of inputs (features, attributes) and outputs (corresponding label). The goal of supervised learning is not to only learn from the input-output examples, but to derive a function that is able to approximate their relationship. This function should be able, as commented in the brief example above, to map well to new and unseen instances of input.output pairs. This property is called generalization [8]. At the same time, this function must be able to avoid overfitting, which is defined as the production of an analysis that corresponds too closely (or exactly) to a particular set of data, making it, most likely, fail to fit additional data or predict future observations in a reliable way [12]. Formally, supervised learning attempts to derive a function f:X→Y, given a set of N training examples {(x1, y1), ..., (xn, yn)}; where X and Y are the input and output spaces, respectively; xiis the feature (input) vector of the i-th instance, and yiis its corresponding set of labels. There are two core steps in a supervised learning task: training and testing. In this first part, a set of samples, composed attributes and their corresponding labels, is presented to the program and it derives the function fbetween these attributes and labels. The second core step, the testing, focuses around using f to 5 make predictions of the labels of unknown input data, which means that the machine will receive instances without their label and it has to assign one to each example. In this phase, is necessary to validate the generalization capability of f, as well as ensure that it is avoiding overfitting of the data. Is a common practice the usage of a new independent (test) dataset in this phase, this is tightly connected to the concept of being able to label “unseen” data. A common, and simple, way of validating data in supervised learning is through the calculation of the accuracy, the percentage of test samples that are correctly predicted by the trained function f. If the accuracy is within an acceptable range, we can consider that f is good to predict new data samples. There are a good variety of different algorithms that serve to find and represent this function, each one has its own way of processing the inputs and modifying the parameters to allow it to make a good match between the given labels of our training instances and the real function we are attempting to approximate. Some of these algorithms are artificial neural networks, case-based reasoning, decision tree learning, random forests, Gaussian regression, naive Bayes classifiers, k-nearest neighbours and support vector machines. This variety of supervised learning algorithms exists, partly, because of the fact that there is no single learning algorithm that works as the best solution on all supervised learning problems. The data type of the label (output) also determines the type of the supervised learning approach that can be applied. There are 3 main types of supervised learning algorithms that can be identified based on this. Firstly, classification algorithms are those that attempt to predict categorical class labels (discrete or nominal), an example of this is the one presented previously about the iris plants. Secondly, we have the metric regression, where the output data comes as an interval. Finally, preference learning makes predictions of ordinal outputs, like ranks, and attempts to derive the underlying global order that characterizes those ordinal labels. Some examples of input-output pairs that one can find in games and may draw attention to supervised learning approaches are [8]: •{player health, own health, distance to player}→{action (shoot, flee, idle)} •{player’s previous position, player’s current position}→{player’s next position} •{number of kills and headshots, ammo spent}→{skill rating} •{score, map explored, average heart rate}→{level of player frustration} Figure 1: Supervised Learning workflow summary. 6 The second is Unsupervised Learning [8]. This type of machine learning algorithms have a key difference in contrast to the ones of supervised learning, the only available data are the features of the instances. As there are no labels, these algorithms have a different focus and utility in contrast to some of those commented before. In general, unsupervised learning attempts to discover associations of the input by searching for patterns throughout all the data attributes, without having access to corresponding target output (labels). The main goal ends up being the study of the intrinsic structure of the data. Unsupervised learning techniques can be aimed at two main types of problems: Clustering and Dimensionality Reduction [13]. The first one has as objective to identify different groups in the elements of the data. The way to do that is by finding the structure of the data so that elements of a same group (or cluster) are more similar (their attributes have some level of similarity) to each other than to those of different clusters. There are a lot of clustering algorithms, each one with different features and complexity, depending on the possible structure of the data, some of them may yield results that may be easier to analyze. Some of these algorithms are: K-means, hierarchical clustering, density scan based clustering (DBSCAN) and Gaussian clustering model [13]. The second one, dimensionality reduction, is directly related to the amount of features, attributes or characteristics that are available in the data. Usually, all that information is represented as columns, and the target of this type of these algorithms is to reduce the amount of columns that have to be processed (it can also be considered a way to preprocess the data). Considering all attributes, there is a possibility that some of them are correlated between each other which means that some information is redundant and, therefore, increases the noise in the dataset. As this noise can negatively affect the ML model’s training and performance, the usage of dimensionality reduction methods can become of high relevance. They present a useful way to reduce complexity and avoid overfitting [14]. A hybrid category between supervised and unsupervised learning can also be considered: semi-supervised learning. In it there available data is composed of both, unlabeled data and labeled data, having more cases of the first type in the set. Still, this method can achieve improvements in the final accuracy in comparison to discarding one kind of examples and applying the correspondent learning method. Figure 2: Example of K-means clustering input and output with k= 3. The last of these categories, and the one in which we focus in this project, is Reinforcement Learning (generally abbreviated as RL) [8] [2]. One of the greater inspirations of this approach of ML is the way in which humans and animals learn to make decisions via positive or negative rewards received by the environment. In this kind of approach, 7 there are usually no labelled samples available. Moreover, the training signal comes from the environment based on the way an agent interacts with it. To better describe how RL works, we can consider that at a particular time step t, the agent is in a particular state sand, then, decides to make an action afrom the set of available actions for this current state. Once the agent executes the selected action, the environment sends an immediate reward r. Through the continuous interaction between the agent and its environment, the first one is able to gradually learn to select actions that maximizes the sum of rewards received. A lot of studies have been made from a variety of disciplinary perspectives focusing on the application of RL. Some of these disciplines include operations research, game theory, information theory and genetic algorithms. RL has also been successfully applied in problems that involve a balance between short-term and long-term rewards, under this category fall examples like robot control and games. Describing RL in a more formal way, the aim of the agent is to discover a policy, π, for selecting actions that maximize the measure of a long-term reward, like the expected cumulative reward. A policy is the strategy that the agent follows when choosing actions based on the state in which it is found. If there exists a function that characterizes the value of each action or the agent is able to learn a function with those characteristics, then we can say that the optimal policy π∗can be derived by selecting the action with the highest value. As previously mentioned, we consider each time step t={0,1,2, ...}where the agent interacts with the environment. We can model this as a Markov Decision Process (MDP) which is defined as a tuple {S, A, T, R}: •S: a set of states {s1, ..., sn} ∈ S. The environment states are a function of the information that the agent has about its surroundings, the state is the agent’s input. •A: a set of possible actions {a1, ..., am} ∈ Ain each state s. The actions represent the ways in which the agent can act in the environment, what it may do to try to change it. •T(s, s0, a): are the transition probability from state sto state s0given action a.T gives the probability of reaching state s0after picking a when the agent is in state s. This probability follows the Markov property implying that future states of the process depend only on the current state and not on the sequence of events that came before it. As a consequence, the Markov property of Tmakes predictions of 1-step dynamics possible. •R(s, s0, a): is the reward function when transitioning from state sto state s0given action a. When the agent is in state sand chooses to execute action a, transitioning to state s0, it receives an immediate reward rfrom the environment. Tand Rare the ones that define the world model and represent the dynamics of the environment (T) and the long-term reward (R) for each possible policy. We can find that the world model is known and available, so there is no need to learn to estimate the transition probability and reward function. In this case, we can directly calculate the optimal policy using model-based approaches. There is also the possibility that the world model is unknown, in which case we will have to approximate the transition and the reward function by learning estimates of future rewards given by selecting action a when 8 killed, the off-tank could pick an item that had to be used to lower the pressure from said NPCs if used correctly. Keeping players active and thinking about how to face the different mechanics has to be balanced in a way to not overwhelm the players and not leave them with nothing to do. Some of the aspects that are commented area thinking in skills or events during the fight that force this emergent behaviour out of the human characters. Around this, one can think about mechanics that force movement or force a change of position or distribution of the players. At the same time this also conditions the way in which the play, when they are clustered, some spells with area effects become more useful, so one may want to force a dispersion or just focus a high amount of damage over one player, forcing the usage of single target healing for that case. Movement to change the position of the boss is a way to modify the flow of the encounter, while moving some players cannot use their spells, as they require for them to be static while casting; other players have movement abilities that may increase the efficiency of their displacement. Some drastic, and temporary, changes during the fight are also of interest, the usual way to see this reflected is through phases, where some of the mechanics of the encounter may change, sometimes drastically; the usage of transitions or interphases, where a new set of abilities or mechanics are introduced for a short duration, is another way to manipulate and change the flow of the fight, as well as connecting the boss phases together in a consistent way. Thinking in ideas and using them to design and build a boss fight can prove to be a complex task. There are a lot of elements to consider, many which are not that easy to understand or some which are not trivial to implement. There are also ideas that may contradict each other or not work out as one expected in the first place. Making a good fight is hard. Figure 7: Madness of Deathwing, final fight of the Cataclysm expansion which received mixed critiques (left) and Ragnaros a boss from the previous tier of the same expansion as Deathwing, a fight that still is considered one of the best ones in WoW’s history (right) 3.1 Project’s Problem Definition As commented before, there are a lot of different games, they can be classified based on different characteristics. Some of the games most commonly used to test RL are a set from the Atari 2600 platform. These last years, the strategy game StarCraft II has been on the focus of researchers trying to create an AI able to compete on high or pro level matches, resulting in the creation of AlphaStar [5]. Albeit the progress done, there are various areas within video games where ML has not been applied nor tested. The search 15 to generate emergent behaviour for NPCs in a way that they improve the gaming experience of players is hard. The main source of complexity for this comes from finding the balance between keeping under control the AI, whilst giving it freedom to make choices; in other words, avoiding unfair or nonsensical behaviours. RL may suffer from this kind of uncertainty, mainly during the early training phase, when the agents can be considered “dumb”. Still, it offers a very strong potential as a tool to obtain the kind of emergent behaviour that some developers may want to put into their games. As the application of RL as a core part of the AI system of an NPC to enhance the player experience in the game is not an area that has been researched in-depth, this project will try to provide some information about that. More specifically, the idea of the project is to design a boss NPC that will battle against a group of players. The agent will have as its objective to defeat (kill) all the enemy players, while these will have as their objective to reduce the health of the agent below a certain threshold. The main objective is to create an NPC that has a set of actions that it can use, and for that NPC to be able to select the best action depending on the situation in which it finds itself in. The target is to try to measure the viability and efficacy of RL as part of the mechanics and as a tool to create emergent behaviour and enhance the experience of the players. To handle some of the early problems that RL may find, a controlled, scripted behaviour script will also be added to take control of the agent during the early training phase. This project not only includes the design and implementation of the AI, but it also covers other parts of the process of creation of a boss encounter: NPC design (thinking in a consistent story to introduce the NPC into the world in a, more or less, consistent way; also referred as lore behind the character), boss mechanics design (skills that the boss can use, as well as other encounter features, like other NPCs that may help the boss) and implementation of the encounter. Some further changes in some of the values in order to balance the boss based on feedback of the users is, also, very possible; as well as solving any other bug or issues that may arise. 16 Figure 8: Yogg-saron encounter in Ulduar. During phase 1 (top left) the boss is hidden. In phase 2 (top right) it appears but is immune to damage. Players have to go through visions (bottom left) and kill the brain. Once the brain is dead, phase 3 (bottom right) begins and players have can damage the boss directly. 3.2 Scope of the Project The objectives of this project cover a wide range. The first one is to design a boss encounter with an adequate level of complexity and difficulty for the chosen amount of players, also considering balance to give them a chance to win the fight. The design and ideas for the NPC also includes some backstory to make it fit in the universe of the game and its theme. The second is to implement all the designed concepts and ideas using the available tools and resources, polishing the result as best as possible, avoiding bugs and errors. The third objective is to plan and design a RL AI model to apply to this kind of environment, considering available data, possible state attributes and actions that the NPC will be able to execute. The fourth objective is the implementation of the designed AI within the system that handles the game itself, and all its elements. Finally, the fifth and last objective is to make a live test of the project on an online server with users. Data will be extracted from the experiment, as well as from the users, expecting to find relevant data and feedback about the AI and the encounter. The reach of the project covers a lot of different areas that are part of the entire process of creation and deployment of this type of content, although, usually, each part is done by specialized groups that are in contact with each other. In order to get good results in the general scope of the project, it is relevant to evaluate the current state of the art of, or gather data about, the corresponding parts. In particular, the most important elements to consider are the current state of the art for RL algorithms, as well as the current state of the art in application of RL in video games. As complementary data, some information about bosses and raid design in World of Warcraft can help to identify specific elements 17 that this project may need as we aim to offer this as a new experience for players. 3.3 State of the Art Is necessary to investigate the current state of the art algorithms and applications of RL. More specifically, I review the current state-of-the-art DQN extension, and one of the latest, and more interesting, projects of RL applied to a video game. 3.3.1 Rainbow Algorithm In relation to RL, there has been a lot of research in different algorithms, each having its own strong points depending on the environment to which they are applied. As previously mentioned, DQN allowed to make great progress in processing environments with high dimensional states and actions combinations. The first Deep Q-Networks algorithm started by making a combination of Q-learning with convolutional neural networks and experience replay. All those elements allowed DQN to be able to learn to play many Atari games achieving a human-level performance, only with an input of raw pixels and game score. From that point, up to these days, the deep reinforcement learning community has proposed many extensions to improve the algorithm, enhancing its speed or stability. Double DQN, or DDQN, is an extension that addresses an overestimation bias of Q-learning, by decoupling the selection and evaluation process from the bootstrap action. Prioritized experience replay is an enhancement that aimed to improve data efficiency by replaying more often transitions from which there is more to learn. Dueling network is an architecture change that helped in generalizing across actions by separately representing state values and action advantages. A3C is an example where we can see that learning from multi-step bootstrap targets shifted the bias-variance trade-off and helped to propagate newly observed rewards faster to earlier visited states. Distributional Q-learning is a method that learns a categorical distribution of discounted returns, instead of estimating the mean. Noisy DQN is an approach that uses stochastic network layers for exploration. These or only some of the many improvements made over the first DQN proposal. These algorithms, each one by itself, enable substantial performance improvements. As each one makes enhancements addressing very different issues, and since the basis upon which they are built is shared, the researchers Matteo Hessel, Joseph Modayil, Hadon van Hasselt et al. that made the paper [2] considered plausible to combine the methods previously commented. The result was the Rainbow algorithm presented in that document. 18 Figure 9: Median human-normalized performance across 57 Atari games. The Rainbow integrated agent (rainbowcolored) was compared to DQN (grey) and six other published baselines. Curves are smoothed with a moving average over 5 points. In particular, Rainbow is conformed by combining DQN with double Q-learning, prioritized replay, dueling networks, multi-step learning, distributional RL and noisy nets. The integration of the agent in the paper starts with the replacement of the 1-step distributional loss with a variant that allows multi-step processing. The authors construct the target distribution by contracting the value distribution in St+nin accordance to the cumulative discount, and shifting it by the truncated n-step discounted return. The following step was tom combine the multi-step distributional loss with double Q-learning through the use of the greedy action in St+nselected according to the online network as the bootstrap action a∗ t+n, and evaluating such action using the target network. All distributional Rainbow variants prioritize transitions by the Kullbeck-Leibler loss, this method was considered as possibly more robust to noisy stochastic environments, in comparison to using the absolute TD error, because the loss was able to continue decreasing even when the returns were not deterministic. Rainbow has a dueling network architecture adapted for use with return distributions. The network has a shared representation fξ(s), which is then fed into a value stream vηwith Natoms outputs, and into an advantage stream aξwith NatomsxNactions outputs where ai ξ(fξ(s), a) will denote the output corresponding to atom iand action a. For each atom zi, the value and advantage streams are aggregated, like in dueling DQN, and then they are passed through a softmax layer to obtain the normalized parametric distributions used to estimate the distributions of the returns. Finally, the authors replaced all linear layers with their noisy equivalente, using, within them, factorized Gaussian noise to reduce the number of independent noise variables. In the end, the authors were able to show a way in which these different ideas could be integrated, and that they ended up being largely complementary. This combination resulted in the current state-of-the-art results on the benchmark suite of 57 Atari 2600 games from the Arcade Learning Environment, considering, both, data efficiency and final performance. The authors also made an ablation study to research the contribution of each of the extensions to the Rainbow agent. The process was to take out one of the extensions out of 19 the entire combination, at a time. This method ended up giving the results shown in Fig. 10, where one can observe the drastic changes that taking out certain extensions could cause to the performance of the agents in the environments. [2]. Figure 10: Median human-normalized performance across 57 Atari games, as a function of time.The integrated agent (rainbow-colored) was compared to DQN (gray) and to six different ablations (dashed lines). Curves are smoothed with a moving average over 5 points. 3.3.2 Asynchronous Methods for Deep RL Another interesting approach to study is the one focused on a parallel type of RL using several instances of the agent, each one in its own version of the environment. Research about this topic can be found in the paper by Volodymyr Mnih, Adri`a Puigdom`enech Badia, Mehdi Mirza et al. [4], in which the authors study and experiment with asynchronous methods for deep RL. Initially, it was thought that the combination of simple online RL algorithms with deep NN was fundamentally unstable. To face this issue, many different solutions were proposed to stabilize the algorithm. All those approaches shared a common idea, focused on the fact that the sequence of the observed data encountered by an online RL algorithm was non-stationary, and, at the same time, the updates of online RL algorithms were strongly correlated. Experience replay memory gave a way to reduce non-stationarity and decorrelate the updates by aggregating over the memory through batching or randomly sampling the agent’s experience data from different time-steps. On the other hand, using that extension limited the methods to off-policy RL algorithms. In general, deep RL algorithms based on experience replay achieved unprecedented results in complex domains, such as the ones of Atari 2600. Still, as experience replay presented various drawbacks, like the usage of more memory and computational power per real interaction, the authors provided a different paradigm for deep RL. They decided to asynchronously execute multiple agents in parallel, each one in its own instance of the environment. The parallelism decorrelated the agents’ data into a more stationary process. This was due to that each parallel agent was going to be experiencing a variety of different states at any given time-step. That idea enabled a much larger spectrum of on-policy RL algorithms, like Sarsa and actor-critic methods, as well as off-policy RL 20 algorithms, like Q-learning, to be applied robustly and effectively together with the use of deep NN. One of the practical benefits offered by this paradigm was that it removed the common rely on heavily specialized hardware, like GPUs or massively distributed architectures, as the experiments were run on a single machine with a standard multicore CPU. The authors did experiments using 5 different algorithms using their deep RL paradigm and making the correspondent modification to allow them to work properly with it. The methods developed were: asynchronous one-step Q-learning, asynchronous onestep SARSA, asynchronous n-step Q-learning and asynchronous advantage actor critic. All of the methods end up being able to successfully train neural network controllers as the parallel actor-learners have a stabilizing effect on training. [4]. Figure 11: ”Learning speed comparison for DQN and the new asynchronous algorithms on five Atari 2600 games. DQN was trained on a single Nvidia K40 GPU while the asynchronous methods were trained using 16 CPU cores. The plots are averaged over 5 runs. In the case of DQN the runs were for different seeds with fixed hyperparameters. For asynchronous methods, the researchers, averaged over the best 5 models from 50 experiments with learning rates sampled from LogUniform(10−4,10−2) and all other hyperparameters fixed.” 3.3.3 RL in Starcraft II: AlphaStar For this project is interesting to make an overview of the latest researches and applications of RL in the area of video games. A very complex project was started by the DeepMind AI research group, whose aim was to create an agent capable of playing the strategy game StarCraft II at the same level of humans, targeting those at professional level. This project changed and evolved throughout the years, until it was finally able to show their best results with the presentation of AlphaStar, an AI that was able to reach the Grandmaster level of skill [5]. The domain of StarCraft has become an important challenge within AI research because of its representative and enduring status among some of the most difficult professional e-sports, as well as its relevance to the real world in terms of its raw complexity and multi-agent challenges. A high number of agents have been proposed over the course of a decade and numerous competitions, although the strongest ones have simplified important aspects of the game, utilized superhuman capabilities, or employed hand-crafted sub-systems. Even with those advantages, none of the agents was able to come near the level of skill of top StarCraft players. The DeepMind team addressed the challenge of StarCraft II using general purpose learning methods that are, in principle, applicable to other complex domains, the approach was a multi-agent RL algorithm that used data from both human and agent games within a diverse league of 21 continually adapting strategies and counter-strategies, each represented by deep NN. They made the evaluations of their agent, AlphaStar, in the full version of the game StarCraft II, through a series of real online games against human players, using the services granted by the game itself. AlphaStar was able to reach Grandmaster level, as commented before, using all 3 available StarCraft races, whose gameplay can vary greatly, and be rated above 99,8% of all the officially ranked human players. Figure 12: ”Training setup. a, AlphaStar observes the game through an overview map and list of units. To act, the agent outputs what action type to issue (for example, build), who it is applied to, where it targets, and when the next action will be issued. Actions are sent to the game through a monitoring layer that limits action rate. AlphaStar contends with delays from network latency and processing time. b, AlphaStar is trained via both supervised learning and reinforcement learning. In supervised learning (bottom), the parameters are updated to optimize Kullback–Leibler (KL) divergence between its output and human actions sampled from a collection of replays. In reinforcement learning (top), human data are used to sample the statistic z, and agent experience is collected to update the policy and value outputs via reinforcement learning (TD(λ), V-trace, UPGO) combined with a KL loss towards the supervised agent. c, Three pools of agents, each initialized by supervised learning, were subsequently trained with reinforcement learning. As they train, these agents intermittently add copies of themselves—‘players’ that are frozen at a specific point—to the league. The main agents train against all of these past players, as well as themselves. The league exploiters train against all past players. The main exploiters train against the main agents. Main exploiters and league exploiters can be reset to the supervised agent when they add a player to the league.” Extract from [5] 22 4 World of Warcraft and the MMORPG Framework In this section I talk about the game of World of Warcraft, detailing and commenting on some of its game mechanics and game design aspects. I also make an introduction to the MMORPG frameworks, also known as emulators, that are used within the community of WoW Private Servers. 4.1 The World of Warcraft In the previous sections, the game World of Warcraft received an overview comment about features and design elements that can be found in it, as well as, mostly, in other MMORPG games. The game code itself is not available to anyone outside of the owner enterprise Activison Blizzard [18], whether if we talk about the server-side code or the client-side code. It is an online game that processes certain parts of the game in centralized servers that manage the realms, which are the instances of the entire world where the players can connect. There are multitudes of realms, each one limited to their own region: Americas, Europe or Asia. [19]. The game client is the one the players use to connect to their desired realm. Once the connection is established, the corresponding server constantly exchanges and updates information with the client. The server uses the data sent by the clients to update each of the corresponding players in the world, at the same time, the server also sends certain type of update packets to the clients, so they can reflect certain changes or effects on their side, for example: the sky of an area is handled by the client, so the server can send data to modify it; this can be done for only one player, as well as more at the same time. The way the server and client work with each other has been changing with each expansion, some events that originally were managed by the server are now handled by the client, allowing to derive computational power to other interesting elements. On the other hand, this also increased the workload and resources requirements on the side of the user. As time passed, the popularity of World of Warcraft grew at the same time that the expansions were shipped. The game reached a peak of around 12 million subscribers for the end of the “Wrath of the Lich King” expansion, although the numbers started to dwindle from there on. After a new expansion was launched, the previous one became deprecated, the content was kept but the difficulty was inherently reduced and the rewards became more of an aesthetic accessory, or as items for collection. This was due to the player receiving an increase in the limit of their growth, during “Vanilla” the maximum level was 60, during the next expansion, “The Burning Crusade”, it became 70, and so on. The values of the attributes of the players, also, became inflated more and more to increase the perceived difference between one expansion and the next. The limits were not the only thing that changed, the way players worked also had modifications with each expansion, some attributes were removed and other introduced, some others were fused into one too. The changes in talent specializations and player spells was another of the big elements revised by the developers, as they changed the way the characters felt and were played. The way in which the world worked also had changes, for example: players had to go to a trainer NPC to pay currency and learn or rank up spells, in “Cataclysm” this was changed so the players would automatically learn and rank up spells while levelling 23 up. The biggest change in the world, and that affected a lot of players, was the “old world revamp” done for the “Cataclysm” expansion. Each expansion usually expanded over the old content, adding a new “continent” for players to explore and challenge. Each continent had various zones and areas, and each area had quests and stories for players to experience. With the “old world revamp”, all the areas of the original two continents, “Eastern Kingdoms” and “Kalimdor”, were updated, the environments changed, in some cases drastically, and most of the quests were redone, with many new ones introduced and many old ones removed. The changes that have been done to the game, for users that played expansions prior to the current one, as well as the accessibility to the game, as it is a subscription based game (requires to pay a monthly subscription or pay for “game time”), may be strong factors that influence some people to look for alternatives to the official version of the game, also known as “retail”. Figure 13: World of Warcraft subscriptions chart from early 2004 to the end of Q1 2015 [20] 4.2 MMORPG Frameworks: Emulators The project known as MaNGOS appeared in 2005, it is an educational project made by volunteers. It is one of the original MMORPG projects that built a “Full-Featured, Free, Massive Network Game Object Server Suit” (MaNGOS). The project gives an easy to use Authentication and World content server daemons (Fig. 14 with a lot of features, everything aimed for World of Warcraft [21]. MaNGOS was used as the base for one of the currently most well-known and used MMORPG frameworks for “World of Warcraft”, also known as “emulator”, mainly for its most popular expansion, “Wrath of the Lich King”: TrinityCore [22]. TrinityCore, oftenly abbreviated as TC, is an MMORPG framework built using C++, together with a back-end of MySQL to manage the database content. It was, at one point, a merger of MaNGOS with many patches submitted by the users, but it later transformed into an individual project with a diverse development team. Over time, TC has been cleaned up a lot, and is still being cleaned up and optimized. The project receives updates and is worked on by their team, as well as by other contributors. The relevance of TC comes from the fact that is a free software, it allows redistribution with or without modifications under the terms of the GNU General Public Licence [23], as published by the Free Software Foundation [24]. MaNGOS and TC are some of the mostly used frameworks to create “World of Warcraft” servers with different objectives in mind. There are many projects that have used the frameworks and then expanded 24 Figure 18: The Maelstrom within the Dragon Soul Raid (left) and the Temple Summit within the Black Temple (right), both are boss fight areas in their respective raids. The first one is an iconic place known from the early times of the game. It became available in the “Cataclysm” expansion and is the place where the final boss of that expansion, “Deathwing” is defeated [32]. It has an imposing nature, but the fight is limited to the 4 platforms that can be seen, as they are the ones used in the encounter that takes place. The possibility of creating a platform with an object of the game was considered, but was scrapped after it was confirmed that the movement of player related NPCs didn’t work correctly. The second area was the chosen map. It is also an iconic zone of the game revealed during one of the last patches of the “The Burning Crusade” expansion. In this zone, the players fight against “Illidan Stormrage, the Betrayer” [33], a relevant lore character introduced in the “Warcraft III” real-time strategy game [34]. The area is at the summit of a temple located in “Outland”. The area has a strange charm thanks to the combination of the lighting, the architecture and the night sky with the planet of “Azeroth” on sight (the origin planet of most of the races from “WoW”). Other aesthetic choices were done with regards to the environment in future iterations, like changing the music, lighting and sky depending on the state of the fight. 5.1.2 Encounter Design specification of the mechanics for the boss encounter, each one having a different basis, some introduced later into the development cycle. The cooldowns, cooldown timers or reuse time, are set considering the complexity that the mechanic has to be managed by the players, and the impact that it has on the them and environment. A mechanic can only be used if it is not in cooldown (cooldown timer is 0) and the boss is in the corresponding phase. When the boss executes a mechanic, it will go on cooldown based on the times specified. Initially, the boss started using a linear phase design, which means that the boss had “main phases” with a predefined order and that was always going to be respected. Much later, considering the limitations of this, with respect to the usage and learning of the mechanics of the later phases, I decided to apply a “cyclic” design. This is another way of organizing the stages within the fight and consist of starting with a random “main” phase, and then connecting with the others through controlled intermissions or transition phases. With this, I can ensure that a phase of the encounter that is required to happen for the AI to learn, will happen with some probability in each episode. From the boss design perspective, the fights that use this cyclic stages approach are usually short, for example “Hagara the Stormbinder” [35], because very repetitive and deterministic fights 31 can become tedious for players that experience them too many times. The final list of mechanics, phases and elements of the “The Existential Will” encounter is as follows: •Souls of the Abyss – Description: A vortex of spirits and souls that can be seen at the temple’s summit. When the boss is in combat, the players that get too near will be consumed and will die instantly. When a player dies to the soul vortex, the boss is healed by 20% of its maximum health and receives 100 energy. The “Souls of the Abyss” are an element that tries to generate a shock on players that meet the boss for the first time. It uses the visual effects from the iconic last phase of the “Lich King” encounter. During the battle, it limits the paths of players around the terrain, punishing them heavily if they try to cross through a “forbidden” area. The “Soul Vortex” can be seen right behind the boss in Fig. 17. •Shrapnel – Description: The boss launches a piece of Shrapnel onto its current main target, inflicting between 195000 and 205000 physical damage, ignoring armor. – Cooldown: 30 seconds. This spell is a heavy damaging skill that causes physical damage and ignores the armor of the target (physical damage is reduced by armor, as the spell ignores it the damage is assured to be threatening). This simple single target skill is used to force the player in using any defensive skill to reduce the incoming damage. Those are usually spells with long cooldown times (they cannot be used frequently). This forces the raid to have at least 2 tank players, as well as to think carefully about how to manage and use the raid’s defensive spells to ensure survivability. This is a mechanic present throughout both of the boss’s main phases. •Light Vortex – Description: the boss emits a wave of holy energy, inflicting between 15600 and 16400 holy damage. Players hit by this wave are pulled to the center of the Souls of the Abyss. – Cooldown: 10 seconds. This mechanic was added in a later iteration in the design process and is also present in the 2 main phases of the encounter. It is used as a way to force players to keep an eye on where they are positioned in relation to the dangerous soul vortex. The pull effect also can be combined with other displacement mechanics that the boss can use. Phase 1: This is the first main phase of the encounter, the reference “1” comes from the previously used linear phase design. Most of the energy generation mechanics in this phase revolve around individual responsibility by correctly handling the absorption of Core Essence and soaking the Dancing Magma. 32 •Twilight Instability – Description: the boss selects a target and starts to channel twilight energy over 5 seconds. During that time, the target will be afflicted by Corruption of the Earth, suffering 20% of its max health as shadow damage every 2 seconds. Once the channeling ends, the boss shoots a Twilight Instability to the target, dealing 1100000 shadow damage distributed between all players in a radius of 30 yards. If the target is not alive when the channelling ends, the boss will use Unleash Essence, afflicting all players and dealing 10% of their maximum health every second for 10 seconds. – Cooldown: 45 seconds. This mechanic has many concepts behind. The main idea is to cause heavy raidwide damage, increased if the players are not grouped up within a certain degree. This gives them range to position themselves around the battlefield, but they must still be prepared to be able to move within the 30 yards radius of the randomly chosen target. The second idea behind this mechanic is tied to the need to give players some time to react. As the spell causes such a great amount of damage, a text warning and a visual signal are used to make players realise that something is about to come. During that time, the players have to position themselves to meet the requirements, ready themselves for the area damage and focus strong healing spells to keep the main target of the mechanic alive throughout those 5 seconds. This can end the fight in an instant if the corresponding attention is not given. It’s a kind of mechanic that tests various capabilities of the players simultaneously, as well as one that can be very difficult to balance correctly. •Shadow Core – Description: the boss summons 2 Shadow Cores at the position of the 2 farthest players. The cores emit Negative Energy dealing from 943 to 1057 shadow damage every second to 5 random players. Periodically, the core will whisper to 3 random players, perturbing their minds, inflicting low shadow damage and stunning them for 5 seconds. Those whispers also corrupt affected players, increasing the shadow damage they receive by 30%, stacking up to 4 times. Moreover, the whispers afflict the players with a Curse of Torpor for 15 seconds, increasing the cooldown of abilities by 15 seconds. After 6 seconds of being summoned, the core will become sensible to the presence of nearby players, if one is found, the core will pick it up and start moving towards the center of the Souls of the Abyss. – Cooldown: 20 seconds. This type of mechanic revolves around the summoning of “adds”, other NPCs that work together with the boss or have special kinds of interactions. They have the duty to interfere with the players’ target of damaging the boss, as well as forcing them to change their positions in relation to their own. The adds will draw attention 33 of the players, even if for a short span of time, and can aim to cause great damage through constant and small interferences. These NPCs have relatively low health and enable another game mechanic when they die. •Core Essence – Description: when a Shadow Core is destroyed by a player, the killer will be affected by Swarming Shadows, which will make the player to periodically spawn Shadow Flames, which cause from 2313 to 2687 shadow damage per second to those within them, at its position. At the same time, a growing Core Essence will be summoned at the position where the Shadow Core died. The essence inflicts from 9250 to 10750 shadow damage to players in contact with it, afflicting them with Deep Corruption, which stacks anytime the player is healed and explodes for 46250 to 53750 shadow damage after reaching 5 stacks. If the boss gets in contact with the essence, she will start to absorb it, receiving 2 energy every .5 seconds and reducing the size of the pool until it’s completely gone. The Core Essence was added during later iterations of the Shadow Core mechanic. The idea was to add an element that had to be handled by the tanks of the raid by moving the boss herself. This is a mechanic that also forces a player to move away while spawning another source of damage, it becomes an individual responsibility to leave those flames away from other raid members. The Deep Corruption debuff was added to punish bad positioning in a harder way, but leaving space to avoid harsh consequences by controlling the healing done to the afflicted character or its position, as the explosion only affects allies, not the afflicted one. The only member that is forced to suffer from that effect is the tank responsible to move the boss as soon as possible to the position of the essence, so the pool does not grow too much and can be eliminated quickly. If left unchecked, the essence can grow up to a 100 times its size, most likely exhausting even more the energy of players and allowing the boss to gather much more energy. This mechanic was added to increase the responsible handling of Shadow Cores. •Dancing Magma – Description: the boss summons a pool of Dancing Magma at a random point around her. The pool will stay inactive for 9 seconds. Once that time passes, if there are no players nearby, the magma will explode into a Magma Geyser causing from 53625 to 56375 fire damage to players within 7 yards and knock them back. The erupting magma shoots Fireball Barrages to the position of each raid member every 0.5 seconds. The fireballs deal between 20400 and 27600 fire damage within the impact zone. If a player is within the area of the Dancing Magma, the pool will start to shoot Lava Bolts all around it, each molten rock deals between 14137 and 15862 fire damage in the zone of impact. The Dancing Magma also shoots a Lava Bolt to 34 the nearest player every second, dealing 10000 fire damage. After 10 seconds have passed since the activation, the Dancing Magma erupts creating a Pillar of Flame that deals from 69375 to 80625 fire damage and launches nearby players into the air. If during the pool’s active state a player fails to be inside the lava, it will become a Magma Geyser. When the Dancing Magma becomes a Geyser, the boss acquires 20 energy. If the magma is kept under control, the boss will receive 5 energy upon the explosion. – Cooldown: 35 seconds. This mechanic is replication of the common “soaking” mechanics that are seen nowadays. These mechanics revolve around giving individual responsibility to members of the raid to position themselves correctly depending on the visual markers. In this case, I tweaked the idea a little bit and only summoned one NPC, as with other similar mechanics, there will be various seconds to give time to a player to get into position. The main difference with other soaking mechanics is that the Dancing Magma has to be soaked for a longer time, meaning that the player will have to be aware of other mechanics of the encounter (for example, the Light Vortex) for the duration. At the same time, the other players will have to consider assigning other candidates to help or support the main “soaker”, as well as guarantee that all involved players will be kept alive. To keep certain interactions fair, the players near the Dancing Magma cannot be chosen as targets of Gravity Crush. •Gravity Crush – Description: the boss channels for 6.5 seconds and traps 3 players in a bubble of gravity, rising them up into the air and crushing them, dealing 9% of the maximum health as arcane damage every 0.5 seconds. When the effect ends, the targets are left to fall to the ground. The boss receives 10 energy points upon usage. – Cooldown: 20 seconds. This spell is straightforwardly taken from another boss fight of the game. It has various elements that turns it into a good mechanic for this encounter. First, is the disabling effect on some players, the effect also applies to the boss that has to remain still while doing the channeling. Second, the high damage that forces the raid to focus their healing spells onto the targets, they will have to be kept relatively high health to avoid dying from fall damage. Third, the synergy with Light Vortex, as rising players over a certain height makes the pull effect of the vortex become a risk even when far away. On the other hand, if players near the vortex are risen, the pull will make them go past it, so it arises a certain amount of consideration about where the players are standing when this mechanic is available for use. 35 Figure 19: The boss in a test of phase 1 during development. The channeling of Twilight Instability and the Shadow Flames, triggered from the death of a Shadow Core, can be seen in the image. Phase 2: This is the second main phase of the encounter and has 4 different main mechanics than those of the other main phase. Most energy generation in this stage is through baseline usage of mechanics, lessening control onto whether the raid wants to force a flight intermission. To increase the energy rate generation the players will have to handle higher raid-wide damage. •Purifier – Description: when this phase starts, the boss summons a Purifier, an orb of holy energy that slowly moves to the position of the farthest player. Every 5 seconds, the Purifier casts Purifying Blast, which knocks back nearby enemies, deals from 48750 to 51250 holy damage and summons a purified area that causes players within it to take between 29250 and 30750 holy damage every second. Each player hit by Purifying Blast receives 2 stacks of Righteous Shear, causing them to deal 7500 holy damage per stack to nearby allies every second, for 30 seconds. The boss also receives 5 energy for each player hit by Purifying Blast. This NPC started as an idea to make a clear difference between the 2 main phases. The Purifier is a big, slow moving, ball of light that traces a straight line between itself and its farthest player. This allows the raid to manipulate where the NPC will go, and, more or less, control where the “void zones” (small areas that deal damage to players within them, pools and void zones are usually separated based on their visual effects) are left. This mechanic punishes players that do not react quickly enough, with an area of effect damage to allies nearby, limiting the available space for the raid. This damage over time can be removed using dispels, so by sacrificing some energy (“paying a price”) the raid can avoid the previously commented penalization. The management of this NPC may become quite important to keep this phase 36 under control while facing the other mechanics. The Purifier cannot be attacked or damaged, at the same time it has a high health pool to keep it safe from possible damaging effects that may bypass its theoretical immunity. This forces the players to handle it by manipulating the position to which it will move, instead of just trying to blow it up as fast as possible to remove it. •Earth Tremors – Description: the boss emits tremors through the ground at least 5 times over at least 10 seconds. The waves create Earth Spikes under 7 random players and other random points around the boss. The spikes erupt after a couple of seconds and deal from 14295 to 15705 nature damage to players above them at the time of the eruption. Each tremor grants 3 energy to the boss. – Cooldown: 25 seconds. This mechanic is oriented into generating some chaos in the formation of the raid. Random players will be constantly forced to move, while others may or may not have to. The raid has to be able to efficiently handle each of the waves that creates spikes, as well as keeping an eye in the other abilities that may disrupt them, mainly the Light Vortex, which can pull a player right over a spike. •Earthen Vortex – Description: the boss teleports to 1 out of 3 possible positions around the soul vortex and traps all players in an Earthen Vortex. Trapped players receive 5% of their maximum health as nature damage every second for 5 seconds. All affected players are also displaced to a random position near the boss. Earthen Vortex gives 15 energy to the boss when used. – Cooldown: 45 seconds. Similarly to Gravity Crush, Earthen Vortex is a spell and mechanic used by an existing boss of the game. Its target is to move players around and leave them grouped around the boss, while inflicting some raid-wide damage for the duration. The main difference with the original version is that, here, the boss first teleports to 1 out of 3 predefined positions, so the mechanic can change drastically the area of the battlefield used by the players. This forces the raid to keep an eye on the possible zones to which the boss can move, so they are not full of “void zones” from the Purifier. This mechanic has explicit synergy with that NPC. It still leaves space for the players to regroup anywhere else once they are freed from the vortex. •Ice Tomb – Description: the boss throws a frost orb to trap the 2 nearest players into a crystalline ice prison. The boss will launch new frost orbs targeting all players within 7.5 yards of the players that are hit and encased in the ice tomb. Trapped players receive 5% of the maximum 37 health as frost damage every second, until they are freed. Every player trapped in an ice tomb grants the boss 4 energy. – Cooldown: 20 seconds. This mechanic is a combination of 2 well-known spells, with the same name, from 2 different bosses. The Ice Tomb from the “Sindragosa” [36] encounter trapped players within resistant tombs that blocked vision of players, not allowing them to be healed, and were used to cover from frost explosions used by the boss. After some time, the players inside started to asphyxiate losing health quickly. The main element of that mechanic was that when the orb impacted, it froze all players within a small radius. On the other hand, the Ice Tomb from “Hagara the Stormbinder” [35] did not freeze players in the area of impact, only the one targeted. At the same time, players trapped in these other tombs did not block vision and could be healed. For this project design, I made the spell work as a mix of both. I used as basis the second Ice Tomb mentioned, but the boss was also going to throw new orbs to those around the player when the tomb was formed. This mechanic aims to test the reaction of the targeted players, as they are the 2 nearest ones to the boss. Those players will have to find a way to get away from the others to avoid a chain reaction. At the same time have to be careful to not interfere with the handling of the other mechanics. This is a spell that can ramp out of control easily, but is easier to handle than the first Ice Tomb version commented previously, mainly because the tombs do not have a lot of health, and those affected by the chain will also be given time to move away. •Lightning Rod – Description: the boss turns 3 players into Lightning Rods. After at least 7 seconds, she shoots a Chain Lightning to the marked players, which deals between 28500 and 31500 nature damage, jumping to nearby players, increasing damage dealt 100% for each jump. Players hit by Chain Lightning are afflicted by Static Overcharge, which makes them deal from 9250 to 10750 nature damage every 2 seconds to nearby allies for 10 seconds. Each player hit by Chain Lightning also grants 4 energy to the boss. – Cooldown: 15 seconds. This mechanic is another similar to that of an existing boss. It is very effective in how it works, as it clearly warns and marks players that are going to be targeted, giving them time to react. At the same time, is a type of spell that can also ramp up pretty easily, becoming deadly after a few jumps to other players. This version has the added functionality of putting a damage effect that does not allow players to instantly regroup, and this effect cannot be avoided. The idea is simple, this mechanic also encourages the player to separate from the rest of the group, but together with other mechanics the space to spread can be drastically reduced. Even more, the right combination of mechanics in this phase can create a chain reaction that may throw the players in disarray. 38 Figure 20: The boss in a test of phase 2 during development. The Purifier and the Earth Spikes from the Earth Tremors can be seen. The first phase of the encounter revolves around handling mechanics with a fixed amount of targets involved. Those mechanics give more weight into individual responsibilities, with support from the rest of the raid. The mechanics of the second phase have the capacity to scale. If managed correctly, they won’t cause many troubles. But, if handled incorrectly the spells can start to jump all over the players. The Earthen Vortex is thought to throw the players into a dangerous situation, forcing them to react and avoid the worst case scenario. Energy and Flight Phases •Description: he boss uses “Corruption” as energy. Hitting enemies with certain spells or triggering certain effects increase the amount of energy that the boss has. When it reaches 100 energy, the boss will become immune and start a flight phase. The Flight Phases, or intermissions, are moments of the encounter in which the players will have to face a complete change in the environment. The boss will become immune to damage and disengage from battle. After that, she will fly to the center of the soul vortex and stay there for the remainder of the intermission, although she will also participate in it by using a new set of spells unique to the correspondent flight phase. There are 2 intermissions: Flight Intermission: Song of the Ancients: When the boss reaches her position over the soul vortex, she will Asphyxiate all players, stunning them and aiming to reduce their health to 1. Once the channeling of this spell finishes, she summons 2 NPCs, the intoners “Devoru” and “Poporu”. •Devoru and Poporu They are 2 mechanical humanoid NPCs that stand still throughout the intermission. They have a high health pool, although it is shared, which 39 means that if one receives damage, the other one receives it too. This flight phase will end once these adds are killed. •Reverberating Hymn – Description: the boss, Devoru and Poporu intone a Reverberating Hymn that deals damage and stacks every 4 seconds, dealing 400 damage per stack and gaining 1 stack per intoner. If the hymn finishes, each intoner will launch a Disrupting Roar. The boss’s roar deals between 92625 and 102375 physical damage, while the other intoners’ roar deals from 47500 to 52500 physical damage. The roar interrupts any player hit while spellcasting. This spell is used to cause a constant and increasing income of damage for the raid. The maximum damage is not too high considering similar mechanics, but the main idea is for this to pose as a looming threat. The final element serves as a way to force players to finish up the phase within the expected time, as the combined roars damage can bring the entire raid down. •Modulation – Description: the boss modulates and launches a sonic wave that deals between 48750 and 51250 shadow damage to all players. – Cooldown: 20 seconds. The modulation spell is a straightforward raid-wide damage that adds up together with the constant damage of the Hymn. This mechanic comes in a regular and welldefined interval of time, which can allow players to prepare to handle its damage. •Resonance Bomb – Description: the boss launches a Resonance Bomb to a random position around her, avoiding the active area of the soul vortex. The amount of launched bombs is based on the amount of noisy players, with a minimum of 1 bomb. If it falls to the ground and there are no noisy players in the landing zone, the bomb will trigger a Modulation and will launch Sonar Bombs to the position of all players. If a noisy player is within the landing zone of the bomb at the moment of impact, it will disappear and remove the noisy effect from the nearest player. – Cooldown: 15 seconds This mechanic is thought to give players some level of choice, as well as to measure their level of organization. It is based on the Disruptor mechanic which makes players “noisy”. Players can choose to ignore the bombs and focus on keep damaging the intoners, just healing through the raid-wide damage. This is a risky choice, as an unfortunate chain of area damage can lead to the death of some members. The harder, but better, way to control this mechanic ends up being to send noisy players 40 Figure 22: Entropic Oblivion phase during testing. The mechanics that can be seen are: Thundering Fissure (on the left), Falling Fragment (rock pillars), Distortion Bomb (on the right), Acid Rain and Electric Instability (the lightning striking the player character). Overall, this phase is focused into giving the feel of a “last stand” against this powerful entity. Balancing this kind of concept is hard, mainly because the chaotic nature of mixing so many spells is something desired. On the other hand, many of these mechanics can have an unexpectedly strong synergy that makes this phase impossible to beat. This last thing is something that is not desired at all, as there is no sense in fighting an unbeatable enemy, a thought that will discourage players from even trying and testing the encounter. 5.1.3 Entering the World of Warcraft With the boss encounter ready, I focused into how most of it could fit within the world of the game. Some of the ideas were conceived throughout the development and the starting point was the basic information given to the boss NPC. From there, I expanded its lore, its history, and thought of a way to connect players to the need to fight the boss. A common way to introduce players to a new story is through the quest system of the game. Quests are given by other NPCs and ask the players to accomplish certain deeds. To give the players a quest, I needed to think of an NPC that worked as intermediary. Hence, I used the initial concept of the dragon boss and transformed it to be the NPC that handled the quest to the players. Dragons are an iconic element of this world, they are considered powerful beings with certain allegiances depending on the faction to which they pertain, denominated as Dragonflights. I made a simple story in which this mysterious and powerful being comes to ask the players for help. The explanation is that another entity took part of her powers, so she could not handle it, as there was a risk of her losing her remaining energy. Because of that, the players are sent to face that unknown entity. The twist at the end is that this entity is the very same dragon, Aru, but with another form, and just wanted to play. Aru is exactly that, a mysterious and powerful being that wants to entertain herself. This concept has been used in many stories, throughout many mediums. In the very end, “The Existential Will” starts with an unknown name “?????” until the final phase where she reveals, without knowing, that she is Aru. 47 Figure 23: Aru dragon NPC. The exclamation above indicates that the NPC can grant a quest to the players. This one is located over the Ironforge mountain in Dun Morogh for Alliance players. 5.2 AI Design The main way of developing the AI system in this game is through scripting. This means that all the actions of the NPC are completely controlled and well-defined. The only main sources of uncertainty are the random timers and the random targeting. Events (spells, mechanics) are controlled with timers that are set within the script, at each update step, the timers tick and, if a set has to be executed they go through that process sequentially. Some bosses have very specific timers, but there are others that add some uncertainty through choosing a random value between a range of minimum and maximum amount of seconds. This can make certain abilities of the boss to end up being used in a synergistic and more powerful way simulating some kind of emergent behaviour, although is a matter of probability. The selection of targets for certain mechanics, or spells, is another source of uncertainty. Most of the time, the boss will have to make a choice of a position or a target where to execute a certain mechanic. The most common way to pick a target is doing it at random, for example a random point around the boss or in the battlefield, or a random player from the raid. Some spells are designed to have an instant effect on the raid, considering them something unavoidable. Those mechanics are usually non-lethal, and are used to exhaust the players’ resources over time. On the other hand, we can find mechanics or spells preceded by a warning, giving time to the players to react to them. The danger that these mechanics can represent varies widely, but is a trait mostly found when the spells can cause high or lethal damage to the raid. At the same time, this also allows players to handle it in a way that they reduce most of the raid-wide impact, lowering the damage overall or just having the player involved receive damage, high or low, but not lethal. Scripting AI is simple and easy to control, but in this project I am interested in trying to integrate support to use an AI with learning capabilities using RL. As commented in section 3, when talking about RL, defining the problem as an MDP is an important part 48 of the work. To do this, I have to think about the data needed for this purpose: State, Actions,Transitions and Rewards. First, I have to consider the data of the environment that is accessible and available. From there, I have to analyze and build a set of attributes based on the data that is relevant for the agent. The final state used in the project has the following structure: •Players data: general player data that gives information about the composition of the raid and their current situation. The amount was limited to 15 entries as a mid-point between the known 10man and 25man versions used in this version of the game. This was done, mainly, to reduce the computational cost of processing each player for both, the AI and the mechanics defined in the script. – Role: he roles of the players based on their specialization, it can be 0: Tank; 1: Healer; or 2: DPS (Damage dealer). – Health: the current health percentage of the players, it is a floating point value between 0.0 and 100.0. A value of 0 is equivalent to saying that the player is dead (so it would be redundant to have a boolean attributes to know if the player is dead or alive). – Distance to Center: how far away the player is from the position of the soul vortex. This may be relevant as there are spells that move raid members towards the vortex, and those within a certain distance are instantly killed. – Distance to Boss: how far away a player is from the position of the boss NPC. This can be relevant to identify when players are grouped around the boss, which can improve the effectiveness of some of the actions. •Raid Health Percentage: the mean health percentage of the entire raid. It is useful to measure how impactful, at a raid-wide level, an action has been. The lower becomes the value, the nearer is the boss to the target of defeating all players. •Alive Players: the amount of players that are still alive. Can be used together with the previous attribute to enhance the measuring of the impact of an action. Mainly, because even if the raid mean-health becomes low does not necessarily means that a player was killed. •Mean Distance between Raid Members: a way to relatively measure the degree of clustering of the raid. If this value is low, that means that most of the players are near each other, possibly enhancing the effectiveness of certain area of effect mechanics. •Boss Phase: he phase in which the boss is. This is relevant because the phase imposes a condition onto what actions can be executed. •Boss Cooldowns: the cooldowns, in milliseconds, of the different mechanics managed by this AI. They are relevant because each cooldown also constraints the usage of their respective action. The order in which the cooldown times are presented is: –Shrapnel 49 –Light Vortex –Twilight Instability –Shadow Core –Dancing Magma –Gravity Crush –Earthen Tremors –Lightning Rod –Ice Tomb –Earthen Vortex Once the state is defined, is necessary to comment about the set of actions that the AI will be able to use. This game allows for a wide variety of actions, each with its own range of possible values. As this project scale and resources are not huge, this is not at the level of something like AlphaStar [5], I decided to trim and limit the set of actions to the mechanics that are of interest. At the same time, this choice also limited the phases that would be used for the RL part, as some of them required a certain level of control and deterministic nature in the actions, for example the intermissions. In the end, I decided to choose the 2 main phases of the boss as targets to train and use, so the 10 mechanics involved in them were chosen as the set of actions. The final phase was not included as it was more of a roleplay element, and would have not been practical to use. To the current set of actions an 11th one was added to handle the possibility of not having any viable choice: “No Action”. The final set of actions that the agent was able to use was of 11 elements: •Shrapnel •Light Vortex •Twilight Instability •Shadow Core •Dancing Magma •Gravity Crush •Earth Tremors •Lightning Rod •Ice Tomb •Earthen Vortex •No Action 50 In this kind of project, the set of transitions is not too easy to define, as the amount of combinations of states and actions become too big to handle. Still, there are some rules regarding actions and certain elements from the state. More specifically, I am referring to the phase and cooldown attributes, which impose hard constraints over the actions, as commented before. Based on the previous specifications, when the agent has a value of “Phase” equal to 1, then the set of possible actions is reduced to: Shrapnel, Light Vortex, Twilight Instability, Shadow Core, Dancing Magma and Gravity Crush. Moreover, depending on which action is chosen there are elements of the state that will change in a predictable way, while others may have a lot of variation. For example, if the action chosen is “Dancing Magma” then: “Boss Phase” will remain as 1; “Dancing Magma cooldown” will change from 0 to 35000 (it may be a lower value as the next time that the state is checked may be some seconds in the future, and not instant); cooldowns of other mechanics will be 0 or will have a value lower than the one they had in the previous state; Data regarding players can vary a lot or not, as they act based on their will or need. The rewards is another part that cannot be exactly defined for the pairs of state and action, but it can be thought as a function to evaluate. In this case, the reward defined was entirely based on monitoring the difference between the raid mean health of the current state with the one of the next state: ∆h=hs−1−hswhere sis the state and s+ 1 is the state of the next step. Another version considered was one that evaluated that, together with giving an extra reinforcement based on the difference of alive players between both states. The idea behind these reinforcement functions is for the boss to learn based on the impact of the spells towards the target of reducing players health, killing them as consequence (which ends up in a reduction of the maximum value of the raid mean health). Based on all these considerations, an approach based on DQN seems to be the best choice, as the networks always grant an answer and they are good to handle huge state spaces. The way in which a boss encounter works makes it ideal for an episodic RL approach. The start of an episode can be easily defined by the moment in which the raid engages the boss. This is, when the players commit an aggression against the boss NPC while it is out of combat, as consequence the combat state is enabled for all, the boss and the raid. To define the end of an episode there will be some things to take in consideration. The trivial case is when the bosses wins by killing all the players (also known as raid wipe or simply wipe), in which case it will reset, get out of combat, move to its original position and regenerate health entirely. That point marks the end of an episode in which the NPC wins. On the other hand, I have to change some rules for the case in which the boss loses. As the phases used for training and learning are only the 2 main phases, I decided to end a lost episode once the boss’s health is reduced to 20% and enters the final phase, which is controlled by script. The reasoning for that is that if the players reach that point of the battle is because the boss was not able to defeat them. The intermissions are a special case, as they are not controlled by the RL AI, but it would not be practical to cut an episode when they happen, as they can be triggered various times throughout the battle. Because of this, when the boss starts an intermission, the action used to finish the episode (if the raid wipes in this phase) is the last one: “No Action”. The reinforcement over this action would be applied using one of the non-main phases into the state, so I consider 51 that the probability of this influencing the action to become better than other mechanics is low. In the end, an episode for the agent starts when a raid of players engages the NPC. After that, it can end in 2 ways: •The boss wins and resets, ending the episode. •The raid gets the boss to 20% of her health and into the final phase. The episode is considered as finalized with a defeat. To ensure that the boss has a, somewhat, solid start when it starts to make choices using what it has learned, I decided to start the training by using a scripted version of the encounter. This version uses all the mechanics, respecting the constraints, based on predefined timers. In this way, the boss’s behaviour would start being more predictable, but this takes advantage of the inexperience of players too. The scripted AI will execute actions and, before executing any other, will then process the state to evaluate the impact of the action and learn from it. As the amount of episodes played increases, the raid and the boss will have learn from one another. After gaining some early experience, I am going to gradually increase the usage of the trained AI to make the choice of what to do, until all is completely controlled by it, during the corresponding phases. As the boss is localized within an instance, it is possible to have many versions of the NPC existing, and even working, at the same time. Each boss will be related to an instance identifier and is going to have its own unique identifier, assigned when the entity was constructed. The usage of asynchronous implementations of RL algorithms was something possible and useful, as seen in section 3. Taking advantage of this, I decided to also apply some asynchronous support for the RL system. In this case, after debating over the matter, I went with the option of making copies of the current agent, and using those copies to choose the action to execute. Even if the copy networks were the ones making choices, the reinforcement learning was going to be applied to the main network, while the experience was going to be saved within the memory of the main agent. After some episodes, each agent will update their network. As I briefly commented beforehand, in this game the entities are able to make a lot of different actions. Some of them may be moving around within the 3 different axes (X, Y and Z) freely, choosing a unit (friend or foe) as target for an action, like moving or attacking, or explicitly attacking, and so on. All actions, excepting those related to the RL AI specified below, are going to be controlled by scripting and will have a well-defined behaviour. 5.3 Boss Encounter Implementation When creating a script for an NPC using the MMORPG framework there are various concepts that are well-known throughout projects. These concepts define the elements that are relevant to define the behaviour of the NPC, when and how something has to happen. The first set of relevant concepts are those related to “game entities”, in our case these are: 52 •WorldObject: the most general representation of any kind of world game entity (being the most general the Object, that is used as superclass of WorldObject and Item). All examples below are also WorldObjects. •Unit: a specified version of a WorldObject. Represents all entities that can move and interact with other game elements in an active way. Units cannot be collided with (cannot be obstacles in the world), nor they can block vision (units can “see” through other units). All examples of Creatures or Players are Units. •Creature: a specific version of an Unit. All NPCs with an AI assigned are Creatures. Examples of this are both “Aru” NPCs, the dragon that handles the quest, and the boss. •Player: a specific version of an Unit. Only player characters compose this kind of entity, there are no NPCs in this category. •GameObject: these WorldObjects are a specific type of interactive element. They can affect the environment in various ways, like they can be obstacles and/or obstruct vision. They are usually fixed and cannot do anything if they are not interacted with, if possible. The Falling Fragments of the final phase are an example of GameObject used within this project. •DynamicObject: another instance of WorldObject mostly used for persistent area effects. They are similar to units, but they lack the ability to actively choose to change the environment. These objects are spawned and most of their effects are handled by another core game mechanic. The Distortion Bomb from the final phase is an example of this kind. Figure 24: Definitions of the classes listed above. All units are able to have a UnitAI, which specifies the behaviour. This is used to create the specifications in Creatures using a CreatureAI. Players can also have an AI assigned, but this is used for a few gameplay elements, like the “mind control” which involves another player, or an AI if it is used by an NPC, to take control of the actions of the target. CreatureAI has other functionalities and additions with respect to UnitAI. Moreover, there is another definition that adds more elements usually used for boss fights: the BossAI. This is the one used to model the scripted AI structure of the boss, while other NPCs use the CreatureAI as base. In the case of GameObjects, they can also have 53 an AI to handle certain events, but this is not relevant in our case. DynamicObjects cannot have specific behaviours specified, their functionality is handled by spells and auras. Precisely, the second concept that I have to introduce is that of spells and auras. Spells are special abilities that units can use, they can have a great variety of effects and targets, they also come with a specific structure that allows to script, control and customize their effects. Auras are a specific type of spell effect, when they hit a target they add themselves to a list that handles auras applications to the unit. Auras also have a lot of effects and are mostly used to trigger constant or periodic effects onto their targets, also referred as owners. Depending on if the aura has a beneficial or detrimental effect, it can be categorized as a buff or debuff, respectively. Auras and Spells is one of the main ways in which this game controls many events and conditions, being able to find spells that apply auras that have no visual effects but are used as a condition for certain things to become visible or to even happen. Now that these important elements have been explained, I will talk about the parts and concepts that are usually used, and found, within the scripts of NPCs, more specifically, in boss’ scripts. These concepts are: Events, Actions and Script Hooks. Events are defined using numerical identifiers, usually through a C++ enum structure. The ids have to be unique for each event related to a creature, but the values can be reused for other creatures. Events are related to things that the NPC does after a timer runs out. The identifiers are used to register the events in a C++ Multimap of events that is processed at every update step. A while loop is the common way to process all the events that have to be executed within the same timestep, when their timer reaches 0. Actions work in a similar way to Events with regards of the numerical identifiers, but their concept on when they are executed differs entirely. Actions are thought as something that is executed instantly when called. They are not bound to timers, but executed on demand. Figure 25: First fragments of C++ enum structures for Actions (left) and Events (right). The script hooks are member functions within the AI structure that can be overwritten and customized. These hooks can be called when something specific happens or in a periodic way. The most relevant of these, a subset of the ones that are used in the project, are: •EnterCombat: called when the entity is out of combat and is the target of an aggression, or detects a hostile enemy nearby (this depends on the reaction mode set for the entity). This function is usually used to define everything needed by the entity to enable its combat mode. For example, in the case of bosses, here where timers for mechanics of the first phase are set. 54 •Reset: used to initialize, or restart, the state of the entity and its related environment. This is called under 2 circumstances. The first is when the entity is created and placed within the world. The other is when the entity goes out of combat without dying, for example if it kills its targets or if they escape in some way, and the environment has to be entirely cleaned-up (removing auras related to the encounter or adds summoned by the entity). •DoAction: this hook is usually used to execute actions in the same update step. The real utility of this function comes from the fact that it can be called by entities other than the one with the AI. •UpdateAI: the update hook is executed at every world update tick. It receives the difference in milliseconds between the current and previous update to control everything time related. In this hook the timers and events are updated and processed as needed. This is where most of the behaviour of any scripted AI is controlled. •SpellHit and SpellHitTarget: hooks that are called when the entity is hit by someone else’s spell, or when a spell casted by the entity hits a target, respectively. This can be used to trigger certain effects, actions or events based on predefined conditions. For example, with the Lightning Rod mechanic, the application of the debuff “Static Overcharge” on the players hit by “Chain Lightning” is done through the SpellHitTarget hook. •DamageTaken: a hook that is called every time the entity receives damage. This has practical applications like neutralizing incoming damage when needed, although the most common use is to control the health dependent phase transitions of the encounter. •JustDied: this hook is called when the entity’s health reaches 0 and it dies. This is usually used to trigger events once the NPC is defeated, or to simply save the progression of the group, for the case of bosses within instances. The hook is also used to make a clean-up like what is done in Reset. 55 Figure 26: Declarations of the AI script hooks listed above. Some are declared within UnitAI, while others are within CreatureAI. Using all the available tools and resources I developed the different scripts and necessary elements to put every designed concept into the game. The design step defined a lot of the spells that had to be used for the fight, but to that list I added many others that increased the visual impact of some of the mechanics. Throughout the process, many enumeration structures were created to keep organized that data, like Events, Actions, SpellIds, Texts, Creatures, between others. Other definitions related to the different phases or general data were also defined (like the Shadow Nodes’ connections of the Shadow Path intermission). The boss and other NPCs that needed specific behaviours had a custom script created for each one. Moreover, various other NPCs were created and scripted to enhance the visual impact of some of the game mechanics. Figure 27: Aru, the Existential Will boss script. Most is contained within the BossAI section that is hidden. The entire structure is contained in the C++ script: the existential will.cpp The scripting in C++ is not the only element that I have to handle. As said before, this framework works using SQL to manage the back-end database content. This means 56 C. In the case that the update counter reaches the target update value defined as hyperparameter, then the target network is updated and the model saved to disk. v. Increase learning steps (frames) counter of the main agent. (f) If this is done through the function to get a mechanic identifier to execute for the C++ script: i. All actions values are taken from the corresponding agent’s network. ii. The identifiers are paired with their priority output value from the network. iii. Based on the phase from the state, invalid actions are removed from the list of candidates. iv. The list is filtered once more, removing mechanics that are still on cooldown. v. The best action is returned to the C++ script. Bugs and Issues Throughout the implementation of the project, some issues arose as consequence of the integration of the Python RL framework to the MMORPG framework. Some of them were detected and found out as simple human errors that could be fixed. For example, in the data object building function within the C++ script, many considerations had to be taken, and some of the versions got errors like passing a “NaN” as the current raid mean health, or null objects in the mechanics’ cooldowns. The first happened in the specific case of only one player within the raid, which could happen if other dead players released spirit beforehand; the second case was because the index to build the object was increased by an erroneous amount. On the other hand, there were some other issues that are more complex, and the source of them could not be found. Fortunately, some workarounds were able to allow the program to work correctly. The main example of this kind of problems that were faced during development is how when using the function to call a Python defined function within a module, defining torch tensors using numpy arrays froze the server entirely. This was avoided by building the tensors after transforming the numpy arrays into python native structures. 5.5 Points of Interest There are 2 relevant aspects of the project that can be emphasized in this part. The first is related to the design and implementation of the RL AI system. This is implemented using an adaptation of the Rainbow algorithm [2] done in Python. Using this method within a new game environment is not the only point that can be emphasized. The adaptation and usage of the Rainbow algorithm in an asynchronous way is one of the spotlights of this project. There are now registered implementations of an asynchronous version of Rainbow, which makes this project interesting to research about the possibility of applying this state-of-the-art algorithm in a new way. 63 The second main aspect of this project that can be emphasized is the way in which the early versions of the AI are trained. Is common to say that an agent with few, or without, days of training is ”dumb”. It literally does not know anything about the environment and the actions it may choose would be random. To solve this issue, the boss relies in the basic AI system that is supported by the MMORPG framework: scripting. This means that the agents will have 2 ways of choosing and executing mechanics. The script allows to schedule timed events that have to be managed within the update ticks. These events will mostly be the mechanics that the boss does during the encounter. The second way relies in a hybrid way, which uses one event that asks the network to choose an action and is only used when the choices are entirely handled by the trained network. The focus is put on the first way, where the consequences of the actions chosen have to be checked and used to train the AI. Each mechanics respects its corresponding constraints regarding phase and cooldown. The events are registered when the phase starts and then executed. When this happens the mechanics is executed and, after a bit of time, the environment is processed and the data is sent to the Python script to execute a training step. During the usage of scripting, processing a training step after a mechanic is executed is assured to happen before the next mechanic is managed. This is an innovative way to face the early training phases, as it guarantees a correct and controlled behaviour when the network can create a very inconsistent one. The drawback comes from the fact that the developer has to create a basic, or not, script that should be able to make all the actions that need training. If the range of possible options increases this method would need revision, for example: adding free movement would imply some knowledge of the terrain, as well as of the entities placed around. 64 6 Experiment 6.1 Experimental Procedure The application of RL within the field of MMORPG games is not common. Moreover, this project include the combination of various elements that have not been tested altogether. This new AI application is going to be tested in the “World of Warcraft: Cataclysm” server called Vulcania, of the NaerZone community. The server administrators contracted the service offered by OVH [37] in 2016. The dedicated server has an Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 32GB of RAM memory, 480GB of SSD disk and Debian 8 (Jessie) operating system. The boss will be made accessible to a server with a mean of 100 users online. Most of the other hyperparameters of the Rainbow algorithm are left as they were, set based on the original paper, with the exception of the input and output dimensions: •Gamma discount factor: 0.99 •PER alpha for prioritization: 0.2 •PER beta for importance sampling: 0.6 •PER priority to ensure that every transition is used: 10−6 •Categorical DQN minimum value of support: 0.0 •Categorical DQN maximum value of support: 200.0 •Categorical DQN unit number of support: 51 •N-Step Learning step number to calculate TD error: 3 The other hyperparameters that have to be defined for the agent were set as follows: •Number of frames for training: 100000 •Memory size: 1000 •Batch size for sampling: 32 •Target update, period for target model’s hard update: 15 In particular, the number of frames is set as a high number expecting to obtain a high amount of training steps. Still, in this kind of environment one would expect to have a constant training that never ends. The other parameters are set to keep an updated version of the network available and saved. This is done to avoid losing any progress in the case that any error or crash happens during the experiments. The structure of the network was left untouched with regards of the original code. The network was constructed using one RelU “feature layer” (connected inputs to 128 units), one “advantage layer” (hidden noisy layer of 128 inputs and outputs; and another noisy layer with 128 inputs and as much outputs as actions times the number of atoms) and one final “value layer” (a hidden noisy layer of 128 inputs and outputs; and another noisy layer with 128 65 inputs and as many outputs as actions). The boss will begin in full script mode which, as explained before, will keep its behaviour controlled. As days go by and players try and learn the boss, the AI will also be gaining experience. The possibility of using the trained network to pick actions will be increased gradually, until the training phase is entirely over. At that point, the only scripted event that the boss will have is the one to call the Python function and obtain a new action from the trained network. At the same time, the network will still be trained with the new experience obtained by applying the chosen actions, in full RL fashion. During the entire process, the boss will be storing data about the number of episodes, the score at each episode and the result (1 : V ictory; 2 : Defeat). To monitor the training of the network, the step and the corresponding loss value will also be stored. To evaluate the level of acceptance of the design of the encounter, as well as the usage of RL as game mechanic seen from the perspective of users. I created a form that those that made tries to the NPC had to answer. The questions are divided into 2 parts, one that tries to analyze some behaviour elements perceived by the players, and other that tries to retrieve an opinion or evaluation of the quality of the encounter. The questions from the form based on the AI are: 1. Reinforcement Learnig is an AI technique that tries to make the NPC to learn how to use its available actions to fulfill its objective. Did you hear/read about its usage as part of the AI of an NPC in an MMORPG game (like World of Warcraft)? Yes - No. 2. How much interest it generates you to know that a complex AI method is being used as part of the design of a Boss? 0: Nothing - 5: A lot. 3. How consistent do you think the behaviour of the Boss was based in your perception? 0: Nothing - 5: A lot. 4. The Boss started by only using scripts (like any other boss in the game) and, over time, the probability of using the RL AI that was being trained was increased. Did you notice any change in the behaviour throughout the days? Yes - No. 5. If the previous answer was ”Yes”. How much do you consider that change to be for better? 0: Nothing - 5: A lot. 6. How noticeable was the change in behaviour throughout the days? 0: Nothing - 5: A lot. 7. Do you think that this kind of techniques have a future as a relevant part of a game’s mechanics even if the behaviour may be difficult to control? Yes - No - N/A. The questions of the form regarding boss design are: 1. How balanced do you consider the Boss to be? (Health, damage, attack speed) 0: Too Unfair - 5: Too Easy. 66 2. How balanced do you consider the phase 1? (Twilight Instability, Shadow Core, Dancing Magma, Gravity Crush) 0: Too Unfair - 5: Too Easy. 3. How high would you qualify phase 1? 0: Very Bad - 5: Very Good. 4. How balanced do you consider the phase 2? (Ice Tomb, Earthen Spikes, Earthen Vortex, Lightning Rod)0: Too Unfair - 5: Too Easy. 5. How high would you qualify phase 2? 0: Very Bad - 5: Very Good. 6. How balanced do you consider the flight intermission: Song of the Ancients? 0: Too Unfair - 5: Too Easy. 7. How high would you qualify the flight intermission: Song of the Ancients? 0: Very Bad - 5: Very Good. 8. How balanced do you consider the flight intermission: Shadow Path? 0: Too Unfair - 5: Too Easy. 9. How high would you qualify the flight intermission: Shadow Path? 0: Very Bad - 5: Very Good. 10. How balanced do you consider the final phase? 0: Too Unfair - 5: Too Easy. 11. How high would you qualify the final phase? 0: Very Bad - 5: Very Good. 6.2 Experiment Results After one week of leaving the boss available for users in the server the amount of data collected regarding episodes is less than expected. Results regarding the progression of the training loss are a bit more interesting and can encourage further research in this kind of approaches. Firstly, in Fig. 30 (left) I show the scores of the completed episodes. The way in which the scoring works is that a score of 100 means that the entire raid was defeated, while any value lower than that implied that at the end part of the raid was alive, meaning that the agent lost. The plots are also divided into the ranges in which the agent had a different, and increasing, probability of using the trained network to make decisions. As shown in the legend, the scripting training phase took most of the time, as it covered only using the script, up to using the network with a probability of 70%. The last few episodes registered were done only using the network to choose boss mechanics to execute. The second plot of Fig. 30 (right) shows the progression of the cumulative sum of victories and defeats suffered obtained by the agent throughout all the episodes. In Fig. 30 (bottom left and bottom right) the cumulative sum within each of the levels of usage of the network are displayed. 67 Figure 30: Experiments scores throughout the episodes and gradual increase of the usage of the network (top left), the cumulative sum of victories and defeats throughout the episodes (top right), the victories (bottom left) and the defeats (bottom right) cumulative sums divided by the level of usage of the network. In Fig. 31 I plot the progress of the training loss throughout the steps executed. Is important to remember that the loss is not calculated, and the network does not proceed to train, until the replay memory has enough entries to start batching. Situations in which the memory had to be refilled with experience can be seen in the graph, like the jump during the phase with 15% of probability of using the network. Figure 31: Training loss progression throughout the training steps and gradual increase in the probability of using the network to pick a mechanic. Even if the participation of users was lacking when asked to answer the questions of the form, I consider that it may be mildly interesting to see the results of the few answer received. Only 4 people answered, so this data does not suffice to draw any clear conclusion. 1. Reinforcement Learnig is an AI technique that tries to make the NPC to learn how to use its available actions to fulfill its objective. Did you hear/read about its usage 68 as part of the AI of an NPC in an MMORPG game (like World of Warcraft)? Figure 32: Answers to the 1st AI behaviour related question. 2. How much interest it generates you to know that a complex AI method is being used as part of the design of a Boss? Figure 33: Answers to the 2nd AI behaviour related question. 3. How consistent do you think the behaviour of the Boss was based in your perception? Figure 34: Answers to the 3rd AI behaviour related question. 4. The Boss started by only using scripts (like any other boss in the game) and, over time, the probability of using the RL AI that was being trained was increased. Did you notice any change in the behaviour throughout the days? Figure 35: Answers to the 4th AI behaviour related question. 69 5. If the previous answer was ”Yes”. How much do you consider that change to be for better? Figure 36: Answers to the 5th AI behaviour related question. People that answered ”No” to the previous question had to give answer to this, which was wrong. The ones that answered ”No” previously, also answered ”No” in this case, so the true ratio is of 50% 6. How noticeable was the change in behaviour throughout the days? Figure 37: Answers to the 6th AI behaviour related question. 7. Do you think that this kind of techniques have a future as a relevant part of a game’s mechanics even if the behaviour may be difficult to control? Figure 38: Answers to the 7th AI behaviour related question. The answers to the part of boss design yielded the next results: 1. How balanced do you consider the Boss to be? (Health, damage, attack speed) Figure 39: Answers to the 1st design related question. 70 2. How balanced do you consider the phase 1? (Twilight Instability, Shadow Core, Dancing Magma, Gravity Crush) Figure 40: Answers to the 2nd design related question. 3. How high would you qualify phase 1? Figure 41: Answers to the 3rd design related question. 4. How balanced do you consider the phase 2? (Ice Tomb, Earthen Spikes, Earthen Vortex, Lightning Rod) Figure 42: Answers to the 4th design related question. 5. How high would you qualify phase 2? Figure 43: Answers to the 5th design related question. 6. How balanced do you consider the flight intermission: Song of the Ancients? Figure 44: Answers to the 6th design related question. 71 7. How high would you qualify the flight intermission: Song of the Ancients? Figure 45: Answers to the 7th design related question. 8. How balanced do you consider the flight intermission: Shadow Path? Figure 46: Answers to the 8th design related question. 9. How high would you qualify the flight intermission: Shadow Path? Figure 47: Answers to the 9th design related question. 10. How balanced do you consider the final phase? Figure 48: Answers to the 10th design related question. 11. How high would you qualify the final phase? Figure 49: Answers to the 11th design related question. 72 [13] Roman, Victor (2019) ”Unsupervised Machine Learning: Clustering Analysis” [Online] [Consulted on: 18th of June, 2020] Available at: <https://towardsdatascience.com/ unsupervised-machine-learning-clustering-analysis-d40f2b34ae7e> [14] Roman, Victor (2019) ”Unsupervised Learning: Dimensionality Reduction” [Online] [Consulted on: 18th of June, 2020] Available at: <https://towardsdatascience. com/unsupervised-learning-dimensionality-reduction-ddb4d55e0757> [15] Poole, David; Mackworth, Alan (2010) ”On-Policy Learning” [Online] [Consulted on: 18th of June, 2020] Available at: <https://artint.info/html/ArtInt_268.html#: ~:text=An%20off%2Dpolicy%20learner%20learns,agent%2C%20including% 20the%20exploration%20steps.> [16] Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David; et al. (2015) ”Humanlevel control through deep reinforcement learning” Nature 518, 529–533 (2015) [Online] [Consulted on: 18th of June, 2020] Available at: <https://www.nature.com/ articles/nature14236> [17] Hazzikostas, Ion (2013) ”Ion Hazziokostas’s Twitter account” [Online] [Consulted on: 19th of June, 2020] Available at: <https://twitter.com/watcherdev> [18] Activision Publishing, Inc; Blizzard Entertainment, Inc (2020) ”Activision Blizzard” [Online] [Consulted on: 19th of June, 2020] Available at: <https://www. activisionblizzard.com/> [19] Blizzard Entertainment, Inc (2020) ”Global Play - Blizzard Support” [Online] [Consulted on: 19th of June, 2020] Available at: <https://eu.battle.net/support/en/ article/7558> [20] chaud (2015) ”WoW Down to 7.1 Million Subscribers” [Online] [Consulted on: 21st of June, 2020] Available at: <https://www.mmo-champion.com/threads/ 1774731-WoW-Down-to-7-1-Million-Subscribers> [21] getMaNGoS (2020) ”The home of MaNGoS” [Online] [Consulted on: 19th of June, 2020] Available at: <https://www.getmangos.eu/> [22] TrinityCore (2020) ”TrinityCore” [Online] [Consulted on: 19th of June, 2020] Available at: <https://www.trinitycore.org/> [23] Free Software Foundation, Inc (1991) ”GNU General Public License” [Online] [Consulted on: 19th of June, 2020] Available at: <https://www.gnu.org/licenses/ old-licenses/gpl-2.0.txt> [24] Free Software Foundation, Inc (2004) ”Free Software Foundation” [online] [Consulted on: 19th of June, 2020] Available at: <https://www.fsf.org/> [25] NaerZone (2019) ”About Us - NaerZone App” [Online] [Only available in spanish, consulted on: 19th of June, 2020] Available at: <https://naerzone.com/ sobre-nosotros.php> 79 [26] Kaivax (2011) ”4.3: Hour of Twilight Patch Notes” [Online] [Consulted on: 19th of June, 2020] Available at: <https://worldofwarcraft.com/en-us/news/3964563/ 43-hour-of-twilight-patch-notes> [27] Aokromes (2018) ”Installation Guide - Trinity Core” [Online] [Consulted on: 19th of June, 2020] Available at: <https://trinitycore.atlassian.net/wiki/spaces/ tc/pages/2130077/Installation+Guide> [28] Python Software Foundations (2004) ”Python Homepage” [Online] [Consulted on: 18th of June, 2020] Available at: <https://www.python.org/> [29] PyTorch Community (2020) ”PyTorch” [Online] [Consulted on: 18th of June, 2020] Available at: <https://pytorch.org/> [30] NumPy ”NumPy” [Online] [Consulted on: 18th of June, 2020] Available at: <https: //numpy.org/> [31] Blizzard Entertainment, Inc (2014) ”World of Warcraft: Looking for Group - Documental” [Online] [Consulted on: 20th of June, 2020] Available at: <https: //youtu.be/4d3M6N5vMoQ?t=1519> [32] Fanbyte (2020) ”Deathwing - NPC - World of Warcraft” [Online] [Consulted on: 20th of June, 2020] Available at: <https://www.wowhead.com/npc=56173/deathwing> [33] Fanbyte (2020) ”Illidan Stormrage - NPC - World of Warcraft” [Online] [Consulted on: 20th of June, 2020] Available at: <https://www.wowhead.com/npc=22917/ illidan-stormrage> [34] Blizzard Entertainment, Inc (2019) ”Warcraft III: Reforged” [Online] [Consulted on: 21st of June, 2020] Available at: <https://playwarcraft3.com/en-gb/> [35] Fanbyte (2020) ”Hagara, the Stormbinder - NPC - World of Warcraft” [Online] [Consulted on: 20th of June, 2020] Available at: <https://www.wowhead.com/npc= 55689/hagara-the-stormbinder> [36] Fanbyte (2020) ”Sindragosa - NPC - World of Warcraft” [Online] [Consulted on: 20th of June, 2020] Available at: <https://www.wowhead.com/npc=36853/ sindragosa> [37] PVHcloud (2020) Web hosting, cloud and dedicated servers” [Online] [Consulted on: 21st of June, 2020] Available at: <https://www.ovh.es/> [38] Fanbyte (2020) ”Champions’ Cache - Object - World of Warcraft” [Online] [Consulted on: 20th of June, 2020] Available at: <https://www.wowhead.com/object= 195631/champions-cache> 80