Hybrid Machine Learning/Simulation Approaches for Logistic Systems Optimization

FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Hybrid Machine Learning/Simulation Approaches for Logistics Systems Optimization Francisco Alexandre Lourenço Maia FINAL VERSION Mestrado Integrado em Engenharia Eletrotécnica e de Computadores Supervisor: Américo Lopes de Azevedo Second Supervisor: João Pedro Tavares Vieira Basto July 22, 2020 c Francisco Alexandre Lourenço Maia, 2020 Resumo Hoje em dia, tem-se testemunhado um abrupto crescimento e desenvolvimento da indústria, refletido no elevado grau de complexidade e inteligência que os sistemas de produção correntes apresentam, onde se destacam os sistemas logísticos. Esta incessante procura pela inovação e melhoramento contínuo são muito recorrentes na época atual, traduzindo-se em constantes transformações no conceito da qualidade de um produto. Deste modo, emerge a necessidade em otimizar os layouts fabris conduzindo a um aumento da flexibilidade face aos seus comportamentos dinâmicos. Neste seguimento surge a imprescindibilidade de aprimoramento do comportamento do veículo autónomo associado, com vista a finalidades comuns como o aumento da produtividade e minimização de custos e lead times. Neste âmbito, o objetivo desta dissertação é a combinação de técnicas de Reinforcement Learning com abordagens de simulação para a otimização de um sistema logístico job-shop, no que à produtividade diz respeito. Para além da implementação do modelo de simulação do sistema logístico, esta dissertação desenvolve também numa fase inicial comportamentos elementares a aplicar ao veículo, implementadas no próprio ambiente de simulação. Posteriormente, dado que a área de Machine Learning tem obtido tanto sucesso noutras áreas tecnológicas, surgiu o desafio da introdução do conceito de rede neuronal, através da criação de uma nova entidade designada Agente e caraterizada pela técnica de aprendizagem baseada em Reinforcement Learning. Por fim, nesta dissertação, para além de se concluir que a abordagem baseada em Reinforcement Learning proporcionou os melhores resultados de produtividade, retiraram-se ainda conclusões no que à robustez destes modelos diz respeito, a fim de avaliar a sua flexibilidade quando sujeitos a diferentes contextos, simulando um ambiente real. i ii Abstract Nowadays, we have been witnessing an abrupt growth and development of the industry, reflected in the high level of complexity and intelligence that the current production systems present, in which the logistics systems stand out. This incessant search for innovation and continuous improvement are very common today, reproducing into constant changes in the product quality concept. In this sense, the need to optimize the factory layouts emerges, leading to an increase in flexibility because of their dynamic behaviours. In this segment, there is an essential need to improve the behaviour of the associated autonomous vehicle, to reach common objectives such as increasing the productivity and minimizing costs and lead times. In this context, the objective of this dissertation is the combination of Reinforcement Learning techniques with simulation approaches for the optimization of a job-shop logistics system, regarding productivity. Beyond the implementation of the simulation model of the logistics system, this dissertation develops, in an initial phase, elementary behaviours to be applied to the vehicle, implemented in the simulation environment itself. Subsequently, given that the Machine Learning area has been so successful in other technological areas, the challenge of introducing the concept of the neural network appears, through the creation of a new entity called Agent and characterized by the Reinforcement Learning technique. Finally, in this dissertation, in addition to concluding that the Reinforcement Learning-based approach provided the best productivity results, conclusions were also drawn regarding the robustness of these models, in order to assess their flexibility when subject to different contexts, simulating a real environment. iii iv Acknowledgements My first words go to Engineer João Basto and Professor Américo Azevedo for all their availability, comprehension and support during this master’s dissertation. Their help was fundamental in this whole process and I am very grateful to them for that. Secondly, I would also like to acknowledge the support of the Engineers Romão Santos and Narciso Caldas for all the availability and incentive during the dissertation. In this segment, I also have a word addressed to INESC-TEC, in particular to the Centre of Enterprise Systems Engineering and all its elements for the facilities offered throughout the dissertation and for the excellent environment provided. I would like to thank the Faculty of Engineering of the University of Porto and its professors for everything that has been transmitted to me over these 5 years, not only academically but also personally. I have to highlight all the friendly relationships I have created in these 5 years, namely Carlos Carvalho, with whom I shared this experience in the INESC-TEC during this final semester, as well as Fábio Queirós, Eduardo Caldas, Francisco Pires, André Oliveira, André Cipriano, Lídio Ribeiro, Maria Pereira, Alexandra Santos and Artur Almeida, among many others that I take with me for the rest of my life. I also thank my family for all the support and motivation that helped me to go through this journey, namely my parents Francisco Maia, Maria Isilda and sister Carolina Maia. Finally, I address a word of appreciation to all my friends for all the encouragement and stimulation during this period of my life, not only in the bad moments but also in the good ones. Francisco Maia v vi A moment of pain is worth a lifetime of glory. Louis Zamperini vii xiv LIST OF TABLES 5.19 Productivity (in parts) analysis referring to different production mixes, for a time horizonof52hours................................. 71 5.20 Productivity (in parts) in stochastic environments, for a time horizon of 36 hours . 72 5.21 Productivity (in parts) in stochastic environments, for a time horizon of 52 hours . 72 Abbreviations and Symbols AGV Automated Guided Vehicle AI Artificial Intelligence FIFO First In, First Out IB Input Buffer IBWSx Input Buffer of Workstation x JIT Just-in-Time MHS Material Handling System OB Output Buffer OBWSx Output Buffer of Workstation x PPO Proximal Policy Optimization RL Reinforcement Learning VRP Vehicle Routing Problem WIP Work-in-Progress WS Workstation xv Chapter 1 Introductory Analysis 1.1 Contextualization Lately, there has been an abrupt growth and development of the industry, in which the concept “Industry 4.0” appeared, which allowed the exchange of information between a variety of equipments in a factory. Namely regarding the optimization of internal processes, as well as a company’s products and even services, this paradigm is revolutionizing the industry worldwide. Consequently, the current need and demand for innovation and improvement follow the model based on continuous improvement. [A year without improving is a year won by competitors - J. M. Juran]. [8] In this sense, the concept of Lean Thinking, founded by Taiichi Ohno and Eiji Toyoda, which combines the elimination of waste (Just-in-time - "Any activity that the customer is not willing to pay" - Taiichi Ohno) with the immediate reaction to any problem that could arise in during a process (Jidoka - Japanese term), has emerged in this context. The main objective of this idea is to increase customer satisfaction, creating significant changes in the manufacturing processes that contribute to their better functioning (Kaizen), increasing productivity and efficiency. [9] And what is the reason for this incessant search for innovation and optimization of industrial processes? The answer is pretty simple, customers are the main reason. Their demands and needs have also been increasing over the past few years, both in terms of variety and quality. At the beginning of the study of these subjects, it was considered that quality would only be related to the product specifications ("Quality is conformance to the specifications" - Philip Crosby). However, this concept has been constantly updated, considering that the primary factor is the satisfaction of the customer’s needs ("Quality is fitness for use" - Joseph Juran). For this, it is firstly necessary to infer the product specifications, followed by the identification of the probable errors that may arise during the processes and their causes, before proceeding to their elimination. [10] The need to increase quality, decrease costs and reduce delivery times, led to the creation of several types of factory layouts, namely the functional (process-oriented, job-shop), line (linear flows, flow-shop) and fixed (the product cannot be moved). In this sense, the industries started to adopt different modes of production, namely the jobshop, characterized by the existence of specialized areas by function, and flow-shop, defined by the 1 2Introductory Analysis production lines. Overall, the layouts are designed to minimize Material Handling costs, eliminate bottlenecks, reduce cycle times, eliminate waste and increase process flexibility. Currently, many industries make use of the job-shop production mode, because it is related to a high diversity of products produced in low volume (production to order). It also allows the increase of the flexibility of production processes, however, they present high WIP and queues. [11] In order to optimize these logistics systems, Material Handling is in great focus at the present. In order to obtain shorter cycle times and lower costs in transporting raw materials between workstations (WSs), there is a need to develop new optimization algorithms. For this, it is necessary to take into account the time and space (of the warehouse, for example). In other words, it is necessary to coordinate all the tasks of each workstation in order, for example, to satisfy all orders with the shortest possible lead time. [12] Directly associated with these material management systems are the AGVs (Automated Guided Vehicle) that allow the materials to be transported between stations, and are generally unmanned. Currently, vehicle routing problems are in the spotlight and their objective is to travel the shortest distance possible, minimizing costs. This type of problem has some common characteristics to the "Traveling Salesman" problem, in which it is supposed to visit a certain number of cities covering the shortest possible distance. In the case of industries, each vehicle has a maximum capacity. Most of these problems can be solved using the Milk-Run system, which is a delivery system that allows us to reduce stocks, reduce waste and optimize routes. It is a delivery system in which one product is deposited and another one is collected right after, in order to save time. It should also be noted that the AGV’s route is fixed. The collection of products from suppliers is carried out on a scheduled basis, in stipulated quantities, making cycle times more predictable. [13] Besides heuristics such as Milk-Run, other approaches, based on metaheuristics, have been also developed to solve this kind of problems. [14] Allied to these approaches, the concept of Machine Learning emerges. Since it has been quite successful in other areas of technology, why not make use of this tool and apply it to production systems? It is precisely these issues that are currently being studied and developed. 1.2 Motivation The growing evolution of the industry due to the continued increase in competitiveness has led logistics management to be in vogue these days. In other words, its processes have undergone significant changes, both financially and temporarily, in addition to the objective of making the system more robust and secure. The importance of reducing human-made failures has led to the use of AGVs, which, besides the safety and precision issues, also contribute to the automatization of production systems, with regard to the sequence of operations. For this purpose, certain heuristics were created and developed that allowed, for example, to minimize the cycle times of a production line as well as the costs associated with Material 1.3 Objectives and Research Questions 3 Handling. It should also be noted that a large part of the lead time is spent on transport, with only a minority focusing on production processes. One of the most important formulations that emerged, taking into account the need to accelerate the flow of materials between locations, was the concept of Milk-Run systems. These ones allow that, in a delivery system, when a certain product is delivered to a stipulated location, another one is also collected, saving time in transportation. In short, these types of systems contribute to the integration between the logistics systems and supply chains. [15] That said, the Milk-Run concept was later transported to a factory layout context, in which each location corresponds to a workstation. As mentioned, these heuristics ensure the resolution of problems such as Job-Shop Scheduling and Material Handling. In order to solve these optimization problems, some tools like Machine Learning (learning algorithms such as neural networks) can be combined to solve material movement problems. This tool has a great prominence nowadays and allows to make predictions taking into account data collected previously, even in highly complex environments. In this segment, one of the paradigms of Machine Learning that emerges is Reinforcement Learning (RL). Based on the environment in question and the decisions taken by the Agent, this learning method allows receiving feedback that indicates the best decision made so far. [16] In summary, the study of the influence that these methods have on solving dynamic vehicle routing problems associated with the transport of materials between workstations in a factory is very interesting and allows to create very effective and adaptable algorithms. These algorithms based on Reinforcement Learning can be applied regardless of the complexity of the system, and there is no need to change the factory layout. In addition to all of this, the investment is low and the results achieved are satisfactory. In conclusion, the constant need to optimize both production times and transport, improving the routes, leads us to question the fact that, if Machine Learning has been so successful in other areas, why not adapt it to the productive systems and combine it with simulation approaches for the optimization of logistics systems. [17] 1.3 Objectives and Research Questions The main objective of this dissertation is the combination of techniques based on Machine Learning with simulation approaches for logistics systems optimization. Firstly, it is necessary to model the problem, understand the factory’s operation (layout), create the simulation model (through the Flexsim software tool) and, finally, define simple decision rules to command the AGV. Subsequently, it is possible to integrate training algorithms based on Reinforcement Learning techniques, which define a completely dynamic behaviour of the logistics system, to increase the productivity and minimize the makespans (the total time to complete a sequence of tasks). These algorithms, made available by OpenAI Baselines, are considered the state-of-the-art of Reinforcement Learning, nowadays. 4Introductory Analysis Finally, it is essential to test whether the algorithm adapts to changes in the factory, with respect to changes in the production mix or in the stochasticity of processing times. In order to be able to accomplish these objectives, we need to answer the following research questions (RQ): •RQ 1: How can we model a factory’s operation in a simulation model with simple decision rules to command the AGV? •RQ 2: How can we integrate training algorithms based on RL techniques with a simulation model to study the factory’s productivity and makespan? •RQ 3: How can we evaluate the robustness of the proposed approaches? 1.4 Methodological Approach This dissertation follows the study of a set of hybrid approaches of simulation and Machine Learning, whose consequent purpose is the optimization of a logistics system, leading to an increase in its productivity and minimization of makespan. The initial phase of the dissertation is allocated to the construction of the simulation model of the job-shop layout, using the Flexsim software, in which an automated guided vehicle will also be incorporated. Posteriorly, there are two elementary decision rules that will be implemented, which provide the indication of the workstation (WS) where a load of an entity will be performed. The first rule, referring to the inaugural simulation model, is based on the First-in First-out (FIFO) algorithm, while the second model is a Milk-Run optimized transport system. Then, the concept of communication between the simulation environment and an external program will be introduced, which will allow the creation of a distributed system, because the last two rules addressed were implemented in the simulation model. This external program, called Server, uses the Python language and it will communicate with the simulation environment, becoming responsible for AGV decision making. This new stage also requires that the communication between the two entities has to be preliminarily established, using the TCP communication protocol. At the beginning of the final stage, the concepts of Machine Learning will be introduced and discussed, namely the introduction of a third entity called Agent, which implements a neural network for decision making. This network will be responsible for defining the WS where the AGV will load the respective entities. For that, the Agent receives from Flexsim, through the models subsequently developed, a set of relevant information, namely observations of the current state of the plant and the current location of the AGV. However, it should be noted that the Server is still present, assuming the role of an intermediary between the Agent and Flexsim. Consequently, through a Reinforcement Learning algorithm, called Proximal Policy Optimization (PPO), the neural network will determine the respective WS where the AGV has to load an entity. A pre-training process for the Agent will also be studied and applied, which will allow the 1.5 Dissertation Organization 5 improvement of its learning phase and, consequently, lead to a possible maximization of productivity ("Base Model") and minimization of makespan ("Makespan Model"). Finally, the production mixes will be changed and the concept of probability distribution will be introduced. In this sense, the models previously discussed will be applied to a stochastic environment, in order to verify if they present a high level of robustness so that they can be applied in a real context, where the processing times are subject to constant changes. All these phases are represented in Figure 1.1. Figure 1.1: Dissertation approach method 1.5 Dissertation Organization With regard to the organization and structure of this document, it is divided into six chapters. The first chapter pretends to introduce the theme of the dissertation, including its contextualization, motivation, objectives and research questions, as well as the schematic of the methodological approach followed. Chapter 2presents the state-of-the-art of the subjects covered by the dissertation, like the industrial production systems, in particular the job-shop type, Material Handling Systems, Machine Learning concepts and discrete-event simulation. The description and characteristics of the problem, as well as the description of its methodology, are covered in chapter 3, even as the implementation of the first three transport systems considered throughout the dissertation. Chapter 4reflects two additional transport systems, introducing Reinforcement Learning techniques in combination with simulation approaches. The robustness of the systems is related to the production mixes and processing times, which are also addressed. 6Introductory Analysis The results obtained are exposed and treated in chapter 5with regard to the five transport systems developed and their robustness. Finally, chapter 6presents the conclusions of the dissertation project and the suggestions for a future work. Chapter 2 State-of-the-Art This chapter aims to present the results of a bibliographic search, in order to internalize, in a simple way, the concepts that will be addressed throughout the dissertation. Firstly, in section 2.1, the concepts of industrial production systems will be introduced in general and, later, in section 2.2, the job-shop production systems will be addressed. Then, section 2.3 presents some interpretations of the logistics systems associated with Material Handling in industrial environments. In section 2.4 the question of Material Handling Systems (MHS) will be addressed, and in the next section (2.5) the principles referring to Milk-Run systems will be introduced. Section 2.6 introduces the subject of Machine Learning, with special reference to the Reinforcement Learning technique. Finally, section 2.7 presents the main concepts associated with simulation approaches. 2.1 Industrial Production Systems The constant evolution of the market means that, nowadays, society seeks for differentiated products, which follows a perspective of diversity rather than quantity. In this way, the specialized industrial organizations have as main objective the increase of the effectiveness and efficiency of their production processes. About typologies, there are systems whose objective is to produce products on a large scale with a low degree of variety and are also characterized by its high productivity, low qualification of their operators, reduced complexity of factory management and reduced flexibility. This type of system is called as product oriented. On the other hand, there are systems that are oriented to the process and to the customer, in which its main goal is to satisfy the customer’s needs, according to a pull perspective, and to value the quality of the product. This type of system gives more importance to the product variety instead of quantity, and it is considered more flexible and denoted by greater complexity of factory management. 7 14 State-of-the-Art •Determined time periods assignment problem - known routes and times. Figure 2.7: Representation of the categories of milk-run in-plant distribution problems, adapted from [4] 2.5 Milk-Run A Milk-Run System is directly related to the concept of transportation in the industry. The need to reduce stocks and costs in transport was the reason for the creation of this model. This idea of cost reduction follows the JIT philosophy of Lean fundamentals. Briefly, this is a delivery system that allows us to optimize routes and reduce wastes between workstations in a factory plant (in-plant). With regard to the Milk-Run system, Baudin [22] states: “This concept allows to move small quantities of a large number of different items with predictable lead times and without multiplying transport costs" [22] - (Baudin, 2004 [13])” All these aspects led Lean manufacturers to choose to organize their transport according to fixed times and routes, in the form of a Milk-Run system. The design of these systems involves a higher complexity, which according to Meyer [21] can be described in the following three factors: •Transport of materials; •Frequency of transports; •Route scheduling. 2.5.1 Advantages of a Milk-Run System Comparing to the traditional approach, shown in Figure 2.4, the Milk-Run system presents the following advantages: 2.5 Milk-Run 15 Inventory Reduction As Figure 2.8 shows, the Milk-Run typology, in cases X and Z, allowed the inventory level to be reduced by 1/3, because the frequency of transport was increased by three units. In case Y, the frequency of transport increased twice, resulting in a decrease in the stock of 2/3 of the level of the point-to-point typology. Figure 2.8: Comparison between the inventory levels of a Point-to-point and Milk-Run typology, adapted from [5] Replenishment with Predictable Lead Times On a daily basis there are thousands of products with different transport frequencies. However, through Figure 2.8, which presents only three different products (X, Y, Z), it is possible to generalize for other cases. So, it can be concluded that their lead times are predictable. Because we have access to data which indicates an increase or decrease in the frequency of supply, it becomes predictable to estimate the stock variations that will occur in the future. Better Inventory Visibility In the case of the point-to-point deliveries, there may be a case where only the product X is transported on a large scale, causing an almost total emptying of the respective shelves. However, this situation does not represent any type of anomaly in the inventory. In the case of the Milk-Run system, because the quantities transported are practically the same for all products, any significant variation in the amount of any type of product present on the corresponding shelf is an immediate sign of the presence of an anomaly. So, the Milk-Run typology has a better visibility of the inventory, allowing to act quickly in case of abnormality. 16 State-of-the-Art Improve Communication Skills with Suppliers Using Milk-Run enables suppliers to be in regular contact with the customers, because of the frequency of supply. Therefore, it is possible to obtain feedback from consumers about the quantities and quality of the delivered products, which makes it possible to improve the delivery system and the future quality of the products. [13] 2.6 Machine Learning The Milk-Run systems need to adapt since the current reality of the factory layouts has led this system to become dynamic so that it is necessary to develop decision algorithms. So, the concept of Machine Learning was introduced in the scope of the transport of materials in a job-shop environment, more specifically the Reinforcement Learning technique. This concept argues that learning from interactions is a fundamental idea subjacent to almost all theories of learning and intelligence. [23] Basically, Machine Learning aims to learn based on previous data and make predictions or decisions for the future. [24] According to Arthur Samuel, pioneer in artificial intelligence: “Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed" - Arthur Samuel, 1959 [23] Enumerating some of its applications [23]: •Analyse product images on a production line to automatically classify them; •Detect tumours through brain scans; •Summarize long documents automatically. The Machine Learning systems can be classified into three categories, according to its learning processes: •Supervised Learning A series of examples (inputs) with the correct answer (outputs) is provided by an external supervisory Agent and, based on training, the implemented algorithm generalizes the correct answer to another set of inputs, afterwards. [25] •Unsupervised Learning The developed algorithm tries to identify similarities between the inputs, categorizing them. One of the best-known techniques is the clustering. [25] •Reinforcement Learning It is located between Supervised Learning and Unsupervised Learning. The algorithm is informed about the quality of the response, but it is not informed about how to correct it. 2.6 Machine Learning 17 So, it is necessary to explore and experiment other different possibilities until the Agent discovers how to obtain a higher quality response. In this particular dissertation, it is pertinent, with regard to the transport of materials in a job-shop environment, to analyse, essentially, the Reinforcement Learning field. 2.6.1 Reinforcement Learning Learning how to control Agents directly from high-level sensory information, such as vision and speech, is one of RL’s longstanding challenges. [26] The Agent is not specifically told what actions to take, unlike what happens in other forms of Machine Learning, like Supervised Learning. Therefore, the Agent will have to find out which actions, deliberated so far, allowed him to obtain greater rewards, at the end of the learning phase. Interestingly, the actions taken in the present will affect the respective reward, as well as the future ones. The concepts of "trial and error search" and "delayed reward" are the two most important characteristics of RL. Unlike Supervised Learning, which is a way of learning based on examples provided taking into account the knowledge of an external supervisor, this is not appliable to an interactive learning. This is because, in interactive problems, in most cases, it is impossible to obtain examples of the desired behaviour that are correct and represent all the situations in which the Agent needs to act. Hence, the RL allows the Agent to decide what action to take, taking into account his own experience. The Agent will have to check the decisions he has made in the past and find out if, in fact, he obtained a beneficial reward, so that he can then later carry out his action. In this sense, a new paradigm appears, in which the Agent, in addition to exploring knowledge that he already has from previous situations, also needs to explore new decisions never made before, to see if he gets a higher reward. Neither of these paradigms is considered better than the other because in both cases, the Agent will fail (obtain a lower reward) and the solution states in the critical capacity of the Agent, so he has to perform several tests, in order to find the solution that provides him with a final value corresponding to the biggest reward. [6] 2.6.2 Reinforcement Learning Characteristics Agent The Agent is the entity that it is responsible to make the decisions and it is called “learner” and “decision maker”. More specifically, the Agent and the environment interact with each other in the form of discrete time intervals (t=0,1,2, ...). As Figure 2.9 presents, for each t, the Agent receives a representation of the state of the environment st, such that st∈Srepresents the set of all possible states. Consequently, the Agent receives a response in the form of a reward rt+1∈Rand a new state of the environment, st+1, which is a feedback for the next decision to make at+1. [6] 18 State-of-the-Art Environment The environment is responsible for informing the Agent of the current state and the reward obtained for the action taken previously. It also tells the Agent a set of all possible states. Action The action is the result of the decision made by the Agent. The objective is to find the best solution, which corresponds to choosing the action that allows him to obtain the highest reward because each action originates different reward values. Reward The reward is a feedback in which the Agent evaluates the consequences of his action taken in the previous state. It is important to refer again that the objective is to obtain the greatest possible accumulation of rewards, keeping in mind that a large reward obtained in a given state does not necessarily mean that the final accumulation of rewards will be the best. This is because, although a specific reward in a given state is the largest one, it may lead to a non-ideal situation in the future and influence negatively the following rewards. [27] Policy π It is a mapping strategy that allows the Agent to decide the next action to take, in order to obtain a good accumulated reward in the long term. The RL specifies how the Agent can change his policy, taking into account his experience. The Agent can also be classified according to policy, value function and model. A policy, π, is a mapping of states s∈Sand actions a∈A(s), for the probability π(s,a)of taking an action at the time of a state s. The value of the state sunder a policy πis still denoted by Vπ(s). [6] Figure 2.9: Agent-Environment interaction diagram, adapted from [6] 2.7 Simulation Modelling is a tool that allows us to solve real context problems. Most of the time we cannot afford to experiment and test real objects, in order to obtain the best solution, since these objects are, in general, expensive and even scarce. Thus, the simulation assumes a fundamental role in this 2.7 Simulation 19 context, in order to be able to solve problems in a practical way and without collateral damage, confirmed by the following quote adapted from [28]. “Modeling consists of finding the path of the problem to its solution, in a risk-free world where we can make mistakes, undo things, go back in time and start again.” [28] In addition to the purpose of modelling, there are more benefits of using this method, such as [29]: •Simulation models allow us to analyse systems and find solutions in which the analytical models fail; •It allows testing new policies, operating procedures, decision rules, information flows without making changes to the real system; •Discover the bottlenecks. However, despite all these gains, the final results can sometimes be difficult to interpret. The fact that the simulation requires a lot of training is one of the least favourable points. 2.7.1 Components of a System In order to better understand the constitution of a simulation system, this section describes its main elements. [29] System A set of entities that interact with each other in order to achieve the outlined objectives. Model An abstract representation of a system, which allows it to describe in terms of its state, entities, processes, events, activities, and others. System State A necessary set of variables to describe the system. Entity An object of interest which requires an explicit representation. Attribute An entity property. 20 State-of-the-Art Event An instant occurrence that can change the state of the system. Activity The time duration that a task needs to be executed, known at the moment it starts. Delay The excess time interval, only known when it ends. Clock A variable that represents the simulated time. A method is a structure used to map real systems in simulation models. With regard to the simulation modelling methods, this can be organized into three categories: Agent-Based, System Dynamics and Discrete-Event Modelling. The use of each one of these methods depends on the system to be implemented and its objectives. Agent-Based It is part of the class of computational models for simulating actions and interactions between autonomous Agents (individual or collective). Despite the lack of knowledge of the system’s behaviour and inability to represent the process flow, the main objective is to verify and study the effect of the Agents on the system as a whole. [28] System Dynamics John Sterman [28] states that this model is a perspective and a set of conceptual tools that enables the understandment of the structure and dynamics of complex systems. This model is also a rigorous modelling method that allows to build formal simulations of complex systems and use them to design more effective policies and organizations. Discrete-Event Modelling It allows conceiving the modelling of a system as a discrete sequence of events in time. Each event occurs at a particular time and causes a change in the state of the system. This system requires that modelling has to be seen as a process so that it is a sequence of operations performed by Agents. [28] 2.7 Simulation 21 2.7.2 Discrete-Event Simulation Nowadays, this simulation method is widely used in the modelling and analysis of problems in the area of logistics systems, as it allows the study of aspects such as processes, scheduling and resource allocation. Health, business processes and military applications are also areas covered by this modelling method. Consequently, all these advances have led to a software development. [30] In the specific case of this dissertation, the simulation will be used to evaluate the internal material flow of a manufacturing plant, in order to make conclusions regarding the factory’s productivity and values of makespan, so we are able to identify possible bottlenecks. Therefore, this tool is used as a method for analysing and solving the following problems, associated with job-shop environments: •Evaluate the effect of changing material transport routes between workstations; •Analyse the phenomenon of resource allocation and bottleneck prevention; •Assess the impact of changing the elements in the layout on performance. Through published articles dedicated to the study of this area of simulation, it is possible to draw examples of applications in distribution and transport systems. Hugan (2001) elaborated a study that allowed him to evaluate the internal traffic of a General Motors automobile plant, in the USA, which was based on the JIT model of Lean manufacture. The simulation allowed to improve the internal routes for each type of product, as well as to estimate the average time spent by a product in the factory, from its entry to its exit. Kuo, Chen, Selikson and Lee (2001) used the simulation of discrete-events to study the flow of materials also in a manufacturing plant, which allowed them to have a deeper knowledge of operations and logistics processes. [30] 2.7.3 Construction Steps of a Simulation Model Problem Formulation The problem must be well formulated so that there is no doubt. There are still cases where it is necessary to proceed with a total or partial reformulation of the problem. [29] Establishment of Objectives and General Project Plan The objectives will be the questions to be answered through the simulation. After deciding which simulation method is the most suitable, it is necessary to list a series of alternatives to the simulation and find ways to evaluate the effectiveness of these same solutions. Besides the objectives defined for the end of each state, it is also important to mention in the plan the number of people involved, as well as the associated costs and the estimated time for each phase of the project. [29] 22 State-of-the-Art Model Conceptualization According to Pritsker (1998), although it is not possible a priori to define the instructions that will lead to a successful model, there are some points of view that must be followed for the model to be successful. It is necessary to start by defining a simplistic model and, from there, make the necessary changes step by step to obtain good results. [29] Data Collection Data collection is directly associated with the construction of the model, since the data collected will serve as input to the model. As this collection fills large intervals of time and this is a very important aspect in the elaboration of the model, it is essential to start the collection as soon as possible. [29] Model Translation This phase consists of converting the model into a simulation language using specific software programs. In the specific case of this dissertation, the software used is the Flexim program. [29] Verification and Validation This step enables to check if the program is prepared for the simulation model, carrying out verification and debugging tests. By comparing the model with the behaviour of the current system, it is useful to use this feedback in order to improve the model. The process is repeated iteratively until the result obtained is satisfactory. [29] Experimental Design The alternatives previously defined in the “Establishment of Objectives and General Project Plan” phase that must be simulated, must be determined. [29] Production and Analysis of Results After several tests of the model, with different data, the resulted analyses are used to estimate the performance of the system that was simulated. [29] Documentation After the simulation, it is necessary to report two types of data: program and progress. If the program is used by others in the future, the program documentation indicates the modes of operation and behaviour of the program, as well as other fundamental aspects. In relation to the progress report, this is essential for the model to obtain credibility and certification. [29] 2.7 Simulation 23 Implementation The success of the implementation will depend on each phase previously referred. [29] 30 Problem and Methodology Figure 3.4: Graphical representation of a WS – Flexsim 3.3.4 Transport As previously indicated, the transport of the entities is carried out by an AGV, which is created from an object of the Task Executer class (Flexsim Library). Thereby, it is only necessary to indicate which stations the AGV will collect (OB) and deliver (IB) parts. This information can be found in detail in the previous chapters of Processing, 3.2.2, Sequencing, 3.2.3 and Production Plan 3.2.4. It is also worth noting that the AGV speed will be 1.50 m/s for all simulation models. 3.3.5 Load All the parts come in into the system through the Source, however, in this particular case, this resource has been replaced by a Queue, called "Source1", which admits the same behaviour. The main reason for using a Queue object instead of a Source is that it is possible to observe the accumulation of parts throughout the simulation. At last, the resources where the parts are loaded are the Source and the output buffers. 3.3.6 Unload The final unloading of the entities is carried out at Sink, and the intermediate unloads between WSs take place in the respective input buffers (IBWS). 3.3.7 Processes There are essentially two tools that allow the building of a simulation model: 3D model and Process Flow. Regarding Process Flow, it always and in any circumstance overlaps the 3D model. However, the cooperation between both is essential, although, in this particular situation, it is in the Process Flow that the flow of entities and resources will be defined. In general, all transport systems involve three large blocks: Creation of Parts (Figure 3.5), Processing of Parts (Figure 3.6) and Transport of Parts. Both phases of creation and processing of parts are common to all models covered in this dissertation. 3.3 Discrete-Event Simulation Model – Flexsim 31 3.3.7.1 Creation of Parts The "Source" block defines the time when the production orders will be released, using an Excel file that contains the information for this purpose. Subsequently, the “Create Object and Type” block allows the creation of the entities and assigns them to the corresponding part type, taking into account the Label referring to the part type of the Production Plan table (Table 3.4). Taking into account the 5 different part types, there is a need to define colours for each part type, in order to distinguish them during the performance of the simulation, accomplished by the “Change Color” block: •1 - Aqua; •2 - Red; •3 - Blue; •4 - Yellow: •5 - Lime. Considering the type of the current part, the block “Determine Line” allows to go through the Routings Table (Table 3.5) and identify the line in question, in order to follow the WS sequence of the part type in evidence. Subsequently, after identifying the WS where the first unload will be performed, it is necessary to acquire the AGV in order to allocate it, so that the initial load can be executed in the Source and the unload in the IB of the following WS. At the end of the process, the resource is released ("Release"). Figure 3.5: Representation of the “Creation of Parts” Block 3.3.7.2 Processing of Parts When an entity enters the IB of a WS, it is necessary to move it to the WS itself, then operate it and, finally, move it one more time to the respective OB. However, it must be noted that the entity will be just moved from the current IB to the corresponding WS if there are no parts in 32 Problem and Methodology the respective WS, because its maximum capacity is unitary. These restrictions are defined by the creation of zones, focused on each one of the existing WS. At the end of the processing, the block “Exit Zone” will release the respective WS from the due entity already processed, and it will be available to receive a new part. Then the line in the Routing Table (Table 3.5) is incremented by one, in order to follow the pre-defined sequence of the part. Figure 3.6: Representation of the “Processing of Parts” Block 3.4 Implementation of a FIFO Transport System Amongst all the transport systems covered in the dissertation, this model is considered the simplest one since the decision rule to apply to the AGV follows a First-in, First-out paradigm. This means that the AGV will decide to take forward the transport of a part that was processed first, instead of a part that has been processed more recently. According to the Process Flow method, in addition to the two blocks discussed in subchapters 3.3.7.1 (Creation of Parts) and 3.3.7.2 (Processing of Parts), the block that mentions the transport of parts is also essential (3.4.1). 3.4.1 Transport of Parts Apart from the initial transport from Source to the IB of the first WS of each production order, addressed in the "Creation of Parts" block (3.3.7.1), this block includes all the other transports. In this segment, as soon as an entity reaches the OB of any of the 7 WSs, the AGV will be requested and, as soon as available, it will transport the part from the current WS to the next one, respecting the particular sequences. Finally, the AGV will be released, originating the creation of a new transport process. 3.5 Implementation of an Optimized Milk-Run System 33 Figure 3.7: Representation of the “Transport of Parts” Block – FIFO Model Table 3.5: Routings Table Line Part Type Workstation Processing Time (in minutes) 1 1 WS2 2 2 1 WS1 9 3 1 WS7 7 4 1 WS6 8 5 1 Sink1 0 6 2 WS4 10 7 2 WS2 9 8 2 WS5 5 9 2 WS1 5 10 2 Sink1 0 11 3 WS2 6 12 3 WS4 10 13 3 WS7 7 14 3 WS3 7 15 3 Sink1 0 16 4 WS7 3 17 4 WS3 7 18 4 WS2 7 19 4 WS4 5 20 4 Sink1 0 21 5 WS4 10 22 5 WS5 6 23 5 WS1 9 24 5 WS2 2 25 5 Sink1 0 3.5 Implementation of an Optimized Milk-Run System The main difference between this Milk-Run system and the one previously presented in the earlier subsection is the decision rule to be applied to the AGV. While in the previous system there was no priority regarding the transport of entities, following the FIFO paradigm, in this specific context 34 Problem and Methodology the AGV has a predefined WS sequence, (identified in the following Table 3.6 and illustrated in Figure 3.8), that it follows in order to load the entities. In other words, if the AGV is, for example, in WS2, but that same station does not contain entities in its OB to perform the load, the AGV will check if the next WS in the AGV Routing Table (WS4) includes parts in the respective OB, and so on. What distinguishes this optimized Milk-Run algorithm from the classic one is the fact that the AGV performs the load directly on the nearest WS, in a clockwise direction, which must necessarily have parts in its OB. On the other hand, what happens in the classic Milk-Run system is that the AGV must visit the nearest WS, also clockwise, even if it does not contain parts in the respective OB. Figure 3.8: AGV Trajectory – Milk-Run Model Table 3.6: AGV Routing Workstation Source WS2 WS4 WS5 WS1 WS7 WS3 WS6 Consequently, using one more time the Process Flow method, three large blocks were also established: Creation of Parts (3.3.7.1), Processing of Parts (3.3.7.2) and Transport of Parts (3.5.1). 3.6 Implementation of the NearestWS Rule 35 3.5.1 Transport of Parts The type of the “Release token.WS” block is the “Schedule Source” and this block allows the definition of the current WS and associates it with a token called token.WS. A token is considered a class, visibly represented by a circle, with the ability to be updated throughout the execution of the Process Flow. The “Next WS?” decision block identifies the next WS where the AGV has to perform the load, more specifically its OB, as well as an indication of the correspondent IB where the entity will be unloaded. After identifying the subsequent WS, it is necessary to reserve the AGV and then identify the oldest “entity” of the next WS. After that, it is essential to load and unload the current part and, finally, update the current WS token. Finally, the resource is released. It should also be noted that when the AGV Routing table is completely crossed and the current line corresponds again to the current WS, it means that none of the WSs has parts in the respective OBs to be loaded and, therefore, the AGV is redirected to a state where it will be waiting until a part is processed ("Wait for Event"). Figure 3.9: Representation of the “Transport of Parts” Block – Optimized Milk-Run Model 3.6 Implementation of the NearestWS Rule This section intends to demonstrate a new heuristic also developed in the dissertation, called NearestWS Rule, whose main objective is to minimize the distances covered by the AGV. In terms of the decision principle, what distinguishes this heuristic from the Milk-Run algorithm (Chapter 3.5) is the fact that the AGV moves to the nearest WS with parts waiting to be transported, not following any sort of route. As Figure 3.11 displays, if the current station is the WS4 and there are only parts already processed waiting for transport in the output buffers of WS2 and WS3, since the closest station to WS4 is WS2, the AGV will load the part present in the WS2 OB. 36 Problem and Methodology Figure 3.10: Illustrative example of the NearestWS Rule One of the differentiating features of this new transport system is the fact that there is an external program that controls the simulation environment, receiving information from it in the form of observations, and sending actions that indicate the OB of the station where the load will be carried out. This approach is completely different from those previously studied, like the FIFO and optimized Milk-Run systems, since Flexsim will not be responsible for making the decisions, but an external program, called Server. This entity is programmed using the Python language and it is represented in Figure 3.11. Figure 3.11: Server-Flexsim relationships The communications between the Server and Flexsim environment, which will be a client, have to be successful. In this sense, the Transmission Control Protocol (TCP) was chosen to accomplish these communications requirements. For this, it is crucial to create Client-Server sockets, that will be discussed in the next Section 3.6.1. 3.6.1 Client-Server TCP Sockets Sockets are an important tool used to send messages over a network. This network can be logical, local to the computer or even physically connected to an external network. 3.6 Implementation of the NearestWS Rule 37 The TCP protocol, in addition to being very reliable, allows that the data sent by the client can be received and interpreted according to the order of sending by the Server. However, regarding the UDP protocol (User Datagram Protocol), it does not guarantee that the data will be received by the Server in the same sequence in which it was sent by the client. [32] Within the scope of this dissertation, the communication between the Server and Flexsim entities requires the creation of sockets to exchange messages, respecting the TCP communication protocol. In this section, Figure 3.12 illustrates the global steps that constitute the creation and development of a Client-Server communication. The Server is considered a passive Agent because it waits for the connection request from the client. On the other hand, the client is characterized for taking the initiative and making a communication request, so it is an active entity in this whole process. [33] Briefly, in a first instance, the Server is responsible for creating the respective socket, and then, an address and a port are immediately associated with it. Subsequently, the Server waits for the moment the client requests the connection. After the connection is successful, it is possible to send and receive messages between both entities, and at the end of this whole process of sending and receiving data, the connections are closed. Figure 3.12: Client-Server TCP/IP communication diagram 3.6.2 Server – External Program From an observation sent by the simulation environment that indicates the current state of each WS in terms of the presence of parts in their respective OB, as well as the current station where the AGV is placed, the Server will then send an action with the respective WS where the AGV will realize the load. To store the current data of the simulation model in the Server, the architecture represented in the following UML diagram in Figure 3.13 was created, where three classes are defined: 38 Problem and Methodology •System_Data - contains the dictionaries referring to the distances between WSs and the own WSs and allows an association with the AGV class, in order to know the current station where the AGV is located; •Workstation - allows to know if a specific WS has parts in its OB; •AGV - contains the AGV current location; Figure 3.13: UML class diagram – NearestWS Rule The observation sent by Flexsim is in the following form: 8 booleans + current WS, "10010000WS1", in which the 8 booleans allow to identify if each of the 7 WSs and Source have parts in their respective OB. Finally, the string allows to identify the current WS. For the specific example mentioned ("10010000WS1"), the conclusions drawn are as follows: Table 3.7: Interpretation of the observation sent by Flexsim Workstation Does it have parts on OB? Current WS WS1 Yes X WS2 No - WS3 No - WS4 Yes - WS5 No - WS6 No - WS7 No - Source No - It is important to note that the current station is updated whenever an unload is performed on one of the WSs, including Sink and excluding Source. In this way, Source will never assume the role of the current WS, as it is only used to perform loads. However, it is essential to know if there are parts to transport in it. Returning to the diagram of the classes that constitute the Server, after extracting the information from the observation, it is necessary to fill all the attributes of the three classes, namely the "boolean_has_part_OB" attribute of the "Workstation" objects, as well as the current WS value, referring to the "AGV" object. Regarding the distance values, they have previously been read from an Excel file and inserted in the distance dictionary of the "System_Data" class. 3.6 Implementation of the NearestWS Rule 39 After the treatment and organization of all this information, it is necessary to proceed with the implementation of the decision rule already mentioned (Figure 3.11), which allows the AGV to carry out the load in the nearest WS that contains parts in its OB. Finally, this information that reflects the output buffer of the chosen station is sent back to the simulation environment, in the form of "OBWS" (for example "OBWS2"). 3.6.3 Flexsim – Simulation Environment 3.6.3.1 Transport of Parts After sending the observation to the Server, Flexsim waits for the response with the action containing the OBWS from the Server. However, if the Server detects that there are no parts in any of the OBs, it sends a message that orders the simulation to go to a wait state ("Wait for parts on OB"), as shown in Figure 3.14. If there are parts in at least one of the stations, the Server sends the action with the OBWS and the Flexsim accesses the oldest object present in that OB. Then, the next WS where the part will be unloaded is found. Finally, in case that the number of parts that enter at Sink reaches the value 2080, which corresponds to the total number of released orders, the simulation ends and its state is sent to the "Sink" block. Figure 3.14: Representation of the “Transport of Parts” Block – NearestWS Rule 46 Implementation of a Dynamic Transport System, using Reinforcement Learning Algorithms Figure 4.6: Neuron general structure, adapted from [7] The main idea of the PPO algorithm is to make sure that the new policy is not too far from the previous one. The version used constitutes an implementation of Stable Baselines and in addition to using vectorized environments, it also allows that the observation and the resulting action can be of a discrete, box, mutidiscrete or multibinary type. In this specific scope, the observation assumes the box type (16,) and the action admits the discrete type (0-7). The next Figure 4.7 represents the network architecture of this dissertation. Figure 4.7: SLP network architecture Within the observation, the first 8 booleans [0-7] indicate the WSs that have or have not parts in their respective OB (Table 4.2). The last 8 booleans [8-15] state the current AGV location, corresponding to a "one-hot encoding" (Table 4.3). Accordingly to the NearestWS Rule (Section 3.6), the stations state is updated just after an unload was performed. But, whenever an action corresponds to a WS that does not contain parts, the AGV goes there in the same way and the current WS is also updated. As an example, if the observation is: [1001000010000000], it means that only WS1 and WS4 have parts in their OB and the current WS is WS1. 4.4 Approach I – Initialization of Weights 47 Table 4.2: Observation segmentation [0-7] Observation [0-7] Existence of parts 10000000 WS1 01000000 WS2 00100000 WS3 00010000 WS4 00001000 WS5 00000100 WS6 00000010 WS7 00000001 Source 00000000 There are no parts Table 4.3: Observation segmentation [8-15] Observation [8-15] Actual WS 10000000 WS1 01000000 WS2 00100000 WS3 00010000 WS4 00001000 WS5 00000100 WS6 00000010 WS7 00000001 Sink 00000000 Source 4.4 Approach I – Initialization of Weights Firstly, it was decided to define a neural network architecture and a set of weights that would allow the replication of the NearestWS Rule. This phase aimed to ensure that an SLP network, with the right set of weights, would be sufficient to obtain a performance equivalent to the results of the NearestWS Rule. For demonstrative purposes, in Figure 4.8 it is possible to observe the behaviour of this practice, taking into account the observation content and the resulting action. Since it is just an example, there were only considered 3 WSs, whose distances between them are also illustrated in Figure 4.8. In order to simplify the visualization, only the weights for the output neurons 1 and 2, corresponding to OBWS1 and OBWS2, are identified in the respective figures. As the Figure 4.8 and Equation 4.3 show, the output neuron that will be activated is the first one, since it obtains the highest output value. As it makes sense, if there are parts in the WS1 and WS3 output buffers, and the current station is the WS1, then the AGV will load the part on the OB of the WS1. Figure 4.8: Neural Network – Example 1 48 Implementation of a Dynamic Transport System, using Reinforcement Learning Algorithms Dmax =3 (4.1) Bias =−Dmax−2=−3−2=−5 (4.2) OB_WS1= (1∗5+0∗0+1∗0)+(1∗3+0∗2+0∗1)+(1∗−5) = 3 (4.3) OB_WS2= (1∗0+0∗0+1∗0)+(1∗2+0∗3+0∗0)+(1∗−5) = −3 (4.4) OB_WS3= (1∗0+0∗0+1∗5)+(1∗1+0∗0+0∗3)+(1∗−5) = 1 (4.5) Like the Figure 4.9 denotes, the only case in which this network would not have the same behavior as the rule would be if the current WS was the Source, where it would be a draw between all stations with a part on their OBs (in this case: WS1 and Source). Figure 4.9: Neural Network – Example 2 Dmax =3 (4.6) Bias =−Dmax−2=−3−2=−5 (4.7) OB_WS1= (1∗5+0∗0+1∗0)+(0∗3+0∗2+0∗1)+(1∗−5) = 0 (4.8) OB_WS2= (1∗0+0∗0+1∗0)+(0∗2+0∗3+0∗0)+(1∗−5) = −5 (4.9) 4.5 Approach II – Neural Network Pre-Training 49 OB_Source = (1∗0+0∗0+1∗5)+(0∗1+0∗0+0∗3)+(1∗−5) = 0 (4.10) The StableBaselines framework, which will be used for Reinforcement Learning algorithms, does not allow the definition of initial network weights. Thus, it was not possible to create a policy with the calculated weight values. Still, this was an important exercise to ensure that this network architecture is capable of achieving performance at least equivalent to the previously defined rule. An alternative to initializing the weights is using a pre-training phase of the network, which is covered in section 4.5. Thus, appears the necessity of studying the influence of the pre-training algorithms to observe its interference in the current problem. 4.5 Approach II – Neural Network Pre-Training This approach allows, through the implementation of an algorithm with results already visible, in this particular case the NearestWS Rule, to have as starting point initial solutions that come from that rule. In this phase, the behaviour that the Agent must have to perform a step in order to replicate the NearestWS Rule is defined. Then, several simulations are run using this Agent to obtain a data set with observation-action pairs. Finally, these data are used to train the SLP neural network according to Supervised Learning algorithms. This process ensures that the neural network starts its learning process with a set of weights that replicate the functioning of the rule. For this, in order to verify the influence of this pre-training in the final solution, two models with different objectives will be approached, namely the minimization of makespan and the maximization of productivity, taking into account the PPO algorithm ("Makespan" and "Base" Models). 4.6 Makespan Model In order to verify the correct behaviour of the Reinforcement Learning algorithm, PPO, the initial objective was to minimize the makespan. Thus, only one release order was defined for a random entity, more specifically a part type of number 4. The main purpose of this model is to verify the correct behaviour of this transport system, observing the tendency of the final solutions to get the optimal solution. Associated with the Agent’s learning process, the total number of time steps defined was 250.000 (steps), so that it is believed to be more than enough to achieve the optimal solution. As a comparable point to the solutions achieved in the optimization process, the optimal solution can be obtained through the NearestWS and optimized Milk-Run rules, because there is just one part in the entire simulation process. 50 Implementation of a Dynamic Transport System, using Reinforcement Learning Algorithms In this case, the pre-training of the network does not apply to this model since the subjacent rule would lead to an ideal behaviour already during the initial phase of the learning process, which is not intended. That said, since the objective is to analyse the behaviour of the optimization curve of the RL algorithm, the idea is that the final makespan values for each simulation performed approach the optimal solution. In this specific case, the ideal solution for a part type 4 is: 1486 seconds. Figure 4.10 shows the Process Flow for the transport process implemented in the Flexsim simulation environment. The “Decide” block defines the next WS where the AGV has to go, taking into account the information sent by the Agent over the various steps it sends. If there are no parts already processed waiting to be transported to the next WS, the AGV goes to a state where it is waiting for the part in question to be processed (given that in this particular model there will only be one part throughout the entire simulation). In the case that the Agent sends a step referring to a WS that does not contain parts in his OB, since the network is in the learning process, the AGV will also go there and the current WS will be updated. As soon as the part enters at Sink, the Flexsim will send the value corresponding to the makespan as a reward to the Server, but with the negative signal (-makespan). This particularity is justified by the fact that the purpose of the PPO algorithm is always to maximize something. In this case, the goal is to maximize (-) makespan, in other words, minimize the makespan. Still in this scenario, the third parameter, called "done", is also sent as a unit, since the part has already been entered into the Sink and the simulation is finished. Figure 4.10: Representation of the “Transport of Parts” Block – Makespan Model After running several simulations, it was found that the model, under the current conditions in which the makespan is only sent at the end of the simulation when the part enters at Sink, does not allow reaching the optimal solution. It was concluded, therefore, that the fact that the rewards always assume null values, with the exception of the response to the last step that includes the makespan, is not enough for the network to be able to make an adequate learning and find the ideal solution within the defined 250k time steps. 4.6 Makespan Model 51 Therefore, some criteria were introduced in order to check whether it is possible to achieve the optimal solution, in particular: •the sending of rewards between time steps with the value of a time interval initiated by a load and ended with an unload (including the processing time), in detriment of sending the total makespan only at the end of the simulation; •the introduction of penalties, if the movements made do not involve the transportation of parts; •the insertion of normalizations, modifying the value of the rewards, converting them in the interval [-1,0]. Later, in order to study the influence of these changes on learning performance, all of these scenarios presented in Table 4.4 were implemented and simulated. Table 4.4: Scenarios of the Makespan Model Scenario Time steps Normalization Normalization Factor Type of Reward Penalty 1 250k No 0 makespan 0 2 250k No 0 makespan -FN/2 3 250k Yes 15k makespan 0 4 250k Yes 15k makespan -0.5 5 250k No 0 between time steps 0 6 250k No 0 between time steps -FN/2 7 250k Yes FN between time steps 0 8 250k Yes FN between time steps -0.5 FN = 4*Travel time between the two most distant stations + 2*Higher Processing Time (Part type no4). For all the cases presented above in Table 4.4, the model converts the solutions obtained to the optimal solution, except for the scenario in which the rewards are sent between time steps without normalization or penalty. The question that immediately arises is related to the time that is necessary to reach the optimal solution and, in this context, it was found that normalization is the factor that allows the model to tend more quickly towards the optimal solution. These results are presented and discussed with more detail in Chapter 5. In the following subsections are presented the steps used in the normalization and penalty methods for the types of rewards: between time steps and makespan. 4.6.1 Normalization The main objective of normalization is to verify whether, by adjusting the values of the rewards to values between [-1, 0], the algorithm will be able to converge better to a good solution and increase its learning capacity. 52 Implementation of a Dynamic Transport System, using Reinforcement Learning Algorithms For this, the normalization factor depends on the type of reward (between time steps or makespan) and application of penalties, characterized by the following Equations 4.11 and 4.12: 4.6.1.1 Type of Reward: Makespan Normalization f actor =2∗7.5k=15k(seconds)(4.11) Reward ∈[−0.5,0],without penalty (4.12) This value attributed to the normalization factor is justified by the fact that it was observed during all the simulations previously carried out that the final value of the makespan, for a single part of type 4, never exceeded the value of 7.5k seconds, with rare exceptions. Therefore, taking into account the subsequent application of penalties (-0.5), the value considered for the normalization factor was 15k seconds, as it is twice 7.5k. Therefore, without applying the penalty, the reward assumes values between -0.5 and 0, since the purpose of optimization is to maximize (-) makespan. 4.6.1.2 Type of Reward: Between Time steps Normalization f actor =4∗T1+2∗T2 (4.13) in which T1=Travel time between the two most distant stations; T2=Higher Processing Time (Part of type 4). Reward ∈[−0.5,0],without penalty (4.14) As in the previous case regarding makespan, the normalization factor applied allows rewards to assume values between -0.5 and 0, without the application of penalties. In the specific case of a part type 4: •T1 = 48.827 seconds; •T2 = 420 seconds. Normalization f actor =1035.308 (4.15) 4.6.2 Penalties Penalties are applied when the AGV performs movements without a part. The objective is, therefore, through the application of this penalty with the value of -0.5, to make it clear to the Agent that, in this specific case, it is not a good practice to make movements without a part, given that it 4.7 Base Model 53 is only one part in the simulation environment. That said, taking into account the type of reward, their values will be included in the following range: Reward ∈[−1,0](4.16) 4.7 Base Model After analysing the Agent’s behaviour towards the model whose objective was to minimize the makespan, it was also decided to analyse the issue of maximizing productivity and create a new model that allows studying the PPO algorithm. With regard to the Process Flow of the simulation model, all the blocks are coincident with the blocks of the previous model, referring to the makespan. The only relevant difference is the issue of the rewards that are sent from Flexsim to the Agent, intermediated by the Server. In response to a step from the Agent, the simulation environment constructs the observation that contains the current status of each WS, in terms of the presence of parts, and the current position of the AGV. However, the rewards assume a unit value whenever a part is entered at Sink. Otherwise, they are assigned a null value. In relation to the parameter “done”, it will be unitary whenever the defined time horizon is reached, ending the simulation. Moreover, whenever a reward assumes a value of 1, there is an internal counter in the simulation environment that is increased, providing an updated indication of the number of parts manufactured so far, to a later comparison of results with the previous productivity models. In this context, unlike what happens with the makespan model, the use of pre-training becomes important in order to verify whether it has had an effect or not. The rule implemented and associated with the pre-training is based on one of the rules previously described in this dissertation, better known as the NearestWS Rule, which allows AGV to perform a load on the nearest WS that contains parts in the respective OB. A not less important issue is to understand the effects of the introduction of penalties on productivity. For this, given that the rewards are already normalized in the interval [0, 1], more specifically assuming the values 0 or 1, the defined penalty was (-) 0.5, similar to the previous Makespan Model. That said, whenever the AGV makes an empty movement, ordered by the Agent, the reward returned will be -0.5. Regarding the number of steps sent by the Agent, after several runs, it was concluded that the value of 10M steps would be the most indicated value and more than enough to reach the final solution. For further comparison of results, the time horizons chosen were 36 and 52 hours. Finally, it should be noted that, whenever a "done" value is assigned to 1, the productivity value related to the simulation in question is also written in a text file, for later analysis. 54 Implementation of a Dynamic Transport System, using Reinforcement Learning Algorithms 4.8 Robustness of the Simulation Models This section was created with the intention of verifying whether the models covered in the previous sections, namely the FIFO, optimized Milk-Run, NearestWS Rule and Base Model, referring to productivity, have an appropriate behaviour against possible changes in the production mix or in the processing times. 4.8.1 Production Mixes With regard to changes in production mixes, the order in which the different part types are launched will suffer some variations and it will be randomly generated, using the Microsoft Excel software tool. Just like what happens in the original mix, in which the probability of each part type being released is nearly 20% (Table 4.5), the same restriction applies to the next 5 different mixes generated. Briefly, the different production mixes are generated so that the part type associated with each production order is determined randomly, but with the same final probability of being created (20%). Table 4.5: Quantity of the different part types – Original Mix Part Type Quantity (units) 1 448 2 410 3 429 4 424 5 369 Therefore, as shown in Chapter 5, there are small differences in the final quantities of parts produced for each one of the mixes, due to the way they were created. 4.8.2 Processing Times The processing times corresponding to each part type and WS were introduced in a stochastic environment, in order to follow, more concretely, a triangular probability distribution. Assuming that Ti j represents the processing time of the part type iin the WS j, it was adapted to respect the triangular distribution, shown in Figure 4.11. In this segment, a variability factor α was also added to simulate possible variations that may occur in a real context. 4.8 Robustness of the Simulation Models 55 Figure 4.11: Triangular distribution function, referring to the processing times In a rigorous way, the respective triangular distribution is defined by the following expressions: f(x|a,b,c) =                        2(x−a) (b−a)(c−a),se a≤x<c 2 (b−a),se x=c 2(b−x) (b−a)(b−c),se c<x≤b 0,otherwise (4.17) Where a, b and c take the next values:        a=Ti j(1-α) b=Ti j(1+α) c=Ti j        (4.18) The results of this procedure, which is based on the introduction of the processing times in a stochastic environment, are described in Chapter 5. 62 Results Analysis Figure 5.5: Makespan related to Scenario 5, for a total of 250k time steps Table 5.8: Numerical interpretation of makespan referring to Scenario 5 (in seconds) Minimum Value Maximum Value Final Value 1630 17202 6645 5.4.6 Scenario 6 – Type of reward: between time steps, with penalties and without normalization Analysing the scenarios 2 and 6, both using penalties, it appears that the two have similar behaviours, with a greater number of simulations carried out in the second scenario. These questions can be observed in Tables 5.5 (scenario 2) and 5.9 (scenario 6). Figure 5.6: Makespan related to Scenario 6, for a total of 250k time steps Table 5.9: Numerical interpretation of makespan referring to Scenario 6 (in seconds) Minimum Value Maximum Value Final Value 1486 3945 1486 5.4 Makespan Model 63 5.4.7 Scenario 7 – Type of reward: between time steps, without penalties and with normalization As in the previous scenario, the optimal value is also achieved (Figure 5.7). Nevertheless, as evidenced in scenarios 2 and 3, the use of the normalization in a singular way allows obtaining the best solution earlier when compared to the isolated application of penalties. Figure 5.7: Makespan related to Scenario 7, for a total of 250k time steps Table 5.10: Numerical interpretation of makespan referring to Scenario 7 (in seconds) Minimum Value Maximum Value Final Value 1486 4096 1486 5.4.8 Scenario 8 – Type of reward: between time steps, with penalties and with normalization As well as the fourth scenario, and as expected, when the normalizations and penalties are applied simultaneously, the optimal solution is obtained in advance in relation to the models in which the penalties or normalization are applied separately. Figure 5.8: Makespan related to Scenario 8, for a total of 250k time steps 64 Results Analysis Table 5.11: Numerical interpretation of makespan referring to Scenario 8 (in seconds) Minimum Value Maximum Value Final Value 1486 4116 1486 5.4.9 Synthesis In short, regarding the models presented in Section 5.4 related to makespan, it is possible to state that the Reinforcement Learning PPO algorithm was correctly implemented, given that the models follow their characteristic curve, allowing to maximize (-) makespan. The results also suggest that the normalization of rewards and the application of penalties improve the learning capacity of the model. 5.5 Base Model Returning to the question of the productivity and taking into account the PPO algorithm, the results achieved for the time horizons of 36 and 52 hours were divided according to the existence or not of a pre-training of the neural network and are represented and indicated, respectively, in Figures 5.9,5.10,5.11,5.12 and Tables 5.12,5.13,5.14,5.15. With respect to the Agent’s learning period, as referred to in Section 4.7, the number of time steps was set to 10M. Regarding the pre-training, it respects the NearestWS Rule, discussed in Section 3.6. For a better understanding of the influence of the pre-training in the learning phase, some sets of observation-action pairs resulting from this same stage will also be presented. For this purpose, a probability function will be applied to the actions, after the learning phase is performed. The objective of this step is to verify in which WS the AGV will carry out the respective load. Subsequently, similarly to the previous Section (5.4) referring to the makespan, the results derived from the application of a penalty factor (-0.5) are also presented. 5.5.1 Without pre-training, for a time horizon of 36 hours Figure 5.9 allows us to infer that the model’s behaviour, with regard to productivity values, follows the PPO optimization algorithm. As can be seen, the model reaches the maximum productivity values in the first simulations. This phenomenon is explained by the weights which are arbitrarily assigned in the initial learning phase of the network. Subsequently, after a certain number of simulations, the model presents lower productivity values, also due to the weights defined by the Agent and because at the beginning of the information collection, the network has little knowledge yet. However, the productivity characteristic curve allows verifying that the productivity values tend to a final maximum solution, which approves that the learning phase was performed correctly. The final value of productivity, which comes from the network’s learning process, is 343 parts that enter at Sink. 5.5 Base Model 65 Figure 5.9: Productivity referring to the Base Model without pre-training, to a total of 10M time steps and a time horizon of 36h Table 5.12: Numerical interpretations of the productivity (in parts) referring to the Base Model without pre-training, to a total of 10M time steps and a time horizon of 36h Minimum Value Maximum Value Final Value 250 347 343 The set of observation-action pairs resulting from the learning phase, without the pre-training, are exemplified below: •Observation-Action pair no1: Parts in WS1 | Current WS: 4 Observation: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [8.3619851e-01, 1.9839258e-04, 7.7551082e-02, 2.0058152e-04, 1.7890350e-04, 6.3502125e-02, 1.9391190e-02, 2.7792549e-03] In this particular situation, the behaviour of the network is as expected, given that the highest probability is associated with the WS1. •Observation-Action pair no2: Parts in WS3, WS4 | Current WS: 4 Observation: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [1.4122830e-03, 1.3475084e-05, 9.6138531e-01, 2.4895007e-03, 7.2637980e-05, 2.9261060e-02, 3.8612273e-03, 1.5044548e-03] With regard to this observation-action pair, the network’s behaviour does not correspond to what would be done by the NearestWS Rule, given that, in this situation, the AGV would load a part in the WS3 and not in the WS4, as the high probability suggests. 5.5.2 With pre-training, for a time horizon of 36 hours Like the model without pre-training, this one also respects the curve referring to the PPO optimization algorithm, reproduced in Figure 5.10. 66 Results Analysis However, in comparison with the model without pre-training, it appears that the results of this model before the learning stage are lower, in terms of productivity. Despite that, similarly to what happened in the optimized Milk-Run and NearestWS Rules, this episode can be explained by the fact that, sometimes, the empty travels or the transport of parts for distant WSs, more common in the model without pre-training, can bring advantages in terms of the final result of productivity, reducing possible bottlenecks and balancing the machine occupancy rates. Figure 5.10: Productivity referring to the Base Model with pre-training, to a total of 10M time steps and a time horizon of 36h Table 5.13: Numerical interpretations of the productivity (in parts) referring to the Base Model with pre-training, to a total of 10M time steps and a time horizon of 36h Minimum Value Maximum Value Final Value 257 277 274 One more time, the examples of the observation-action pairs, resulting from the learning process, evidence the previous accomplishment of the learning phase: •Observation-Action pair no1: Parts in WS1 | Current WS: 4 Observation: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [9.9999833e-01, 6.0297623e-07, 9.3485424e-22, 1.0542382e-06, 3.6167533e-10, 4.3565872e-16, 4.4415426e-18, 2.8845209e-15] The behaviour of the network is the one expected, since the WS1 has the highest probability. •Observation-Action pair no2: Parts in WS3, WS4 | Current WS: 4 Observation: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [9.00684897e-25, 1.09724185e-16, 1.59646764e-13, 1.00000000e+00, 1.18856031e19, 6.28844218e-25, 2.55962060e-27, 2.96594020e-27] In contrast with what happened in the previous scenario in which the pre-training did not occur, the higher probability is now consistent with the expected (WS4), according to the NearestWS Rule. 5.5 Base Model 67 5.5.3 Without pre-training, for a time horizon of 52 hours According to the graph in Figure 5.11, the behaviour of the learning phase, like the Model without pre-training for the 36-hour time horizon, follows the PPO algorithm, despite the initial drop of the productivity values. Regarding the final value of productivity, this is logically higher than the productivity referring to the period of 36 hours. Figure 5.11: Productivity referring to the Base Model without pre-training, to a total of 10M time steps and a time horizon of 52h Table 5.14: Numerical interpretations of the productivity (in parts) referring to the Base Model without pre-training, to a total of 10M time steps and a time horizon of 52h Minimum Value Maximum Value Final Value 398 507 501 For the same observation-action pairs, the solutions are the following ones: •Observation-Action pair no1: Parts in WS1 | Current WS: 4 observation: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [8.6369443e-01, 2.2193830e-04, 4.7343463e-02, 6.8649366e-05, 1.9357287e-04, 7.4317701e-02, 1.2128289e-02, 2.0318190e-03] For this case, the behaviour of the network is the one expected (WS1). •Observation-Action pair no2: Parts in na WS3, WS4 | Current WS: 4 observation: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [2.6672529e-03, 7.9401660e-05, 9.7478318e-01, 2.3247360e-03, 1.2263234e-04, 1.7269300e-02, 2.2482565e-03, 5.0515484e-04] In accordance with the results of the last model (36h), without pre-training, it is concluded that the learning of the network leads the AGV to do the load in the further station (WS3) instead of the nearest one (WS4). 68 Results Analysis 5.5.4 With pre-training, for a time horizon of 52 hours In this last scenario, Figure 5.12 proves the expected behaviour of the model. As Table 5.15 indicates, the final value of productivity is lower than the value obtained in the scenario in which the pre-training did not occur. The most admissible reasons are presented in Section 5.6 that compares the results of all models. Figure 5.12: Productivity referring to the Base Model with pre-training, to a total of 10M time steps and a time horizon of 52h Table 5.15: Numerical interpretations of the productivity (in parts) referring to the Base Model with pre-training, to a total of 10M time steps and a time horizon of 52h Minimum Value Maximum Value Final Value 375 409 404 The following observation-action pairs follow the conjunctures performed previously, similarly with the results obtained in the model with pre-training, referring to the time horizon of 36 hours: •Observation-Action pair no1: Parts in WS1 | Current WS: 4 Observation: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [9.9983370e-01, 1.7993931e-06, 4.0928912e-19, 1.6300578e-04, 1.4105979e-06, 4.8301262e-15, 1.4212710e-17, 2.4972540e-14] •Observation-Action pair no2: Parts in WS3, WS4 | Current WS: 4 Observation: [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0] Action: [8.7494940e-28, 5.5310151e-18, 3.6004825e-13, 1.0000000e+00, 3.2771040e-19, 4.0131407e-26, 4.2842155e-29, 7.9547190e-29] 5.6 Comparison of the FIFO, Optimized Milk-Run, NearestWS Rule and Base Models 69 5.5.5 Application of Penalties After the penalties of -0.5 were applied whenever an empty movement is carried out, it was concluded that the four models follow the expected behaviour in relation to the PPO algorithm, as its curve presents. However, as Table 5.16 expresses, the results are inferior. Since performing empty movements is not necessarily a bad practice, this is one of the reasons for which the productivity values are lower using penalties. Table 5.16: Productivity (in parts) referring to the Base Model, with penalties Base Model Productivity (36h) Productivity (52h) Without pre-training 343 501 With pre-training 274 404 Without pre-training, with penalties 304 454 With pre-training, with penalties 271 403 5.6 Comparison of the FIFO, Optimized Milk-Run, NearestWS Rule and Base Models Initially, comparing all the models previously studied in Sections 5.1,5.2,5.3 and 5.5, with regard to productivity, the Base Model was the one that presented the best results. That said, the use of Machine Learning techniques combined with simulation approaches has enabled us to achieve good solutions. Regarding the Base Model, the difference between the usage or not of pre-training has to do with the fact that, at the beginning of the learning process, the resulting solutions are already or not close to a local optimum. If pre-training occurs, the solutions derived from the learning process will be in close to a local optimum, unlike what happens in the situation where there is no pre-training. Since the solutions start near a local optimum, the Agent has difficulties to explore and find new better solutions than the ones settled outside this region of the solution space. In this sense, it was found that the final productivity results obtained were higher in the case where the pre-training was not carried out (Table 5.17). This is explained by the fact that, in the absence of pre-training, the occurrence of transports to distant stations to the detriment of closer stations allows the reduction of bottlenecks and the balancing of the machine occupancy rates. The fact that there is no pre-training also allows the Agent not to be so easily stagnated in a local optimum given by the pre-training, so there is more freedom to explore new solutions, which will allow the Agent to achieve better solutions, contrarily to what happens in the model where there is pre-training. 70 Results Analysis Table 5.17: Productivity (in parts) referring to the models in study Model Productivity (36h) Productivity (52h) FIFO 177 262 Optimized Milk-Run 266 389 NearestWS Rule 258 377 Base, with pre-training 274 404 Base, without pre-training 343 501 5.7 Robustness of the Simulation Models After analysing the models when applied to deterministic environments, it is pertinent to verify their behaviour when applied to other situations. In particular, it is important to study the influence of different production mixes and processing times, in order to examine the adaptability of the models, but only regarding productivity, which therefore excludes the model related to the study of makespan. It is also important to mention that, according to the Base Model, there was no customized re-learning for these scenarios, because the weights obtained in the base scenario (original mix) were used. 5.7.1 Production Mixes Five different mixes were defined, randomly generated, so that, as described in Section 4.8.1, the number of orders for each part type is balanced (≃20%). The time horizons under study are 36 and 52 hours for comparison principles, and the AGV speed also remains constant at 1.5 m/s, since speed does not constitute a factor under consideration. Tables 5.18 and 5.19 enable us to conclude that, for both temporal perspectives, the Agent when applied to different production mixes ensures that the productivity results remain, on average, close to each other. So, it is possible to validate the Base Model and its adaptability in different production contexts, just like the other models. It should also be noted that the Base Model, without pre-training, remains the one that obtains the best results. Table 5.18: Productivity (in parts) analysis referring to different production mixes, for a time horizon of 36 hours Model Initial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5 FIFO 177 196 179 171 178 174 Optimized Milk-Run 266 261 251 257 261 242 NearestWS Rule 258 252 239 250 252 235 Base, with pre-training 274 273 259 266 278 252 Base, without pre-training 343 341 328 314 325 329 5.7 Robustness of the Simulation Models 71 Table 5.19: Productivity (in parts) analysis referring to different production mixes, for a time horizon of 52 hours Model Initial Mix Mix 1 Mix 2 Mix 3 Mix 4 Mix 5 FIFO 262 271 254 256 268 253 Optimized Milk-Run 389 385 359 379 386 370 NearestWS Rule 377 366 344 362 371 355 Base, with pre-training 404 409 376 388 394 382 Base, without pre-training 501 473 471 473 477 478 5.7.2 Processing Times The second criterion to evaluate the adaptability of these models related to productivity includes the introduction of them in a stochastic environment in which, according to Section 4.8.2, the definition of processing times follows a triangular distribution. The variability factor, α, applied to the processing times of each part was 0.4. It is also important for comparison purposes to maintain the time horizons in 36 and 52 hours and the AGV speed in 1.5 m/s. The following Tables 5.20 and 5.21 provides the results of the 10 simulations generated and individually applied to each one of the implemented models. These results are important to analyse the average and standard deviations of the resulting productivity values. The low standard deviation as well as the similarity between the average values of productivity obtained for each one of the models and the values associated with the original mix, also indicate that all the values are in agreement with each other. So, similarly to what happened with the production mixes, we can conclude that all models present good adaptability when subject to stochastic environments. To conclude, the Base Model, without pre-training, is the one that allows the achievement of the higher productivity values. 78 REFERENCES [14] R. Drießel and L. Mönch, “An integrated scheduling and material-handling approach for complex job shops: a computational study,” International Journal of Production Research, vol. 50, no. 20, pp. 5966–5985, 2012. [15] R. M. Logística, “Milk run: Conceitos, vantagens e ganhos para a operação logística.” https://revistamundologistica.com.br/blog/achiles/ milk-run-conceitos-vantagens-e-ganhos-para-a-operacao-logistica. Accessed in 2019-11-23. [16] R. F. D. Santos, “Estratégias híbridas de machine learning e simulação para a resolução de problemas de escalonamento,” 2018. [17] S. Huang, “Optimization of job shop scheduling with material handling by automated guided vehicle,” 2018. [18] A. de Freitas Ribeiro, “Taylorismo, fordismo e toyotismo,” Lutas Sociais, vol. 19, no. 35, pp. 65–79, 2015. [19] A. Azevedo, “Lean thinking: princípios, fundamentos, principais ferramentas.” Azevedo, Américo, Lean thinking: princípios, fundamentos, principais ferramentas. Sistema de Produção Toyota Parte I, 2013. 28 slides. Support material for the GOPE course of MIEEC of FEUP. Accessed in 2020-01-23. [20] G. Zäpfel, R. Braune, and M. Bögl, Metaheuristic Search Concepts: A Tutorial with Applications to Production and Logistics. Springer Berlin Heidelberg, 2010. [21] A. Meyer, “Milk run design (definitions, concepts and solution approaches). dissertation, karlsruher institut für technologie (kit) fakultät für maschinenbau, kit scientific publishing, karlsruher (2015),” 2015. [22] . S. G. Brar, G. S., “Milk run logistics: Literature review and directions, london, england,” 2011. [23] A. Géron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media, 2019. [24] Y. Li, “Deep reinforcement learning: An overview,” arXiv preprint arXiv:1701.07274, 2017. [25] S. Marsland, Machine learning: an algorithmic perspective. Chapman and Hall/CRC, 2014. [26] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013. [27] V. H. de Sousa Carneiro, “Abordagens híbridas de" machine learning"/simulação para sistemas logísticos dinâmicos,” 2019. [28] I. Grigoryev, Anylogic in three days. Fifth Edit.C, 2018. [29] J. Banks, J. S. Carson II, L. Barry, et al., “Discrete-event system simulationfourth edition,” 2005. [30] A. K. d. Silva, Método para avaliação e seleção de softwares de simulação de eventos discretos aplicados à análise de sistemas logísticos. PhD thesis, Universidade de São Paulo, 2006. REFERENCES 79 [31] V. H. S. Carneiro, Hybrid Machine Learning/Simulation Approaches for Decentralized Production Scheduling. 2018. [32] “Socket programming in python (guide) by nathan jennings.” Accessed in 2020-03-02. [33] M. Almeida, Luís; Santos Pedro; Sousa, “Support information for lab assignment 1a tcp sockets in c.” Support information for lab assignment 1a TCP sockets in C, Luís Almeida, Pedro Santos, Mario Sousa, 18 slides. Support material for the ACI course of MIEEC of FEUP. Accessed in 2020-04-15. [34] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines.” https://github.com/hill-a/stable-baselines, 2018.