A theoretical demonstration for reinforcement learning of PI control dynamics for optimal speed control of DC motors by using Twin Delay Deep Deterministic Policy Gradient Algorithm

A Theoretical Demonstration for Reinforcement Learning of PI Control Dynamics for Optimal Speed Control of DC Motors by using Twin Delay Deep Deterministic Policy Gradient Algorithm TUFENKCI, S.; ALAGOZ, B. B.; KAVURAN, G.; YEROGLU, C.; HERENCSÁR, N.; MAHATA, S. Applied Materials Today Volume 213, Part C, March 2023, 119192, Pages 1-16 ISSN: 0957-4174 DOI: https://doi.org/10.1016/j.eswa.2022.119192 Accepted manuscript © 2022. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/ dspace.vutbr.cz 1 A Theoretical Demonstration for Reinforcement Learning of PI Control Dynamics for Optimal Speed Control of DC Motors by using Twin Delay Deep Deterministic Policy Gradient Algorithm Sevilay TUFENKCI1*, Baris Baykant ALAGOZ2, Gurkan KAVURAN3, Celaleddin YEROGLU4, Norbert HERENCSAR5, Shibendu MAHATA6 1Malatya Turgut Ozal University, Department of Computer Technology, 44100, Malatya, Turkey, sevil[email protected] 2Inonu University, Department of Computer Engineering, 44100, Malatya, Turkey, b[email protected] 3Malatya Turgut Ozal University, Department of Electrical-Electronics Engineering, 44100, Malatya, Turkey, [email protected] 4Inonu University, Department of Computer Engineering, 44100, Malatya, Turkey, [email protected] 5Brno University of Technology, Faculty of Electrical Engineering and Communications, Dept. of Telecommunications, Technicka 12, 616 00 Brno, Czechia, herenc[email protected] 6Dr. B. C. Roy Engineering College, Department of Electrical Engineering, Durgapur, West Bengal 713206, India, [email protected] *Corresponding author: [email protected] 2 Abstract: To benefit from the advantages of Reinforcement Learning (RL) in industrial control applications, RL methods can be used for optimal tuning of the classical controllers based on the simulation scenarios of operating conditions. In this study, the Twin Delay Deep Deterministic (TD3) policy gradient method, which is an effective actor-critic RL strategy, is implemented to learn optimal Proportional Integral (PI) controller dynamics from a Direct Current (DC) motor speed control simulation environment. For this purpose, the PI controller dynamics are introduced to the actor-network by using the PI-based observer states from the control simulation environment. A suitable Simulink simulation environment is adapted to perform the training process of the TD3 algorithm. The actor-network learns the optimal PI controller dynamics by using the reward mechanism that implements the minimization of the optimal control objective function. A setpoint filter is used to describe the desired setpoint response, and step disturbance signals with random amplitude are incorporated in the simulation environment to improve disturbance rejection control skills with the help of experience based learning in the designed control simulation environment. When the training task is completed, the optimal PI controller coefficients are obtained from the weight coefficients of the actor-network. The performance of the optimal PI dynamics, which were learned by using the TD3 algorithm and Deep Deterministic Policy Gradient algorithm, are compared. Moreover, control performance improvement of this RL based PI controller tuning method (RL-PI) is demonstrated relative to performances of both integer and fractional order PI controllers that were tuned by using several popular metaheuristic optimization algorithms such as Genetic Algorithm, Particle Swarm Optimization, Grey Wolf Optimization and Differential Evolution. Keywords: Deep reinforcement learning, DC motor, PI controller, Twin-delayed deep deterministic policy gradient, metaheuristic optimization 1. Introduction Reinforcement Learning (RL) is an effective machine learning method that is designed for learning from experience (Kaelbling et al., 1996; Mnih et al., 2013). In recent years, it has been utilized for intelligent control of real systems ( Mnih et al., 2015; Lillicrap et al., 2016; Rabault et al., 2019). Today, many systems in daily use include DC motors to convert electrical energy to mechanical energy. Therefore, performance improvement in DC motor control contributes to many 3 innovative application areas such as electric vehicles (Wu et al., 2004), Unmanned Aerial Vehicles (UAV) (Solomon et al., 2006; Solomon, 2007), etc. DC motors have been frequently utilized in many application areas (Cui et al., 2012; Berahim, 2014). Their lower price, ease of use, flexibility, and robustness are the main reasons for preferring DC motors in applications. Today, they are used in numerous areas, such as robots, industrial machinery, home equipment, and electric vehicles . In daily life applications, optimal speed control of DC motors has importance in terms of efficiency and comfort. The main objective of DC motor speed control is to produce the desired speed within a certain reference value range in the shortest time and to reject environmental disturbances such as load alterations and changes in operating conditions. The most commonly used DC motor speed control techniques is based the Proportional Integral Derivative (PID) controller family (Sabir & Khan, 2014), which can consider the error signal, the change in the error signal, and the sum of the error signal in order to produce a control signal. The PID controller family is a widely preferred industrial control standard because of their effectiveness and their simplicity in the controller realization (Ekinci & Hekimoglu, 2019). There is a large amount of research collection for theory and practice of PID control systems in the literature (Åström & Hägglund, 1995; Visioli, 2006). However, analytical optimal tuning methods are not commonly considered environmental uncertainties and disturbance impacts. Therefore, the controller tuned analytically may not exhibit optimal design performance when they are applied to a real system. There are many approaches to address the efficiency and performance of the PID controllers in design tasks (Åström & Hägglund, 1995; Visioli, 2006; Young et al., 1999; Alagoz et al., 2015). Especially, the multi-loop model-reference control structures can improve the disturbance rejection performance of the classical PID controller loops (Butler et al., 1989; Alagoz et al., 2020); however, these methods do not guarantee an empirically validated disturbance rejection control performance. In real control applications, unpredictable system perturbations and environmental disturbances may occur. Therefore, the control system must be designed to be robust enough against parametric uncertainty, measurement noise and unpredictable disturbance in order to maintain the desired control performance in practice. A solution to this design issue may be the experience-based optimal tuning of controllers in a realistic control simulation environment that can simulate the stochastic nature of real-world systems by incorporating random system perturbation and disturbances. This simulation environment provides a useful training environment for controller tuning algorithms to reach the best practical performance against uncertainty, noise and environmental disturbances. RL methods are inherently suitable for the experience-based learning and performance improvement according to simulation environment results, and this point was a central motivation for the authors in the current study. We particularly focused on designing a 4 suitable simulation environment for effective RL based controller tuning. Then, we illustrated implementation of the TD3 algorithm in order to manage experience-based learning of robust performance PI controller dynamics for the motor control application in the designed simulation environment. Recently, Traue et al. pointed out the importance of RL environment design for intelligent motor control (Traue et al., 2022). specifically focus on control performance robustness and disturbance rejection improvement problems in the design stage of the RL training environment. To contribute to their perspectives, we addressed the problem of application specific simulation environment design for improvement of robust control performance of PI controllers. Similar to Traue et al., Book et al. implemented the Deep Deterministic Policy Gradient (DDPG) algorithm for RL control of electric motors and presented experimental results (Book et al., 2021). In another recent work, application of the deep Q-networks (DQN) algorithm was demonstrated for the speed control of permanent magnet synchronous motors and compared with performance of classical PI controllers (Song et al., 2021). However, these works employed the trained RL network as a direct RL controller instead of tuning a standard controller. Direct RL control may lead to practical realization difficulties such as the computational complexity of RL training and the stability concerns. Table 1 lists some inherent properties of the metaheuristic optimization and RL algorithms that make them advantageous soft computation tools for control applications. The metaheuristic methods can be advantageous to solve well-defined optimal control problems because of their effective global search capabilities. However, the experience based learning according to the agentenvironment interaction is a critical property to prefer the RL method for controller tuning according to well-designed simulation environments that can meet application specific control requirements. Table 1. Some useful properties of metaheuristic optimization and RL algorithms for control applications Properties Metaheuristic optimization based tuning methods RL based tuning methods Supporting population based global searching High No Supporting local searching High High Supporting off-line controller tuning High High Supporting online controller tuning Low High Suitability of algorithms for the experience based learning from agent-environment interaction Low High Suitability of algorithms to work as controller Low High The RL has also been used to implement a neural controller (Koch et al., 2019) and for tuning classical controllers (Chen et al., 2022; Esmaeili et al., 2017) in control applications. The fuzzy Qlearning method was suggested for tuning the PID controller based on RL and applied for temperature control (Chen et al., 2022). Esmaeili et al. proposed an improved RL-based fuzzy-PID 5 controller for load frequency control of an island microgrid (Esmaeili et al., 2017). The main concern may be the question whether the RL algorithms can learn from whole system dynamics and control them. A suitable implementation of the RL algorithm can manage control of system dynamics based on the training experiences. In DC motor speed control, the PI controller from the PID control family has been widely preferred (Nagarajan et al., 2016; Kanojiya et al., 2012; Sundareswaran & Vasu, 2000). A PI controller can be suitable for DC motor speed control because derivative elements of PID may cause very fast changes and chatter in the control signal and may amplify noise signals in the control loop of DC motors. Therefore, for a smoother and more straightforward control solution, PI control can be preferred for speed control of DC motors. In this study, a RL approach is implemented to learn the PI control dynamics that can provide a desired DC motor control performance within the control simulation environment. The RL method needs a realistic simulation environment to learn from agent experiences. Major practical control complications such as unpredictable external disturbance and measurement noise can be introduced in this simulation environment and contribute to learning an effective and optimal PI control dynamic from experience in the simulation environment. According to RL policies, the agent acting in this environment tries to find the most appropriate action that maximizes the reward (Kaelbling et al., 1996). Therefore, the optimal PI control dynamics that deal with the complication can be learned from the control simulation environment in order to perform the best PI action by using a suitably designed reward and observer mechanism. Historically, the RL strategy, which was founded by Richard Bellman in the 1950s, first gained success in playing a backgammon game. As a result, combining it with an artificial neural network over time achieved superior success and performed better than humans in solving complex problems (Mnih et al., 2015; Graepel, 2016). Later, Q-Learning was first to come out in 1989, which is a special type of RL approach. Watkins used the letter Q for the value function, which is based on the theory of Markov decision processes (Watkins, 1989; Watkins & Dayan, 1992). However, Q-Learning has not attracted interest in its domains until DQN algorithms were developed (Mnih et al., 2013). Afterwards, the Deterministic Policy Gradient (DPG) algorithm, which uses the actor-critic structure, was developed (Lillicrap et al., 2016; Silver et al., 2014). Finally, the TD3 Policy Gradient algorithm was proposed, and the TD3 algorithm can work more effectively than many RL methods in applications (Fujimoto et al., 2018). RL-based methods are widely used in the field of control. It can enhance intelligence in control systems (Lillicrap et al., 2016; Rabault et al., 2019; Chen et al., 2018; Brandi et al., 2020; Satheeshbabu et al., 2019). In the current study, the TD3 algorithm is implemented for learning optimal PI control dynamics from a closed-loop DC motor simulation environment. A reference model introduced by a setpoint filter is used in this simulation environment to define desired step 6 responses within a range of step input amplitudes for the TD3 algorithm. The additive input disturbance model with the random amplitude step disturbance was implemented in the simulation environment to train the RL-based PI control dynamics for improved disturbance rejection. The actor-network was designed to yield the PI coefficients of the learned optimal PI dynamics in this environment. Thus, an RL tuned PI controller was obtained for the DC motor speed control, and its performance was tested and compared with the results of the metaheuristic optimization based optimal PI and fractional order PI (FOPI) tuning algorithm. The main contributions of this study can be summarized as follows: (i) This study demonstrates the design of control simulation environments in order to learn optimal and disturbance reject PI dynamics from a set-point reference model by using the TD3 algorithm. Thus, the reference model response based optimal performance tuning of the PI controller according to the realistic environmental conditions (e.g., random disturbance insertion, random change of input signal in a wide range) was performed by using the reinforced learning strategy. In this learning perspective, the reference model describes a desired control system response (overshoot, settling time etc.) and improves RL based controller tuning process according to control application requirements and specifications. (ii) To investigate performance improvement of the RL based controller tuning method, controller tuning performance of the TD3 algorithm was compared with performances of state-of-art controller tuning methods in the current study. We illustrate the performance improvement of RL-based PI tuning in overshoot-free, smooth control of DC motor speed compared to several metaheuristic PI and FOPI controller tuning methods in the reference model based controller tuning. Also, performance improvement of RL-PI controller with the TD3 algorithm is demonstrated over that of the DDPG algorithm in this DC motor control problem. In the rest of the paper, Section 2 provides preliminary knowledge of RL and briefly introduces TD3 policy gradient method algorithm. After presenting the speed control model of DC motors, which is used in the simulation environment, a Simulink simulation environment for training of the TD3 algorithm to learn an optimal and robust PI controller dynamics for DC motor speed control is explained in Section 3. Section 4 presents a simulation study that compares results of RL based PI controller design and results of PI and FOPI controller designs of other popular methods. Some conclusions are summarized in Section 5. 7 2. Preliminaries and Problem Statement 2.1. Preliminaries of Reinforcement Learning The RL is a machine learning technique that learns from the direct interaction of assets with the environment (Sutton & Barto, 1998). The agent learns the environment and rules through experience to achieve goals in the environment. The reward and punishment mechanisms are used in the training process, and these mechanisms guide the agent's learning (Hoshino & Kamei, 2003; Russell & Norvig, 2003). The main difference from other learning methods is that the system only learns by trial and error in its own environment without using any predefined instruction sets (Sutton & Barto, 1998; Narendra & Thathachar, 2012). The reward motivates the agent to choose the best action in the environment. In order to find the best way to solve the problem, the system's learning process depends on a sufficient number of trials (Narendra & Thathachar, 2012). The agent aims to obtain an optimal policy by maximizing the total reward. This learning method can be assumed as a Markovian process (Bellman, 1957), and its theoretical foundation assumes RL to be a stochastic learning process that can progressively enhance set and trial results by suitably rewarding agent actions in the environment. At each discrete time increment , the agent selects an action according to its policy depending on the state . Then, the agent receives a reward , and a new state is received from the environment for the action (Fujimoto et al., 2018). In the Q-Learning, the quality of a state-action couple is evaluated according to the function , and it expresses expected rewards from an action at the state . The is calculated by using the Bellman equation. The form of the Bellman equation is adapted as . (1) This equation implies that the maximum future reward can be possible with the reward for the current action ( ) plus the maximum of future reward estimations . To solve the equation iteratively, the Bellman error l B eat iteration is defined as . (2) Then, an iterative scheme to reach optimal state-action value was given by using temporal difference learning (Sutton, 1988) as , (3) 8 where, is a discount factor between zero and one, determining the weights of short-term rewards (Luu, 2015). The parameter is the learning rate. The function is the reward for action a and observer state . This formulation defines the RL process as a Markov process. In other words, future situations depend on only present situations in this process. Therefore, conditional probability is widely used to model RL processes. Markov process for a state can be described as . (4) If the current state of a system is considered, previous action from at the state from leads to a new state from and a reward from . The probability density function for the random variables and can be expressed for the stochastic modeling of the RL as , (5) where , ' ,s s S r R and . It can be seen that Equation (1) depends on the state of the reward at the time and the maximum of state-action quality. Therefore, the possibilities of transition to is written by considering the marginal distribution of the state (Morales & Zaragoza, 2011). (6) Accordingly, the expected reward can be written by using a marginal distribution of states as (Morales & Zaragoza, 2011) . (7) To implement this scheme, approximation functions such as neural networks are used to learn actions that maximize rewards from the environment. Progress in RL methods continues and application fields of RL have expanded in many fields such as attitude control for hypersonic reentry vehicles (Y. Liu et al., 2022), zone scheduling optimization of pumps in water distribution networks (Xu et al., 2021), flexible control of discrete event systems using environment simulation (Zielinski et al., 2021) and autonomous navigation of UAV in multi-obstacle environments (Zhang et al., 2022). Kavuran contributed to this discussion by investigating the effect of deep RL on the fractional-order oscillator (Kavuran, 2022). 2.2. A Brief Introduction of Twin Delayed Deep Deterministic Policy Gradient Algorithm The TD3 policy gradient algorithm, which is an improvement of the DDPG algorithm, takes into account the function approach error (Lillicrap et al., 2016; Fujimoto et al., 2018). The objective of the RL is to find the optimal policy that maximizes the expected reward by tuning the 15 with one neuron for the output . The other networks are the actor neural networks (actornetwork) that yield the action ( ) by using observer states ( s ). Based on values, the TD3 algorithm trains critic-networks and actor-networks to maximize the reward as an optimal control objective given by Equation (21) in this study. Action Input State Input Fully Connected Layer Fully Connected Layer Concatenation Layer ReLU Layer Fully Connected Layer ReLU Layer Fully Connected Layer State PathAction Path Figure 5. The architecture of the critic neural network that is implemented in this application The state observer monitors important states of the control system from the simulation environment. To match the weight coefficient of the actor-network with the PI controller coefficients in the Matlab RL control example, the integral of the error signal is observed to implement the integral state ( ), and the error signal itself is observed to implement proportional state ( ) of PI dynamics. The state observer was implemented by monitoring these two states of PI dynamics as . (22) Actor network is designed by using a two-input fully connected layer with a linear activation function. Two observer states feed the actor network. Accordingly, the action of the actor can be easily written by using the observer states as follows . (23) After the training of the actor network is completed, the weights ( , ) of the neural network expresses the learned PI controller coefficients. Here, the weight corresponds to the 16 coefficient, and the weight corresponds to the coefficient of the discrete PI controller implementation. To demon follow: (24) where expresses the discrete-time PI controller function, where the proportional coefficient of the PI controller is , and the integral coefficient of the PI controller is . 4. Simulation Study In order to obtain the desired speed control performance for the DC motor by using the TD3 policy gradient algorithm, the training of the RL system was carried out by using the Simulink simulation environment shown in Figure 3. Control simulations were performed for 20 seconds for each episode, and an error signal from each simulation was used to obtain observer states (Equation 22). Error and control signals from the simulation environment are used to calculate the reward (Equation 19) in the training of the TD3 algorithm. Table 4 lists some hyperparameters used for the configuration of the training process of TD3 agents. The trial and error method was used to set suitable values to these parameters. Table 4. Variables of training process for the TD3 algorithm Maximum Number of Episodes (T) 100 Mini Batch Size 128 Sampling time 0.1 s Learning Rate 0.001 During the training of the RL system, reference step input signals with random amplitudes (within the range of 0-155) were applied to the input of the system in order to improve setpoint control performance, and the step disturbance signals with random amplitude (within the range of [- 5,5]) were used to train the system to obtain an improved disturbance rejection performance. To give a reward according to the optimal control formulation (Equation (19)), parameters and are set to 0.9 and 0.01, respectively. The RL training in the simulation environment (Figure 3) allows the actor neural network to respond similar to a PI controller by adjusting the actor-network weights ( , ). The actornetwork is the fully connected single neuron with two inputs from observed states. Equations (23) and (24) prove that a combination of actor-network and observer block implements a PI controller. The output of the actor-network is the control signal for the DC motor. Figure 6 shows the change 17 of instant values and average values of rewards obtained during the training process in the simulation environment. The average reward is the average of the instant rewards and expresses overall performance during the training process. Figure 6. Change of instant values and average values of rewards in the simulation environment When the training was completed, the p w was obtained 1.6626, which is the equivalent value of a discrete-time PI controller, and the was obtained 2.0830, which is the equivalent of a discrete-time PI controller. According to these training results, the learned discrete PI controller dynamics are equal to 1 0830.26626.1)( z T zC s. (25) In this study, the RL-PI control was rewarded to learn the step response of the setpoint filter 1 1 )( s sFset , and it enables smoothly settling of electric motor speed to the reference speed 100 rpm without any overshoot and ripples. This is an important property for electrical motor control because the overshoots and ripples can cause redundant acceleration and fluctuations in the DC motor's speed responses. These effects result in discomfort and unnecessary energy consumption in control applications, where DC motors are used, for example applications in electrical vehicles, UAVs, etc. To analyze performances of the designed PI controllers, a control system test simulation of 100 seconds duration was carried out, and an additive input disturbance with a step waveform (the amplitude of step disturbance is 20) is applied at the simulation time of 50 seconds. The performance of the RL-PI control is compared with the responses of the PI and FOPI controllers that were tuned by using several metaheuristics such as Genetic Algorithm (GA) (Holland, 1973), Average Reward * Instant Reward o 18 Particle Swarm Optimization (PSO) (Kennedy & Eberhart,1995), Grey Wolf Optimization (GWO) (Mirjalili et al., 2014) and Differential Evolution (DE) (Storn & Price,1997). Similar to the RL algorithms, metaheuristic optimization method can be effectively used for seeking optimal solutions in decision making and scheduling problems such as optimal charging scheduling problem for electric vehicles (Liu et al., 2020), batch-processing machine scheduling problem (Zhou et al., 2021 ), energy-efficient distributed no-idle flow-shop scheduling problem (Zhao et al., 2020; Zhao et al., 2021a; Zhao et al., 2021b). Also, successful use of several metaheuristic methods was shown in controller tuning problems (Tufenkci et al., 2020; Zheng & Pi, 2016; Komathi & Umamaheswari, 2020; Liu & Hsu, 2010). Besides, hyperheuristic approaches that involve high-level heuristics and low-level heuristics, can provide improved performance in a wider application domain owing to high-level tactics (Zhao et al., 2021c). The metaheuristic methods optimized the PI controller coefficients to yield a step response that resembles the response of the setpoint filter . This objective is the same as in the RL-PI control system, where the filter is a supervisor (a reference model) that was used to shape the step response of the control system according to responses of the reference model , which provides smooth and overshoot-free control of the DC motor. Metaheuristic methods have been widely used for optimal tuning of the controller functions. Therefore, we demonstrated the robust control performance improvements of the RL-PI controller with TD3 algorithm compared to results of GA, PSO, GWO and DE algorithms. The solved optimization problem for optimal tuning of PI controller functions ( ) and FOPI controller ( ) via metaheuristic methods were shown in the Appendix. Table 5 shows PI and FOPI controller tuning results of GA, PSO, GWO and DE algorithms. Since the objective of reference model based controller tuning aims to approximate to the response of the reference model 1 1 )( s sFset , the optimal controller coefficients were obtained to close each other. Figures 7 and 8 show step responses of those controllers in comparison with the response of the RL-PI controller with the TD3 algorithm. Figures 9 and 10 show error signals from these controller simulations. Figures 11 and 12 show the change of control signals during these control simulations. Main advantages of the RL-PI controller tuning comes from the design of a suitable simulation environment that involves random amplitude disturbance insertion and random setpoints. The experience based learning of the PI controller dynamics can explore more robust responses for the input disturbances in the RL based tuning. The metaheuristic tuning methods only solved the optimization problem that is defined in the Appendix. Therefore, metaheuristic tuning methods, which are based on analytical solutions of an optimal control problem, cannot directly consider effects of random amplitude disturbance signal or the random amplitude reference signal on control 19 performance (Ozbey et al., 2020). This shortcoming can limit the tuning performance of metaheuristic optimization methods in application specific design. Table 5. Optimal PI and FOPI controller functions ( )(sC ) that were tuned by metaheuristic algorithms Algorithms Controllers Algorithms Controllers GA-PI GWO-PI GA-FOPI GWO-FOPI PSO-PI DE-PI PSO-FOPI DE-FOPI Figure 7. Step responses obtained by GA-PI, PSO-PI, GWO-PI, DE-PI controllers and RL-PI controller with TD3 algorithm Figure 8. Step responses obtained by GA-FOPI, PSO-FOPI, GWO-FOPI, DE-FOPI controllers and RL-PI controller with TD3 algorithm 20 Figure 9. Error signals for GA-PI, PSO-PI, GWO-PI, DE-PI and RL-PI controllers Figure 10. Error signals for GA-FOPI, PSO-FOPI, GWO-FOPI, DE-FOPI and RL-PI controllers 21 Figure 11. Control signals for GA-PI, PSO-PI, GWO-PI, DE-PI and RL-PI controllers Figure 12. Control signals for GA-FOPI, PSO-FOPI, GWO-FOPI, DE-FOPI and RL-PI controllers We also compared performances of two different RL strategies in the PI controller tuning problem. When the training of DDPG algorithm was completed, the discrete-time controller coefficients were calculated as and of a discrete-time PI controller. According to these training results, the DDPG learned discrete PI controller dynamics are equal to 22 . (26) Figure 13 shows the change of instant values and average values of rewards during the training with the DDPG algorithm in the same simulation environment. Figure 14 shows step responses of the RL-PI controllers that were learned by the TD3 algorithm and the DDPG algorithm. The DDPG algorithm has been used as RL strategy in the motor control application (Traue et al., 2022; Book et al., 2021). The same simulation environment and the same neural network architectures were used for these algorithms. Also, parameter settings of the DDPG algorithm were configured to be the same with the TD3 algorithm in the training. Figure 15 and 16 show comparisons of error signals and control signals. Simulation results showed that the RL-PI controller tuned with the TD3 algorithm provided better PI controller performance compared to the RL-PI controller tuned with the DDPG algorithm. In general, the TD3 algorithm can be more consistent in convergence to maximum reward values than the DDPG algorithm, mainly because of using two critic network and actor networks with a delayed update and action noise regularization (clipped noise addition as in the equation (10) for a clipped double Q-learning effect (Fujimoto et al., 2018)), which make TD3 more advantageous to escape from the overfitting to narrow reward peaks and improves the search skill of the TD3 algorithm. The difference in rewards characteristics in Figure 6 (for the TD3 algorithm) and Figure 13 (for the DDPG algorithm) confirms improvements of the TD3 algorithm over the DDPG algorithm in this control problem. Figure 13. Change of instant values and average values of rewards in the simulation environment Average Reward * Instant Reward o 23 Figure 14. Step responses obtained by the RL-PI controller with TD3 algorithm and the RL-PI controller with DDPG algorithm Figure 15. Error signals for the RL-PI controller with TD3 algorithm and the RL-PI controller with DDPG algorithm 24 Figure 16. Control signals for the RL-PI controller with TD3 algorithm and the RL-PI controller with DDPG algorithm Table 6 lists the Mean Square Error ( ,where the is number of sampled data from simulation environment) and Average of Absolute Control signal ( ) performances of the considered algorithms for the PI controller tuning efforts. We performed 5 times independent trial runs for each algorithm. In these tests, to obtain different results, the RL-PI algorithms were initiated at 5 different initial points of the PI controller coefficients. Here, the MSE indicates the dissimilarity between the response of the DC motor control system and responses of the reference model . The AAC expresses the average level of control signal amplitude that is an important indicator related to the energy efficiency of the control actions. Table 6. MSE and AAC performances of the compared PI controller tuning methods Controllers MSE AAC Min Average Max Standard Deviation Average GA-PI 26.28310 26.28360 26.28390 3.16876 10-4 111.8900 PSO-PI 26.25929 26.28388 26.31272 1.91939 10-2 111.8901 GWO-PI 26.27081 26.70522 28.38485 9.39030 10-1 111.8901 DE-PI 26.28387 26.28389 26.28390 1.17834 10-5 111.8900 RL-PI (DDPG) 6.011537 111597.0 557907.5 2.49495 105 315.9496 RL-PI (TD3) 9.053762 14.91569 20.75544 5.66412 111.8495 31 https://doi.org/10.1007/S00521-020-05352-1/FIGURES/9 Storn, R., & Price, K. (1997). Differential evolution a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, 11(4), 341-359. https://doi.org/10.1023/A:1008202821328 Sundareswaran, K., & Vasu, M. (2000). Genetic tuning of PI controller for speed control of DC motor drive. Proceedings of the IEEE International Conference on Industrial Technology, 1, 521 525. https://doi.org/10.1109/icit.2000.854212 Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9 44. https://doi.org/10.1007/bf00115009 Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT press, Cambrige, MA. Traue, A., Book, G., Kirchgassner, W., & Wallscheid, O. (2022). Toward a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Control. IEEE Transactions on Neural Networks and Learning Systems, 33(3), 919 928. https://doi.org/10.1109/TNNLS.2020.3029573 - domain. In Journal of Advanced Research, 25, 171 180. https://doi.org/10.1016/j.jare.2020.03.002 University of Michigan. (2017). Control Tutorials for MATLAB and Simulink - Motor Speed: System Modeling. https://ctms.engin.umich.edu/CTMS/index.php?example=MotorSpeed§ion=SystemModeling Visioli, A. (2006). Practical PID Control. In Practical PID Control. https://doi.org/10.1007/1-84628-586-0 Water Tank Reinforcement Learning Environment Model - MATLAB & Simulink - MathWorks Switzerland. (n.d.). Retrieved March 24, 2022, from https://ch.mathworks.com/help/reinforcement-learning/ug/water-tankreinforcement-learning-environment-model.html Watkins, C. J. C. H. (1989). Learning from delayed rewards. In Robotics and Autonomous Systems, 15(4), 233 235. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3 4), 279 292. https://doi.org/10.1007/bf00992698 Wu, H. X., Cheng, S. K., & Cui, S. M. (2004). A controller of brushless DC Motor for electric vehicle. 2004 12th Symposium on Electromagnetic Launch Technology, 528 533. Xu, J., Wang, H., Rao, J., & Wang, J. (2021). Zone scheduling optimization of pumps in water distribution networks with deep reinforcement learning and knowledge-assisted learning. Soft Computing, 25(23), 14757 14767. IEEE Transactions on Control Systems Technology, 7(3), 328 342. https://doi.org/10.1109/87.761053 Zhang, S., Li, Y., & Dong, Q. (2022). Autonomous navigation of UAV in multi-obstacle environments based on a Deep Reinforcement Learning approach. Applied Soft Computing, 115, 108194. Zhao, F., He, X., & Wang, L. (2020). A two-stage cooperative evolutionary algorithm with problem-specific knowledge for energy-efficient scheduling of no-wait flow-shop problem. IEEE Transactions on Cybernetics, 51(11), 5291 5303. https://doi.org/10.1109/TCYB.2020.3025662 Zhou, S., Xing, L., Zheng, X., Du, N., Wang, L., & Zhang, Q. (2021). A Self-Adaptive Differential Evolution Algorithm for Scheduling a Single Batch-Processing Machine with Arbitrary Job Sizes and Release Times. IEEE Transactions on Cybernetics, 51(3), 1430 1442. https://doi.org/10.1109/TCYB.2019.2939219 Zhao, F., Ma, R., & Wang, L. (2021a). A Self-Learning Discrete Jaya Algorithm for Multiobjective Energy-Efficient Distributed No-Idle Flow-Shop Scheduling Problem in Heterogeneous Factory System. IEEE Transactions on Cybernetics. https://doi.org/10.1109/TCYB.2021.3086181 Zhao, F., Zhang, L., Cao, J., & Tang, J. (2021b). A cooperative water wave optimization algorithm with reinforcement learning for the distributed assembly no-idle flowshop scheduling problem. Computers and Industrial Engineering, 153. https://doi.org/10.1016/j.cie.2020.107082 32 Zhao, F., Di, S., Cao, J., Tang, J., & Jonrinaldi. (2021c). A Novel Cooperative Multi-Stage Hyper-Heuristic for Combination Optimization Problems. Complex System Modeling and Simulation, 1(2), 91 108. Zheng, W., & Pi, Y. (2016). Study of the fractional order proportional integral controller for the permanent magnet synchronous motor based on the differential evolution algorithm. ISA Transactions, 63, 387 393. https://doi.org/10.1016/j.isatra.2015.11.029 Zielinski, K. M. C., Hendges, L. V., Florindo, J. B., Lopes, Y. K., Ribeiro, R., Teixeira, M., & Casanova, D. (2021). Flexible control of Discrete Event Systems using environment simulation and Reinforcement Learning. Applied Soft Computing, 111, 107714. https://doi.org/10.1016/J.ASOC.2021.107714