Transfer learning-enhanced deep reinforcement learning for aerodynamic airfoil optimisation subject to structural constraints

Transfer learning-enhanced deep reinforcement learning for aerodynamic airfoil optimisation subject to structural constraints David Ramos,1Lucas Lacasa,2Eusebio Valero,1, 3 and Gonzalo Rubio1, 3 1)ETSIAE-UPM-School of Aeronautics, Universidad Politécnica de Madrid, Plaza Cardenal Cisneros 3, E-28040 Madrid, Spain 2)Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), 07122 Palma de Mallorca, Spain 3)Center for Computational Simulation, Universidad Politécnica de Madrid, Campus de Montegancedo, Boadilla del Monte, 28660 Madrid, Spain (*Electronic mail: [email protected].) (Dated: July 1, 2025) The main objective of this paper is to introduce a transfer learning-enhanced deep reinforcement learning (DRL) methodology that is able to optimise the geometry of any airfoil based on concomitant aerodynamic and structural integrity criteria. To showcase the method, we aim to maximise the lift-to-drag ratio CL/CDwhile preserving the structural integrity of the airfoil –as modelled by its maximum thickness– and train the DRL agent using a list of different transfer learning (TL) strategies. The performance of the DRL agent is compared with Particle Swarm Optimisation (PSO), a traditional gradient-free optimisation method. Results indicate that DRL agents are able to perform purely aerodynamic and hybrid aerodynamic/structural shape optimisation, that the DRL approach outperforms PSO in terms of computational efficiency and aerodynamic improvement, and that the TL-enhanced DRL agent achieves performance comparable to the DRL one, while further saving substantial computational resources. I. INTRODUCTION Airfoil design is key to improving the performance and efficiency of various aerodynamic systems, such as aircraft wings, turbine blades, and wind turbines1. Such design often predicates on (i) achieving an optimal balance between two aerodynamic forces –where one aims to maximise the aerodynamic lift and minimise the aerodynamic drag– while (ii) maintaining structural integrity2. The aerodynamic balance is often explicitly modelled as the lift-to-drag ratio CL/CD, where CLand CDare the lift and drag coefficients. The optimisation problem can be formally stated as maximising an objective function J(x), with xrepresenting the airfoil design parameters. The two common strategies for airfoil shape optimisation involve, respectively, using gradient-based or gradient-free methods. The former methods rely on the computation of ∇J(x)to iteratively improve the design3,4. This approach is computationally efficient –particularly with high-dimensional design spaces–, making it a popular choice in aerodynamics as highlighted by comparative studies5,6. The well-known adjoint method, introduced in Fluid Dynamics by the seminal work of Pironneau7, is one prominent example of a gradientbased technique; requiring a computational effort that scales independently of the number of design variables. Subsequently, this method was refined by the so-called discrete adjoint method, becoming a widely used choice in this field8,9. While gradient-based methods are generally efficient and very popular in aerodynamic shape optimisation, they only have provable local convergence, and thus for non-convex objective functions their performance is sensitive to the initial design point10 and function continuity3. On the other hand, gradient-free methods offer an alternative to complex, nonlinear optimisation problems that often appear in aerodynamic design3,11. Despite their higher computational cost compared to gradient-based methods, they are more robust at finding global optima and handling challenging objective functions3,4. Popular gradient-free methods deployed in the context of airfoil shape optimisation include Genetic Algorithms (GA)12 or Particle Swarm Optimisation (PSO)13–15. As advanced, these methods are often limited by their computational demands and can be inefficient with very large design spaces6. To partially bridge these issues, some recent efforts to improve the computational efficiency of gradient-free methods include, e.g., parallel implementations, which can make GAs viable even for intensive applications such as those in aerospace engineering16. Recently, deep reinforcement learning (DRL)17 has emerged as a promising (gradient-free) tool for shape optimisation18,19. DRL methods have demonstrated remarkable capabilities in various domains, particularly when dealing with complex high-dimensional data4. Their impact in fluid dynamics studies and aeronautical applications belongs to a recently but rapidly expanding research field that encompasses both general-purpose ML-based fluid mechanics20–22, and their application to several aeronautical problems23,24 with clear industrial impact25. In the task of airfoil optimisation, a significant drawback of training DRL algorithms lies in the computational cost associated with solving the aerodynamic flow problem around the airfoil for a concrete shape: while the most accurate approach to solve the aerodynamic flow involves computational fluid dynamics (CFD) techniques, its intrinsic high computational cost requires opting for alternative, approximate solvers. For instance, some recent studies employ panel methods such as XFoil18,19 to approximate the aerodynamic performance of airfoils, while some other studies26,27 focus on further reducing optimisation costs by constructing surrogate models instead. Although the latter offers faster training times and smoother reward variations with respect to the agent’s actions This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 2 –facilitating the learning process–, these approximate models come at the price of a decrease in accuracy. Interestingly, one significant advantage of recent deep learning methods in optimisation tasks is their potential to enhance their learning efficiency through so-called Transfer Learning28,29: the possibility of pre-training a given model on one task and subsequently fine-tuning it for a related but different task. Transfer learning could also be developed in the context of DRL, accelerating the agent’s learning process by reusing elements such as policies or feature representations and effectively reducing the need for extensive exploration, a particularly valuable trait in aerodynamic optimisation. To the best of our knowledge this approach has been seldom explored, with a few encouraging exceptions for missile control surface and airfoils28,30. The novelty we bring with respect to these two works is a greater reduction of the cost when transfer learning is applied, and the thorough implementation of a suite of transfer learning-enhanced DRL framework for aerodynamic optimisation subject to structural constraints. Additionally, note that most studies on airfoil shape optimisation focus solely on maximising aerodynamic efficiency18,19,26,28,31, without explicitly considering important structural integrity constraints32 related, e.g., to the airfoil’s area or thickness. Accordingly, there is a need to develop shape optimisation frameworks that aim to maximise the aerodynamic efficiency while preserving the structural integrity of the airfoil. This, we argue, can be interpreted as an example of multi-objective optimisation26, which in practice can be reduced to a single-objective one by suitable scalarisation. To bridge these gaps, here we aim to explore the performance of DRL (enhanced with different possible Transfer Learning strategies) as a gradient-free framework for airfoil shape optimisation that considers the concomitant role of aerodynamic efficiency and structural integrity. To that aim, we define a flexible reward function that combines (i) the liftto-drag ratio of the flow around the airfoil and (ii) the airfoil’s maximum thickness. We initially train our DRL agent using XFoil –both for aerodynamic optimisation alone, and combined aerodynamic shape optimisation with structural integrity preservation– and show that the DRL agent is successful at both tasks. At this point results are compared with the ones obtained with Particle Swarm Optimisation, certifying the advantage of DRL over more traditional gradientfree methods. We then consider a Transfer Learning scenario where the DRL agent is pre-trained using NeuralFoil33, a faster-yet-less-accurate neural network surrogate model of XFoil built using domain knowledge. After an initial pretraining phase, we subsequently fine-tune the agent’s performance through transfer learning, incorporating direct training with XFoil. We compare four different transfer learning strategies and demonstrate how this combined approach can be successfully implemented, achieving accuracy levels comparable to those obtained through direct DRL training with XFoil, but at a reduced computational cost. The rest of the paper is organised as follows. In Section II, we introduce the whole methodology. We start by formalizing the original multi-objective optimisation problem where aerodynamic and structural integrity criteria yield two different objective functions, and we suitably combine these into a reward function which effectively scalarises the problem into the framework of single-objective optimisation. Then, we introduce the reinforcement learning method and enhance it with several transfer learning strategies. In Section III, we report the results, comparing DRL agents with PSO (both) for pure aerodynamic shape optimisation and aerodynamic/structural integrity optimisation, trained with vs without transfer learning. Finally, in Section IV, we conclude. II. METHODOLOGY A. Problem under study in a nutshell We consider the two-dimensional geometric optimisation of an airfoil shape under fixed operating conditions. The conditions chosen for the study are an Angle of Attack AoA =2 degrees, a Mach number Ma =0.5, and a Reynolds number Re =106. These represent a typical aerodynamic scenario for subsonic flow, allowing us to investigate the optimisation of airfoil shapes under conditions that are common in practical applications. In general terms, we shall consider an airfoil shape to be ‘better’ than another one when the aerodynamic flow around the airflow shows an enhanced lift-to-drag ratio. Accordingly, from an aerodynamic perspective the objective function to be maximised is such ratio, which is computed by applying an aerodynamic solver to any given airfoil geometry. Note, however, that aiming to optimise this ratio alone can yield geometries whose structural integrity can be compromised. Accordingly, in what follows we will also include structural constraints when we build the reward function. In the next subsections, we detail our proposed geometric optimisation framework and its specifications. We will introduce a Reinforcement Learning paradigm which we enhance by incorporating a Transfer Learning strategy, where the agent suitably switches between using a fast, surrogate aerodynamic solver (NeuralFoil33) to a slightly slower, yet more accurate one (XFoil34) as the method to assess the aerodynamic flow and to compute the agent’s reward. XFoil is a classic model –based on a panel method coupled with a viscous boundary layer formulation– designed for subsonic flows that calculates the pressure distribution on the airfoil and subsequently derives the lift and drag characteristics. It is faster than performing full DNS integration, but is still considered as an actual aerodynamic solver. NeuralFoil on the other hand – implemented in a Python-based library– is a ML surrogate model that can provide representation of the viscous, compressible airfoil aerodynamics for nearly any airfoil, with control surface deflections, across a 360-degree angle of attack, at any Reynolds number. NeuralFoil has been trained with millions of XFoil simulations, enabling rapid calculations of aerodynamic characteristics, albeit with a slight trade-off in accuracy. Unlike XFoil,NeuralFoil avoids non-convergence issues. While training the agent with NeuralFoil yields poorer results than when the agent is trained with Xfoil (see Table V), we will show that we can successfully leverage NeuralFoil to This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 3 reach optimal results in the context of Transfer Learning enhancement. As a final note, the developed DRL geometric optimisation framework is available through the open source library pyLOM35. B. The basic DRL optimisation framework The initial aim is to develop a DRL framework, where the DRL agent is capable of improving the aerodynamic efficiency of a given airfoil, as illustrated in Fig. 1. To that aim, the agent interacts with the environment (the airfoil geometry, which produces a certain aerodynamic flow) and its goal is to find a policy that ultimately maximises the lift-to-drag ratio (CL/CD) of such aerodynamic flow, by adequately performing actions that adjust the design variables. 1. RL specs I: environment, state space and action definition Environment. The initial stage of the process involves the creation of an environment in which the agent can undergo training –see Fig. 1–, and includes (i) choosing an adequate airfoil representation, and (ii) choosing an aerodynamic solver that will compute the aerodynamic properties of the airfoil. Regarding (i), observe that a priori one could simply use the 2D coordinates of the points over the airfoil. However, this choice is delicate as it may lead the agent to explore unusual or even unphysical shapes. A better choice is to use the so-called Class-Shape Transformation (CST) parameters36, a widely-used technique to parametrise geometries based on the curvature of the shape along the chord. This parameterisation indeed generates more realistic shapes and reduces the dimensionality of the design space, allowing for a more efficient optimisation. The implementation of such environment was done using the Gymnasium Python library37, a maintained fork of the OpenAI Gym library, which provides a flexible framework for defining and managing RL environments. State space. The state space spans a total of 18 scalar parameters STATE = (s1,s2,...,s18), where each parameter is bounded in a concrete range si∈[LOWERBOUND(i),UPPERBOUND(i)]. LOWERBOUND and UPPERBOUND are two vectors that define the boundaries of the state space and are defined in Appendix A. These 18 parameters are converted to an airfoil using CST representation, such that eight of them characterise the upper surface, eight of them the lower surface, one for the leading-edge weight, and one for the trailing-edge thickness. Different configuration values of these 18 parameters within their ranges correspond to different airfoil geometries. Action space and state update. To modify the state of the environment (the airfoil), the agent predicts an action, i.e. a vector ACTION = (ACTION(1),ACTION(2),...,ACTION(18)) where ACTION(i)∈[−1,1]∀i, as recommended in the documentation of stable baselines38. This vector is the one performing the change in the parameters that represent the airfoil. Now, to prevent the sequential changes to the airfoil from being too abrupt, we further define a weight vector α= (α1,α2,...,α18), where αi=LOWERBOUND(i)−UPPERBOUND(i) EPISODEMAXLENGTH , where EPISODEMAXLENGTH is a positive integer that represents the maximum permitted length of an episode, defined with more detail in Sec. II B 2. This allows the agent to potentially explore both limits of the state space in a single episode, regardless of the initial state. Accordingly, updating the state based on the action follows STATE(i)←− STATE(i)+α(i)·ACTION(i),i=1,2,...,18 (1) or in vector form STATE ←− STATE +α⊙ACTION,(2) where ⊙is the element-wise multiplication (Hadamard product). 2. RL specs II: Episode definition A single step of the DRL agent consists in, given an action, updating the airfoil geometry according to Eq. 2, assessing how this new geometry improves (or not) the aerodynamic efficiency and checking if the termination condition is met. We have decided that this condition will be reached when a maximum number of steps has been completed or if the aerodynamic solver, in this case XFoil, does not converge. The iteration of many steps until the termination condition defines an episode, and the number of steps taken is called the episode length. Here, we initially define the maximum length of an episode as EPISODEMAXLENGTH =100. Observe that this length needs to strike a balance: if it is too large, the agents will have problems to converge since this would reduce the number of episodes during training and the agents would receive less varied data. Moreover, the number of possible states increases, making the problem more complex. On the other hand, if the length is too short, the values of αin Eq. (2) will be comparatively too large and the airfoil modifications could be too abrupt, further causing instability in the optimisation process. Once an episode is defined, we need to define how the start of a new episode is carried out. A standard approach is to fix the initial state (to a certain airfoil geometry, e.g. naca0012), such that at the beginning of each episode the state of the environment is always reset to the same airfoil. This is, for example, the choice in18. Some other studies19 in turn decide to reset to a different airfoil geometry, the rationale being that in this way the agent is capable of learning to optimise starting from a variety of shapes, i.e. the learned policy is more universally applicable. Here we chose the latter strategy, so that at the beginning of an episode the environment is chosen at random from a pre-compiled set of 20 naca airfoils listed in the Appendix A. This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 4 Environment Initial airfoil RL Agent Action Airfoil representation CST Airfoil Geometry Aerodynamic Solver NeuralFoil / XFoil PredictsUpdates state Computes basic reward (CL/CD) New updated state Figure 1: Basic DRL training diagram. 3. RL Specs III: Reward function Once the episode has been defined, we need to specify the reward function, which estimates the quality of any given action. For pedagogical reasons, we now outline the construction of such reward function in a step-by-step way. As a starting point, we consider the maximisation of the lift-to-drag ratio CL/CD–where both aerodynamic coefficients are computed from the aerodynamic solver–, which yields a purely aerodynamic shape optimisation. As previously mentioned, in this work we will use two solvers: XFoil and NeuralFoil. Observe that, in addition to calculating lift and drag coefficients, NeuralFoil also returns a ‘confidence’ value: a scalar κ∈[0,1]that indicates how confident the surrogate model is after making the lift and drag predictions. In order to penalise weird geometries or geometries whose computed aerodynamic properties have large uncertainty, we will aim to maximise κ·(CL/CD), i.e. we aim at finding geometries that at the same time are (i) improving the lift-to-drag ratio and (ii) keeping the confidence high. With this term κ, we effectively mitigate the risk of the agent relying on potentially inaccurate predictions in regions where NeuralFoil is known to perform poorly. Now, since in RL the agent tries to maximise the cumulative reward18, the actual reward function needs to take a differential form, e.g. κi CL CDi −κi−1 CL CDi−1 .(3) So far, as previously mentioned the reward function denotes a purely aerodynamic optimisation. So at this point we consider as a second objective function the needs to preserve the structural integrity of the airfoil. As a simply proxy, we consider the airfoil’s maximum thickness MT, and our goal is to preserve as much as possible this structural property, i.e. we aim to minimise the mismatch between the initial and final thicknesses. We formalise this by using a Gaussian kernel λ, such that at step i, the kernel reads λi=e−σ(xi−1)2,(4) where xi:=MTi/MT0is the ratio between the airfoil’s maximum thickness at step iand the initial one, and σ>0 is an hyperparameter that controls the strength of the nonlinear penalisation. This Gaussian regularisation is inspired by computational fluid mechanics mesh optimisation methods that minimise error while maintaining low computational cost39,40. For a fixed σ,λsimply penalizes those airfoils whose maximum thickness is not preserved (see Fig. 2for an illustration of how the values of λpenalises geometries with nonconserved maximum thickness, for different values of σ). As such we now have to maximize the term given in Eq. 3and, independently, maximize λin Eq. 4. This, in essence, is a multi-objective optimisation problem: we aim to find airfoil geometries that maximise the aerodynamic properties and maximally preserve the initial maximum thickness (a proxy for preserving the structural integrity). Now, instead of treating the aerodynamic and the structural aspects independently, we choose to scalarize the multi-objective problem by combining the aerodynamic and the structural terms into a single reward function. Finally, we can define such reward at step i as Ri=λiκi CL CDi −λi−1κi−1 CL CDi−1 ,R0:=κ0 CL CD0 .(5) Note that when we use XFoil then κis not defined, in that case we set κ=1 in Eq. 5. Since we now have a single reward function, observe that from a mathematical point of view we have approximated a multi-objective optimisation problem with a simpler, single-objective one which balances aerodynamic and structural integrity constraints. The Gaussian kernel effectively acts as a structural regularisation term. This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 5 In what follows, we will distinguish the special case σ=0 (yielding λi=1,∀i)which will be called the purely aerodynamic optimisation, from the more general case σ=0 where structural integrity enters in the reward function. This latter scenario will be called the hybrid aerodynamic/structural optimisation case, although we need to make clear at this point that optimisation in the structural dimension is simply aimed at minimising the change in maximum thickness. Figure 2: The Gaussian regularisation term λ=exp(−σ[x−1]2)as a function of the hyperparameter σ. This term penalises geometries where the airfoil’s maximum thickness deviates from the initial one, as captured by a maximum thickness quotient xthat deviates from one. 4. RL specs IV: the agent To train an agent with reinforcement learning, one needs to choose the specific DRL algorithm. This choice may depend, among other things, on whether the action space is discrete or continuous. For continuous actions, there are a variety of algorithms, including A2C41, PPO42, SAC43 or TD344. After running some preliminary analysis, we decided to use PPO (Proximal Policy Optimisation), as it shows better convergence properties than A2C. Moreover, we discarded the use of TD3 and SAC since they had difficulties converging. PPO is a popular policy gradient algorithm that, for instance, is behind the post-training with RLHF of popular Large Language Models, underscoring its versatility. In our implementation, PPO employs neural networks to model the policy and value functions. Specifically, PPO uses an actor-critic framework where the policy network (actor) selects the actions, and the value network (critic) evaluates the expected returns of those actions. Figure 3shows a schematic of the neural networks comprising PPO. To train the agent, the Python library Stable Baselines 345 has been used. This is a library that implements different reinforcement learning algorithms and is easy to integrate into Gymnasium environments. C. Transfer learning Finally, we need to explicitly discuss the transfer learning technique. We start by recalling that this is a way of accelerating the training process28, as it has been observed in RL frameworks that an agent initially trained with a (fast) lowFeature extractor Policy feature extractor Value feature extractor Policy neural network Value neural network Policy output layer Value output layer Figure 3: Architecture of the DRL agent that uses Proximal Policy Optimisation. The figure illustrates the transfer learning strategy #3, highlighting the part of the agent’s parameters which are copied and frozen in the fine-tuning phase and the part which is copied and is still trained in during fine-tuning. cost model over a substantial number of iterations and subsequently trained with a high-cost model over a limited number of iterations achieves a comparable performance with respect to training solely with a high-cost model over a longer period. Transfer learning thus reduces the overall computational cost and time required for the training process, something particularly welcome in high-cost applications such as aerodynamic shape optimisation. Here, as previously discussed, we use the surrogate model NeuralFoil as the low-cost model, and XFoil as the more accurate yet more computationally expensive solver. To provide a more effective evaluation of the efficacy of Transfer Learning we use the smallest model available of NeuralFoil, which has a clear speed advantage and a clearly lower accuracy compared to XFoil. In practice, we fine-tune the actor-critic neural network utilised by the PPO algorithm. Fine-tuning is a common technique in machine learning where a pre-trained model is adapted to a new task or domain by continuing the training process with different data. Accordingly, here the initial step is to train an agent with the low-cost model. Subsequently, the weights of this agent are transferred to a new agent that is then trained with the high-cost model. Several different approaches have been considered: • #1 : Share all the parameters of the network and continue with the training. • #2 : Share all parameters except those of the last layer, This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 6 both for the action network and the value network, and then continue with the training. • #3 : Share all the parameters, freeze all of them except those of the last layer of the network, and continue the training. • #4 : Share all the parameters except those of the last layer of the network, both for the action network and the value network, freeze them, and continue the training. To clarify, by ‘freezing of weights’ during fine-tuning we mean that they will not be changed. Fig. 3illustrates how the weight transfer is performed for the specific scenario raised on strategy #3. Finally, note that after pre-training the first agent, an entropy term is added to the PPO loss function, which encourages the new model to explore the state space. This makes it easier for this new agent to move into other areas of the state space. The PPO configurations for each case are described in Appendix B. III. RESULTS Results cover a variety of analysis of the DRL agents, see Appendix Bfor an in-depth study of the hyperparameter selection and the final configurations of the different PPO hyperparameters. We initially compare the performance and training’s computational efficiency of the DRL agent (trained for the purely aerodynamic optimisation and then also trained on hybrid aerodynamic/structural optimisation) with respect to a traditional gradient-free optimisation methodology, namely Particle Swarm Optimisation (PSO). Then we perform a more detailed analysis of the performance of the DRL agent in the single vs multi-objective scenario. Finally, we assess the performance and efficiency gain when the DRL agent is enhanced with different possible Transfer Learning strategies. We recall that each episode has at most 100 steps and that training airfoils are sampled randomly at the beginning of each episode from the set of 20 airfoils specified in previous sections. On top of the computing the reward in every step following Eq. 5, we monitor the average episode’s reward each N_STEPS, see Appendix Bfor details. The episode reward is defined simply as the sum of the rewards Riof each step iin an episode. The criterion to finish training is based on observing a plateau in this metric. Once the DRL agent is trained, we evaluate its performance by running a whole episode on the set of airfoils in the Aerosandbox46 Python library, which is essentially a superset of the UIUC Airfoil dataset. This dataset includes a broad variety of airfoils, which allows for a comprehensive evaluation of the agent’s ability to generalise across different shapes and aerodynamic characteristics. In the evaluation, we always use XFoil as the method to compute the lift-to-drag ratios, so as to have a fair way of comparing agents. Now, since XFoil is known to sometimes have convergence issues, the total number of airfoils evaluated fluctuates a bit between agents. The evaluation metrics include the best lift-to-drag ratio attained within each episode best =maxCL CDiepisode , and the net improvement (in lift-to-drag ratio units) within each episode improvement =best −CL CDinitial . Aditionally, in the hybrid optimisation case we monitor the deviation of the airfoil’s maximum thickness with the percentage ∆MT =|MTopt −MTini| MTini ·100, where MTini and MTopt are the maximum thickness of the initial airfoil and the airfoil with best lift-to-drag ratio, respectively. A. Deep Reinforcement Learning vs Particle Swarm Optimisation This section assesses the proposed DRL methodology and compares it (both for purely aerodynamic and hybrid optimisation) with a traditional gradient-free method: Particle Swarm Optimisation (PSO)47. PSO is a population-based optimisation algorithm inspired by the social behaviour of bird flocks searching for corn. It is widely used due to its simplicity, ability to handle non-differentiable objective functions, and efficiency in exploring high-dimensional search spaces. PSO has been successfully applied in various domains, including image and video analysis applications, engineering design and neural networks, and fluid dynamics optimisation. Furthermore, PSO has proven effective in airfoil optimisation13–15 and seems to better suited than other standard gradient-free choices such as Genetic Algorithms for airfoil optimisation48,49. Here, we use Xoptfoil250 to perform PSO optimisation, since this tool is well designed to compare PSO and DRL as it actually makes calls to XFoil underneath. The chosen PSO parameters are the default ones that this library provides. The results obtained using this tool are systematically compared to those given by the DRL agent trained for 81920 steps using XFoil. Since the main point of this paper is DRL, we do not conduct a thorogh hyperparameter optimisation for PSO, so we cannot guarantee that its results may slightly differ after thorough hypertuning. The configuration of XFoil used in the training of the DRL agent and the PSO optimiser is detailed in Table IX. We start with the purely optimisation case, where the DRL agent is trained to solely improve the aerodynamic efficiency of the airfoil. For illustration, the first 20 steps of the DRL policy, as applied to the initial airfoil NACA0012, is shown in Fig. 4a, whereas the airfoil reaching the highest lift-to-drag ratio is depicted in Fig. 4c (center). We clearly see that the DRL agent’s policy is able to substantially and monotonically This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 7 (a) Optimisation of a NACA0012 airfoil by a DRL agent trained on purely aerodynamic optimisation. Here, x/c and y/c mean the x and y coordinates of the airfoil normalized to the chord length. (b) Optimisation of a NACA0012 airfoil by a DRL agent trained on hybrid aerodynamic/structural optimisation (σ=15). Here, x/c and y/c mean the x and y coordinates of the airfoil normalized to the chord length. (c) Left: initial NACA0012 airfoil. Center: Purely aerodynamic optimisation. Right: Hybrid aerodynamic/structural optimisation (σ=15). Figure 4: Illustration of a DRL policy (20 steps) on NACA0012 trained on aerodynamic and aerodynamic/structural optimisation, respectively. increase the aerodynamic efficiency of the airfoil. Now, to initialy compare the performance of DRL and PSO optimisation, we select 20 airfoils at random from the UIUC dataset –checking that they represent a wide variety of shapes– and subsequently applied the policies from DRL and PSO. The initial and final CL/CDresults of these trajectories are shown in Table I. We find that DRL outperforms PSO in most of the cases, certifying the suitability of DRL for airfoil shape optimisation and suggesting its supremacy over PSO. Then, we tackle the whole UIUC airfoil dataset and proceed to make the full comparison between DRL and PSO. The performance metrics best and improvement (averaged over the whole set of evaluated airfoils) are depicted in Table II, certifying the supremacy of DRL over PSO for aerodynamic optimisation. In a second step, we then focus on the hybrid optimisation case. For illustration, in Fig. 4b we show the performance of a DRL agent’s policy which was trained on hybrid aerodynamic/structural optimisation with σ=15 (see also Fig. 4c Airfoil Initial PSO single DRL single PSO multi DRL multi naca0012 38.32 47.72 249.4 45.91 103.6 naca2412 80.53 216.7 257.1 162.2 184.1 s1223 93.69 239.1 204.4 156.2 115.0 fx63143 76.12 86.67 217.4 63.57 102.1 e387 119.9 209.2 224.8 194.9 197.3 naca4412 113.0 223.0 250.7 163.4 141.0 naca63 -2a015 54.34 158.8 76.5 117.9 104.8 clarky 107.2 215.1 225.5 164.5 181.7 e174 122.0 218.7 231.3 194.4 197.0 mh113 116.9 247.1 254.9 145.3 169.4 ag25 102.7 250.5 249.8 215.1 215.0 raf48 71.33 201.5 248.4 138.0 159.1 naca23015 43.53 146.0 257.3 86.30 167.7 sc20012 31.69 51.24 201.0 45.19 157.0 goe398 100.5 240.5 257.1 142.6 170.7 n64212 75.90 222.5 261.8 166.4 133.0 ls013 44.53 52.51 231.6 45.61 168.0 dae21 155.1 203.3 258.0 259.8 177.8 vr7 94.59 236.3 253.6 153.0 187.0 naca66 -2415 93.72 199.7 240.8 147.5 160.7 Table I: Comparison of PSO and DRL performance for purely aerodynamic and hybrid aerodynamic/structural optimisation (in terms of the highest CL/CDreached in each of the two optimisation paradigms), across a set of different airfoil with different initial CL/CD. On average, DRL reaches better aerodynamic efficiency than PSO. Method Evaluated Airfoils Improvement Best PSO (pure aerodynamic) 1566 105 ±50 198(76) DRL (pure aerodynamic) 1456 141 ±48 243(35) Table II: Comparison of PSO and DRL evaluation for purely aerodynamic optimisation with the whole UIUC airfoil dataset, cerifying the supremacy of DRL over PSO for purely aerodynamic optimisation. The Improvement metric is provided in terms of mean ±one standard deviation (when averaged over the whole UIUC ensemble), whereas the best metric is provided in terms of median (IQR), since its distribution is left-skewed. (right)). The choice of σ=15 is arbitrary, it is not aimed at signalling any Pareto optimality but rather is selected for illustration purposes. The interested practitioner will actually need to select the value of σdepending on the specific priorities of their application. It is clear that the incorporation of the structural regularisation in the reward function has an impact: the lift-to-drag ratio is still increased (albeit reaching smaller values than in the purely aerodynamic case), but now the maximum thickness of the shape seems to be maintained. Interestingly, within Xoptfoil2 it is possible to define geometrical constraints –such as e.g. a target maximum thickness– and therefore a fair comparison between DRL and PSO within the hybrid optimisation case discussed in Sec. III B is possible too. For illustration, in this comparison we set σ=15 (that This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 8 we will later show to achieve a good balance between aerodynamic optimisation and maximum thickness conservation). A summary of the performance of the 20 previously selected evaluation airfoils can also be seen in Table I, whereas summary statistics of the evaluation over all the converged airfoils of the UIUC airfoil dataset are reported in Table III. It is evident that DRL exhibits superior performance in terms of aerodynamic efficiency optimisation. However, PSO demonstrates a more effective capability in preserving the maximum thickness. It’s possible that this might just be flagging the capacity of DRL to explore more design space than PSO, that in that sense only optimises locally, i.e. performs more exploitation. One thing to note from Table Iis that PSO anecdotically optimises certain airfoils more effectively in both pure and hybrid optimisation scenarios. One possible reason for the behavior in these cases is that if during optimisation with DRL one of the early states of the episode did not converge, making the modifications that the agent does to the optimised airfoil too small. This is, for instance, what happens with NACA632a015. Another reason why DRL may perform worse than PSO is because it looks for unrealistic shapes, erring on the side of creativity. This is what appears to happen with airfoil S1223. Finally, there may be airfoils that PSO simply optimises better. Method Evaluated Airfoils Improvement Best ∆MT (%) PSO (hybrid) 1566 64 ±36 150(61) 0.67 ±2.21 DRL (hybrid) 1444 89 ±37 179(46) 11.5 ±9.4 Table III: Comparison of of PSO and DRL performance for hybrid aerodynamic/structural optimisation (σ=15) on the whole UIUC airfoil evaluation dataset. Whereas DRL clearly allows to reach higher aerodynamic efficiency, PSO makes a better job at preserving the maximum thickness. Pointwise statistics of the Improvement and best metrics are provided as in Table II. Finally, we provide a comparison of optimisation time per airfoil (averaged over the whole UIUC evaluation set) for DRL and PSO, as performed on an Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz+. For concreteness, we focus on the purely aerodynamic case (where the maximum thickness is not necessarily preserved, e.g. σ=0), although the results are qualitatively similar in the hybrid optimisation case. The DRL agent, using 1 process and 1 CPU thread, takes about 23 seconds to optimise all 1566 airfoils, i.e. an average of 0.0147 s/airfoil. Comparatively, PSO was notably slower; actually, too slow to directly perform the optimisation in a single core, so MPI was used to accelerate the evaluation (and then the results per process were inferred). The CPU used has 48 cores, 47 used by worker processes plus the one by master process, and with this setup, optimising all the evaluation set took 22 hours and 26 minutes. Assuming linear acceleration, this means 47·(22·3600+26·60)/1566 =2424 seconds on average per airfoil (using a single process). Accordingly, PSO takes about 1.65·105times more time than DRL to optimise each airfoil using a single process. This is because, once trained, the DRL agent does not need to make any solver call to optimise an airfoil. Since training is a one-time cost, DRL becomes increasingly advantageous when optimising multiple airfoils, as its inference is virtually instantaneous. This highlights the drastic efficiency advantage of DRL, making PSO impractically slow by comparison. B. Hyperparameter exploration in the hybrid optimisation case Here we provide additional insights on the performance of the DRL agent trained on the hybrid optimisation case (i.e. σ>0 in Eq. 5). We systematically vary σ–where larger values of σprovide more weight to the structural optimisation over the aerodynamic one), and assess (i) the number of training steps needed to reach training convergence, (ii) the lift-todrag ratio performance metrics improvement and best evaluated on all the airfoils present in the Aerosandbox library and (iii) the variation in the maximum thickness, as a function of the regulariser’s hyperparameter σ. Results (for the converged airfoils) are depicted in Table IV, which serves as a sensitivity analysis of σ. It can be observed that the number of necessary training steps varies (typically increases) with σ. At the same time, results confirm the evidence illustrated in Fig 4and show that the DRL agent (i) is able to improve the aerodynamic efficiency while (ii) also maintaining the structural integrity of the airfoil. Finally, results also certify that forcing the maximum thickness to be preserved comes at the cost of a smaller improvement of the aerodynamic efficiency. σImprovement Best ∆MT (%) Airfoils Evaluated Training steps 0 140 ±49 241 (37) 64 ±22 1982 81920 2 108 ±41 201 (49) 22 ±16 1965 120832 5 99 ±39 190 (52) 21 ±18 1960 120832 10 95 ±42 190 (53) 15 ±10 1909 120832 15 91 ±37 180 (46) 12 ±11 1952 120832 20 93 ±37 186 (45) 12 ±10 1933 120832 30 94 ±41 186 (57) 8 ±11 1938 501760 100 84 ±42 176 (59) 5 ±5 1922 1001470 1000 54 ±35 137 (46) 7 ±8 1897 501760 Table IV: Hybrid optimisation agent evaluation with UIUC for different values of σ. Pointwise statistics of the Improvement and best metrics are provided as in Table II, and ∆MT statistics over the whole UIUC is provided in terms of mean ±one standard deviation. C. Enhancing the DRL-based airfoil optimisation with Transfer Learning To round off, here we compare the training efficiency and evaluate the performance of the trained DRL agent when training is done using a single aerodynamic solver (XFoil alone or, for completeness, NeuralFoil alone) with respect to the case when the agent is pre-trained with NeuralFoil and fine-tuned with XFoil). For the sake of exposition, we separate the purely aerodynamic optimisation (σ=0 in Eq. 5) and the hybrid one (σ>0). This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045 9 Purely-aerodynamic optimisation case. Results for all DRL agents in the purely aerodynamic case (with and without transfer learning) are summarised in Table V, reporting (i) the total number of training steps, (ii) the total solver time, (iii) the number of airfoils evaluated, and the metrics (iv) pointwise statistics of best and (v) improvement over all the airfoils in the evaluation set. Pointwise statistics of best reported in the table are given in terms of median (IQR) as the distributions are not Gaussian (Shapiro-Wilk test) and highly skewed, whereas in the case of improvement we report mean ±std as the distributions are fairly more symmetrical. Note that to compute the total solver time we simply multiply the number of training steps by the execution time per call to the particular aerodynamic solver, which according to NeuralFoil’s GitHub amounts to 73 ms for a call to XFoil and 4 ms for a call to the smallest version of NeuralFoil, which is the one we actually used. Although the total solver time is not equivalent to the total training time, it is nonetheless an objective way of comparing the computational effort of the agent using vs. not using transfer learning. As previously mentioned, the number of airfoils evaluated may differ slightly between agents because XFoil, which is consistently used for evaluation, occasionally fails to converge. In such cases, we excluded the corresponding airfoils from the analysis. First, out of the four transfer learning strategies depicted in Sec. II C, we found that strategy #1 had the best performance, followed by #3. Strategies #2 and #4 did not converge and are therefore not reported in the table. This lack of convergence indicates that the information present in the final layers of the neural network plays an important role in shaping the agent’s behaviour. Second, from the table we observe that the number of necessary training steps for the agent solely trained with Xfoil (81920) is notably larger than the total number of steps needed for the DRL agents with transfer learning, where the pre-training (NeuralFoil) and fine-tuning totally amounts to 26312 +10240 =36552 training steps. The cost of the steps performed with NeuralFoil could be considered negligible, so the important aspect here is that the number of steps required with Xfoil is reduced by a factor of 8. Additionally, it can be observed that the total number of iterations decreases. This is because the agent converges with fewer iterations when trained with NeuralFoil, something that can be attributed to its continuous nature and its lack of non-convergence problems. This is already evidence that transfer learning reduces the computational effort. Third, not only the number of steps, but also the total solver time during training (5980 seconds vs. 105 +748 =853 seconds) is notably shorter for agents trained using transfer learning. On this respect, we have also compared the total training time with and without transfer learning. If we define the Time Reduction (%) as Time Reduction (%) =TL-free training time −TL-enhanced training time TL-free training time ·100 .(6) we find Time Reduction ≈86%, reaching higher reductions than in other settings30. Fourth, the performance in the evaluation set of the agent trained solely on XFoil seems statistically superior (MannWhitney U test p<10−9), but the DRL agent trained with Transfer Learning (strategy #1) shows a very similar performance (paired sign test p≈0.006, see boxplots of the best performance metric in Fig. 5), confirming that transfer learning indeed increases computational efficiency while maintaining good performance. Finally, in Fig. 6we scatter plot the best lift-to-drag ratio obtained for each evaluated airfoil as a function of the initial estimate of the airfoil shape for the four DRL agents. These scatter plots show that while there are some specific shapes for which the agent cannot improve the aerodynamic efficiency, in most of the cases all DRL agents can improve it, with varying degrees of performance, which are in good agreement with those previously reported. Interestingly, we find that all airfoils with initial CL/CD>150 seem to be greatly improved by the DRL agent trained solely with XFoil and the one trained with the Transfer Learning strategy #1 (see the clear exclusion area in panels (A) and (D)). Hybrid optimisation case. Results of the DRL agent (with TL) trained on a combined aerodynamic/structural optimisation are depicted in Table VI. Comparing these with the equivalent results without TL reported in Table IV, we find that (i) the procedure involving TL substantially reduces the number of training steps as well as the training time, reaching Time Reduction as large as 94%; (ii) fixing the structural hyperparameter σ, the improvement in aerodynamic efficiency is only slightly lower for every value of σ(around a 10%) in the TL-enhanced DRL agent than the one without TL, and (iii) the preservation of the maximum thickness is quite similar in both techniques. In summary, TL-enhancement also works in the hybrid optimisation case. IV. CONCLUSIONS While traditional airfoil shape optimisation typically considers aerodynamic efficiency, the optimisation can produce airfoils whose structural integrity might be compromised; in other words, there is a need to preserve some structural constraints –such as the airfoil’s maximum thickness– within the optimisation process. In this paper, we have developed a technique for airfoil shape optimisation –flexible to accomodate purely aerodynamic or hybrid (aerodynamic/structural) shape This is the author’s peer reviewed, accepted manuscript. However, the online version of record will be different from this version once it has been copyedited and typeset. PLEASE CITE THIS ARTICLE AS DOI: 10.1063/5.0274045