Monte Carlo simulation applicable for predictive algorithm analysis in aerospace

Monte Carlo Simulation Applicable for Predictive Algorithm Analysis in Aerospace Jorge Bautista-Hernández1,2(B)and María Ángeles Martín-Prats1 1Electronics Engineering Department, University of Seville, 41092 Seville, Spain [email protected], [email protected] 2Aerospace Electrical Engineering Department, Airbus Poland, 02-256 Warsaw, Poland Abstract. Safety investigations about electrical wiring harness caused by failures in electrical systems establish that origin of these accidents are related to electrical installation. Predictive techniques which mitigate and reduce risk of the occurrence of errors to enhance safety shall be considered. The development of machine learning has evolved towards the creation of innovative predictive algorithms which show high performance in data analysis and making predictions in the context of artificial intelligence. The Monte Carlo approach is used to validate the model performance. In this paper, Monte Carlo simulation was used to evaluate the level of the uncertainty of the selected parameters over 1000 runs. This study analyzes the reliability of the predictive algorithm in order to be implemented as an automatic error predictor in aerospace. The results obtained are within the expected range suggesting that the model used is accurate and reliable. Keyword: Monte Carlo Simulation ·Predictive Algorithms ·Sensitivity Analysis ·System Reliability ·Automatic Error Predictor 1 Introduction Safety is a pillar in our lives. It has been evaluated in aerospace that major accidents are consequential from human errors which can contribute to up to 80% of the total accidents. Human errors can never be eliminated completely but they can be reduced to the minimum by implementing predictive and automatic algorithms which are focus upon the risk rather than on the error elimination. An analysis of dataset is encouraged to be performed in order to better identify relevant indicators and situations which are vulnerable to create an error in order to implement measures to avoid potential failures. Human errors in aerospace are considered as a multi-event and can be mainly generated from design (e.g. models errors), manufacturing (incorrect procedures), installation issues (incorrect assembly) and operational errors of the aircraft (miscommunication or poor decisions). Furthermore, some of these errors are likely to generate a hazard. The complex process for error generation needs to be better analyzed in order to show a holistic view of one indicator towards the creation of the error [1]. Based on the aviation authorities investigation the main cause for accident creation was the failures generated in the electrical harness installation. Thus, quantification of the main parameters that © The Author(s) 2023 L. M. Camarinha-Matos and F. Ferrada (Eds.): DoCEIS 2023, IFIP AICT 678, pp. 243–256, 2023. https://doi.org/10.1007/978-3-031-36007-7_18 244 J. Bautista-Hernández and M. Á. Martín-Prats exist in the electrical system of an aircraft and used predictive techniques to predict uncertainties are necessary. Advanced technology such as cyber-physical systems and automation are effective strategies to prevent errors [2]. Cyber-physical systems (CPS) are based on computational and physical elements which can be used to monitor processes in order to prevent errors before they occur. For example, in electrical manufacturing, the risk matrix can be used as an outcome of the predictive algorithm in order to detect in real time any anomalies before they cause a failure. Additionally, automation can be used to prevent errors and reduce the risk of human error creation. The main goal is to implement an innovative methodology to keep the aerospace industry at the greatest level of safety and potentially analyses its applicability to other disciplinessuchasenergy, health care, transportation or infrastructure. Thus, the research question established in this paper is: What can be the benefit of introducing this novel methodology in the industry? To answer to this question requires to give evidences to the following assumptions: •Time assessment during the creation of the manufacturing engineering processes, •Mitigation errors in the end-to-end process, •Positive impact in safety in order to keep aviation standards at the highest level. The quantification of the successful results have generated a decreased of 93% in manufacturing time and 90% in potential errors creation during creation of applicable manufacturing engineering processes. This specific paper aims to assess reliability and validation for the model system engineering represented in Fig. 1by using a Monte Carlo simulation. This technique can be used as a proper method to assess reliability for a system engineering. The model in Fig. 1presents a level of uncertainty in the three input parameters selected in this simulation. The Monte Carlo method will solve this uncertainty by running 1000 samples for each parameter and representing the result using the probabilistic function. This function represents the probability of the possible outcomes values are below a threshold. The aim of the study is to investigate the level of sensitivity of the key parameters used for the prediction to consider them as good indicators [3,4].The impact of varying these parameters on model with 1000 runs simulation will define the robustness of the model as a tool to be implemented as an auto-failure detector. Thus, the model uncertainties are identify towards a more reliable system in order to improve predictions in the future [3]. The remainder of this paper is as follows. The review of the relationship to connected cyber physical spaces is in Sect. 2, Sect. 3addresses the research methodology related to the validation model. The main results are presented in Sect. 4. Discussions are in Sect. 5and finally to summarize in Sect. 6conclusions and future work. Monte Carlo Simulation Applicable for Predictive Algorithm 245 2 Relationship to Connected Cyber Physical Spaces The motivation to implement this novel and innovative methodology based on the proposed automatic and predictive algorithm in the electrical manufacturing processes is fundamental in order to keep aerospace safety at the greatest level. The advanced technologies such as Big Data and the increasing system complexity together with the necessity that the data needs to be fast analyzed to provide the best solution, enable the creation of a Cyber Space Model to respond to this necessity [2]. Cyber-Physical systems provide feedback in real time and present very good adaptive and predictive capacity. The latest research in the aerospace framework shows these systems not only have a positive impact on the aviation safety but also enhancing efficiency, integration and autonomy of the next generation aerospace systems. The solution approach proposal based on this hybrid and predictive algorithm contributes to connect Cyber Physical Spaces through the use of machine learning techniques. The automatic manufacturing processes and predictive tasks enable engineering systems to execute activities independently with minimum human actions. Therefore the mitigation error is guaranteed. Additionally, the assessment in real time of the error creation by using the risk matrix not only decrease the probability of create a failure, but it also ensures the correct decision provided by the automatic system. This situation generates a positive impact in the electrical harness built process. Thus, automation process and predictive tasks are references towards connected Cyber Physical Spaces. This emerging and innovative methodology convergences on the new technologies used in the new dynamic manufacturing industry. The required multi-interaction to satisfy the demand of systems complexity establish better collaboration between academic engineering disciplines and industry since cyber-physical systems also requires more elements to be inter-connected and easy adaptability to this new technology and future applications. 3 Methods 3.1 The Algorithm Overview The innovative procedure using predictive algorithms has been developed in aerospace not only to mitigate the errors but also to predict the error creation in the electrical manufacturing engineering processes in the aerospace context by using innovative machine learning techniques. The Fig. 1represents the algorithm structure and is based on the following elements. The risk matrix mechanism that assess the probability of error creation in each specific harness. It determinates five main categories from ‘very low’ to ‘very high’ probability of error creation [5]. The automation tool is developed to avoid manual tasks during creation of the engineering documentation. Thus, it mitigates creation of an error during the engineering process. The dendrogram is using the hierarchical agglomerative method which creates groups with similar objects within the data set. The clusters data provides information about the critical groups which require special attention. The logistic regression estimates the parameters by establishing relationships between the input data and the outcomes according to a mathematic criterion. The confusion matrix will 246 J. Bautista-Hernández and M. Á. Martín-Prats define the accuracy of the algorithm by table of contingency. Finally, the computation time will be the minimum necessary to define the optimal number of iterations using the method of gradient descent after regression logistic is applied to the categorical variable. Fig. 1. Diagram representation of the predictive algorithm and Monte Carlo simulation 3.2 The Input Parameters The first step in the creation of the predictive algorithm is to define the parameters. These are based on the electrical configuration of the harness to be manufactured andeventually installed on the aircraft. These are the number of zones (Z), number of wires (H) and number of electrical components (N) which define the risk matrix F. Such criteria defines each harness category based on these parameters. The risk matrix function defines the probability of creating an error during the manufacturing process of electrical harness as part of the outcomes from predictive algorithms. The risk matrix as a function can be expressed as follows: φ=φ(Z,H,N)(1) Scores assigned to each parameter will be evaluated on a scale from 1 to 5, being 1 the simplest geometry and 5 the most complex. A detailed description of the model has been recently proposed by Bautista Hernández and Martín Prats assessing its performance towards introduction in aerospace applications [6]. The baseline parameters for the Monte Carlo simulation will be based on the following relevant metrics, p1for wiring length, p2for number of electrical components and p3for the protective sheath quantities present in the ‘bill of material’ on each harness for a military aircraft. The Monte Carlo function determinates the performance of the model for implementation purposes. The generic function for Monte Carlo simulation λ, is expressed as follows: λ=λ(p1,p2,p3)(2) Monte Carlo Simulation Applicable for Predictive Algorithm 247 TheC295aircraftdatasetwithatotalof 221 electrical harness tofullydefinetheelectrical system of this aircraft. In total in this case study, 157 harness considered are presenting the following parameters, 18523.95 m of the total wiring length, 21200.55 electrical components and 250.84 m of protective sheath length. The baseline run Monte Carlo simulation was executed 1000 times across a range between maximum and minimum values of meters of wiring length (0.29–2374.79), units of number of components (4– 2243) and meters of protective sheath (0.06–22.59). 3.3 Sensitivity Analysis The sensitivity analysis was carried out to understand the correlation between the input parameters and indicate the importance for the outcomes. Additionally, this analysis evaluates the model performance by considering the response of the input variables parameters after simulation [7]. The high correlation between the variables indicates that increasing of their values will enhance outcomes values. Thus, the probability of creating an error will be also higher [8]. The outcomes presented were analyzed to understand the impact of the maximum and minimum values within the dataset suggesting that the model performs well. In this medium-light size aircraft the predicted outcomes after 1000 runs were within the range of expected values. The box-plots performed for the chosen parameters represented in Fig. 2aim to analyze the data set and demonstrate the spread of numerical data through their quartiles follows one of the known distributions probabilistic. The Fig. 2shows the dataset distribution associated to each parameter defined by the generic function Monte Carlo in a logarithmic scale. In the first case related to the p1, the first quartile Q1=e0.9 =2.45 m marks one quarter (25%) of the ordered dataset and the value 1.5 IQR (-) below the first quartile is 1.5 IQR (-) =e0.1 =1.01 m. The maximum value in the dataset is 2374 m and the value for 1.5 IQR (+) =e 6.3 =544.21 m above the Q3=e3.2 =24.52 m, which marks three quarter (75%) of the ordered dataset. The maximum value is above the 1.5 IQR (+), so in this case the maximum is an outlier which may indicate the measurements are not in the center of the data. In the second case related to the p2, the first quartile Q1=e2.4 =11.02 components and the value 1.5 IQR (-) =e1.4 =4.05 components. The maximum value of the dataset is 2243 components. In this case, this maximum value is above the 1.5 IQR (+) =e6.9 =991 components showing outliers. In the third case related to the p3, the first quartile Q1=e0.1 =1.1 m and the 1.5 IQR (+) =e2.3 =9.97 m below the maximum value of this parameter in the dataset which is 22.59 m. The analytical results estimate the input data for the three parameters selected can be fitted to a normal distribution. 248 J. Bautista-Hernández and M. Á. Martín-Prats Fig. 2. Box-plots representation for input metrics p1,p 2,p 3on the Monte Carlo function λ 3.4 Monte Carlo Simulation The Monte Carlo simulation is a method to assess reliability of an engineering system. The simulation uses as input parameters the p1,p 2,p 3values. These values are randomly assigned 1000 times variating within the range of the maximum and minimum values defined from the entire dataset in a logarithmic scale shown in the Table 1. This simulation is used to obtain outcomes estimation from a set of stochastic trial from proper definition quantities [4]. Table 1. Minimum, maximum, mean and standard deviation values of the input parameters used in Monte Carlo simulation Description parameter min max mean std units Wiring length p10.1 7.77 2.46 1.90 m Number of components p21.38 7.71 3.66 1.36 – Protective sheath p30.1 3.11 0.44 0.74 m Such statistical distributions are used within the range of the lower and the higher end value for each parameter to estimate the outcomes across a Monte Carlo simulation with 1000 runs. Probabilistic distributions are considered for probability calculations within the dataset. The assignment for calculation of the probability in each individual run was carried out using the normal function from the random Python xlwings library within the given interval. As the input dataset variable X is log-normally distributed then the input values were transformed to a distribution Y =ln(X) for representation [9]. The simulation is defined by generation of a stochastic trial to each of the input parameters on each of 157 electrical harness on a C295 military aircraft following a probabilistic function normally distributed. After simulation using random combination of the input variables the output is calculated for each parameter. Finally, the outcomes are represented with the probabilistic function distribution and the cumulative function of density. 3.5 Method of Gradient Descent The regression logistic used to predict the categorical variable needs 144 iterations until the algorithm converges. This situation requires a high computation time. The method Monte Carlo Simulation Applicable for Predictive Algorithm 249 of gradient descent can be used to find the optimal solutions by proper adjustment of the parameters which minimize the function in order to reach the global minimum faster and the convergence of algorithm [10]. Being f :⊂Rn→R, this method can find θ⊂/f(θ)min.TheEqs.3,4and 5 define the best solution path in order to find the optimal solution. θ∈arg min f(θ)(3) θt+1=θt−δt∇f(θt)(4) ∇f(θt)=(∂f/∂x1(θ),...,∂f/∂xn(θ)) (5) δtis the learning rate which determinates the number of iterations until the algorithm converges. The convenient value of δguarantees minimize the number of iterations improving the computation time. The use of this method allows the algorithm to perform faster. Thus, the number of iterations decrease to 41 generating significantly benefit in the computation time. 4 Results The tendency of each parameter presents a linear relation between the parameters chosen for the Monte Carlo simulation (length, number of electrical components and protection sheaths). The correlation of these parameters is quantified by the correlation matrix shownintable2. A positive value of the coefficient means an increase of this parameter necessary involves an increase in the other parameter. The strongest correlation occurs when the absolute value of the coefficient is as close as possible to 1. From the correlation matrix we observed that p1and p3, as well as parameters p2with p3have lower positive correlation coefficients, but still indicate some degree of correlation between these variables. Table 2. Correlation matrix for the input parameters used for the Monte Carlo simulation Description Parameter Length Components Protection Wiring length p11.0000000 0.928046 0.470757 Number of components p20.928046 1.0000000 0.471076 Protective sheath p30.470757 0.471076 1.0000000 The Monte Carlo simulation has performed 1000 runs and the output parameters as a result of the new predicted values for the wiring length, numbers of electrical components and protective sheath are normally distributed. The cumulative distribution function shows the probability that each parameter value stays below specific threshold. 250 J. Bautista-Hernández and M. Á. Martín-Prats 4.1 First Output - Wiring Length The Eqs. (6–7) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a first output of the new parameter for the wiring length, the p1will be between 2.01 and 3.00 m with a probability of occurrence between 50 and 90% (Fig. 3). P(0.5)→X1=e0.7=2.01 m (6) P(0.9)→X2=e1.1=3.00 m (7) Fig. 3. Results for p1outcomes on the wiring length prediction 4.2 Second Output – Number of Components The Eqs. (8–9) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a second output of the new parameter for the number of components, the p2will stay between 45 and 245 units with a probability of occurrence between 50 and 90% (Fig. 4). P(0.5)→X1=e3.8=45units (8) P(0.9)→X2=e5.5=245units (9) 4.3 Third Output – Protective Textile Sheaths The Eqs. (8–9) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a third output of the new parameter protective sheath Monte Carlo Simulation Applicable for Predictive Algorithm 251 Fig. 4. Results for p2outcomes on the number of electrical components prediction length, the p3in a new harness will be between 1.49 and 4.48 m with a probability of occurrence between 50 and 90% (Fig. 5). P(0.5)→X1=e0.4=1.49 m (10) P(0.9)→X2=e1.5=4.48 m (11) Fig. 5. Results for p3outcomes on the protective sheath prediction The reliability assessment at each parameter level is needed to calculate the probability that a new component does not exceed a limit defined through the cumulative distribution function [11]. The previous figures refer to the Monte Carlo simulation represent the random values of the parameters which are variating from maximum to the minimum within the three parameters. The distribution function shows the outcomes predicted are normally distributed. The y-axis shows the frequencies where a parameter appears and the x-axis the value of that parameter at that frequency. The cumulative distribution function estimates