Machine learning-based test oracles for performance testing of cyber-physical systems: An industrial case study on elevators dispatching algorithms

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/360864207 Machine learning‐based test oracles for performance testing of cyber‐ physical systems: An industrial case study on elevators dispatching algorithms ArticleinJournal of Software: Evolution and Process · May 2022 DOI: 10.1002/smr.2465 CITATIONS 14 READS 313 7 authors, including: Aitor Gartziandia Mondragon University 4 PUBLICATIONS33 CITATIONS SEE PROFILE Aitor Arrieta Mondragon University 95 PUBLICATIONS1,011 CITATIONS SEE PROFILE Jon Ayerdi Mondragon University 12 PUBLICATIONS213 CITATIONS SEE PROFILE Miren Illarramendi Mondragon University 37 PUBLICATIONS133 CITATIONS SEE PROFILE All content following this page was uploaded by Aitor Arrieta on 31 May 2022. The user has requested enhancement of the downloaded file. Received: Added at production Revised: Added at production Accepted: Added at production DOI: xxx/xxxx ARTICLE TYPE Machine Learning-based Test Oracles for Performance Testing of Cyber-Physical Systems: an Industrial Case Study on Elevators Dispatching Algorithms† Aitor Gartziandia1| Aitor Arrieta*2| Jon Ayerdi2| Miren Illarramendi2| Aitor Agirre1| Goiuria Sagardui2| Maite Arratibel3 1Information and Communication Technologies, Ikerlan, Mondragon, Spain 2Software and Systemcs Engineering, Mondragon University, Mondragon, Spain 3Control Systems, Orona, Hernani, Spain Correspondence *Aitor Arrieta, Email: [email protected] Present Address This is sample for present address text this is sample for present address text Summary The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re-used at different test phases. To deal with these issues, we propose DARIO, a test oracle that relies on regression machine-learning algorithms to detect both functional and non-functional problems of the system. The machine-learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non-functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% to 98%, across the different scenarios proposed. For the context of non-functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranging between 83% to 87%. KEYWORDS: Test Oracle, Regression Testing, MachineLearning, Performance Testing 1 INTRODUCTION Cyber-Physical Systems (CPSs) combine digital cyber technologies with physical processes [21]. The software of such systems has an increasingly long life-cycle, requiring maintenance over several years. This is the case of Orona, one of the largest elevator companies worldwide, whose software needs to be constantly maintained and updated to deal with (1) new functional and non-functional requirements, (2) the correction of bugs, (3) legislative changes and (4) hardware obsolescence and system degradation [7]. In a system of elevators, one critical subsystem is its traffic master, which is in charge of managing the passenger flow in a building. It is composed of several software modules, such as the dispatching algorithm, which is the component in charge of deciding which elevator must attend each passenger by considering several aspects (e.g., estimation of the waiting time of each passenger or estimated time required to arrive to destination). New generations of these algorithms are also starting to consider additional aspects, such as energy consumption. The algorithms also include certain functionalities based on the building, such as †This publication is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871319. Jon Ayerdi, Aitor Arrieta, Miren Illarramendi and Goiuria Sagardui are part of the Software and Systems Engineering research group of Mondragon Unibertsitatea (IT1326-19), supported by the Department of Education, Universities and Research of the Basque Country. 2 Gartziandia et al controlling access to disable specific floors to unauthorized passengers or special calls for handicapped persons (e.g., by assigning additional space in the elevator or assigning them the closest elevator). Orona, one of the leading elevator companies in Europe, has a large suite of dispatching algorithms that are in constant maintenance and evolution to solve different customers’ demands. When changes are performed in these algorithms, Orona has a well established systematic verification and validation process at different test phases (Software-in-the-Loop (SiL), Hardware-in-the-Loop (HiL) and Operation). At SiL and HiL, two kinds of tests are executed. On the one hand, unitary tests aim to test specific scenarios and verify that the functional requirements of the system are met.1Such tests are relatively short, lasting from 30 seconds to 5 minutes. On the other hand, Orona executes what is internally known as full-day passenger tests, which have the objective of verifying the functional performance of a systems of elevators (what in the elevation industry is known as the Quality of Service (QoS) of a system of elevators [11]). In such case, the objective is to detect functional performance failures (i.e., the failures that cause a degradation in the QoS of the system). An example of a typical functional performance metric in this domain is the Average Waiting Time (AWT), which measures the average time that passengers need to wait until an elevator attends their call. To determine whether the quality of an algorithm is satisfactory or not, a regression test oracle is used. Specifically, the same traffic profile (i.e., test input) is executed in a previous version of the dispatching algorithm, and compared with the dispatching algorithm version under test. If the performance metric under study is better or similar, the test is classified as PASS. Conversely, if the performance metric under study is not good enough, the test is classified as FAIL. This regression test oracle has two main problems. Firstly, the test needs to be executed both in the algorithm under test as well as in a previous version of the algorithm. This can be time-consuming as the execution is performed using simulation-based testing at system level and the test inputs usually simulate full-day passenger traffic profiles. This problem is exacerbated at the HiL test phase, where the simulation is performed in real-time, and thus, in order to simulate a full-day traffic profile, two full-days are required (i.e., one full-day for the new software version under test and another one for the previous software version). The second problem involves that the test oracle cannot be re-used for testing the new version “on-the-fly” in operation. This is a problem because the faults not detected in the validation phases can manifest in the real installation. The last problem relates to the difficulty of detecting non-functional bugs, which pose significant challenges as compared to functional bugs; these type of bugs typically result in an overuse of computational resources [20], which are scarce in CPSs, and detecting them is starting to become a priority inside Orona, our industrial partner. The barrier between functional and non-functional bugs in CPSs is blurry, and therefore, we make a distinction between functional and nonfunctional performance bugs. On the one hand, the purpose of the elevator dispatching algorithm is outputting optimal assignments. Consequently, a sub-optimal assignment (as opposed to being slow at issuing the assignment) is considered an incorrect output for the dispatcher, and is therefore classified as a functional performance failure. This is analogous to a GPS providing a valid but not optimal route between two points. On the other hand, non-functional performance bugs are those that do not cause a functional failure, but result in an overuse of computational resources [20]. Unfortunately, CPSs are not exempt of non-functional performance bugs, which can occasionally lead to more severe consequences, such as functional failures. In our conference-version paper published on Automation of Software Test (AST) conference [3], we explored the use of machine-learning to tackle the test oracle problem. Specifically, we proposed DARIO (Dispatching AlgoRIthm Oracle), a test oracle that relies on regression learning algorithms to automatically validate elevators dispatching algorithms. Instead of using regression oracles, DARIO trains a machine learning algorithm by using data from a previously tested dispatching algorithm version to detect functional performance bugs. In this paper, besides functional bugs, we explore whether DARIO is capable of detecting non-functional performance bugs. Our approach provides several advantages. Firstly, the training process of the employed regression learning algorithms is much faster than employing a regression test oracle. While the regression learning algorithms used in DARIO take a few seconds to train, the regression test oracles used in practice may take from minutes to hours at the SiL test phase and days at the HiL test phase. Secondly, DARIO can be used as streamlined oracle at SiL, HiL and in operation, permitting the detection of potential inconsistencies within the real system. Thirdly, DARIO returns a quantitative verdict value over the simulation time, which provides information of how close the system was from failing at each simulation step or how severe a fault was. This opens the possibility to include new verification and validation paradigms in the context of dispatching algorithms, including falsification-based test generation [51, 56]. It can also be used for on-line testing, which could save a significant amount of testing time by stopping the simulation if a severe fault is detected, as proposed by Menghi et al. in a recent study [52]. This paper is an extension of our AST 2021 paper[3]. Specifically, we build upon the conference-version paper [3] from the the following perspectives: •We adapt DARIO, our test oracle that works on top of machine-learning algorithms to detect both, functional and non-functional performance bugs. We show that our approach is effective in both cases by carrying out a new empirical evaluation based on performance mutation testing. 1Notice that unitary tests are different to those unit tests, which aim at testing individual software modules Gartziandia et al 3 •We significantly extend the background and related work sections. Specifically, in the background section we provide a distinction between two types of performance metrics for CPSs (i.e., non-functional performance and functional performance). In addition, we include a new section to explain background theory related to machine-learning algorithms. •We extract new lessons learned and open challenges, which can be used by future researchers as starting point for devising novel approaches. We can summarize our contributions as follows: •We propose DARIO, a novel approach for building test oracles for performance tesing of dispatching algorithms. DARIO works on top of regression machine-learning algorithms. These algorithms aim at predicting the reference performance metric (either, functional (e.g., AWT) or non functional (e.g., execution time) values of a system of elevators based on the passenger traffic data. •We perform an empirical evaluation by using an industrial case study provided by Orona, one of the leading elevator companies in Europe. In this evaluation, we used mutation testing to determine how accurate DARIO was when compared to traditionally employed regression test oracles. We compared five regression learning algorithms. For functional performance bugs, the Regression Tree algorithms performed the best overall. For non-functional performance bugs, there was no a clear winner, but all the algorithms performed well. •We provide a set of lessons learned and open challenges from applying DARIO in practice. The rest of the paper is structured as follows: Section 2 summarizes the relevant background on performance testing and machine learning. Section 3 provides a description of the case study and the current software testing process at Orona. Section 4 presents our approach to automatically test elevators dispatching algorithms. Section 5 evaluates our approach by means of an empirical evaluation with an industrial case study. Section 6 discusses the results, the lessons learned and the open challenges based on the performed evaluation and the discussions we had with domain experts. We position our work with the current state of the art in Section 7. Lastly, Section 8 concludes our paper. 2 BACKGROUND 2.1 Performance metrics for cyber-physical systems The analysis of the system performance has not been as widespread as the analysis of functional requirements, as deviations in performance may not be as harmful as functional failures in most cases. Furthermore, the performance of a system is often harder to verify due to the difficulty to evaluate this type of properties, combined with a lack of specific requirements in many cases [77]. This makes it necessary to integrate performance analysis activities in all of the system’s life-cycle stages, from the system design to operation [17], by employing mechanisms such as static code analysis, performance testing, and run-time monitoring [58, 43]. Ideally, performance requirements should be defined in a requirements document in a concrete, verifiable manner, such that the compliance of the system can be evaluated accurately. Nevertheless, in practice, these requirements are often not defined or ambiguous. Some challenges of performance requirements specification are establishing requirements that can actually be checked in a concrete and precise manner and selecting the appropriate inputs to use when verifying each requirement [77]. The lack of verifiable specifications makes it difficult to assess whether the performance of the system is acceptable or not. This difficulty in verifying the validity of the observed system behavior is known as the test oracle problem, and it is recognized as one of the fundamental problems of software testing [12]. We make a distinction between two types of system performance: Non-functional performance and functional performance. While nonfunctional performance metrics only measure the efficiency of a system’s implementation, functional performance metrics are derived (indirectly) from the system output. We argue that both types of properties share many aspects in common in the context of testing and verification, and that performance testing techniques can be applied to either of them in most cases. In this case, this can be considered either functional or nonfunctional testing based on whether we use functional or non-functional performance metrics. Although performance testing is usually considered non-functional testing, we must consider that the functional performance properties of the system are derived from its output. 2.1.1 Non-functional performance Non-functional performance refers to the usage of computational resources of the system, such as response time, CPU usage, memory usage or energy consumption [20]. The non-functional performance of the software in domains such as web pages or mobile apps is very relevant, as performance issues can affect the usability of the system, causing a loss of users and/or money. For CPSs, this type of properties are an even larger concern, since some non-functional performance issues can escalate to major functional failures, and even safety implications; For instance, an 4 Gartziandia et al Unmanned Aerial Vehicle (UAV) may run out of batteries during its flight due to an excessive energy consumption, resulting in a loss of functionality, and potentially some additional damage from a crash if there are no additional safety measures. 2.1.2 Functional performance Functional performance refers to the properties derived indirectly from the output of the system, rather than the system’s efficient usage of the computational resources. This type of properties are typically used for evaluating the adequateness or quality of a system’s output due to the lack of a more direct way to evaluate its behavior. As an example, an optimizing compiler’s output can be evaluated based on the non-functional performance of the generated program (e.g. execution time), but such properties are actually derived from the functional behavior of the compiler (the code it outputs), and not its non-functional performance (e.g. the time it takes for the compiler to execute). Similarly, CPSs such as multielevator systems are typically evaluated based on Quality of Service (QoS) metrics such as the waiting times for the passengers, which are mostly derived from the functional behavior of the system (call assignments to elevators) rather than its non-functional performance. 2.2 Machine-learning algorithms Machine-learning is a sub-field of Artificial Intelligence (AI) that provides the software with the ability to “learn” from observed data with the goal of making decisions without being explicitly programmed to do so [44]. These approaches construct models by observing data using statistical analyses, where learning begins with the search for patterns inside the data used for training the algorithm [25]. Machine-learning can be categorized in the categories of supervised, unsupervised and reinforcement learning [2]. In supervised learning, the training data consists of some input vectors with their respective labels, which indicate the output expected for each input vector [2]. The objective of these algorithms is to learn to predict the corresponding output for the given inputs, which could have been unforeseen to the algorithm before, by modifying their internal values [44]. Within this category we can distinguish two subcategories: (1) classification algorithms, where the aim is to map input values to a finite number of categories and (2) regression algorithms, where the output values consist of continuous variables. Unlike supervised learning algorithms, unsupervised learning algorithms do not have labeled data. Their goal is to cluster the data by identifying patterns on the similarities and dissimilarities [25]. Lastly, reinforcement learning approaches aim at selecting actions with certain goals [25]. They use feedback on the effect of the action taken (i.e., rewards), which is later used to improve the estimation and iteratively perform more effective actions [25]. In our study, we use supervised learning algorithms aimed at solving a regression problem. 3CASE STUDY 3.1 The System Under Test Elevators are CPSs composed of different subsystems that collaborate to transport passengers vertically within a building. Figure 1 shows the architecture of a system of elevators from Orona. Each elevator is managed by a controller, which is responsible for managing both vertical (floor to floor) and horizontal (doors opening and closing) movements of each elevator. The traffic master is the software system that coordinates the controllers to handle the calls made by the passengers. This system is responsible for (1) the execution of the dispatching algorithm (i.e., the allocation of passenger calls to the available cars), (2) the overall system signaling (i.e., registration of the calls, information to the passengers, etc.), and (3) additional functionalities such as access control (i.e. permission for the passengers to access certain floors) or management of special operating modes. Locally, the communication among the different controllers is done through a Controller Area Network (CAN) bus. The CAN bus is an industrial field bus, originally developed by Bosch for the automotive domain, which allows for a real-time communication for distributed embedded systems. In addition, the traffic master can be connected through Ethernet to external computers in order to enable some advanced features (e.g., passenger access control, remote system configuration, etc.). The System Under Test (SUT) of this paper is the traffic dispatching algorithm. This algorithm is executed periodically to select the optimal elevators to serve the active floor calls. It receives the passenger calls, access control information, and the status (position, direction, etc.) of each elevator as inputs, and then determines which the best elevator to attend every call is. This is done by considering different criteria, such as the reduction of the Average Waiting Time (AWT), the Journey Time (JT) or energy consumption. The AWT refers to the average time that passengers wait until they enter in the elevator, and the JT is the average time that passengers wait inside the elevators until they reach their destination. Both are functional performance metrics used to measure (and test) new versions of the traffic dispatching algorithm. Depending on the algorithm, already allocated floor calls can be reallocated (assigned to another car) to optimize the overall cost function. Due to its effect on the aforementioned fitness criteria, this component is critical to ensure that an elevator installation performs correctly. Gartziandia et al 5 FIGURE 1 Overview of the architecture of an elevator installation from Orona [27] The traffic algorithm should allocate the complete set of active landing calls within a limited time frame (response time), but it has limited computational resources and is executed within a task that shares these resources with other tasks that provide the functionalities mentioned above (signaling, access control, etc.). This fact, alongside with situations such as a highly demanding traffic profile or a big building with many floors and elevators, can result in the consumption of the computational resources available for the allocation of landing calls. This means that, when developing new dispatching algorithm versions, it is relatively easy to introduce non-functional bugs. Consequently, new approaches to detect these kind of bugs are necessary, and is increasingly becoming an internal priority for Orona. 3.2 Software Testing Process at Orona Figure 2 shows an overview of the software development process that Orona has established for dispatching algorithms [7]. The entire development process consists of a total of ten steps divided in different phases: Phase 0: In the initial phase, the process of developing a new dispatching algorithm starts by taking the necessary pre-development actions to define the characteristics of the new algorithm. The main step during this step is the requirement elicitation. •Step 1 – Requirements elicitation: The first step in the software development and maintenance process refers to elicitation of the requirements for the new dispatching algorithm version. This is carried out by the dispatcher manager and her team. Usually, requirements are defined based on (1) new legislative standards from the elevation domain that require changes and (2) new customer needs. Customer needs might be different depending on the type of building where the elevator needs to be installed. For instance, a hospital might require a special emergency button that will enable transporting critical patients through the building. After this requirement elicitation process, a validation plan is obtained as an output. This plan is later used throughout the testing process, ensuring the compliance of the newly developed system. Phase 1 – Software-in-the-loop phase: In this phase the new algorithm is developed according to the established requirements and when it is ready for execution it is tested in SiL environment, where all the components of the plant are simulated, permitting a fast execution to find errors in an early stage. •Step 2 – Development: The second step refers to the development of the dispatching algorithm. The dispatcher manager assigns a specific task to one or more developers, which usually consists in implementing some change in one of their existing dispatching algorithms. This development could be either due to (1) a change in the requirements (taken from a previous step) or (2) due to a required change that has been identified (e.g., a bug or a potential improvement). Usually, while the engineer develops the new software version, small (manual) 6 Gartziandia et al validations are carried out with a tool named Elevate [59]. This tool allows simulating executions of the software dispatching algorithms at the system level. •Step 3 – Testing (SiL): When the engineer considers that the new software version implements the change correctly, the change needs to be checked for issued (e.g., bugs, performance regressions, etc.). At this stage, the tests are executed at the Software-in-the-Loop test phase. In this case, the SUT is the real dispatching algorithm, but the rest of the system (i.e., cabins, engines, communication buses, etc.), are simulated in a local Personal Computer (PC). Two kinds of tests are performed in this step: (1) short scenario tests and (2) full-day tests. In the short scenario tests, specific functional properties are tested in the most isolated way possible; the expected outcome of a test in this case is obtained by either implementing simple assertions or manually. In the full-day tests, scenarios that mimic a normal full-day (or subscenarios of it) in the life-cycle of the system of elevators are executed. The expected outcome of these tests is related to certain functional performance metric values over time windows obtained by re-executing the test with a different algorithm or with an older version (i.e., a regression oracle). The objective at this stage is to detect functional performance bugs in the code. Phase 2 – Hardware-in-the-loop phase: In this phase the simulation environment is substituted by an hybrid environment where some hardware components are included so that the SUT can interact with them. This results in a more complex scenario which should be thoroughly configured and the execution of the SUT is in real-time. •Step 4 – HiL Infrastructure Configuration: After testing the software at the SiL level, the next phase refers to the HiL testing. At the HiL level, the software is integrated with the real-time infrastructure, including a Real-Time Operating System (RTOS) and the real target microprocessor and communications. Unlike at the SiL phase, the physical part of the system of elevators is emulated in real-time with appropriate HiL test benches. The fourth step is the first step at this phase, and encompasses the configuration of the HiL infrastructure. For instance, the necessary controllers need to be configured depending on the type of building that the test needs to be executed in. •Step 5 – SUT & Test Artifacts Deployment: After the HiL infrastructure has been configured, the SUT and the test artifacts need to be deployed. On the one hand, the SUT is deployed and integrated with the remaining hardware infrastructure. On the other hand, the test artifacts (e.g., test oracles, test inputs, etc.) are deployed. •Step 6 – Testing (HiL): The sixth step refers to the execution of the test at the HiL test bench. The tests at this step are performed in realtime, and their goals are, on the one hand, to detect non-functional bugs and, on the other hand, to detect functional bugs that are related with features that were unverifiable at the SiL level due to them not being present in that environment (e.g., VIP calls). As in the third step, both short-scenario tests and full-day tests are being carried out at this stage. When all the tests have been carried out at the HiL phase, the next steps consist in the deployment and testing of the new software version in operation. Phase 3 – Operation phase: In the last phase, the dispatching algorithm is deployed and executed in the real installation, and its performance is monitored and analyzed. •Step 7 – System deployment over real scenario: Within the seventh step, the new software version is deployed in the real system. To that end, a maintainer travels to the installation and deploys the new software version. After that, a set of manual tests are carried out by the maintainer in order to ensure that everything is functioning properly. •Step 8 – System Operation: After that, the system operates normally. Within this phase, there are no specific actions taken by Orona’s personnel. •Step 9 – Monitoring and feedback: The ninth phase relates to monitoring the behavior of the system at run-time. In addition, the maintainer might travel to the installation to check how the system behaves at specific high-traffic hours, noting issues such as queues being formed in the lobbies, etc. •Step 10 – Analysis of feedback data: Feedback is sent back to production, and the dispatcher manager analyzes it to decide whether specific corrective actions need to be taken. The test oracles play an integral role in three main stages. First, in the stage 3, where different test cases are executed at the SiL test phase. Second, in stage 6, where the tests are carried out at the HiL test phase. Lastly at stage 8, where the system is in operation, but some simple rules can be checked. A test case in the context of Orona for testing a dispatching algorithm version refers to the following fields: •Building installation: It is the context configuration at which the SUT is being executed. It has different fields, such as the number of floors the building has, the number of elevators, which floor is served by each elevator, etc. For each elevator, there are also different fields, such as the maximum load, the energy it consumes, dynamic information (e.g., speed, acceleration and jerk), etc. Gartziandia et al 7 FIGURE 2 Software development process of Orona’s dispatching algorithms [7] •Test input (call list profile): a test input in this context refers to a *.txt file that includes a list of passengers. For each passenger, this file includes (1) the arrival time (i.e., when the passenger requests an elevator), (2) arrival floor, (3) destination floor, (4) weight of the passenger, (5) capacity factor by mass, (6) the loading time, (7) the unloading time and (8) information related to the behavior of the passenger when not all elevators serve all floors. •Expected output: Based on the input, what should the behaviour of the system be. This differs depending on the type of testing done. At unitary functional level, the expected output is typically related to a functional behavior of the elevator (e.g., elevator number 1 attends calls from passengers 1 and 2, elevator number 2 attends a call from passenger 3). It is important to reiterate that unitary functional level tests are not unit tests. Unit tests are carried out to test specific function, whereas unitary functional level tests are short tests done with the integration of the entire dispatching algorithm. The goal of these unitary tests is to ensure that functional requirements are met. For long full-day tests, functional behaviour is related to certain QoS metrics, which is addressed in this paper. Furthermore, at the HiL and at Operation, the non-functional metrics are also considered. Specifically, in our study, we considered the execution time of the dispatching algorithm. At unitary level tests, the aim is to test specific scenarios and verify that the functional requirements of the system are met. As mentioned, these unitary tests are different to unit tests, which aim at testing individual software modules. The expected output is typically related to a functional behaviour of the elevator (e.g., elevator number 1 attends calls from passengers 1 and 2, elevator number 2 attends a call from passenger 3). For long full-day tests, functional behaviour is related to certain QoS metrics, which is addressed in this paper and non-functional metrics are also considered. This paper focuses on the oracles applied for, what is known in Orona internally as “long full-day tests”. In these kinds of tests, the test inputs encompass long full-day passenger data that simulate the passenger flow in an installation. During the execution of these tests, the expected output refers to a time series of functional and non-functional performance metric values over the time. Typical functional performance metrics include the Average Waiting Time (AWT), Average Time to Destination (ATTD) or the consumed energy. Traditionally, the most used performance metric in Orona has been the AWT, because, according to certain studies, it is the most sensitive measure for a passenger to determine whether a system of elevators performs well or not [11]. Regarding non-functional performance, the most critical metric to check in dispatching algorithms is the response time, as these algorithms have a limited time frame to give a response. These test cases can be differentiated into two groups: based on theoretical passenger data and (2) based on real passenger data. On the one hand, theoretical passenger data based test cases provide test inputs based on theoretical studies of passenger flows in buildings. For instance, Figure 3 depicts a graph showing the number of up calls, down calls and inter-floor calls in a time window of five minutes for a simulation of 13 hours based on the Siikonen theory for a building of offices. As can be seen, in the first couple of hours, the number of up calls increases as passengers are arriving at the office. After 5 hours, a pattern is seen with an increase in the number of down calls followed by an increase in the number of up-calls, which represents the lunch time break. At the end of the day, there is another down peak, representing the end of the working day. The advantage of theoretical traffic profiles is that they can be easily obtained by using tools like Elevate [59] just by providing data of the building population (e.g., how many people work at each floor). However, a negative aspect of the theoretical profiles is that they might not accurately represent the results obtained during real operation, where unforeseen situations arise frequently. 8 Gartziandia et al (a) Full day profile collected from real installation by Orona (b) Siikonen Full day profile FIGURE 3 Passengers activity of real installation and theoretical profiles obtained with Elevate On the other hand, Orona uses data obtained from real installations. This helps with the validation of dispatching algorithms from several perspectives, such as the identification of certain traffic patterns not considered in theoretical traffic profiles or a higher customized validation. For instance, this could happen in a building where the canteen is on the top floor. Then, the traffic profile would significantly change as there would be a second peak in the number of up-calls followed then by an increase in the number of down-calls. An algorithm could also perform better than another depending on the traffic profile, and subsequently, the use of test inputs obtained from the field is a powerful method to validate these algorithms. The advantage of using real traffic profiles is that developers can use actual data from what it is happening in operation. This can help identify specific traffic patterns that can be found in a building. On the downside, obtaining real traffic profiles can be a non-trivial task, it requires the system to be in operation, and it is usually not possible to use this data for new installations. In Figure 3, the differences between a real and a theoretical traffic profile for a building in Madrid can be seen. One discrepancy that can be observed is that the interfloor calls are not as frequent in the real passenger profiles when compared to the theoretical ones. In addition, at the lunch-peak, a high peak of outgoing passengers can be seen at around 14:00 in the real passenger data, which is a pattern that is not present in the theoretical data. 4 REGRESSION LEARNING-BASED TEST ORACLE FOR ELEVATORS DISPATCHING ALGORITHMS Figure 4 shows the overall architecture of the proposed approach. The proposed solution is divided into two main phases: (1) the training phase and (2) the testing phase. During the training phase, a regression machine-learning algorithm is trained by using data from previous software versions. This data is the one cataloged by Orona as reference to validate other versions of the software. Subsequently, it is considered that the data used for training is from an error free version of a dispatching algorithm. The machine-learning algorithm yields a model, which is used by DARIO in the testing phase. During the testing phase, a test is executed at certain test phase (e.g., SiL or HiL). For instance, if it is at the SiL test phase, Elevate executes a test case by using test input data, information of the building installation (i.e., speed of elevators, structure of the building, etc.) and the SUT itself. When the test has finished, files containing (1) the passenger traffic data and (2) the performance measures over the simulation time (i.e., the test outcomes, which are either, those related to functional performance (e.g., AWT) or non-functional performance (e.g., execution time)) are collected. It is noteworthy that DARIO is applicable at the SiL test phase for detecting functional performance bugs, whereas at the HiL test phase it can be used for both functional and non-functional bugs. This is because, for the case of non-functional bugs, the SUT needs to be executed in the real target processor, which is not available at the SiL test phase. The performance measures (i.e., test outcomes) may be be affected by a poor implementation of the dispatching algorithm from both a functional or non-functional perspective. From the functional point of view, when testing dispatching algorithms at system level, the main performance measure considered to label a test as PASS or FAIL by test engineers is the Average Waiting Time (AWT). This metric measures the average time each passenger needs to wait until their elevator arrives. The AWT can be a global value that measures the overall AWT for all passengers in the test input or a signal over the simulation time, indicating the AWT of the passengers in the test input for a specific time period. Elevate provides information on both. Regarding non-functional attributes, it is important for the dispatching algorithm to provide a response in a limited time frame, Gartziandia et al 15 •FP: The test oracle (i.e., DARIO) returned a “FAIL” and the regression test oracle returned a “PASS”. The tests were executed in Elevate instead of in the HiL due to practicality (i.e., if the tests were executed using the HiL test bench, the experiments would take around 2 years). 5.1.4 Experimental Scenarios To answer the proposed RQs, a total of four experimental scenarios were designed, where we aimed to analyse the effect of different training and testing data on the results. Note that the names of the proposed scenarios follow the pattern [Training data type-Testing data type]: •Scenario Theoretical-Theoretical: We first used test cases that involve theoretical test inputs. These test inputs were automatically generated by Elevate, and are based on a study performed by Siikonen [67]. In total, 10 of these test cases were used. We employed the 10-fold cross validation to validate DARIO along with all the selected machine-learning algorithms. •Scenario Theoretical-Real: In the scenario Theoretical-Real, the same type of test cases were used for training, but for testing, test cases obtained from the real installation were used. This scenario would emulate (1) how the theoretical passenger data performed in order to train the algorithms during validation when using data extracted from operation and (2) how the theoretical passenger data perform for training the algorithms when the oracle is used in the real installation. •Scenario Real-Real: In this scenario, we used test cases obtained with data extracted from the real installation in operation for training. In total, four test cases were available for the building used in the evaluation. We thus employed the 4-fold cross validation to validate DARIO along with all the selected machine-learning algorithms. •Scenario Real-Theoretical: In this case, we used the same test cases as in scenario Real-Real to train the algorithms, but for testing we used the ten theoretical passenger-based test cases. Table 1 summarizes the main characteristics of the test cases used in these four scenarios, including, the number of up calls, down calls, number of detected mutants by the test case, and the test case duration. TABLE 1 Main characteristics of the used test cases during the experimental scenarios Test case # of Up Calls # of Down Calls # of Detected AWT Mutants # of Detected Performance Mutants Simulation time (h:min) real1 2756 1711 18 13 8:30 real2 3086 2366 18 14 9:10 real3 3438 3117 18 10 11:45 real4 3508 3050 21 12 13:35 theoretical1 3994 3377 20 11 12:55 theoretical2 3950 3379 18 13 12:55 theoretical3 3983 3379 26 14 12:55 theoretical4 3989 3402 18 20 12:55 theoretical5 3989 3387 18 12 12:55 theoretical6 3964 3384 19 11 12:55 theoretical7 3977 3386 21 14 12:55 theoretical8 3919 3433 21 12 12:55 theoretical9 3976 3354 18 15 12:55 theoretical10 3945 3407 20 14 12:55 5.2 Results We now present the results obtained for the four scenarios designed to answer the RQs. 16 Gartziandia et al 5.2.1 RQ1 – Functional performance Table 2 summarizes the obtained results for the four scenarios designed to answer the RQs. In the scenario Theoretical-Theoretical a 10-fold cross validation was performed to measure how DARIO performed when trained ad tested with theoretical traffic profiles. In terms of precision, SVM and regression tree were the techniques showing best results (0.98) followed by stepwiselm, RGP and ensemble. When considering the recall measure, ensemble performed best (0.89), followed by the rest with values around 0.80. In terms of accuracy and F-measure, SVM and regression tree performed well, unlike the remaining three machine learning algorithms, which all dropped below 0.8 in terms of both accuracy and F-measure. Both SVM and regression tree performed best in terms of the Specificity metric, both having a value of 0.99. In contrast, the rest of the algorithms showed a value of less than 0.8. With regards to balanced accuracy, regression tree showed a value of 0.91, followed by SVM, which had a value of 0.88. The remaining algorithms showed a value lower than 0.8. TABLE 2 Summary of results for the four experimental scenarios for functional mutant detection Scenario Metrics SVM Regression Tree Ensemble RGP STEPWISELM Theoretical-Theoretical Precision 0.98 0.98 0.68 0.76 0.83 Recall 0.77 0.82 0.89 0.78 0.79 Accuracy 0.94 0.95 0.69 0.71 0.79 F-1 0.86 0.89 0.70 0.68 0.76 Specificity 0.99 0.99 0.62 0.70 0.79 Balanced Accuracy 0.88 0.91 0.75 0.74 0.79 Theoretical-Real Precision 0.95 0.81 0.41 0.61 0.41 Recall 0.66 0.81 0.86 0.86 0.88 Accuracy 0.96 0.76 0.40 0.58 0.39 F-1 0.76 0.73 0.46 0.60 0.45 Specificity 0.98 0.75 0.27 0.50 0.25 Balanced Accuracy 0.82 0.78 0.56 0.68 0.56 Real-Real Precision 0.82 0.97 0.30 0.59 0.59 Recall 0.88 0.94 1 0.88 0.83 Accuracy 0.93 0.98 0.37 0.58 0.57 F-1 0.85 0.95 0.45 0.62 0.59 Specificity 0.94 0.99 0.17 0.50 0.50 Balanced Accuracy 0.91 0.97 0.59 0.69 0.67 Real-Theoretical Precision 0.25 0.25 0.25 0.30 0.25 Recall 1.0 1.0 1.0 0.99 1.0 Accuracy 0.25 0.25 0.25 0.37 0.28 F-1 0.39 0.39 0.39 0.45 0.40 Specificity 0.0 0.0 0.0 0.17 0.04 Balanced Accuracy 0.50 0.50 0.50 0.58 0.52 As for the scenario Theoretical-Real, where the ten theoretical passenger data based test cases were used for training the machine learning algorithms but real-world data for testing, in terms of the average precision, in this case SVM performed best, followed by regression tree. Results for the ensemble, RGP and stepwiselm algorithms in terms of precision were overall quite low. These three algorithms, however, slightly outperformed SVM and regression tree for the recall metric. Nevertheless, in terms of accuracy and F-measure, SVM showed the best results followed by regression tree, outperforming the remaining three algorithms with significant difference. The specificity and balanced accuracy measures had a similar distribution, with the superiority of SVM followed by regression tree and far from them the rest of the algorithms. Scenario Real-Real in Table 2 shows the results for the 4-fold cross validation when using the real passenger data based test cases both, for training and for testing. As can be seen, in this case, regression tree performed best in terms of precision, accuracy and F-measure. Furthermore, with a recall of 0.94, the regression tree algorithm was the second best, after ensemble. Nevertheless, the precision, accuracy and F-measure values for ensemble were all below 0.5, meaning that this algorithm had a high number of false positives. In terms of specificity and balanced accuracy, the algorithm which performed best is regression tree, with the values 0.99 and 0.97 respectively, followed by SVM, with an specificity of 0.94 and Gartziandia et al 17 a balanced accuracy of 0.91. Far from there, we could found the remaining three algorithms, which showed values below 0.7 for both specificity and balanced accuracy. Scenario Real-Theoretical also used real passenger data based test cases to train the machine learning algorithms. Although all algorithms performed well in terms of recall, meaning that they produced none or a low number of false negatives, their results in terms of precision, accuracy and F-measure were below 0.5. The values were also quite low for both the specificity and balanced accuracy for all the algorithm. All this means that a high number of false positives were produced when following this strategy for training the algorithms but testing the dispatching algorithms with theoretical data. When training with theoretical traffic profiles, the results for all the four measures in scenario Theoretical-Real were lower than those shown in scenario Theoretical-Theoretical. Our hypothesis behind this is related to the difference between the types of passenger traffic flow in the test cases that are based on theoretical traffic profiles and the ones obtained from the real installation. It might be possible that the theoretical passenger profiles do not explore areas which the real passenger profiles actually do. This could be the case, for instance, when the canteen or the bar is on a specific floor. The theoretical traffic profiles also make assumptions that might not hold for all offices building. For instance, the theoretical traffic profiles assume that there are lunch peaks from 12:00 to 15:00, where workers from an office building go to have lunch. Nevertheless, there might be companies and buildings where most of the workers have a continuous work day from 7:00 to 15:00. The hypothesis is similar when training with real traffic profiles. The traffic profiles obtained from the real building installation might not exercise areas or produce situations that are considered in the theoretical traffic profiles. This makes it difficult for the regression algorithms to accurately predict the reference AWT value when trained with real traffic profiles, but tested with theoretical traffic data. Based on the obtained results, we can answer the first RQ as follows: SVM and regression tree algorithms were the algorithms performing best when trained with theoretical passenger profiles based test inputs. When training with real traffic profiles, the regression tree algorithm stood out over the rest. Their results were acceptable when training and testing with the same kind of traffic profile. However, the performance of the algorithms decreased when trained with theoretical traffic profiles and tested with real traffic profiles and vice-versa. 5.2.2 RQ2 – Non-Functional performance Table 3 summarizes the results for the non-functional performance mutant detection. In the scenario Theoretical-Theoretical, where a 10-fold cross validation was performed to measure the performance of DARIO when trained and tested with theoretical profiles, we saw that overall, the five algorithms showed competitive results. In terms of precision, recall, accuracy, and specificity, SVM, Regression Tree, RGP and Stepwiselm showed the most competitive results, with values ranging between 0.81 and 0.87. On the other hand, Ensemble showed the most competitive results when considering the recall metric (0.96). Nevertheless, its precision and specificity were worse than the rest of the algorithms (0.71 and 0.45 respectively), which also penalized the Accuracy and the F-1 measures (i.e., 0.75 and 0.82 respectively), although these results can still be considered acceptable. In terms of Accuracy and F-1, RGP obtained slightly better results than the rest of the algorithms, although the differences are minimal. In the scenario Theoretical-Real, the machine-learning algorithms were trained with theoretical passenger profiles, but tested with real passenger profiles. Similar to what happened in scenario Theoretical-Theoretical, the results for SVM, Regression Tree, RGP and Stepwiselm were similar. All of them showed good performance in terms of precision and specificity (between 0.93 and 0.94 for the former and 0.95 for the latter), but their recall measure decreased (0.57 for SVM, Regression Tree and StepwiseLM, and 0.67 for RGP). Their accuracy and F-1 measure remained relatively high (i.e., between 0.75 and 0.79 for the accuracy metric and between 0.71 and 0.79 for the F-1), although the results were worse than those from scenario Theoretical-Theoretical. Conversely, the Ensemble algorithm obtained different results. While its precision and specificity were lower, but still competitive (0.9 for both), the rest of the metrics were higher: a recall of 0.8, an accuracy of 0.85, and an F-1 measure of 0.85. Within the scenario Real-Real, where real passenger data profiles are used for both training and testing, and therefore a 4-fold cross validation used, all the algorithms showed a similar performance. In fact, all of them showed the same precision and specificity of 0.95, which is highly competitive. In terms of recall, accuracy and F-1 measure, the Ensemble showed slightly better performance than the rest, although the differences were minimal. All these values were over 0.8 for the recall, and over 0.87 for accuracy and F-1. This means that all the algorithms are highly effective within the scenario Real-Real. The last scenario trained the machine-learning algorithms with real passenger data and used theoretical passenger data for testing. All the five machine-learning algorithms showed similar results. As compared to scenario Real-Real, the precision, accuracy, F-1 and specificity metrics dropped, while the recall measure was increased to 0.98. The differences among the algorithms were small, although RGP and Stepwiselm obtained slightly better precision, accuracy and F-1 values when compared with the other algorithms. In general, despite having worse results than in scenario Real-Real, the results of the algorithms were still competitive. 18 Gartziandia et al TABLE 3 Summary of results for the four experimental scenarios for performance mutant detection Scenario Metrics SVM Regression Tree Ensemble RGP STEPWISELM Theoretical-Theoretical Precision 0.87 0.86 0.71 0.87 0.86 Recall 0.84 0.84 0.96 0.87 0.84 Accuracy 0.83 0.83 0.75 0.84 0.83 F-1 0.85 0.85 0.82 0.87 0.85 Specificity 0.82 0.81 0.45 0.81 0.81 Balanced Accuracy 0.83 0.83 0.71 0.84 0.83 Theoretical-Real Precision 0.93 0.93 0.90 0.94 0.93 Recall 0.57 0.57 0.80 0.67 0.57 Accuracy 0.75 0.75 0.85 0.79 0.75 F-1 0.71 0.71 0.85 0.79 0.71 Specificity 0.95 0.95 0.90 0.95 0.95 Balanced Accuracy 0.76 0.76 0.85 0.81 0.76 Real-Real Precision 0.95 0.95 0.95 0.95 0.95 Recall 0.80 0.80 0.82 0.80 0.80 Accuracy 0.87 0.87 0.88 0.87 0.87 F-1 0.87 0.87 0.88 0.87 0.87 Specificity 0.95 0.95 0.95 0.95 0.95 Balanced Accuracy 0.88 0.88 0.89 0.88 0.88 Real-Theoretical Precision 0.75 0.73 0.73 0.75 0.75 Recall 0.98 0.98 0.98 0.98 0.98 Accuracy 0.79 0.77 0.77 0.80 0.80 F-1 0.85 0.84 0.84 0.85 0.85 Specificity 0.52 0.48 0.48 0.53 0.53 Balanced Accuracy 0.75 0.73 0.73 0.76 0.76 In the scenarios where theoretical data was used for training, similar to what happened in RQ1, the overall results of scenario such scenario (i.e., Theoretical-Real) were worse than those from scenario Theoretical-Theoretical. The same happened with scenarios where real data was used for training, where results from scenario Real-Theoretical were lower than those from scenario Real-Real. This means that the training of the algorithms should be done with the same type of passengers profiles that will later be used for testing. Nevertheless, the overall results were not as low as those from the previous RQ. For instance, in scenario Theoretical-Real, the Ensemble algorithm was competitive enough regarding all the five metrics to be usable in practice. In addition, the results for all the algorithms in scenario Real-Theoretical were quite competitive. Furthermore, in this case, there was not one specific algorithm that stood out over the rest, which means that any of them could be used in practice. Therefore, we can answer the second RQ as follows: Overall, all the algorithms performed well when dealing with the problem of non-functional bug detection. Their results were better when training and testing with the same kind of traffic profile. Nevertheless, when training with one traffic profile and testing with the other, the results were still quite competitive. 5.3 Threats to Validity We now discuss internal and external validity threats of the performed evaluation: Internal validity: A potential internal validity threat in our study might be related to the thresholds of the arbiter we designed, which are configurable. To reduce this threat, we discussed the parameters with domain experts to see which thresholds could be appropriate to consider a test as pass or fail. The same concern applies to the parameters of the selected machine-learning algorithms. To reduce this threat we used the default parameters from the MATLAB framework for training the algorithms. Another internal validity threats relate to the usage of a 4-fold cross validation when using the “real-real” scenarios. This was due to the fact that those cases use data obtained from the real buildings (i.e., operational Gartziandia et al 19 data). Unfortunately, our industrial partner did not have more operational data. However, we believe that this scenario is realistic given that the it is the number of test cases that the company has. Moreover, it is important to note that each test input has several data points of full-day traffic profiles, meaning that we have sufficient data to train the algorithms. External validity: An external validity threat in our evaluation is related to using a single benchmark dataset based on test cases for testing dispatching algorithms. To reduce this threat, the dataset was obtained from actual test cases in Orona for testing dispatching algorithms. Furthermore, to avoid bias in the results, we did not use the same dataset for training an algorithm and for testing it, using the appropriate k-fold cross validation techniques in those scenarios where this was necessary (i.e., Exp. 1 and Exp. 3). Another external validity threat relates to the used case study. Although only a single case study was used, it is important to note that it is a real industrial case study, which provides a high degree of complexity to our evaluation. Furthermore, the used dispatching algorithm is the one which is most used in Orona’s elevators. 6DISCUSSION We now discuss the results obtained from the evaluation, which was carried out in an industrial context, the lessons learned from it and the open-challenges. 6.1 Analysis of the Results The evaluation aimed to assess whether DARIO can be an appropriate technique to be applicable in the context of functional and non-functional performance testing of CPSs. To this end, we carried out an empirical evaluation using both functional and non-functional bugs. Traditional mutation operators were used for the former, whereas we used the study by Delgado-Perez et al., [20] for the creation of non-functional performance mutants. Overall, we believe that the results are positive, indicating that our approach can be applicable in our context. For the case of functional faults, we found that the regression tree machine-learning algorithm performed better than the remaining algorithms. On the contrary, for the case of non-functional faults, there was no clear winner among the five machine-learning algorithms we assessed. In addition, we identified the importance of training the algorithms based on the same type of passenger data that was going to be evaluated by the resulting test oracles; when theoretical passenger data is going to be used, the training should be performed with theoretical passenger data too. On the other hand, if real passenger data is going to be verified, we recommend this kind of data to be used for training the algorithms instead. Based on the performed evaluation, we believe that the results of our approach are good enough to be applicable by Orona engineers when performing long full-day tests. Similarly to other works in the field of the test oracle problem [?], we employed the precision, recall, accuracy, F-1 metrics, alongside specificity and balanced accuracy, that are commonly used by the machine-learning community. Nevertheless, a recently performed systematic literature review [25] revealed that many studies used the mutation score metric, which aims at assessing the percentage of mutants killed. In our case, the mutation score mostly depends on the type of test inputs used, rather than the test oracle itself. In fact, we used the same criterion as the regression test oracle, currently used in the context of Orona, for determining whether a mutant is killed or not by a test case. It is important to note that test oracles are prone to false positives and negatives [9, 72], and therefore, we have given priority to the right classification done by DARIO, our test oracle, rather than the amount of mutants detected by it. 6.2 Lessons learned Based on the results obtained from the evaluation section and the discussion with domain experts about the applicability of the approach, we extracted the following lessons: Lesson 1 – Training data: The use of the right data for training the machine-learning algorithms play a critical role in the accuracy of DARIO. Exceptions aside, the results obtained in scenarios Theoretical-Theoretical and Real-Real are much better than the ones of obtained in scenarios Theoretical-Real and Real-Theoretical, as it is shown in the Section 5.2. Therefore, in order for the oracle to be accurate enough, the training data should be of the same type as used when testing the system. This means that when Orona uses theoretical data for testing their algorithms, they should also use theoretical test data for training DARIO. Conversely, if real data is used for testing their algorithms, they should use real passenger data for training DARIO. However, we saw that the impact of this was larger in the case of functional bugs as compared to the non-functional bugs. A potential reason for this could be that the problem of functional-performance is harder to solve due to a higher variability of the AWT metric to test inputs. Another reason could be the experimental set-up, where we used a larger amount of mutants for the case of functional performance testing than the non-functional performance testing. 20 Gartziandia et al Lesson 2 – Machine-learning algorithms: Our empirical evaluation suggested that in the case of functional-performance, there might be large differences among the selected algorithms. In general, the regression tree performed overall best in the first case. Conversely, for the case of nonfunctional performance bugs, most algorithms performed similarly, with some exceptions in some of the cases; for instance, ensemble was the worst algorithm in the scenario Theoretical-Theoretical, whereas the best one in the scenario Theoretical-Real and competitive in the scenarios RealReal and Real-Theoretical against the remaining algorithms. For the application of the tool in industry, for the first case, the regression tree algorithm is recommended. However, further investigation is required before recommending one of the algorithms for detecting non-functional bugs; this might be based on the data used and a set of preliminary experiments done before pushing one model to production. Lesson 3 – Importance of real passengers data: In the study, we showed that the oracles performed competitively when using real data obtained from the real installation (even if they were trained only with three real passenger profiles). This happened in both cases, functional and nonfunctional performance experiements (i.e. RQ1 and RQ2). In other contexts, such as web-engineering, technologies like DevOps permit using data from operation at design-time to enhance software engineering processes (e.g., testing). The good performance of the proposed approach with field test data shows the importance of researching on adapting design-operation continuum techniques (e.g., DevOps) in the context of CPSs and in domains like elevation. The simulation at design-time of situations seen only in operation is of great advantage for engineers. For instance, this permits the detection of unforeseen situations that can only be seen when the system is in operation. Lesson 4 – Need for dealing with uncertainty in test verdicts: As it could be appreciated in our empirical evaluation, the accuracy of DARIO with certain algorithms is competitive. However, using these oracles in Orona increases the uncertainty in relation to the correctness of the verdicts. This might require re-executing tests in the regression test oracle at certain points in order to confirm verdicts. Although this might increase the cost of testing, the test results can also be used to retrain the algorithms. The increase in test execution costs is further exacerbated at the HiL and at operational levels. Potentially, DARIO could be used as preliminary oracle for testing, and use the regression oracle only at critical stages (e.g., to confirm certain test verdicts). For instance, when a test has failed, but this fail was due to the numerical verdict being close to 0 (i.e., a low severity fault), the same test could be eventually repeated with a regression test oracle. On the other hand, if the test has failed, but the numerical verdict was far from 0 (i.e., high severity fault), the test could be classified as fail without the need for confirming the verdict with a regression test oracle. Lesson 5 – Consequences of mistaken oracle: An oracle may be mistaken when providing a test verdict. That is, the test oracle is subject to false positives (FPs) and false negatives (FNs). On the one hand, a false positive means that a test was cataloged as “fail” when it should have been provided a“pass”. This results into, probably, the need of the developer debugging where the potential fault is located. Since there is no fault behind, this may result into time spent by an engineer in debugging a fault that does not exist. This issue could be mitigated by confirming the test verdict by using a regression test oracle. In the cases of functional bugs, this issue could happen in 2 to 3% of the cases, given that the precision of the algorithm Regression tree is between 0.97 and 0.98. Re-running tests to confirm the presence of faults in 3% of the cases is not an issue for our industrial partner. However, these values increase in the case of non-functional performance faults, having false positives in around 14% of the cases in the case of the “Theoretical-Theoretical” scenario. This suggests that more research is required to increase the robustness of such algorithms. On the other hand, a false negative means that a test was cataloged as “pass” when it should have it done as a “fail”. If this happens at designtime (i.e., at the SiL or at the HiL phases), the bug might be missed and shipped to production. As shown in our experiments, this could happen in 18% of the cases if used a “theoretical-theoretical” test oracle. However, the regression tree algorithm showed a recall of 0.94 in the cases of “realreal” scenarios, which implies that all the many of these faults could be later detected. These faults are in the limit and could be partially solved if we consider the uncertainty that the inputs provoke in the ML models. This will be investigated in the future before fully transferring the method to our industrial partner. The current version could be used as first pass at phases 1 and 2 of Orona’s testing process, while traditional oracles can be used for corner cases or to confirm test verdicts. In addition, our approach can be used for operational testing, which already outperforms the current state-of-the-practice in Orona’s context, where this type of test was not possible up to now. 6.3 Open Challenges In this study we have assessed the performance of DARIO, a test oracle that is built on top of machine-learning algorithms to detect both, functional and non-functional bugs. DARIO is a good alternative to a traditionally employed regression test oracle, as it permits new applications, such as testing of the HMI modules or testing at different test phases (i.e., SiL, HiL and Operation). Deploying DARIO at run-time can be feasible at this point for Orona by using an architecture based on microservices [29]. However, at run-time, new challenges arise, such as the inherent uncertainty produced by the environment at which elevators operate. For instance, when executing a test at the SiL or HiL test phases, the weight of passengers is known beforehand to the oracle, which might impact the performance of the algorithms (e.g., when passengers weight less, more passengers can enter at an elevator). Conversely, this is unknown to the oracle at design-time, and therefore, uncertainty-wise techniques would need to be integrated within DARIO. Another uncertainty related issue at operation time could be the wrong use of passengers of the elevator. For instance, one could be that of calling an elevator, but later not using it. All these uncertain situations need to be classified systematically and integrated in DARIO in order this oracle to be effective at operation-time, where a regression test oracle cannot be used. Gartziandia et al 21 Our approach has been exclusively designed for dispatching algorithms. However, performance metrics, both functional and non-functional, can be found in many other CPSs applications, including those from the automotive [35, 74] or aerospace domains [53]. We believe that another line of research could be the application of DARIO in other CPS domains. Furthermore, DARIO has been intended to be used as test oracle for already defined test cases. However, this approach opens the gate to other testing activities. For instance, the use of the numerical verdict to determine whether a failing test is of critical or low severity could be use for activities like falsification-based test case generation. These techniques are expensive to execute [51], and therefore, a technique like DARIO could be of great benefit in such a context. This technique could also be combined with a similar approach of approximation-refinement testing [51] based test generation, in which the tests are only eventually executed in the SUT, and instead, a surrogate model of the new SUT is used. 7 RELATED WORK In this section we give an overview of the related work along the following dimensions: (1) use of machine-learning for test oracle generation, (2) simulation-based testing of CPSs and (3) testing of systems of elevators. 7.1 Use of machine-learning for test oracle generation The use of machine-learning algorithms for software testing activities has significantly increased in the last few years [23], covering many activities, including test case generation [13, 14], test case selection [70] or test prioritization [70, 10, 41]. Test oracles are one of the core components required to allow full test automation. A recent systematic literature review gathered a total of 22 relevant studies that used machine-learning for the purpose of generating test oracles [25]. Among their findings, they identified that machine-learning was used to (1) generate test verdicts [16, 31, 45, 63], (2) generate metamorphic relations [32, 33, 38, 39, 55, 78] and (3) generate the expected outputs [1, 22, 36, 54, 62, 64, 65, 69, 76, 68, 46, 79]. In this classification, our machine-learning algorithms fall into the category of the generation of expected outputs, although the entire test oracle (i.e., DARIO), provides a final test verdict. Just like all the techniques for generating expected outputs identified by Fontes and Gay [25], our machine-learning algorithms use supervised learning based on prior system executions. In addition, similar to all the studies that used a machine learning component to generate the expected outputs (with the exception of [22]), our approach used regression to determine the expected output. However, there are significant differences between our approach and those studies identified in [25]. Firstly, the type of system that our oracle tackles are CPSs, which have many idiosyncrasies, such as the expected output being a signal over time, or the long test execution time. Secondly, we investigate the performance of our approach in the context of both, functional and non-functional bugs. Thirdly, we compared a total of five machine-learning techniques, including SVM, regression tree, ensemble, RGP and Stepwiselm, unlike most of the studies identified in [25], which use a maximum of two techniques. Lastly, our SUT is a real industrial case-study from the elevator domain (i.e., the Dispatching algorithm for Orona elevators); this last point paves the way towards overcoming one of the limitations identified by Fontes and Gay [25] in the context of using machine learning for test oracle generation, i.e., the use of toy examples. 7.2 Simulation-based testing of CPSs Simulation-based testing has been extensively used to test CPSs [51, 52, 48, 49, 50, 13, 14, 26, 53, 66, 24, 34, 75, 6, 47, 4, 5, 6, 73]. Our approach relies on simulation-based testing for detecting performance problems (caused by either functional or non-functional bugs) of elevator systems. In such a context, there are recent studies that tackle the test oracle problem [52, 71]. Menghi et al. proposed a test oracle generation tool for Simulink models [52]. Their approach consists in a Domain Specific Language (DSL) with sufficient expressiveness to specify signal properties-based requirements. Later, a model-to-model transformation is performed in order to generate Simulink subsystems. Similar to our work, their oracles also provide a quantitative measure for the satisfaction degree of a requirement. However, our study is focused on generating a reference signal by a machine-learning algorithm trained with data from previously tested software versions, and later applying and arbitration mechanism. Although we could use their tool to generate oracles by specifying some requirements, this would be infeasible for the context of performance metrics, as inferring the relation between passenger traffic data and functional performance measures (e.g., AWT) as well as non-functional performance measures (e.g., memory consumption) is nontrivial. Furthermore, this relation is highly dependent on the building installation characteristics, and would therefore require a manual change every time the dispatching algorithm is tested in a different context; In contrast, training DARIO with already available data is straightforward and fast. Stocco et al. proposed a technique for testing self-driving cars which use deep neural networks to determine the driving parameters for the actuators of the vehicle [71]. Similar to our approach, their oracle employs simulation-based testing and determines a confidence value for the system at each step of the execution. However, the oracle they propose uses an unsupervised learning 22 Gartziandia et al technique based on the camera images (i.e., the input) of the self-driving car, whereas DARIO uses supervised regression learning based on the QoS measures of a system of elevators (i.e., the test output in our context). 7.3 Testing of systems of elevators There are some studies that focus on testing software from elevators systems. Nicolas et al., proposed an encoder based on FPGAs for simulationbased testing of elevator controllers in real-time [57]; their goal was to test the position and speed of elevators. Sagardui et al., relied on model-based testing and feature models for testing configurable software systems in charge of controlling the doors of elevators [61]; in this paper, the goal was to test refactored embedded code, and the non-refactored software acted as a golden oracle. In these cases the system was not the dispatching algorithm, but other software components of the elevators. In our previous paper, we used the technique Metamorphic Testing to test the dispatching algorithm of Orona [8], showing promising results. A problem with metamorphic testing was the difficulty to extract effective metamorphic relations. To solve this problem, we proposed a technique that combined mutation testing with genetic programming [9]. The generated metamorphic relations were as competent as those defined with the assistance from domain experts [9]. However, that metamorphic testing approach was mainly designed for short-scenario tests, whereas the technique shown in this paper is designed for long-scenario tests. Our previous work used machine-learning algorithms to detect performance bugs on software updates [28]; nevertheless, the evaluation was not as realistic as in this case, since the execution of the algorithms was performed in a local PC, whereas in this case we used the real microprocessor used by Orona in their elevators. In fact, we noticed that the results were significantly different in this case. Galarraga et al., proposed a test case generation method based on genetic algorithms to the dispatching algorithm under uncertain passenger conditions [27]. While there are many studies in the field of elevators dispatching algorithms where artificial intelligence algorithms adapted to this context are investigated (e.g., ant-colony optimization and neural networks [42], genetic algorithms [15]), to the best of our knowledge, this is the first study that proposes a method for testing them (both from the functional and non-functional perspectives). 8CONCLUSION AND FUTURE WORK In this paper we have proposed DARIO, a test oracle that relies on machine-learning to automatically test elevators dispatching algorithms from the perspective of both, functional and non-functional performance. Compared with the traditionally used regression oracles, which have several disadvantages, DARIO trains machine-learning algorithms with previous test data. This training takes only a few seconds (always less than 3 seconds), whereas executing the regression test oracle takes minutes or hours at SiL (depending on the length of the test case), and hours or days at HiL (not being possible to correctly perform some tests, such as those involving HMI). In our evaluation, where an industrial dispatching algorithm from Orona was used, we tested the proposed approach with two kinds of bugs: functional and non-functional. When using functional bugs, the accuracy of the proposed test oracle when labeling tests as PASS or FAIL ranged between 0.79 and 0.87. When using non-functional bugs, the accuracy ranged between 0.8 and 0.87. The type of training data used impacted these results, although it had a greater impact in the case of the functional bugs. Overall, we believe that these results are competent enough to transfer the tool to practitioners, although further investigation is required to enhance these results. As future research lines, we would like to explore handling the uncertainty in oracles from different perspectives. Firstly, as explained in the lessons-learned section, using DARIO increases the uncertainty related to the correctness of the verdicts. In deep learning algorithms, Kim et al., used the surprise adequacy [40]. Similar metrics adapted to the algorithms used by DARIO could be employed to measure such uncertainty. Other relevant uncertainties could be those that the CPS itself is exposed to, especially in operation. In the context of elevators at operation, in order DARIO to be effective, the uncertainty from several perspectives needs to be considered (e.g., uncertainty in passengers’ behaviors). Another example of uncertainty could be related to the hardware itself, such as the noise of the sensors or delays in the communication systems. References [1] Aggarwal, K., Y. Singh, A. Kaur, and O. Sangwan, 2004: A neural net based approach to test oracle. ACM SIGSOFT Software Engineering Notes, 29, no. 3, 1–6. [2] Alpaydin, E., 2020: Introduction to machine learning. MIT press. [3] Arrieta, A., J. Ayerdi, M. Illarramendi, A. Agirre, G. Sagardui, and M. Arratibel, 2021: Using machine learning to build test oracles: an industrial case study on elevators dispatching algorithms. 2021 IEEE/ACM International Conference on Automation of Software Test (AST), IEEE, 30–39. Gartziandia et al 23 [4] Arrieta, A., G. Sagardui, L. Etxeberria, and J. Zander, 2017: Automatic generation of test system instances for configurable cyber-physical systems. Software Quality Journal,25, no. 3, 1041–1083. [5] Arrieta, A., S. Wang, A. Arruabarrena, U. Markiegi, G. Sagardui, and L. Etxeberria, 2018: Multi-objective black-box test case selection for costeffectively testing simulation models. Proceedings of the Genetic and Evolutionary Computation Conference, ACM, New York, NY, USA, GECCO ’18, 1411–1418. URL http://doi.acm.org/10.1145/3205455.3205490 [6] Arrieta, A., S. Wang, U. Markiegi, A. Arruabarrena, L. Etxeberria, and G. Sagardui, 2019: Pareto efficient multi-objective black-box test case selection for simulation-based testing. Information & Software Technology,114, 137–154, doi:10.1016/j.infsof.2019.06.009. URL https://doi.org/10.1016/j.infsof.2019.06.009 [7] Ayerdi, J., A. Garciandia, A. Arrieta, W. Afzal, E. P. Enoiu, A. Agirre, G. Sagardui, M. Arratibel, and O. Sellin, 2020: Towards a taxonomy for eliciting design-operation continuum requirements of cyber-physical systems. 28th IEEE International Conference on Requirements Engineering, RE 2020, Zurich, Switzerland, 2020. [8] Ayerdi, J., S. Segura, A. Arrieta, G. S. Arratibel, and M. Arratibel, 2020: Qos-aware metamorphic testing: An elevation case study. 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), IEEE, 104–114. [9] Ayerdi, J., V. Terragni, A. Arrieta, P. Tonella, G. Sagardui, and M. Arratibel, 2021: Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 1264–1274. [10] Bagherzadeh, M., N. Kahani, and L. Briand, 2021: Reinforcement learning for test case prioritization. IEEE Transactions on Software Engineering. [11] Barney, G. and L. Al-Sharif, 2015: Elevator traffic handbook: theory and practice. Routledge. [12] Barr, E. T., M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, 2014: The oracle problem in software testing: A survey. IEEE transactions on software engineering,41, no. 5, 507–525. [13] Ben Abdessalem, R., S. Nejati, L. C. Briand, and T. Stifter, 2016: Testing advanced driver assistance systems using multi-objective search and neural networks. Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 63–74. [14] — 2018: Testing vision-based control systems using learnable evolutionary algorithms. Proceedings of the 40th International Conference on Software Engineering, ICSE ’18. [15] Bolat, B., P. Cortés, E. Yalçin, and M. Alişverişçi, 2010: Optimal car dispatching for elevator groups using genetic algorithms. Intelligent Automation & Soft Computing,16, no. 1, 89–99. [16] Braga, R., P. S. Neto, R. Rabêlo, J. Santiago, and M. Souza, 2018: A machine learning approach to generate test oracles. Proceedings of the XXXII Brazilian Symposium on Software Engineering, 142–151. [17] Brunnert, A., A. van Hoorn, F. Willnecker, A. Danciu, W. Hasselbring, C. Heger, N. Herbst, P. Jamshidi, R. Jung, J. von Kistowski, A. Koziolek, J. Kroß, S. Spinner, C. Vögele, J. Walter, and A. Wert, 2015: Performance-oriented DevOps: A Research Agenda. URL http://arxiv.org/abs/1508.04752 [18] Chan, W. K., J. C. Ho, and T. Tse, 2010: Finding failures from passed test cases: Improving the pattern classification approach to the testing of mesh simplification programs. Software Testing, Verification and Reliability,20, no. 2, 89–120. [19] Deisenroth, M. P., A. A. Faisal, and C. S. Ong, 2020: Mathematics for machine learning. Cambridge University Press. [20] Delgado-Pérez, P., A. B. Sánchez, S. Segura, and I. Medina-Bulo, 2020: Performance mutation testing. Software Testing Verification and Reliability. [21] Derler, P., E. A. Lee, and A. Sangiovanni-Vincentelli, 2011: Modeling cyber-physical systems. Proceedings of the IEEE (special issue on CPS),100, no. 1, 13 – 28. [22] Ding, J. and D. Zhang, 2016: A machine learning approach for developing test oracles for testing scientific software. SEKE, 390–395. 24 Gartziandia et al [23] Durelli, V. H., R. S. Durelli, S. S. Borges, A. T. Endo, M. M. Eler, D. R. Dias, and M. P. Guimaraes, 2019: Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability,68, no. 3, 1189–1212. [24] Etxeberria, L., F. Larrinaga, U. Markiegi, A. Arrieta, and G. Sagardui, 2017: Enabling co-simulation of smart energy control systems for buildings and districts. IEEE 22nd Conference on Emerging Technologies and Factory Automation (ETFA2017), 1–4. [25] Fontes, A. and G. Gay, 2021: Using machine learning to generate test oracles: a systematic literature review. Proceedings of the 1st International Workshop on Test Oracles, 1–10. [26] Gaaloul, K., C. Menghi, S. Nejati, L. C. Briand, and D. Wolfe, 2020: Mining assumptions for software components using machine learning. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 159–171. [27] Galarraga, J., A. Arrieta, S. Ali, G. Sagardui, and M. Arratibel, 2021: Genetic algorithm-based testing of industrialelevators under passenger uncertainty. 2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE. [28] Gartziandia, A., A. Arrieta, A. Agirre, G. Sagardui, and M. Arratibel, 2021: Using regression learners to predict performance problems on software updates: a case study on elevators dispatching algorithms. Proceedings of the 36th Annual ACM Symposium on Applied Computing, 135–144. [29] Gartziandia, A., J. Ayerdi, A. Arrieta, S. Ali, T. Yue, A. Agirre, G. Sagardui, and M. Arratibel, 2021: Microservices for continuous deployment, monitoring and validation in cyber-physical systems: an industrial case study for elevators systems. 2021 IEEE 18th International Conference on Software Architecture Companion (ICSA-C), IEEE, 46–53. [30] Genç, A. E., H. Sözer, M. F. Kıraç, and B. Aktemur, 2019: Advisor: An adjustable framework for test oracle automation of visual output systems. IEEE Transactions on Reliability. [31] Gholami, F., N. Attar, H. Haghighi, M. V. Asl, M. Valueian, and S. Mohamadyari, 2018: A classifier-based test oracle for embedded software. 2018 Real-Time and Embedded Systems and Technologies (RTEST), IEEE, 104–111. [32] Hardin, B. and U. Kanewala, 2018: Using semi-supervised learning for predicting metamorphic relations. 2018 IEEE/ACM 3rd International Workshop on Metamorphic Testing (MET), IEEE, 14–17. [33] Hiremath, D. J., M. Claus, W. Hasselbring, and W. Rath, 2020: Automated identification of metamorphic test scenarios for an ocean-modeling application. 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), IEEE, 62–63. [34] Hou, Y., Y. Zhao, A. Wagh, L. Zhang, C. Qiao, K. F. Hulme, C. Wu, A. W. Sadek, and X. Liu, 2015: Simulation-based testing and evaluation tools for transportation cyber–physical systems. IEEE Transactions on Vehicular Technology,65, no. 3, 1098–1108. [35] Jahangirova, G., A. Stocco, and P. Tonella, 2021: Quality metrics and oracles for autonomous vehicles testing. 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE, 194–204. [36] Jin, H., Y. Wang, N.-W. Chen, Z.-J. Gou, and S. Wang, 2008: Artificial neural network for automatic test oracles generation. 2008 International Conference on Computer Science and Software Engineering, IEEE, volume 2, 727–730. [37] Just, R., D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, 2014: Are mutants a valid substitute for real faults in software testing? Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, 654–665. [38] Kanewala, U. and J. M. Bieman, 2013: Using machine learning techniques to detect metamorphic relations for programs without test oracles. 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), IEEE, 1–10. [39] Kanewala, U., J. M. Bieman, and A. Ben-Hur, 2016: Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels. Software testing, verification and reliability,26, no. 3, 245–269. [40] Kim, J., R. Feldt, and S. Yoo, 2019: Guiding deep learning system testing using surprise adequacy. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE, 1039–1049. [41] Lachmann, R., S. Schulze, M. Nieke, C. Seidl, and I. Schaefer, 2016: System-level test case prioritization using machine learning. 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, 361–368.