Robustness Testing of an Industrial Road Object Detection System

Robustness Testing of an Industrial Road Object Detection System Anne-Laure Wozniak∗†, Ngoc Q. K. Duong‡, Ian Benderitter‡, Sarah Leroy∗, Sergio Segura§and Ra´ ul Mazo† ∗Kereval, Rennes, France †Lab-STICC, ENSTA Bretagne, Brest, France ‡Lacroix Impulse, Rennes, France §SCORE Lab, Universidad de Sevilla, Seville, Spain Abstract—As AI-based critical systems are expected to operate in dynamic environments, it is crucial to ensure their reliability under various operational conditions. In computer vision, one way to achieve this is by testing the system’s robustness to input image perturbations. However, while many methods have been proposed and evaluated in academic settings, their effectiveness and applicability in practice remain largely unknown. In this paper, we report the results of testing the robustness of an industrial case of an AI-based road object detection system, in a black-box setting. By defining relevant perturbations and metrics, we analyse the system’s response to changes in its hardware and software environment, and identify areas for improvement through retraining with data augmentation. We address the key challenges encountered during this evaluation and provide insights that may help practitioners in performing similar tests and guide future research on robustness testing of AI-based object detection systems. Index Terms—robustness, software testing, object detection, artificial intelligence. I. INTRODUCTION The development of artificial intelligence (AI) based systems has made significant progress in recent years, with these systems performing at or above human levels in many tasks, such as natural language processing [1] or image classification [2]. But with the emergence of real-time critical systems using AI, such as autonomous vehicles, there is a need to carefully evaluate the reliability of the AI algorithms, during their entire life-cycle and under different operating conditions, to avoid potentially fatal failures. A common approach to assess the reliability of AI systems is by testing its robustness, this is, assessing the “ability of an AI system to maintain its level of performance under any circumstances” (ISO/IEC TR 24029-1:2021). In the context of computer vision—the scope of this paper— this assessment is usually done by generating images altered by perturbations. Thus, various perturbations on images have been proposed to ensure the robustness of the system in different scenarios (e.g., adversarial attacks, or changes in weather or in the settings of its hardware setup) [11], [18], [22]. This work has been supported by the French National Agency of Research and Technology (ANRT) [CIFRE N°2020/0754]; by grant PID2021-126227NB-C22, funded by MCIN/AEI /10.13039/501100011033/FEDER, UE; and grant TED2021-131023B- C21, funded by MCIN/AEI/10.13039/501100011033 and by European Union “NextGenerationEU”/PRTR. Most of the approaches to generating perturbations have, however, been proposed and evaluated in academia, in settings that are sometimes quite different from what can be encountered in practice [3]. There is thus a need for industrial case studies to better understand the strengths and limitations of current approaches for assessing the robustness of AI-based systems. In this paper, we report the results on robustness testing of an industrial case of a safety-critical machine learning-based object detection system. More specifically, we aim to test the robustness of the system against common perturbations, encountered in its normal functioning. To this end, we apply metamorphic testing, a widely adopted testing approach in the field of AI [4], [5]. We thus propose thirteen transformations representative of changes in the sensor and digital processing parameters and study the overall behaviour of the system when facing them. By analysing the sensitivity of the system to transformations, we identify areas of improvement that can lead to a retraining of the system with data augmentation. We then study the benefit of such retraining, with regard to the robustness of the system. Finally, we report a set of lessons learned to help practitioners and to guide research on the subject. A preliminary version of this paper was presented in [6]. This paper extends our previous work in several directions. First, we detail the testing method followed including the key steps, metrics and decisions made when testing the target robustness of AI-based road object detection system. Second, we present a larger evaluation including new experiments on the metrics used to assess the robustness and novel findings related to model retraining. Third, we provide a detailed list of lessons learned to guide testing of similar systems and future research on the topic. The rest of the paper is structured as follows. Section II introduces in detail the industrial system under test, while Section III presents the test approach and the metrics used. Section IV reports the main results of the tests and Section V details the lessons learned. Finally, we discuss related works in Section VI and conclude the paper in Section VII. II. ROAD OBJECT DETECTION SYSTEM In this section, we introduce the system under test and present its context of use, the architecture and the training 82 2023 IEEE International Conference On Artificial Intelligence Testing (AITest) process of the object detection model, as well as the data used to perform training and testing. A. Context The system has been developed by Lacroix1, a French industrial leader in the field of embedded AI for the automotive and smart city domains. It has been investigated within the scope of road monitoring in cities where cameras are strategically placed at road junctions to detect target objects for traffic regulation (see Figure 4(a) for an example). In the following, we describe the model architecture and the training with Lacroix in-house dataset. Note that, during the robustness testing of the system, testers were not aware of the model architecture and training details as the system was in a black-box setting. Such a choice was motivated by the need to outsource the testing without having to share sensitive information at that time. B. Model architecture The object detection model includes a backbone for visual feature extraction, adaptation layers, and detection heads for bounding box detection and object classification. The overall architecture is described in Figure 1. Fig. 1. Architecture of the tested road object detection system. For the backbone, the EfficientNet architecture is used, motivated by its performance in computer vision benchmarks [7]. More specifically, Lacroix chose the EfficientNet-Lite (model 2)2architecture to further optimise the network for embedded 1https://www.lacroix-group.com/ 2https://paperswithcode.com/lib/timm/tf-efficientnet-lite systems. This model significantly reduces the computational power required compared to the original EfficientNet by (1) introducing novel activation functions, (2) removing squeeze- and-excitation blocks, and (3) fixing the stem and head while scaling models up in order to reduce their size and computations. The EfficientNet-Lite2 backbone has thus only 6 million parameters and an accuracy of 77.48% on the ImageNet dataset [8]. After the EfficientNet-Lite2 backbone, Lacroix has added three deconvolution layers to increase the input feature size. Each deconvolution layer is followed by a BatchNorm layer and a Scale layer, as in the EfficientNet architecture. The output feature dimension after three deconvolution layers is 136 ×136. It is then passed through a ReLU layer before entering detection heads. The detection heads are implemented following those proposed in CenterNet [9], where each object is detected as a triplet, rather than a pair, of key-points to improve both precision and recall. The CenterNet head offers fast one-stage object detection, with performances comparable to more computational demanding two-stage approaches such as Faster- RCNN [10]. Three branches are implemented for object center regression, object height/width regression, and object classification, respectively. C. Dataset and training Lacroix has built an in-house dataset for the road object detection and tracking system. This dataset includes over 33,000 high resolution images (1920x1080, saved in JPEG format) extracted from short videos captured in different cities in France and Vietnam to avoid regional bias. For better generalization in real-world situations, videos were also captured with different camera parameters as well as various day/night times. Images were annotated by a specialized partner where all human faces and vehicle license plates were first anonymized. Six distinct object classes (“person”, “car”, “bicycle”, “motorcycle”, “truck”, and “bus”) were annotated with bounding box coordinates. Overall, about 23% of the annotations concerns objects captured in France and the remaining 77% of the annotations concerns objects captured in Vietnam. More characteristics of the dataset (class, object size, and day/night distributions) are described in Figure 2. The dataset is split into a training set (over 30,000 images) and a test set (2,645 images). To maximize the model performance in a considered deployment scenario (i.e., road junctions), only images captured by fisheye cameras are used in the test set named DJANGO. A first training was performed, using simple data augmentation based on random contrast, exposure and saturation alterations, along with random crops, flips and rotations. The model resulting from this training is the one tested primarily in this article, referred to as SUT v1. A second training was then performed, motivated by the results of our sensitivity analysis, which included data augmentation based on random gamma alterations, defocus blurring and jittering—we refer to this version obtained after retraining as SUT v2 (see Section IV-B car 21% truck 2% bus 1% person 47% bicycle 7% motorcycle 22% 19% 11% 9% 36% 24% 33% 54% 48% 50% 46% 57% 51% 27% 40% 41% 17% 19% 16% CAR TRUCK BUS PERSON BICYCLE MOTORCYCLE small objects medium objects large objects 96% 100% 98% 90% 98% 96% 4% 0% 2% 10% 2% 4% CAR TRUCK BUS PERSON BICYCLE MOTORCYCLE day night Fig. 2. Dataset characteristics. Left figure: distribution of classes; right figures: distribution of object sizes as defined in COCO dataset (top) and distribution of day and night samples (bottom). for details). In both cases, all images are resized and cropped to 544 ×544. The backbone EfficientNet-Lite2 is pre-trained with ImageNet dataset, then the whole network is fine-tuned for 70 epochs with the training set and model with highest performance measured by AP0.5for the test set is saved for testing. The training is done with Adam optimizer, batch size is 12, and learning rate is 0.0001. Finally the network is converted to be executed on edge platform with low computational capability via post-training quantization and other hardwarecompatible acceleration techniques. III. TESTING METHOD An overview of the test method is presented in Figure 3. It consists of two main phases: test design, in which we define the test objectives and strategy, and test execution, in which test cases are generated and the robustness of the system is assessed. In what follows, we describe the goal and research questions, as well as the test design and test execution phases. The test method is based on metamorphic testing, where the definition of transformations on input images to generate follow-up test cases is guided by domain knowledge. The choice of such an approach among related work is motivated (i) by the absence of white-box access, thus the impossibility of using most of coverage metrics, and (ii) by the absence of detailed specifications, thus the difficulty of establishing properties on which to base the generation of test cases. A. Goal and research questions The main objective of our study is to test the robustness of the detection model under different circumstances, in a black-box setting. To do so, we classified circumstances into three categories: those relating to the hardware or software environment of the system (e.g., change in the parameters of the image signal processor), those relating to the observed environment (e.g., change in weather), and those relating to a potential external adversary (e.g., adversarial attacks). In this work, we report the results of testing the robustness of the system to hardware and software changes, such as using different camera technologies. This is a priority for the company since the system is intended to be deployed in Define transformations Generate follow-up test datasets Compare circumstances Analyse the sensitivity to perturbations Select metrics Evaluate the performance of the SUT for each transformation for each original and follow-up test dataset for each original and follow-up test dataset Retrain the SUT Test design Is the robustness assessment satisfactory? SUT can be improved Test execution Need to test new transformations Fig. 3. BPMN diagram of the overall test method. several cities that will not all use the same equipment. The company is also interested in assessing the potential impact of retraining with data augmentation on the robustness of the system. Hence, our study aimed to answer the following research questions: RQ1: How robust is the SUT against changes in its hardware and software environment? We aim to evaluate the extent to which a change of camera technology or image TABLE I TYPES OF TRANSFORMATIONS. Step Parameter Category Acquisition Lens material Blur (camera) Lens distortion Image distortion Focus Blur Resolution Blur Aperture Colour, Blur Shutter speed Blur ISO Sensitivity / Gain Noise Dead pixels/lines Faulty pixel Processing Dematrixing Colour (ISP) Tone mapping (HDR) Colour Sharpening Blur Contrast Colour Brightness Colour Noise reduction Noise White balance Colour Lens shading Colour Color mapping Color Encoding/Compression Blur Black level Noise signal processor calibration can have an impact on system performance. RQ2: To what extend retraining with data augmentation has an impact on the robustness of the SUT? Data augmentation is a well-known method to improve a model’s ability to generalize by increasing the amount of data used during training with slightly modified data. We thus aim to study the impact of such data augmentation on the robustness of the system. B. Test design In the following, we detail the way transformations and metrics were chosen during the test design phase. 1) Define transformations: As previously mentioned, we rely on domain knowledge to define relevant transformations. We have thus selected image transformations from the literature [11] and mapped them to the parameters that can affect the different stages of the image acquisition process (see Table I). This ensures that the transformations are representative of reality. Moreover, this mapping between the real phenomena and the types of transformations on the images allowed us to define five new transformations that were not included in [11], such as unsharp mask and chromatic aberration. This gives a total of 13 transformations, divided into three different categories. The blur category includes the different blurring techniques (Gaussian blur, zoom blur, defocus blur), but also the inverse transformations (e.g., pixelation). The colour category includes transformations that affect the histogram of the image in the broadest sens, such as colour, contrast and brightness. Finally, the noise category includes Gaussian noise, Speckle noise, Salt noise, Pepper noise, and the Salt and Pepper noises combination. See Figure 4 for examples of some transformations. 2) Select metrics: In the following, we discuss the choices of the metrics to assess performance, compare the circumstances and analyse the overall robustness of the system. Table II summarises the set of metrics used during the tests. a) Measuring performance: To measure performance, we use the COCO detection evaluation metrics3, including 6 average precision (AP) and 6 average recall (AR) metrics. These 12 metrics are widely used in object detection, as they take into account both localisation and classification performance. One metric of particular interest is the average precision over all classes, for Intersection over Union (IoU) thresholds between 0.5 and 0.95. This metric is the primary COCO challenge metric and rewards detectors with better localization. Its value range from 0 to 1, and the better the performance, the higher the value. According to the COCO challenge leaderboard4, the latest and best performing detection models have an AP between 0.5 and 0.6. b) Comparing circumstances: In order to assess robustness, we need to compare the performance of the system under different “circumstances” and therefore be able to compare them. In our case, not all of the transformations have the same level of severity or represent the same difficulties for the system. To overcome this issue, we therefore seek to construct a common scale that would allow us to measure the impact of each transformation on the images used by the system. In practice, we exploit the fact that the observed scene is globally always the same, since the camera is fixed, and that each transformation is applied separately to the whole image: in this configuration, the use of full-reference quality metrics is relevant. In the following, we use the structural similarity index (SSIM) [12] to compare transformations. This choice is motivated by our test objectives: we wish to evaluate the system against realistic transformations, and the performance of this type of system is often compared to human vision. The choice of a metric that corresponds more to the quality perceived by humans therefore seems more relevant to us. The SSIM measures the similarity between an original image and its altered counterpart, in terms of luminance, contrast, and structure. Its value ranges from 0 to 1, where a SSIM of 1 indicates that the two images being compared are identical. Based on a measure of structural distortion, to which the human vision system is more sensitive, it is closer to human perception of quality than other common quality metrics such as mean squared error (MSE) and peak signal- to-noise ratio (PSNR) [13]. It should be noted, however, that its use is highly dependent on the use case and the transformations one wishes to study. For example, as it is highly sensitive to rotation, spatial shift and scaling, it is important to ensure that the camera is fixed in time, if one does not want to distort its value. c) Analysing the sensitivity: The system is said to be robust if it maintains the same level of performance across multiple circumstances. We therefore define “sensitivity” as 3https://cocodataset.org/#detection-eval 4https://cocodataset.org/#detection-leaderboard (a) Original (b) Pixelate (c) Zoom blur (d) Gamma correction darker (e) Speckle noise (f) Chromatic Aberration Fig. 4. Examples of transformations applied to an original image. TABLE II SUMMARY OF METRICS.S=SMALL,M=MEDIUM,L=LARGE. Purpose Metric Range Description and notations Performance Average Precision [0,1] Area under the precision-recall curve, averaged over all categories. We distinguish between: AP across IoUs – AP, AP0.5,AP 0.75 AP across object scales – APS,AP M,AP L Average Recall [0,1] Maximum recall given a fixed number of detections per image, averaged over categories and IoUs. We distinguish between: AR across detections – AR1,AR 10,AR 100 AR across object scales – ARS,AR M,AR L Circumstances SSIM [0,1] Similarity between two images, in terms of luminance, contrast, and structure. Robustness Sensitivity [0,∞[Absolute value of the slope of the performance curve as a function of SSIM, obtained by linear regression. the absolute value of the slope of the performance curve as a function of SSIM, obtained by linear regression. In this context, it is a direct measure of the robustness. The sensitivity is computed separately for each of the transformations. Its value is greater than or equal to 0, where a sensitivity of 0 corresponds to the case where the system is perfectly robust to the perturbation considered. To facilitate comparison, we have normalised the sensitivity between 0 and 1 across all graphs. C. Test execution The first step in the test execution is the generation of a follow-up test dataset for each of the transformations, based on the original dataset. Each of these datasets is then used to evaluate the performance of the system against perturbations, through the metrics defined in the previous section. In parallel, the average SSIM is computed for each of the follow-up datasets using the original one as a reference. These results are then synthesised to compute the sensitivity of the system to each of the perturbations. At this stage, several choices are possible. Firstly, it is possible to restart the process from the beginning and iterate using new transformations to complement the prior sensitivity analysis. Secondly, the sensitivity analysis can give indications on how to improve the model and lead to a re-training by integrating new perturbed data in the dataset. Indeed, data augmentation is an effective and recommended method for improving the results of models and, in particular, their ability to generalise [14]. Thus, methods have been developed to include data obtained with data augmentation in training sets, in order to improve the robustness of the models [15]. Following the example of these solutions, we can use images altered by the transformations used in the previous tests to re-train the model and improve its robustness to these same transformations. In the following section, we study the case where we proceed to the re-training step. Three transformations have been selected by Lacroix developers to perform data augmentation: defocus blur, jitter (that is part of the chromatic aberration transformation), and gamma correction. According to them, these transformations correspond to the most likely or problematic phenomena for their use case, also taking into account our initial robustness assessment. Table III summarises the transformation parameters used and the proportions of modified data in the training dataset. Note that the modified TABLE III DATA AUGMENTATION PARAMETERS. Transformation Parameters Distribution Defocus Blur No change 50% 0.02 (radius) 25% 0.03 (radius) 25% Jitter No shift 33% (Chromatic Aberration) Left shift 33% Right shift 33% Gamma Correction No change 50% 0.3 (gamma) 12.5% 0.5 (gamma) 12.5% 2.0 (gamma) 12.5% 2.5 (gamma) 12.5% images are chosen randomly. IV. RESULTS In this section, we report the results of our study and how they answer our research questions. A. RQ1: Robustness assessment Figure 5 shows the sensitivity results obtained for SUT v1 against the transformations defined in the previous section. In order to compare and validate our results, we also provide baseline results for a common object detection model, called Faster R-CNN. This model was trained and evaluated on the BDD100k dataset [16], consisting of 100,000 videos of driving scenes recorded under different conditions (change of weather, time of day, and city). Overall, it can be observed that SUT v1 is more robust to Noise transformations (e.g., Gaussian or Salt and Pepper noises), than to Blur or Colour transformations (e.g., Gaussian blur, chromatic aberration, or pixelation): on average, the sensitivity of the SUT to noise is 0.28, while it is 0.47 to blur and 0.55 to colour changes. 0 0.25 0.5 0.75 1 GammaCorrectionBrighter GammaCorrectionDarker UnsharpMask SpeckleNoise GaussianNoise SaltNoise SaltPepperNoise PepperNoise ChromaticAberration DefocusBlur GaussianBlur ZoomBlur Pixelate (a) Faster R-CNN (b) SUT v1 (c) SUT v2 Sensitivity Fig. 5. Sensitivity to transformations of (a) Faster R-CNN, (b) SUT v1, and (c) SUT v2. From a model perspective, these results can be explained by the high sensitivity of object detection models to edges and contours. In terms of the hardware and software environment, this means that the model will be particularly sensitive to a degradation in the quality of the camera, and in particular its lens. A poorer quality lens will generally produce images with less sharpness in the corners and the edges, more chromatic aberration, and less contrast. While blurring effects are difficult to correct upstream of the model, the ISP nevertheless allows the adjustment of colour and brightness perturbations. Our results thus highlight the need to monitor and calibrate it regularly to adapt to the conditions of use of the model. In the following, these results are discussed in more detail, focusing on object classes and sizes. Figure 6 shows the sensitivity of the model to transformations for each of the classes in the test dataset, as well as for different object sizes. Note that only the car, truck and person classes are present in the test dataset. Also, object sizes are defined as in the evaluation metrics used by COCO: small objects have an area of less than 32² pixels, large objects have an area of more than 96² pixels, and medium-sized objects are all those whose area lies between these limits. The area considered here is that of the bounding box formed by the object. It can be seen that not all classes and sizes of objects are sensitive to the same transformations. Overall, the person class seems less sensitive to transformations than the car and truck classes. In the same way, small and medium-sized objects seem to be more robust to transformations than large ones, regardless of their class. For small objects, the sensitivity results should be treated with caution. This category includes pedestrians that are far away from the camera, and vehicles largely obscured by background elements, at the edge of the region of interest. Due to their size and lack of detail (pedestrians look more like a monochrome silhouette), these objects are difficult for the system to detect and quickly erased by the transformations applied to the images. This results in almost zero AP values, regardless of the SSIM, and therefore a sensitivity close to zero. On the other hand, the Unsharp Mask transformation preserves their appearance, the AP values vary more according to the SSIM, and we therefore find a higher sensitivity. Furthermore, this is a limitation of the sensitivity metric, and the assessment of robustness cannot be done solely based on the sensitivity measurement. It is important to put this measure in context and relate it to the AP value. Response to RQ1 The SUT shows higher sensitivity to blurring and colour changes compared to noise, indicating a high vulnerability to degradation of the camera lens quality and changes in some ISP parameters, such as brightness. In addition, there is a variation in sensitivity depending on the class and size of the objects: large cars and trucks are the most vulnerable to such changes. B. RQ2: Retraining with data augmentation Before discussing the sensitivity results in detail, we compare the performance for the original model (SUT v1) and the re-trained version (SUT v2). We measure that, overall, the AP declined from 29% to 27% on the original test dataset. 0 0.25 0.5 0.75 1 SpeckleNoise GammaCorrectionDarker PepperNoise DefocusBlur GaussianNoise SaltPepperNoise SaltNoise ChromaticAberration Pixelate GaussianBlur UnsharpMask GammaCorrectionBrighter ZoomBlur Car Truck Person Small Medium Large Sensitivity Fig. 6. Sensitivity to transformations of SUT v1, per class (car, person, truck) or object size (large, medium, small). This is, however, a normal phenomenon, also observed in the literature [17]. Increasing the training set with similar but slightly perturbed images adds complexity to the problem that the model must optimise. It also increases the mismatch between the training data and the original test set. On the other hand, comparing the sensitivities of the SUT before and after re-training in Figure 5, we observe that the system is clearly more robust to some specific transformations: the sensitivity to Defocus blur, used in the data augmentation, decreases from 0.31 to 0.10, while that to Gaussian blur decreases from 0.54 to 0.21 and that to Unsharp Mask decreases from 0.60 to 0.42. There are also smaller improvements in the sensitivity of the Noise transformations with, on average, a 20% decrease in sensitivity. The same conclusion can be drawn for the other Blur and Colour transformations with decreases ranging from 3 to 9% in sensitivity, with the exception of the Chromatic Aberration transformation which sees its sensitivity increase by 8.5%. A more detailed study of the performance curves as a function of the SSIM nevertheless allowed us to determine that this was linked to an outlier. Overall, while these results seem to confirm the favourable impact of such re-training, it is surprising that we do not observe as strong an improvement in sensitivity to the transformations used for data augmentation as for the Defocus Blur. This may be due, however, to the parameters used for data augmentation or the distribution of modified images in the training dataset. Further experiments are needed to conclude on the subject. Response to RQ2 Retraining with targeted data augmentation improved the robustness of the system against 12 out of 13 transformations, with improvements ranging from 3% to 68% in sensitivity. V. LESSONS LEARNED Lesson 1: The method is effective, but further guidance is necessary. The results of our study suggest that the test method, presented in Figure 3, is appropriate in practice for assessing robustness against changes in hardware or software settings. Nevertheless, more guidelines are desirable at the crucial stages of test design and results analysis. Those could include, without being limited to, systematic methods for choosing metrics or for performing sensitivity analyses, with the aim of providing a systematic robustness indicator. Lesson 2: Effectiveness of retraining with image perturbations. Our results show that retraining with targeted data augmentation is an effective approach for improving the robustness of AI-based road object detection systems against hardware and software issues. Future work should investigate how these image perturbations should be selected in a systematic way to further optimise the robustness improvement. Lesson 3: Domain knowledge is required. Evaluating the object detection system was challenging due to the need, despite the black-box setting, for domain-specific knowledge at different stages of test design and execution. When defining the relevant transformations, it is crucial to determine which ones are representative of real-world circumstances. Unfortunately, transformation benchmarks from the literature may not be sufficient, either because they lack specific transformations or because not all transformations are relevant to the system. Specific domain knowledge is also required to interpret the sensitivity results, in order to fully understand the impact on the system, determine the causes of the failures, and define relevant thresholds to formally decide on the system’s robustness or on a retraining. Lesson 4: Need for additional metrics beyond conventional ones. While average precision (AP) is the most widely used metric to measure object detection models’ performance, Lacroix developers provided feedback that the analysis of their system’s robustness could be limited by relying solely on this metric. For example, the number of false negatives is a critical indicator for their use case and could be used separately in the sensitivity analysis since its value is not explicit in the AP. Thus, other metrics must be chosen according to the need and constraints of the application case. Lesson 5: A methodology is required that lies between agnostic and specific. The above discussions show that each system comes with its own specificities and constraints, whether it is the definition of the circumstances of use of the system or the choice of relevant metrics. It is therefore necessary to adapt the testing method, particularly through domain knowledge. However, over-specification of the test method has several disadvantages: it becomes difficult to compare the results from one system to another, even within the same domain, and the method may be complex to transpose to another particular case. There is therefore a need to guide practitioners on the method and metrics to be used, without closing the door on specific use case constraints. VI. RELATED WORK In computer vision, the robustness of AI-based systems is usually assessed by generating images altered by perturbations [3], [4]. The aim is to evaluate the ability of the system to maintain its performance against uncertainties in its operational environment, which are reflected in the input data. These inputs can be separated into two categories: adversarial inputs and natural inputs. Adversarial inputs are images created with perturbations imperceptible to humans, but which can cause AI models to make incorrect predictions. In recent years, several methods for generating adversarial inputs in object detection systems have been proposed [18]. Such inputs are nevertheless often considered to be unrealistic in practice. Alternatively, natural inputs are created by applying common perturbations to images to simulate real-world scenarios. The definition of these perturbations can be guided by knowledge of the application domain [19], coverage metrics [20], or properties that the system must satisfy, such as metamorphic relations [21]. In addition, several perturbation benchmarks have been proposed in the literature to assess robustness [11], [22]. Few of these methods have been evaluated in practice, in an industrial context or on real world cases. Such a context generally implies strong constraints, such as a black box access. As complement to related work, we propose an industrial case study to better understand the strengths and limitations of current approaches to assess the robustness of AI systems. VII. CONCLUSION In this paper, we share our experience in testing the robustness of an industrial AI-based road object detection system. We describe in detail the testing approach, with the definition of relevant transformations on the input images of the system in order to evaluate its performance against changes in its hardware and software environment. In this approach, we also identify areas for improving the robustness of the system through retraining with data augmentation. We identify several challenges in the design and execution of the tests in practice and report some insights that we believe may be useful to practitioners in testing the robustness of AI-based object detection systems. REFERENCES [1] A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan, “Speech recognition Using Deep Neural Networks: A Systematic Review,” IEEE Access, vol. 7, pp. 19143–19165, 2019. [2] K. He, X. Zhang, S. Ren, J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” in Proc. Int. Conf. on Comput. Vision (ICCV), 2015, pp. 1026–1034. [3] V. Riccio, G. Jahangirova, A. Stocco, N. Humbatova, M. Weiss, and P. Tonella, “Testing machine learning based systems: a systematic mapping,” Empirical Softw. Eng., vol. 25, pp. 5193-–5254, 2020. [4] J. M. Zhang, M. Harman, L. Ma, and Y. Liu, “Machine Learning Testing: Survey, Landscapes and Horizons,” IEEE Trans. on Softw. Eng., vol. 48, no. 1, pp. 1–36, 2022. [5] T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic Testing: A Review of Challenges and Opportunities,” ACM Comput. Surv., vol. 51, no. 1, pp. 4:1-4:27, 2018. [6] A.-L. Wozniak, S. Segura, R. Mazo, and S. Leroy, “Robustness Testing of a Machine Learning-based Road Object Detection System: An Industrial Case,” Proc. 1st International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI), 2022. [7] M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in Proc. Int. Conf. on Mach. Learn. (ICML), 2019, pp. 6105–6114. [8] J. Deng, W. Dong, R. Socher, L. -J. Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. Int. Conf. on Comput. Vision and Pattern Recognit. (CVPR), 2009, pp. 248–255. [9] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “CenterNet: Keypoint Triplets for Object Detection,” in Proc. Int. Conf. on Comput. Vision (ICCV), 2019, pp. 6568-6577. [10] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards realtime object detection with region proposal network,” in Proc. Advances in Neural Inf. Process. Syst. (NIPS), 2015, pp. 91-–99. [11] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in Proc. Int. Conf. on Learn. Representations (ICLR), 2019. [12] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process., vol. 13, no. 4, pp. 600–612, 2004. [13] Z. Wang and A. C. Bovik, “Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures,” IEEE Signal Process. Mag., vol. 26, no. 1, pp. 98–117, 2009. [14] P. Y. Simard, D. Steinkraus, and J. C. Platt, “Best practices for convolutional neural networks applied to visual document analysis,” in Proc. Int. Conf. on Document Analysis and Recognit., 2003. [15] X. Gao, R. K. Saha, M. R. Prasad, and A. Roychoudhury, “Fuzz Testing based Data Augmentation to Improve Robustness of Deep Neural Networks,” in Proc. Int. Conf. on Softw. Eng. (ICSE), 2020, pp. 1147– 1158. [16] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning,” arXiv preprint, arXiv:1805.04687, 2018. [17] L. Rice, E. Wong, and J. Z. Kolter. “Overfitting in adversarially robust deep learning,” in Proc. Int. Conf. on Mach. Learn. (ICML), 2020. [18] A. Serban, E. Poll, and J. Visser, “Adversarial Examples on Object Recognition: A Comprehensive Survey,” ACM Comput. Surveys, vol. 53, pp. 1—38, 2020. [19] A. H. M. Rubaiyat, Y. Qin, and H. Alemzadeh, “Experimental Resilience Assessment of an Open-Source Driving Agent,” in Proc. Pacific Rim Int. Symp. on Dependable Comput. (PRDC), 2018, pp. 54–63. [20] Y. Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars,” in Proc. Int. Conf. on Softw. Eng. (ICSE), 2018, pp. 303–314. [21] S. Wang and Z. Su, “Metamorphic object insertion for testing object detection systems,” in Proc. Int. Conf. on Automated Softw. Eng. (ASE), 2021, pp. 1053-–1065. [22] B. Zhao, S. Yu, W. Ma, M. Yu, S. Mei, A. Wang, J. He, A. Yuille, and A. Kortylewski, “OOD-CV: A Benchmark for Robustness to Out-of- Distribution Shifts of Individual Nuisances in Natural Images,” in Proc. Eur. Conf. on Comput. Vision (ECCV), 2022, pp. 163-–180.