Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance

Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance Manuel Carranza-García ⇑ , Pedro Lara-Benítez, Jorge García-Gutiérrez, José C. Riquelme Division of Computer Science, University of Sevilla, ES-41012 Seville, Spain article info Article history: Received 9 July 2020 Revised 1 February 2021 Accepted 2 April 2021 Available online 6 April 2021 Communicated by Zidong Wang Keywords: Autonomous vehicles Anchor optimization Class imbalance Convolutional neural networks Deep learning Object detection abstract Object detection has been one of the most active topics in computer vision for the past years. Recent works have mainly focused on pushing the state-of-the-art in the general-purpose COCO benchmark. However, the use of such detection frameworks in specific applications such as autonomous driving is yet an area to be addressed. This study presents an enhanced 2D object detector based on Faster R- CNN that is better suited for the context of autonomous vehicles. Two main aspects are improved: the anchor generation procedure and the performance drop in minority classes. The default uniform anchor configuration is not suitable in this scenario due to the perspective projection of the vehicle cameras. Therefore, we propose a perspective-aware methodology that divides the image into key regions via clustering and uses evolutionary algorithms to optimize the base anchors for each of them. Furthermore, we add a module that enhances the precision of the second-stage header network by including the spatial information of the candidate regions proposed in the first stage. We also explore different reweighting strategies to address the foreground-foreground class imbalance, showing that the use of a reduced version of focal loss can significantly improve the detection of difficult and underrepresented objects in two-stage detectors. Finally, we design an ensemble model to combine the strengths of the different learning strategies. Our proposal is evaluated with the Waymo Open Dataset, which is the most extensive and diverse up to date. The results demonstrate an average accuracy improvement of 6.13% mAP when using the best single model, and of 9.69% mAP with the ensemble. The proposed modifications over the Faster R-CNN do not increase computational cost and can easily be extended to optimize other anchor-based detection frameworks. Ó2021 Elsevier B.V. All rights reserved. 1. Introduction Developing robust machine learning models that can accurately detect and classify multiple objects in an image remains a core challenge in computer vision. Object detection has attracted the interest of many researchers due to its application to multiple real-world problems such as autonomous driving [1,2], robotic vision [3], security surveillance [4], or land monitoring [5].In recent years, the latest advancements in this field have been achieved thanks to the development of deep convolutional networks. Deep learning has proven to be a very powerful tool for learning abstract hierarchical representations of raw input data [6]. With the increase of availability and quality of remote sensing data collected by different sensors (higher resolution RGB cameras, LiDAR and radar data, etc.), deep learning models have pushed the state-of-the-art in many visual recognition tasks [7]. In particular, object detection is one of the main perception problems that the advanced driver assistance system (ADAS) of autonomous vehicles faces. The multi-modal sensors equipped in these vehicles provide valuable data to be used with popular deep learning-based object detectors. Nevertheless, the perception systems of self-driving vehicles need to be accurate and robust enough to operate safely in complex scenarios such as mixed urban traffic, adverse weather conditions, unmapped roads, or areas with unreliable connectivity [8]. Under these circumstances, it is still hard for existing detectors to perceive all traffic participants (vehicles, pedestrians, traffic signs, etc.) accurately, robustly, and in realtime. The impact of autonomous driving in the future promises to be important due to its potential to improve road safety, reduce traffic, and decrease pollution [9]. However, many aspects require significant progress before this technology can fully substitute human driving. In this work, the aim is to enhance the 2D object detection accuracy in the images obtained from the on-board cameras of autonomous vehicles. The goal of this detection task is to determine https://doi.org/10.1016/j.neucom.2021.04.001 0925-2312/Ó2021 Elsevier B.V. All rights reserved. ⇑ Corresponding author. E-mail address: [email protected] (M. Carranza-García). Neurocomputing 449 (2021) 229–244 Contents lists available at ScienceDirect Neurocomputing journal homepage: www.elsevier.com/locate/neucom the presence of objects from given categories and return the spatial location of each instance through a bounding box [10]. In recent literature, the main trend in object detection is to develop increasingly sophisticated architectures to improve the performance over the general-purpose COCO (Common Objects in Context) benchmark [11]. However, the effectiveness of such generic object detectors when applied to particular applications is still far from optimal [12]. For this reason, this study proposes several modifications to the popular Faster R-CNN detection framework to better adapt it to the specific context of self-driving cars. The novelty of this work lies in two main aspects that are considered for improving the performance of the original model: optimizing the anchor generation procedure and modifying the learning process to improve accuracy over minority instances. The uniform anchor generation procedure of Faster R-CNN is not suitable for the autonomous driving scenario. The default configuration, which has proven to be effective for generic object detection, produces anchors with the same scale and aspect ratios at each location of the feature map. However, due to the perspective projection of on-board cameras in the vehicles, the scale of objects has a strong correlation with their position in the image in this context. This implies that in regions where objects tend to be very large, producing small-scale anchors is not appropriate, and vice versa. To overcome this anchor mismatch issue, our proposal is to divide the images into several regions and optimize each of them independently. With the help of a clustering study, key regions in the images that have objects with significantly different dimensions are obtained. Then, a methodology based on evolutionary algorithms is presented in order to search for optimal values of scale and aspect ratio for the prior anchors of each region. Furthermore, we modify the second-stage header network introducing spatial properties extracted from the region of interest (ROI) proposals of the first stage. The spatial features of ROIs (size and position in the image) are concatenated to the convolutional features extracted from the backbone network to improve localization accuracy. Another important issue is that the default training scheme of Faster R-CNN results in a significant performance drop in minority classes and difficult instances. In the literature, the learning process of detectors has been given less attention compared to the development of architectures [13]. However, a balanced learning scheme is crucial for this multi-class scenario due to the presence of elements, such as pedestrians or cyclists, that are less frequent than vehicles. In this study, an extensive analysis of different approaches to address foreground-foreground class imbalance is performed. Several alternatives are explored, such as assigning different weights according to the class distribution and the use of focal loss, which has been traditionally used in one-stage detectors. Finally, an ensemble model based on non-maximum suppression and test-time augmentation is designed, combining the different training strategies to increase the robustness of the detector. The recently released Waymo Open Dataset [14] is used for evaluating the proposal, which is the largest and most diverse up to date in terms of geographic coverage and weather conditions. Waymo is 15 times more diverse than other existing benchmarks such as KITTI [15]. In this dataset, the object detection task has objects from three classes (vehicles, pedestrians, and cyclists) that are divided into two difficulty levels. To the best of our knowledge, this is the first study that addresses anchor optimization and class imbalance in a multi-class 2D detection problem in the context of autonomous vehicles. The proposed methodology can easily be extended to other anchor-based detection frameworks with different backbone networks, since it does not rely on the specific implementation carried out in this study. In summary, the main contributions of this work can be compiled as follows: A novel region-based anchor optimization methodology using evolutionary algorithms for 2D object detection for autonomous driving A module that enhances the second-stage header network of Faster R-CNN by including the spatial features of ROIs produced by the region proposal network. A thorough study of different training procedures to address severe foreground-foreground class imbalance in two-stage object detectors. An ensemble model combining the strengths of the different learning strategies to improve detection precision. The rest of the paper is organized as follows: Section 2presents a review of related work; in Section 3the materials used and the methods proposed in the study are described; Section 5reports and discusses the results obtained; Section 6presents the conclusions and potential future work. 2. Related work Recent progress in the object detection field has been driven by novel methodologies based on deep learning, as it has happened in many computer vision tasks [16]. Existing image object detectors in the literature can be mainly divided into two categories: twostage detectors such as Faster R-CNN [17], and one-stage detectors such as SSD [18]. Generally speaking, the strength of one-stage detectors lies in their higher inference speed, while two-stage architectures obtain higher localization accuracy. The pioneering two-stage detector was the Regions with CNN features framework (R-CNN) [19]. R-CNN used the selective search method to crop box proposals from the image and feed them to a convolutional network classifier. This external proposal generation was very costly and inefficient. Faster R-CNN solved this issue by sharing features between the region proposal network (RPN) and the detection network [17]. This approach improved accuracy and speed and has led to a large number of follow-up works. For instance, R-FCN proposed a position-sensitive ROI cropping that respects translation variance [20]. Later studies have focused on exploiting the multi-scale properties of feature extractors, such as the feature pyramid networks (FPNs) with lateral connections proposed in [21]. Other works have tried to improve this detector by including rotation-invariant and Fisher discriminative regularizers on the CNN features [22]. Due to the cost of manually labeling bounding boxes, there are also many studies on weakly-supervised object detection that only work with image-level labels [23]. Currently, the COCO leaderboard is led by Cascade R-CNN methods. Cascade models build a sequence of detectors trained with increasing intersection-over-union (IoU) thresholds, to be sequentially more selective against false positives [24]. In contrast, one-stage architectures predict class probabilities and bounding box offsets directly from the image, without the region proposal step. The first YOLO (You Only Look Once) architecture was proposed in [25], achieving real-time inference rates but with high localization errors. SSD combined ideas from YOLO and RPN to improve the performance while maintaining high speed [18]. With the help of default bounding boxes, SSD detects objects at different scales on several feature maps. More recently, an important step forward in the one-stage family was achieved by solving the foreground-background class imbalance problem. RetinaNet proposed a novel focal loss function that focuses on difficult objects by down-weighting the importance of well-classified samples [26]. Other interesting approaches are the anchor-free one-stage detectors, which do not require pre-defined anchor boxes. FCOS [27] and CenterNet [28] use the center of objects to define positives and regress the four distances that build the M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 230 bounding box from that point. Models such as ExtremeNet [29] or CornerNet [30] generate the boxes by locating several keypoints first. Anchor-free detectors are more flexible than RetinaNet and can achieve similar performance. The backbone network that acts as a feature extractor and its capacity to extract quality features play a very important role in all detection frameworks. Rather than the VGG network [31] used in the original Faster R-CNN paper, deeper and more densely connected architectures have been recently proposed. Some examples include the ResNet [32] used in the Mask R-CNN detector [33], ResNeXt [34], Res2Net [35], or HRNet [36]. The improved YOLOv3 detector includes multi-scale predictions and a new feature extractor, DarkNet-53, which uses residual blocks and skip connections [37]. Other works such as NAS-FPN [38] introduce neural architecture search to learn optimal feature fusion in the pyramid and build a stronger backbone. However, these complex networks lead to slower inference speed. Since this is undesired in real-time applications, other researchers have focused on designing lightweight backbones such as MobileNets [39], which are also less accurate. In general, finding the optimal speed/accuracy balance in a backbone architecture is a difficult task that highly depends on the problem to be addressed [40]. The interest in the autonomous driving field has risen significantly in recent years. Many high-quality datasets are becoming available for the research community to push the state-of-the-art in problems such as object detection. After the popular KITTI benchmark [15], other datasets have been released such as NuScenes [41] or PandaSet [42]. A complete overview of existing selfdriving datasets is provided in [43]. Waymo has recently released the most extensive and diverse multi-modal dataset up to date [14].In[44], the authors evaluate the trade-off between accuracy and speed of several state-of-the-art detectors over the Waymo dataset. However, there are still few works that have addressed the optimization of existing 2D object detectors in the context of autonomous vehicles. RefineNet proposed extra regressors to further refine the candidate bounding boxes for vehicle detection [45]. The work in [46] presents an anchor optimization methodology and ROI assignment improvement over two-stage detectors but also focusing only on vehicles and not on other traffic participants. A CNN-based methodology to improve vehicle detection in adverse weather conditions was presented in [47]. There are other important perception problems in autonomous driving being addressed with deep learning such as 3D detection using LiDAR point clouds [48,49] and object tracking [50,51]. 3. Materials and methods This section presents the dataset used for the study and the methodology proposed to enhance object detection in the context of autonomous vehicles. Firstly, the anchor optimization procedure and the modifications applied to the Faster R-CNN architecture are described. Secondly, the different learning strategies studied to address class imbalance are explained. Thirdly, the proposed ensemble model is presented. In the final section, the remaining implementation details are provided to allow reproducibility. The complete source code can be found at [52]. 3.1. Waymo Open Dataset The Waymo Open Dataset [14] consists of 1150 driving video scenes across different urban areas (Phoenix, San Francisco, and Mountain View) and at different times of the day (day, night, and dawn). Each scene captures synchronized LiDAR and camera data for 20 s, resulting in around 200 frames per scene. The problem addressed in this study is 2D object detection. This task is to assign 2D bounding boxes to objects that are present in a single RGB camera image. Waymo’s vehicle is equipped with five high-resolution cameras (Front, Front Left, Front Right, Side Left, and Side Right). Frontal cameras obtain images with a resolution of 1920x1280, while lateral cameras have an image size of 1920x886. All cameras have a 25:2  horizontal field of view (HFOV). An example of the images captured by all cameras in a single frame is presented in Fig. 1. As it is done in Waymo’s online challenge, the images obtained from all cameras are considered a single dataset for evaluation purposes. The dataset contains around 10 million manually annotated labels across all cameras. Three different classes are considered for this problem: vehicles (which includes any wheeled motor object such as cars or motorbikes), pedestrians, and cyclists. Fig. 2a shows an example of the labeled data provided, which are tightly fitting bounding boxes around the objects. Furthermore, Waymo provides two different difficulty levels for the labels (Level 1 and 2), which are illustrated in Fig. 2b. Level 2 instances are objects considered as hard and the criteria depends on both the human labelers and the object statistics. For the evaluation, the level 2 metrics are cumulative and also include all objects belonging to level 1. The count of objects of the different classes is presented in Table 1. As can be seen, there is a significant class imbalance between vehicles and pedestrians, and the number of cyclist labels is minimal. The dataset is divided into 1000 scenes for training and validation (around 1 million images), and 150 for testing (around 150 k images). Waymo provides an online submission tool to evaluate the models over the testing set since those labels are not publicly available. The scenes composing the test set are from a different geographical area, which ensures that the capacity of generalization of trained models is properly evaluated. 3.2. Faster R-CNN architecture Faster R-CNN has been extensively used in the recent literature as a general-purpose object detection framework [17]. This detector follows a multi-task learning procedure, combining classification and bounding box regression to solve the detection problem. It uses a convolutional backbone (e.g. VGG, ResNet) to extract hierarchical features from the images, and consists of two stages: a region proposal network (RPN) and a Fast R-CNN header network. Fig. 3 shows the Faster R-CNN architecture, illustrating the complete two-stage process. In the first stage, the RPN uses features from an intermediate level of the feature extractor to predict class-agnostic box proposals (object or background). This is achieved by predicting multiple candidate boxes at each location using multi-scale reference anchors. Afterwards, a limited number of these proposals (typically 300) are selected as regions of interest (ROIs) and pass to the second stage. The selected ROIs are used to crop features from the same intermediate feature map using a ROI pooling operation. Those cropped features are then fed to the remaining layers of the backbone network to predict a class and perform a box refinement for each proposal. In Faster R-CNN, convolutional features are shared between both stages, which improves accuracy and speed. However, its bottleneck is the number of ROIs proposed by the RPN, since the computation of the second stage is run once per proposal. In this study, the ResNet-101 backbone network is used, since it provides a good speed/accuracy trade-off [53]. It is also the network provided for the baseline results published with the Waymo dataset, which allows a fair comparison. As in the original implementation [32], features used in the RPN are extracted from block 3 of the ResNet. In the second stage, ROIs are cropped and resized to 14x14, and then max-pooled to 7x7 before being fed to block 4. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 231 However, our proposal does not rely on this specific backbone and can be used with other existing anchor-based two-stage detectors. Despite its advantages, there are several aspects of Faster R- CNN that can be improved to better adapt it to the characteristics of the problem addressed in this study, which is object detection in the autonomous driving scenario. The improvements proposed to the original architecture are highlighted in red in Fig. 3: the perregion anchor generation optimization in the RPN; and the ROIs spatial features concatenation in the header classification network. The details of these modifications are provided in the following sections. 3.3. Anchor generation optimization The first stage of the Faster R-CNN is the region proposal network (RPN) that selects the ROIs to be forwarded to the detection stage. The ROIs are generated using a convolutional sliding window over some intermediate feature map of the backbone network. These proposals are parametrized relative to reference boxes known as anchors. In order to detect different-sized objects, the RPN predicts multiple region proposals at each location by using multi-scale anchors. There are kanchor boxes with different scales and aspect ratios centered at each pixel of the feature map. The scale ratio is defined with respect to a base of 256, and the aspect ratio is the width over the height of the box. The size and shape of these anchors have to be manually defined and are critical for the success of the detector [54]. In this self-driving scenario, the shape of objects captured by the cameras can be significantly different depending on their class and position in the image. The shape of pedestrians tends to be tall and narrow, while vehicles are often wider and more squared. Furthermore, the perspective projection of the cameras equipped in autonomous vehicles is an important factor that must be considered for object detection [46]. The original RPN implementation in Faster R-CNN using ResNet proposes multiple anchors with different scales ratios ð0:25;0:5;1;2Þand aspect ratios ð0:5;1;2Þat each location [32]. This uniform configuration was found to work well for general-purpose object detection, but it is far from optimal in this particular application. Those pre-defined scales and aspect Table 1 Count of labeled objects in the Waymo dataset. Vehicle Pedestrian Cyclist Count 7.7 M 2.1 M 63 K Percentage 78.07% 21.29% 0.64% Fig. 1. Images from a single frame obtained by the five cameras of Waymo’s vehicle. Fig. 2. 2D object detection problem in the Waymo Open Dataset. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 232 ratios do not coincide with the size of the objects seen from onboard cameras, hence resulting in many invalid anchors and useless computation. Therefore, in this study, the aim is to find a better configuration for those values of scale and aspect ratio. Due to the perspective of the cameras in the vehicle, the size of the captured objects highly depends on their position in the image. Therefore, one of the first steps of this study is to analyze the relationship between bounding box dimensions and their location in the image. Fig. 4 displays the distribution of all objects in the dataset with respect to two variables: the vertical position of the center of the object (y-axis center), and the object’s height. Due to the different dimensions and nature of frontal and lateral cameras, the analysis is separated for both of them to check for any significant particularities. As can be seen in the figure, there is a strong correlation between both variables. For frontal cameras, there is a 0.67 Pearson correlation coefficient, while for lateral cameras the correlation is 0.69. This positive correlation implies that objects of larger size tend to appear at the bottom part of the image, while smaller objects are more often at the top part of the image, as they are further away. These findings confirm the fact that a uniform anchor generation across the image is not optimal for this context. This anchor dimension mismatch can significantly decrease the performance of the detector. The next steps of the paper focus on how the anchor generation process has been modified to adapt it for this specific problem and improve the detection accuracy. Our perspective-aware proposal is divided into two steps: the division of the images in key regions using a clustering analysis, and the per-region anchor optimization using evolutionary algorithms. 3.3.1. Region division using clustering Given the different characteristics of objects depending on their location in the image, the objective is to find a better anchor generation methodology that accounts for the perspective projection. In contrast to the default uniform generation method, the proposal is to divide the image into several key regions and obtain the best configuration for each of them independently. In order to find the optimal division in regions, a clustering analysis is performed with respect to the aspect and scale ratio of all 10 million objects in the dataset. When the K-Means algorithm is applied, the best division is obtained with two clusters after analyzing internal validity indices such as the Silhouette index. Fig. 5 shows the distribution of elements in the cluster according to the values of both features. As can be seen, while the aspect ratio is similar for the elements in both clusters, there is a significant difference in the scale ratio. The distribution plot shows that the K-Means clustering algorithm assigns to cluster 1 the majority of the larger scale elements (those that have a scale ratio greater than 1) and to cluster 0 the smaller scale objects (those that have a scale ratio below 1). Fig. 3. Faster R-CNN architecture. The improvements proposed in this study are highlighted in red. Nrefers to the number of ROIs passing to the second stage. Fig. 4. Correlation between the object size and its location in the image. The (0,0) point refers to the top left corner of the image. The regression line illustrates the strong positive correlation. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 233 With the clustering results, the region division can be obtained by analyzing the position in the image of the objects belonging to each cluster, which is illustrated in Fig. 6a. This figure plots the center of objects in both x and y-axis (horizontal and vertical position). Frontal and lateral cameras are combined for this clustering study, hence a normalized value is used for the vertical position of the objects. As can be seen in the figure, the majority of elements of cluster 1 (larger objects) are in the bottom part of the image. In contrast, elements in cluster 0 are spread across the middle and upper part of the image. The spatial bounds that delimit the clusters can be found with the help of the density distribution of the position of elements. For each cluster, we define the interval delimited by two values a and bthat contains 99% of the elements, in order to allow for the presence of outliers. Those two values define the interval in which the majority of elements of each cluster are positioned in the image. Fig. 6a displays the a and bvalues with dotted red lines. For cluster 0, the bounds are at positions 0.188 and 0.691. For cluster 1, only the first bound (0.392) is displayed since the elements spread until the bottom of the image. If those bounds are combined, we can establish four key regions in the image that will have objects with significantly different characteristics. The final result of the region division study is depicted in Fig. 5. Distribution of the aspect and scale ratio of the elements belonging to each cluster.. Fig. 6. Division of the camera images into four regions according to the clustering study. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 234 Fig. 6b, in which some example images are shown with the bounds for the four regions (R1, R2, R3, and R4). These images clearly illustrate the difference in scale between objects of different regions. 3.3.2. Evolutionary algorithm to optimize anchors per region Once the region division has been defined, an evolutionary algorithm (EA) is designed to find the optimal values for scale and aspect ratios for each of them. An evolutionary algorithm minimizes a fitness function by exploring a population of possible solutions over a specific number of generations. The population is formed by individuals that are represented as a chromosome, which are a combination of genes. For this problem, the chromosome is encoded as a set of 7 floating-point numbers representing the parameters that define the anchors. The first three numbers correspond to the aspect ratios and the remaining four correspond to the scale ratios. The objective is to search for the best value of each gene within defined boundaries and with a specified decimal precision. The EA creates a random initial population of a specific size. From the initial population, the algorithm combines the best individuals using single-point crossover and creates a new generation. The crossover is applied separately to the aspect ratio and scale ratio genes, so that there is no interference between the two parameters. Additionally, a low mutation probability is added to maintain genetic diversity over generations. The complete parameter configuration of the proposed EA is presented in Table 2. The gene boundaries are defined considering the size of images and the limitations of using ResNet-101 as a backbone network. Given the base anchor scale of 256, the specified boundaries imply that the maximum possible scale is 1024 pixels and the minimum is 16 pixels (which is the receptive field unit of the network). An individual can be represented as X¼ðA;SÞ, where A¼ða 1 ;a 2 ;a 3 Þ defines the three possible aspect ratios and S¼ðs 1 ;s 2 ;s 3 ;s 4 Þdefines the four possible scale ratios. The decoded individual defines a set of 12 anchor configurations B a ij , which are conformed by the cartesian product between aspect and scale ratio (B a ij ¼AS). Therefore, the whole decoding process of an individual Xcan be expressed as follows: X¼fB a ij g;where B a ij ¼ða i ;s j Þfor i2f1; ::; 3g;j 2f1; ::; 4gð1Þ The goal of the proposed EA is to maximize the intersection between the anchors generated from the decoded individual and the ground truth boxes in the images. This is measured using the Intersection over Union (IoU) metric, which is presented in Eq. (2). The IoU is defined as the ratio of area-of-overlap to area-of- union of a ground truth bounding box B gt and a proposed base anchor B a : IoUðB gt ;B a Þ¼areaðB gt \B a Þ areaðB gt [B a Þ ¼areaðB gt \B a Þ areaðB gt ÞþareaðB a ÞareaðB gt \B a Þð2Þ The IoU metric is used for defining the EA fitness function (Eq. (3), which evaluates the quality of the solutions provided by individuals in the population. For an individual X, the decoded chromosome generates the 12 possible anchors (4 scales and 3 aspect ratios). For each ground truth bounding box, the fitness function finds the maximum overlap obtained with the proposed anchor boxes and averages the result among all objects. A logarithmic factor is applied in order to ensure that there are fewer ground truth boxes with a maximum IoU lower than 0.5 [55]. In the equation, Krefers to the number of ground truth objects. FitnessðXÞ¼ 1 K X K k¼1 1maxIoUðB gt k ;XÞ  2 log maxIoUðB gt k ;XÞ   maxIoUðB gt k ;XÞ¼ max i¼1::3 j¼1::4 IoUðB gt k ;B a ij Þ  ð3Þ Table 3 presents the optimized parameters that are obtained for each region. As can be seen, the optimal values found by the evolutionary algorithm are significantly different from the original configuration. Regions at the top of the image with smaller objects have lower scale ratios. The scale in R1 and R2 ranges from values close to zero and up to a maximum of 0.5, which means that default anchors of scales 1 and 2 are not suitable. In contrast, R4 accounts for the presence of larger objects with values ranging from 0.6 until 2.9. Fig. 7 presents an example that shows the difference between the uniform anchor generation (red boxes) and our optimized proposal (blue boxes). The dotted red boxes illustrate useless default anchors that have a great size mismatch with objects in different regions. In regions with small objects, the original strategy generates useless large-scale anchors. Analogously, in the bottom part of the image where there are objects closer to the camera, it generates very small inefficient anchors. Our proposal generates anchors with a higher matching precision, which results in a more effective detector. In the experimental study, the parameter optimization process is also analyzed separating frontal and lateral cameras, in order to check if there is a further improvement. 3.4. ROIs spatial features concatenation As it was seen in the previous Section 3.3, the spatial location of objects in this context plays an important role in the detection problem. Considering this fact, we propose a slight yet effective modification to the second stage of the Faster R-CNN framework. The information used by the second stage only comes from the convolutional features cropped from an intermediate feature map of the backbone network, which are then max-pooled to a 771024 map. This implies that the spatial characteristics of each box proposal are lost in the ROI pooling process. The size (width and height) and position of the proposed bounding box Table 2 Parameter configuration of the evolutionary algorithm Parameter Value Gene boundaries [0.06, 4] Gene precision 10 3 Crossover probability 0.8 Mutation probability 0.2 Population size 100 Num. Generations 50 Table 3 Optimized anchor configuration found with the evolutionary algorithm. The values for scale and aspect ratios for each region are provided. Parameter Region Values Scale Ratio Original 0.25, 0.5, 1, 2 R1 0.074, 0.158, 0.250, 0.414 R2 0.082, 0.155, 0.254, 0.500 R3 0.095, 0.189, 0.518, 1.557 R4 0.598, 1.101, 1.800, 2.852 Aspect ratio Original 0.5, 1, 2 R1 0.344, 0.672, 1.801 R2 0.461, 0.853, 2.246 R3 0.473, 0.905, 2.497 R4 0.314, 0.805, 2.136 M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 235 with respect to the complete image are not taken into account by the Fast-RCNN header network. In this multi-class object detection problem with images from on-board cameras of a vehicle, the position of the box has a significant correlation with its dimensions. Therefore, including this information in the class-specific prediction and box refinement done in the second stage can enhance the localization accuracy. For these reasons, our proposal is to construct those spatial features and concatenate them to the convolutional features of each ROI obtained from the cropped map. This process is illustrated in the flowchart displayed in Fig. 8, and in Fig. 3 with more detail. For each of the Nselected ROIs, cropped features are fed to the fourth block of the ResNet backbone. Then, the network is divided into two branches for different tasks: classification and boundingbox regression. In each branch, the convolutional features are flattened using a spatial average pooling, which converts the N772048 maps into N2048 neurons. At this point, four features are concatenated for each ROI proposal in both branches: the width of the box, the height of the box, the horizontal position of the center of the box (X center), and the vertical position of the center of the box (Y center). This results in a layer of N2052 neurons. Those combined features are then used to predict the class and refine the boxes by means of a fully connected (FC) layer. 3.5. Learning strategies to address class imbalance Class imbalance is an important issue that may severely degrade the performance of detectors if it is not properly addressed [56]. In object detection, there are two different class imbalance problems: background-foreground and foreground-foreground. The background-foreground imbalance is associated with the small number of positive examples (bounding boxes matching an object) compared to the number of negatives boxes (background of the image). The foreground-foreground problem refers to the imbalance between object classes of a dataset, which often leads to overfitting on the over-represented class. Since the foregroundbackground imbalance is unavoidable and does not depend on the specific dataset, it has attracted more interest in the recent literature. However, in the self-driving dataset used in this study, there is a significant imbalance between vehicles (80%) and the other type of objects (pedestrians 20% and cyclists less than 1%). In order to alleviate the foreground-background imbalance, Faster R-CNN uses hard sampling techniques based on heuristic methods. In the first stage, random sampling is used for training the RPN. To avoid a bias in the learning towards negative samples, 128 positive and 128 negative examples are randomly selected to contribute to the loss function. In the second stage, the employed strategy is to limit the search space. For the Fast R-CNN detection network, only the best NROIs are selected according to their objectness scores, while maintaining a 1:3 positive–negative ratio. However, this methodology presents the problem that all examples are equally weighted once they are sampled. The original Faster R- CNN implementation does not take into account the presence of under-represented objects, which limits performance under high imbalance. Therefore, we aim to design a better training procedure that addresses the foreground-foreground imbalance in this scenario. Our proposal is to study how the loss function of two-stage detectors can be modified using cost-sensitive re-weighting techniques, which has been one of the main approaches in the litera- Fig. 7. Difference between the uniform anchor generation procedure and our anchor optimization proposal. The complete image is displayed in the center, with three zoomed areas. The dotted red lines illustrate cases with an important anchor size mismatch when using the default anchor configuration. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 236 ture for class imbalance problems [57]. These techniques are based on assigning relatively higher costs to minority instances, hence building a better class-balanced loss. In this work, we explore two re-weighting alternatives that can improve overall detection accuracy: balance the loss function with weights that are proportional to the class distribution; and the use of focal loss, which has been traditionally used for one-stage detectors. 3.5.1. Weight assignment based on class distribution The first approach is to assign different weights to instances of each class depending on the class frequency distribution. This simple yet effective method has been widely used in many computer vision tasks [58]. As stated before, in Faster R-CNN, the training is framed as a multi-task learning problem that combines classification and bounding box regression. We propose to modify the loss function by adding a weight w i parameter to the classification and regression terms. The complete loss function with weights assignment is expressed in Eq. (4): Lðp i ;t i Þ¼L cls þL reg ¼ 1 N cls X i w i L cls ðp i ;p  i Þþ k N reg X i w i p  i L smooth 1 ðt i t  i Þ L cls ðp i ;p  i Þ¼CEðp i ;p  i Þ¼p  i logp i ð1p  i Þlogð1p i Þ ð4Þ where p i is the predicted probability of proposal ibeing an object, and p i is the groundtruth label (0 or 1). t i are the four predicted box coordinates, and t i are the ground truth coordinates. w i is the weight assigned depending on the ground truth class. N cls and N reg are normalization terms which are set to the RPN mini-batch size, which is typically 256. L cls is the binary cross-entropy function and L reg is the smooth L1 loss function. Note that we have described only the RPN loss function for simplicity, but the modification is done in both stages. The loss function in the second stage is the same, except that in the RPN the classification problem is binary (background vs object) while in the Fast R-CNN header network the classification is multi-class. This implies that the multi-class cross-entropy function is used. In this case, the normalization parameters are set to the number of ROI proposals that pass to the second stage. In order to obtain the best possible performance, a grid search is performed to find the optimal weight value for each class. The search involves experimenting with different sets of weights, considering four possible values ð0:3;0:5;0:7;0:9Þthat can be assigned to vehicles and pedestrians. The maximum weight ð1:0Þis kept for cyclists since they are extremely underrepresented. 3.5.2. Reduced focal loss The second re-weighting alternative that is explored is focal loss, which was introduced in [26] as an improvement to the RetinaNet one-stage detector. The focal loss function belongs to another line of work that assigns weights according to sample difficulty. These methods assume that training a detector with hard examples improves performance. A method of this family that has been used in two-stage detectors is Online Hard Example Mining (OHEM) [59]. However, OHEM requires more memory consumption and increases training time. Although focal loss was found to be more effective than OHEM for imbalance problems, it has not been used extensively in two-stage detectors. In this work, we study the application of the focal loss function to the Faster R- CNN framework and propose a modification to better adjust it to its characteristics. Focal loss (FL) assigns higher weights to hard examples, with the aim of alleviating the high background-foreground imbalance. As is shown in Eq. (5), focal loss modifies the standard crossentropy equation by adding a factor ð1p i Þ c . When c >0, the relative loss for easy and well-classified samples is reduced, putting the effort on the classification of hard examples. FLðp i Þ¼ a ð1p i Þ c logðp i Þð5Þ Although this function was introduced to reduce the influence of easy background examples, it also has an effect on the foreground-foreground imbalance. Instances from minority classes often have higher losses since they are rare and their features are usually poorer. However, the direct application of focal loss in two-stage detectors presents several problems. Firstly, focal loss can have as a side-effect that the learning is biased towards noisy or mislabeled data, which is also hard to classify [57]. Furthermore, focal loss contradicts slightly the behaviour of two-stage detectors: RPN aims to maximize recall (allowing false positives), while the job of Fast R-CNN header network is to classify proposals correctly. An extreme focus on hard samples can reduce the recall of RPN, which implies that lower-quality proposals pass to the second stage. Therefore, inspired by the work in [60], we propose a modified version called reduced focal loss (RFL). Reduced focal loss aims to perform hard example mining but softening the effect of difficult samples. This is achieved by applying the factor only to instances with losses that are above a certain threshold. The loss of samples that are below the threshold remains unaltered, which means that is the same as the original cross-entropy loss. This approach is formulated in Eq. 6. RFL is applied in the classification term of the loss function in both stages of the Faster R-CNN detector. RFLðp i Þ¼ a RFðp i ;thÞlogðp i Þ RFðp i ;thÞ¼ 1ifp i <th ð1p i Þ c th c if p i Pth (ð6Þ Following [60], the thresholds are fixed to 0.5 and 0.25 for the RPN and Fast R-CNN respectively. RFL is more suitable than the original focal loss for two-stage detectors, and can significantly Fig. 8. Flowchart illustrating the ROIs spatial features concatenation process. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 237 [51] X. Dong, J. Shen, D. Wu, K. Guo, X. Jin, F. Porikli, Quadruplet network with oneshot learning for fast visual object tracking, IEEE Transactions on Image Processing 28 (7) (2019) 3516–3527, https://doi.org/10.1109/ TIP.2019.2898567. [52] M. Carranza-García, P. Lara-Benítez, Faster R-CNN optimization for Waymo 2D object detection, Available online: URL:https://github.com/carranza96/ waymo-detection-optimization.git, (Accessed 4 July 2020) (2020).. [53] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, K. Murphy, Speed/accuracy trade-offs for modern convolutional object detectors, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 (2017) 3296–3297, https://doi.org/10.1109/ CVPR.2017.351. [54] M. Ahmad, M. Abdullah, D. Han, Small Object Detection in Aerial Imagery using RetinaNet with Anchor Optimization, in: 2020 International Conference on Electronics, Information, and Communication (ICEIC) (2020) 1–3. https://doi. org/10.1109/ICEIC49074.2020.9051269. [55] M. Zlocha, Q. Dou, B. Glocker, Improving RetinaNet for CT Lesion Detection with Dense Masks from Weak RECIST Labels, CoRR abs/1906.02283 (2019). arXiv:1906.02283. [56] K. Oksuz, B.C. Cam, S. Kalkan, E. Akbas, Imbalance problems in object detection: a review, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020) 1, https://doi.org/10.1109/TPAMI.2020.2981890. [57] Y. Cui, M. Jia, T. Lin, Y. Song, S. Belongie, Class-balanced loss based on effective number of samples, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019 (2019) 9260–9269, https://doi.org/10.1109/ CVPR.2019.00949. [58] C. Huang, Y. Li, C.C. Loy, X. Tang, Learning deep representation for imbalanced classification, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 (2016) 5375–5384, https://doi.org/10.1109/CVPR.2016.580. [59] A. Shrivastava, A. Gupta, R. Girshick, Training region-based object detectors with online hard example mining, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 (2016) 761–769, https://doi.org/10.1109/ CVPR.2016.89. [60] N. Sergievskiy, A. Ponamarev, Reduced focal loss: 1st place solution to xview object detection in satellite imagery, CoRR abs/1903.01347 (2019). arXiv: 1903.01347. [61] J. Xu, W. Wang, H. Wang, J. Guo, Multi-model ensemble with rich spatial information for object detection, Pattern Recognition 99 (2020), https://doi. org/10.1016/j.patcog.2019.107098 107098. [62] J. Hosang, R. Benenson, B. Schiele, Learning non-maximum suppression, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017 (2017) 6469–6477, https://doi.org/10.1109/CVPR.2017.685. [63] Á. Casado-García, J. Heras, Ensemble methods for object detection, ECAI (2020), https://doi.org/10.3233/FAIA200407. [64] J. Huang, V. Rathod, C. Sun, Tensorflow object detection api, Available online: URL: https://github.com/tensorflow/models/tree/master/research/object_ detection, (Accessed 19 June 2020) (2020).. [65] Z. Zou, Z. Shi, Y. Guo, J. Ye, Object detection in 20 years: a survey, CoRR (2019). arXiv:1905.05055. Manuel Carranza García received his Computer Science Master’s degree from University of Seville (Spain) in 2019 and is currently pursuing his Ph.D. in Computer Science at the University of Seville, Spain. His research interests include deep learning, computer vision, and remote sensing. Pedro Lara Benítez received his Computer Science Master’s degree from University of Seville (Spain) in 2019 and is currently pursuing his Ph.D. in Computer Science at the University of Seville, Spain. His research focuses on deep learning, time-series forecasting, and online learning. Jorge García Gutiérrez received the Ph.D. degree in Computer Engineering from the University of Seville, Spain, in 2012. He has been working for the Department of Computer Science of the University of Seville since 2008 where he is currently Lecturer Professor. His primary areas of interest are machine learning techniques, big data, remote sensing, data fusion and evolutionary computation. José C. Riquelme received the M.Sc. degree in Mathematics and the Ph.D. degree in Computer Science from the University of Seville, Spain. Since 1987 he has been with the Department of Computer Science, University of Seville, where he is currently Professor. His primary areas of interest are data mining, machine learning techniques and evolutionary computation. M. Carranza-García, P. Lara-Benítez, J. García-Gutiérrez et al. Neurocomputing 449 (2021) 229–244 244