Deep learning-based object detection for autonomous driving

UNIVERSITÀ DEGLI STUDI DI PARMA DOTTORATO DI RICERCA IN “TECNOLOGIE DELL’INFORMAZIONE” CICLO XXXIII Deep Learning-based Object Detection for Autonomous Driving Coordinatore: Chiar.mo Prof. Marco Locatelli Tutore: Chiar.mo Prof. Massimo Bertozzi Dottorando: Andrea Zinelli Anni 2018/2021 Contents Introduction 1 1 Prior Art 7 1.1 What are Deep Neural Networks? . . . . . . . . . . . . . . . . . 7 1.1.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . 7 1.1.2 Linear Basis Function Regression . . . . . . . . . . . . . 9 1.1.3 Parametric Basis Functions: The most basic Neural Network............................. 10 1.1.4 From Shallow to Deep Models . . . . . . . . . . . . . . . 12 1.1.5 Training Neural Networks . . . . . . . . . . . . . . . . . 13 1.2 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . 15 1.2.1 The convolutional operator . . . . . . . . . . . . . . . . 16 1.2.2 Convolutional Neural Networks . . . . . . . . . . . . . . 17 1.3 The First Large Scale Models for Image Classification . . . . . 20 1.4 Semantic Segmentation . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Objectdetection .......................... 26 1.5.1 2D Object Detection on Images . . . . . . . . . . . . . . 27 1.5.2 Tradeoff between One-stage and Two-stage techniques . 32 1.6 3D Object Detection . . . . . . . . . . . . . . . . . . . . . . . . 33 1.6.1 LiDAR-based 3D Object Detection . . . . . . . . . . . . 34 1.6.2 Image-based 3D Object Detection . . . . . . . . . . . . 41 ii Contents 2 Object Detection for Parking Slot Detection 47 2.1 Prior Art and Motivation . . . . . . . . . . . . . . . . . . . . . 47 2.2 FasterR-CNN............................ 49 2.2.1 Training Procedure . . . . . . . . . . . . . . . . . . . . . 53 2.3 Parking Slot Detector . . . . . . . . . . . . . . . . . . . . . . . 54 2.3.1 Training Procedure . . . . . . . . . . . . . . . . . . . . . 57 2.3.2 Dataset Construction and Data Preparation . . . . . . . 60 2.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . 61 2.4.1 Semantic Shift Problem . . . . . . . . . . . . . . . . . . 63 2.4.2 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . 65 2.4.3 Model Simplification and Sparsification . . . . . . . . . 68 2.4.4 Prediction Directly on Spherical Images . . . . . . . . . 73 2.5 Discussion.............................. 77 3 Monocular 3D Object Detection via Generalized Intersectionover-Union Minimization 79 3.1 Prior Art and Motivation . . . . . . . . . . . . . . . . . . . . . 79 3.2 BaselineModel ........................... 82 3.3 3D Detection Module . . . . . . . . . . . . . . . . . . . . . . . . 83 3.4 Model Optimization . . . . . . . . . . . . . . . . . . . . . . . . 87 3.4.1 Generalized Intersection-over-Union . . . . . . . . . . . 87 3.4.2 Training Procedure . . . . . . . . . . . . . . . . . . . . . 89 3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . 92 3.5.1 The KITTI Dataset . . . . . . . . . . . . . . . . . . . . 92 3.5.2 Comparison with the State of the Art . . . . . . . . . . 93 3.5.3 Comparison with other Loss Formulations . . . . . . . . 98 3.5.4 Qualitative results . . . . . . . . . . . . . . . . . . . . . 102 3.6 A case study: 3D GIoU applied to Frustum-PointNets . . . . . . . . . . . . . 105 3.6.1 3D detector: standard optimization . . . . . . . . . . . . 106 3.6.2 3D detector: proposed optimization . . . . . . . . . . . . 107 Contents iii 3.6.3 Experimental Results . . . . . . . . . . . . . . . . . . . 108 3.7 Discussion.............................. 110 4 3D Object Detection on LiDAR scans via Voting and SelfAttention Mechanisms 113 4.1 Prior Art and Motivation . . . . . . . . . . . . . . . . . . . . . 113 4.2 Votenet for 3D Object Detection on Driving Scenarios . . . . . 115 4.2.1 PointNets for point cloud processing . . . . . . . . . . . 115 4.2.2 Baseline Votenet Model for 3D Object Detection . . . . 119 4.2.3 Modifications to the baseline for Autonomous Driving Scenarios .......................... 122 4.3 Enhancing SA layers via Self-Attention . . . . . . . . . . . . . . 127 4.3.1 The attention mechanism . . . . . . . . . . . . . . . . . 128 4.3.2 Self-attention applied to SA layers . . . . . . . . . . . . 131 4.4 TrainingSetup ........................... 135 4.4.1 KITTI LiDAR Dataset . . . . . . . . . . . . . . . . . . . 135 4.4.2 Model architecture . . . . . . . . . . . . . . . . . . . . . 135 4.4.3 Training Routine . . . . . . . . . . . . . . . . . . . . . . 138 4.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . 139 4.5.1 Ablation study on the design choice . . . . . . . . . . . 140 4.5.2 Ablation study on attention type . . . . . . . . . . . . . 142 4.5.3 Comparison with State-of-the-Art Systems . . . . . . . . 144 4.5.4 Qualitative results . . . . . . . . . . . . . . . . . . . . . 147 4.6 Discussion.............................. 150 Conclusions 153 Bibliography 157 Introduction Recent advancements in parallel computing hardware and frameworks, coupled with the enormous increase in publicly available annotated datasets, has caused a resurgence in interest from the research community on the topic of machine learning and, in particular, neural networks. In the last decade alone, an extremely wide variety of different neural architectures and models have been proposed to tackle the most disparate problems, including but not limited to sentence translation, image generation, image classification, detection, localization and planning. In particular, the release of increasingly powerful Graphical Processing Units (GPUs) allowed for the development of more complex neural network designs, giving rise to the research field that is currently known as deep learning. Amid these designs, perhaps the most important and notorious one is represented by Convolutional Neural Networks (CNNs), a category of neural network that utilizes multiple stacked layers of learnable convolutional kernels to build powerful hierarchical representations of input images and carry out complex visual tasks [1, 2, 3, 4, 5, 6, 7, 8]. All these solutions have shown remarkable performance, often surpassing "traditional" algorithms by wide margins while being more efficient due to their high parallelism. As a result, many "classical" algorithms are gradually being replaced by deep learning-based solutions, bringing about the so-called age of Software 2.0: the software engineer no longer hand-designs the logic of the algorithm directly, but rather designs the neural network model that learns the most suitable logic from the data. 2 Introduction Perhaps one of the industries most affected from this shift in paradigm is automotive: powered by this emergent technology, extensively researched and applied to the field of computer vision, Advanced Driver Assistance Systems (ADAS) and Autonomous Driving solutions have benefitted from tremendous leaps in performance. Among all the tasks that are required for successful autonomous navigation, one of the most critical is perception, which consists of interpreting the signals captured by the sensor suite displaced on the vehicle such that they can successfully be used for the subsequent tasks of tracking, planning and control. Most of the research on deep learning-based perception in the field of autonomous driving concentrates on two specific kinds of sensors, RGB cameras and LiDARs, which are characterized by complementary strengths and weaknesses. Cameras provide a much denser and richer type of signal, while also being considerably cheaper than the LiDAR solutions currently available on the market; on the other hand, they do not provide any information about the distance of the perceived objects from the sensor, and thus require stereoscopic setups to be able to infer the scene depth. Conversely, LiDARs provide an accurate reconstruction of the surrounding environment in the form of a point cloud, but such signal is considerably sparser and does not contain additional semantic information such as color. Given this data, many deep learning models and techniques have been developed that carry out different types of perception tasks. A particularly important one towards fully automated navigation is object detection, which can be defined as follows: given an input, which is the signal returned by one or more sensors, the objective is to identify all entities of interest inside such input and determine their state. Such state can represent any properties of the detected object: image-based 2D detectors [9, 10, 11], for instance, usually estimate an image-aligned bounding box that contains said object, as well as the class it belongs to. 3D detectors [12, 13, 14, 15, 16, 17] estimate 3D bounding boxes that minimally enclose each object, which are represented by their position in the world, their dimensions and their orientation. By integrating additional information, like radar data or past measurements [18], other quan- Introduction 3 tities can be estimated, such as object velocity or trajectory. By summarizing the input signal via a finite set of elements, each one representing the state of a particular detection (e.g. an obstacle on the road), object detectors provide an output that is very simple and compact, and therefore immediately useful to the subsequent planning phase, as it requires little to no additional post-processing. Moreover, deep learning-based object detectors are extremely flexible, allowing to detect almost any category of interest, as long as there exist suitable training data for model optimization. Given its central role in contemporary autonomous driving pipelines, this thesis focuses on the topic of deep object detection. More specifically, I propose three different systems, each one working on different input data and tackling a distinct detection problem. The first system consists of an deep learning model for parking space detection and vacancy classification on surround-view images [19]. This approach builds upon the existing two-stage object detector Faster R-CNN [9], modifying its structure and logic to adapt it to the peculiar nature of the task of parking slot detection. Parking slots can be of different types (e.g. rectangular or slanted) and can be observed from different angles, rendering the original implementation, which predicts axis-aligned bounding boxes, ineffective for accurate slot prediction. As such, I propose a variation to the original model, removing the anchor-based region proposal and allowing generic quadrilaterals to be estimated instead of axis-aligned boxes. To train and evaluate the model, I collected and annotated a small dataset depicting different road scenes and parking lots, stitching together the bird’s eye view projections of four fisheye cameras to construct surround-view images. Several experiments show the effectiveness of the proposed formulation in unobserved situations, as well as under noise and different observation conditions. The second work consists in an extention of Faster R-CNN to the task of monocular 3D car detection. This is accomplished by introducing a simple additional neural network module into the original architecture which is tasked to perform image-based 3D detection, essentially by learning the mapping be- 10 Chapter 1. Prior Art linear basis function regression. In this case, each input is transformed into a feature vector by a set of predetermined functions, known as basis functions Φ(x)=[φ1(x), ..., φM(x)], before being fed to the linear model: f(x)W,b=Φ(x)·W+b.(1.10) The use of this intermediate representation allows to model more complex relationships between input and output. For instance, if we know beforehand that xand yare both scalars and the process ggenerating yfrom xis a polynomial of degree n, we could choose Φ(x)=[x,x2, ..., xn]as basis functions, allowing the model to represent polynomial functions up to degree n. Optimizing via the minimization of Eq.1.3 would yield the polynomial that best fits the dataset D. Despite no longer being linear with respect to the input, this new model is still linear with respect to the weights, which means that a closed form solution can be obtained using the normal equations if the quadratic loss (Eq. 1.4) is adopted: the only difference is that now the design matrix Xdoes not contain the input values, but rather their feature representations obtained through the basis functions. Note that simple linear regression can be obtained by setting Φ(x) = x. 1.1.3 Parametric Basis Functions: The most basic Neural Network For complex data it is often unfeasible to determine the nature of the underlying function g, and therefore a suitable set of intermediate basis functions. These cases can be approached by utilizing parametric basis functions [28]: Φ(x) = [φW1,b1 1(x), ..., φWM,bM M(x)].(1.11) Here, each φWi,bi i(x)is commonly a scalar function composed of a parametric linear mapping followed by a non-linear function: φWi,bi i(x) = σ(x·Wi+bi).(1.12) 1.1. What are Deep Neural Networks? 11 The non-linear function σ(·), commonly referred to as activation function, is necessary to allow the set of basis functions to model non-linear relationships between input and output. In this formulation, the set of parameters {(Wi,bi)}M iis also optimized, allowing for the most suitable intermediate representation to be determined directly from the training data. Contrarily to the previous two approaches, however, this model is no longer linear with respect to the parameters, due to the presence of the activation function σ(·): as a result, the corresponding minimization problem (Eqs. 1.3, 1.4) is no longer convex. This has two implications: on the one hand, there is no closed form solution for the set of parameters; on the other hand, there is no guarantee that gradient descent techniques will reach the global minimum, as the loss function might now contain local minima and saddle points. This formulation represents the most basic form of neural network: •the set basis functions Φparametrized by {(Wi,bi)}M i, is commonly referred to as hidden layer. This name stems from the fact that the intermediate representation produced by Φis generally abstract and not immediately interpretable by an external observer; •each Wi∈RIrepresents a weight term, and each bi∈Rrepresents a bias term; •each element φWi,bi i(x)of the feature vector is commonly referred to as neuron, and the number of neurons in the hidden layer is called width; •the hidden layer Φas defined in Equation 1.11 represents a fully-connected layer, since each output neuron is a function of the entire input; •the outward linear mapping fW,b, parametrized by {W,b}, represents the output layer of the network. The Universal Approximation Theorem [29, 30] states that this neural network is theoretically capable of approximating any continuous function, under the 12 Chapter 1. Prior Art assumption that the activation function σ(·)is non-constant and continuous and that the width of the network (i.e. the number of basis functions) is sufficiently high. Given that any algorithm can be represented by a function (e.g. classifying an image can be seen as a function that maps an input matrix of pixels to a probability distribution), this model could be used to solve any task. For many practical applications, however, obtaining a sufficiently accurate approximation of the desired function would require an extremely high width, which leads to prohibitive computational and memory costs. Moreover, this kind of model exhibits a tendency to overfit the data, expecially when the dataset is limited. Overfitting is a phenomenon where the model, instead of learning an approximation of the unknown underlying function generating the data, it memorizes the training set instead. As a result, the model performs well on samples belonging to the training set, but exhibits poor results on new, unobserved data points. 1.1.4 From Shallow to Deep Models The basic idea in Deep Learning and Deep Neural Networks consists of extending basic neural networks by applying multiple hidden layers in a cascaded way, leading to the following formulation: f(x)W,b=WT·ΦL◦ΦL−1... ◦Φ1(x) + b,(1.13) where Φl(x)=[φWl,1,bl,1 1(x), ..., φWl,Ml,bl,Ml M(x)] represents the l-th hidden layer (i.e.: the l-th set of basis functions) of the model. The number of hidden layers that comprise this model is commonly referred to as the depth of the neural network. A vast amount of research on a wide variety of different tasks highlighted how deeper models are capable of attaining comparable performance to shallow networks (i.e. networks with one or only few hidden layers) with considerably less parameters, while also being less prone to overfitting. The common consensus on why this is the case is attributed to how deep models extract information from the input compared to shallow ones. Despite the intermediate representation produced by a network with a single 1.1. What are Deep Neural Networks? 13 hidden layer is enough to ensure the model is a universal approximator, it is believed to be highly inefficient [31]. Conversely, stacking a sequence of hidden layers one after the other allows a deeper network to learn a representation of the input at multiple scales and levels of abstractions, which in turn allows for more complex function families to be represented more efficiently. Moreover, this hierarchical structure of intermediate representations actively contributes to preventing overfitting, as it is more likely to capture meaningful patterns in the input rather than just memorizing the data points. Increasing the depth, however, is inevitably met with an higher difficulty in optimizing the model. Increasing the number of hidden layers leads to more cascaded non-linear activations, which in turn renders the cost function highly non-convex. Early on, common choices for the activation function used to be the logistic function, or sigmoid σ(x) = (1 + e−x)−1, and the hyperbolic tangent function tanh(x) = (1 −e−2x)/(1 + e−2x). When increasing the depth of the model, however, these functions often cause optimization issues: this is due to the properties of the gradients of these functions, which is always ≤1(in case of sigmoid ≤1/4) and quickly saturates to zero when moving away from the origin. As a result, chaining multiple hidden layers that use this kind of nonlinearity leads to gradients that get progressively smaller towards the early layers. This phenomenon is known as the vanishing gradient problem, and it hinders optimization as the parameters of these layers cannot update properly. As a result, most modern deep learning pipelines have gravitated towards nonsaturating functions. The most widely adopted choice is the Rectified Linear Unit ReLU (x) = max (0, x)[32], whose gradient is constant to 1 in the active part of the function, thus avoiding the vanishing of the gradient. In order to have useful training signal for every input value, variations of the ReLU have been proposed, such as the Leaky ReLU, the PReLU [33] and the SELU [34]. 1.1.5 Training Neural Networks Despite the non-convexity of the loss function, gradient descent techniques can still be utilized to optimize Neural Networks: while there is no guarantee that 14 Chapter 1. Prior Art the optimization will converge towards the global minimum of the function, the hope is that the process will result to a local minimum that is appropriate enough for the task to perform. In order to compute the gradients to use for gradient descent, backpropagation [35] is often employed: this method relies on the ability to determine the gradient of simple functions, as well as on the chain rule of differentiability, to progressively determine the numerical value of the gradient of each weight with respect to the loss function. For instance, given the loss function value L, ∂L/∂fcan be computed directly, which can then be used to recover the derivatives of the weights using the chain rule: ∂L ∂W=∂L ∂f·∂f ∂W,∂L ∂b=∂L ∂f·∂f ∂b.(1.14) Here, ∂f/∂Wand ∂f/∂bare trivial to compute given the linear relationship between the weights and biases and the function f. Given these quantities, the chain rule can then be enforced iteratively to compute the gradients with respect to the hidden layer parameters: ∂L ∂WL =∂L ∂f·∂f ∂ΦL·∂ΦL ∂WL ,(1.15) ∂L ∂Wl =∂L ∂f·∂f ∂ΦL·∂ΦL ∂ΦL−1·. . . ∂Φl ∂Wl l= 1, . . . , L −1(1.16) . Again, all the partial derivatives in the above equations can be computed analytically, as the corresponding functions are either linear or depend on the non-linearity σ(·)whose derivative should be known. Modern Neural Networks are often computationally expensive and they are usually optimized over very large datasets. As a result, applying gradient descent directly for their optimization is often intractable. This is due to the fact that gradient descent requires the model to be evaluated on all training samples (i.e. Eq. 1.4) every time a parameter update (Eq. 1.9) is to be performed, which leads to a prohibitive amount of time necessary to reach the local minimum. As a result, most current models are trained using Stochastic Gradient 1.2. Convolutional Neural Networks 15 Descent (SGD) instead: in this variant, each optimization step is performed using only BNrandomly sampled data points at a time, where each group of Belements is often referred to as batch and Brepresents the batch size. Consequently, performing each parameter update is significantly faster, as it requires only a fraction of the data each time. Once the entire training set is used to perform updates, an epoch is completed, the training set is re-shuffled and new batches are created for further training. The theoretical drawback of this method is that it does not guarantee that even a local minimum will be reached during optimization. In pratice, however, SGD tends to reach a satisfactory value for the cost function rather quickly and, by introducing noise into the training process by randomly sampling batches of data each time, often acts as a regularizer, reducing overfitting and leading to models that perform better on new data samples. Many variations of SGD have been proposed, with the objective of improving and speeding up training. Adding a momentum term [36] allows for a faster convergence towards a minimum and helps avoid oscillations in badlyconditioned regions of the loss function. Adagrad [37], instead of updating each parameter using the same learning rate, adapts the learning rate individually to each parameter. ADAM [38], one of the most used optimizers currently together with SGD with momentum, improves upon the idea of Adagrad while also keeping track of past gradient updates like in momentum SGD. 1.2 Convolutional Neural Networks Convolutional Neural Networks (CNNs) represent perhaps the most well-known and widely-used category of deep learning models. These neural networks are designed to harness the properties of certain classes of signals of being organized in hierarchies and presenting local patterns. Examples of these kinds of signals include: •natural images, in which neighboring pixels are correlated (the value of each pixel usually depends on its vicinity) and are organized to form 16 Chapter 1. Prior Art spacial patterns and structures; •videos, where the correlation takes place not only in space but also through time; •natural text, where neighboring characters form words and neighboring words form sentences. 1.2.1 The convolutional operator Convolution is a fundamental mathematical operator which is widely used in image processing. It can be interpreted as a way to perform multiplication between two signals having different numbers of elements. For instance, given an input image x∈RH×W×C, where H×Wrepresent its spatial dimensions and Cits number of channels, the convolution of this image with a kernel W∈RK1×K2×Ccan be defined as follows: y(h, w)=(x∗W)(h, w) = X (k1,k2)∈S x(h+k1, w +k2)·W(k1, k2),(1.17) where S=−K1 2,...,K1 2×−K2 2,...,K2 2 (1.18) The output y∈RH0×W0preserves the number of spatial dimensions of the input, and each element (h, w)is the result of a scalar product between the kernel and the neighborhood of (h, w)in the original image. In other words, convolution is a function that operates locally and that returns a map of local responses of the kernel to the input. Due to the fact that convolution cannot be computed for those positions in which the kernel is not completely inside the input, the spatial dimensions H0×W0of ydo not necessarily coincide with those of the input. A commonly adopted technique, especially in deep convolutional models, to preserve the spatial dimensions consists of padding the input, commonly with zeros, such that every location can be processed. Note that, although Eqs. 1.17 and 1.18 describe a bidimensional convolution, 1.2. Convolutional Neural Networks 17 this operator can be applied to signals having any arbitrary number of spatial dimensions by choosing the kernels appropriately. Even before CNNs, convolution has been widely used in computer vision for many different tasks. For instance, convolution is central in the well-known Canny edge detector algorithm [39] as it is used both for image smoothing (using gaussian kernels) and for gradient computation (typically done using Sobel kernels). Another direct application of convolution in computer vision is image sharpening, which makes use of high-pass kernels. 1.2.2 Convolutional Neural Networks A Convolutional Neural Network (CNN) can be defined as a Neural Network in which at least one of its hidden layers is convolutional instead of fullyconnected. A convolutional layer is a layer in which the basis functions in Φ={φWi,bi i}M i=1 take the following form: φWi,bi i(x) = σ(x∗Wi+bi).(1.19) The differences with respect to a fully-connected layers are as follows: •each weight term Wi∈RK1×K2×Cis now a kernel, applied to the input x∈RH×W×C(in case of bidimensional convolution); •each bias term bi∈Ris a scalar value which is added independently to each output of the convolution; •the activation function σ(·)is applied independently to each element resulting from the above transformation. As a result, the output of each function φiis no longer a scalar, but rather a matrix in RH×W(assuming proper padding is used), and the output of the convolutional layer Φis in RH×W×M. A bidimensional convolutional layer can therefore be interpreted as a layer that takes a signal having spatial dimensions H×Wand Cchannels as input, and produces a new signal having the same spatial size and Mchannels in which each element is a feature representation 18 Chapter 1. Prior Art of the neighborhood of the same element in the input. This output signal is commonly referred to as feature map, while each of its elements represents a neuron. The convolutional kernels used to compute the feature maps, instead of being hand designed to perform specific operations (e.g. smoothing or gradient computation) are learned during optimization, yielding the set of local transformations that are most appropriate for the task at hand. Convolutional layers bring several advantages over their fully-connected counterpart: •the resulting model contains considerably less parameters. For example, if the input to a fully-connected layer contains Ielements, the weights of each basis function must contain Ielements. For high dimensional inputs, such as images, this would quickly lead to prohibitively large models. The number of weights in convolutional layers, on the other hand, depends only on the amount of channels of the input, which are generally limited compared to their spatial dimensions; •the convolutional operation is equivariant to translations in the input. In other words, if the input signal is shifted along one or more of its spatial dimensions, the corresponding output value does not change but it is subject to the same shift. This property is very powerful, as it implies that convolution is able to capture local patterns within an input independently of where those patterns are. This is in contrast to fullyconnected layers, in which a shift of the input signal leads to an entirely different output. The depth of the model plays an especially important role for CNNs, as stacking multiple convolutional layers one after the other allows the neurons of the model to progressively expand their receptive field over the input. The receptive field of a neuron can be defined as the portion of the input that neuron is a function of. Suppose that the first hidden layer of a CNN is comprised of kernels having spatial size 3×3. The receptive field of the neurons produced 1.2. Convolutional Neural Networks 19 by this layer would be 3×3, as each neuron is a function of a 3×3region of the input. If we now apply a second convolutional hidden layer, again with kernel size 3×3, to the feature map produced by the first layer, the resulting neurons would have a receptive field of 5×5over the input (IMAGE). By stacking multiple convolutional layers, it is possible to progressively increase the receptive fields of the neurons arbitrarily, potentially allowing each one to "see" the entire input image. This has proven to be very powerful in practice as it allows for very "strong" hierarchical representations to be learned by the model: the neurons of the early hidden layers learn to recognize low-level general patterns, such as edges, dots or other basic shapes. Neurons from later layers become progressively more specialized and abstract, as they have access to more contextual information, and learn to recognize high-level structures that are useful for solving the specific task [40, 41]. This phenomenon is one of the most important contributors to the robustness of this class of models to overfitting, as well as their ability to perform well even when trained on limited data. In practice, modern deep convolutional models do not keep the spatial dimensions constant throughout the entire model, but rather they progressively reduce them, generating increasingly smaller feature maps. This is achieved either by using strided convolutional layers or pooling operators. Strided convolutions operate exactly like normal convolutions, with the difference that some input locations are skipped. In particular, the stride of this operator determines how many positions the kernel "moves" when computing the next value: for instance, is the stride is equal to 2 this means that every other element in the input will be ignored, leading to an output whose spatial size is half of that of the input. Pooling operators can be seen as a special case of strided convolutions in which the kernels are not optimized, but rather perform a specific kind of operation over the input. The most widely used are Max Pooling and Average Pooling, in which the kernels perform a max and mean operation respectively. Normal convolutions can also be seen as a special case of strided convolution where stride is equal to 1. The advantage of progres- 26 Chapter 1. Prior Art The current state-of-the-art for segmentation models is represented by the DeepLab family of architectures [4, 57, 58]. In DeepLab [4], the authors adopt an ImageNet-pretrained VGG16 model as base for the segmentation. To deal with the limitation that VGG16 produces a feature map at 1/32 the original resolution (which in turn would produce very coarse segmentation maps), the authors remove the last two pooling layers of the model, and employ dilated convolutions in the following layers to make up for the loss of receptive field. The resulting segmentation map is then upsampled from 1/8 the resolution to the original input size using bilinear interpolation, and Conditional Random Fields (CRF) are used to further improve the result and better segment the finer details. DeepLabv2 [57], extends DeepLab by integrating ResNet as additional backbone for better performance and by introducing the Atrous Spatial Pyramid Pooling (ASPP) module, where multiple kernels having multiple dilation rates are used to capture objects at different sizes and scales. DeepLabv3 [58] augments the ASPP module with image-level features and removes the costly CRF post-processing, while still performing considerably better than its predecessors. 1.5 Object detection Semantic segmentation provides very powerful information for autonomous driving systems. For example, it can be applied to determine the free driving space, or to locate traffic lanes or other kinds of horizontal traffic signs. It represents, however, redundant information when the targets for detection are obstacles such as cars or pedestrians or more complex entities such as vertical traffic signs. In these cases, since semantic segmentation does not provide an instance-level distinction but only returns the class each pixel belongs to, additional post-processing steps would be required to determine the exact location of each single object. A class of problems that can be considered complementary to semantic segmentation is object detection. Object detection can be defined as follows: 1.5. Object detection 27 given an input x(e.g. an image, a LiDAR scan etc...), the goal is to locate within this input all the entities of interest as well as to describe their states {Si}N i=1. This state might include the position of the entity, its class and other properties such as size or velocity. What makes object detection particularly interesting and challenging is that the number of detections, and therefore the number of outputs to be returned by system, greatly varies depending on the input. For instance, one image might display several tens of cars to be detected or none at all. This is in contrast with the previously presented problems, where the dimension of the output was independent of the content of the input: in classification the output is always a single probability vector; in image segmentation the output is always a probability vector per image pixel. To deal with this complication, the general approach adopted in deep learning consists on estimating a very high number of possible objects and progressively suppress redundant and negative information to obtain the final set of high-quality estimations. 1.5.1 2D Object Detection on Images Probably the most researched variant of object detection in deep learning is 2D object detection on images. In this case, the state for each object is represented by its class and by the smallest axis-aligned 2D bounding box containing it, that is Si= (ci, xi, yi, wi, hi), where cirepresents the class of the i-th object, xi and yithe center of its bounding box in the image and hiand wiits dimensions. Pioneering research in the direction of fully deep learning-based object detection is represented by R-CNN (Region-based CNN) [5]. In this work, object detection is achieved via a combination of Selective Search [59] and ImageNetpretrained AlexNet. Selective Search is an algorithm that, given an image, returns a set of bounding boxes that are likely to be located in interesting regions of the image. The generated set of boxes has high recall (it is likely to contain all interesting objects), low precision (most of the boxes are wrong, inaccurate or duplicates) and is generally very large (around ∼2000 boxes are generated for a 600x600 image). Moreover, the algorithm is class-agnostic, that 28 Chapter 1. Prior Art is it does not specify what kind of object is within each of the boxes. RCNN uses Selective Search to determine an initial set of detections, called region proposals; each proposal is then used to generate a new image, of size 227 ×227, by cropping and reshaping its content, that is subsequently given as input to a pretrained AlexNet model, which is tasked to classify the proposal content as well as to compute a correction to the box location and size. In particular, the last fully-connected layer of AlexNet is replaced with two new parallel layers: a layer having N+ 1 outputs, which represents the probability vector for the desired Nclasses plus the background class for negative boxes, and a layer for bounding box regression, which returns four values corresponding to the box corrections in position and dimensions. While surpassing all other methods by a large margin, however, RCNN is extremely slow, requiring over 40 seconds per image. This is due to the fact that AlexNet must be applied to each of the 2000+ region proposals, resulting in a system that is impractical for autonomous driving situations, where high frame rates are essential. A step towards improving upon this limitation is the follow-up work Fast R-CNN [60]. The major contribution of this work consists in feature sharing among different proposals: here the convolutional backbone, which is now VGG16, is applied to the entire input image, obtaining a feature map that is shared for all objects. Then, for each region proposal returned by Selective Search, the corresponding area in the feature map is determined by projecting the box and warped to a standard 7x7 size using the so called RoI Pooling operation. This pooled set of features is finally used for classification and box regression. This formulation has two major advantages over the original design: on the one hand, applying the convolutional backbone to the entire image instead of once per proposal results in a considerably faster computation, over 200 times faster than RCNN. On the other hand, sharing the same set of features for all proposals results in a stronger intermediate representation that is more aware of the context, which leads to improved performance. Despite being considerably faster than RCNN, Fast R-CNN is still bottlenecked by Selective Search, which requires around 2 seconds per image to 1.5. Object detection 29 generate the proposal boxes, slowing down the entire pipeline considerably. A solution to this problem is introduced by Faster R-CNN [9]. In this work, a specific subnetwork, called Region Proposal Network (RPN), is used instead of Selective Search to estimate the set of proposal boxes. RPN generates the proposal boxes by assigning to each pixel of the feature map generated by the backbone a set of predefined boxes, called anchors. For each anchor, it then estimates whether it is an interesting or a background box, as well as computing a correction for it. The proposal boxes are finally obtained by removing the background anchors and by filtering duplicates via Non-Maximum Suppression. These proposals are then processed as in Fast R-CNN to obtain the final detections. Further details about this approach will be reviewed in the next chapter, as Faster R-CNN is central to both the proposed Parking Slot Detection network as well as the 3D Object Detection network. By avoiding Selective Search, and performing both region proposal and bounding box estimation in a unified forward pass of the model, Faster R-CNN is considerably faster than its predecessors, allowing for more than 15 images to be processed per second. Moreover, it achieves top performance, making it one of the current state-of-the-art approaches. This method is further improved in a subsequent work [11], where a Feature Pyramid Network is used as backbone in order to extract feature maps at multiple scales. Anchors and proposal boxes are then assigned to the proper scale depending on their size. The R-CNN approaches are widely regarded as two-stage approaches. This is due to the fact that the final detections are obtained through two separate phases: first, a set of proposal boxes is generated. Then, each proposal is analyzed individually (i.e. through the pooling of its features) in order to compute its class and refine its state. Another class of object detection techniques is represented by single-stage methods: here, the final detections are obtained directly, without exploiting an intermediate set of object proposals. One of the first works in this direction is represented by YOLO (You Only Look Once) [6]. Here, object detection is cast as a single convolutional (ndr: there are 2 FC layers tho) neural network that estimates the bounding boxes 30 Chapter 1. Prior Art directly from the input image. To achieve this, the input image is divided into an S×Sgrid of cells, and the network is tasked to estimate Bpotential boxes per cell. To do this, the input is processed by a stack of convolutional and pooling operations such that an S×S×(B∗5 + C)feature map is obtained. Each pixel of this map is responsible for returning the boxes for the objects whose centers lie in the corresponding grid cell of the input: in particular, the network returns Bpossible boxes per cell with the respective confidences, center coordinates and dimensions, as well as a C-class probability vector that indicates the category of the detected object. The unified design adopted by this approach allows for extremely fast predictions: the base model, which is a modified GoogLeNet architecture, is able to perform detection at 45 frames per second, while the light model, which is similar to the base model but with less layers, reaches 155 frames per second. Similarly to YOLO, SSD (Single-Shot Multibox Detector) [7] approaches object detection using a single, unified, convolutional neural network. Instead of predicting the boxes directly, however, in SSD the detections are obtained as refinements of a set of predetermined boxes (similarly to the anchors in Faster R-CNN). Moreover, instead of using a single feature map to perform detection, multiple feature maps at different scales are used for prediction: the low-resolution maps are adopted for bigger predetermined boxes, while the high-resolution ones are used for smaller predetermined boxes. Harnessing features at multiple scales for differently sized boxes allows the resulting detections to be more accurate, considerably improving performance over YOLO. YOLO was subsequently improved, leading to two new models: YOLOv2 [10] and YOLOv3 [61]. YOLOv2 is similar to its predecessor with a few tweaks and changes aimed at improving recall and localization accuracy of the boxes. Firstly, they improve upon the network architecture: they adopt a new model similar to VGG16 as base, called Darknet-19, and they integrate Batch Normalization, which speeds up training and improves performance. Moreover, they strengthen the pretraining on ImageNet. Secondly, they adopt an anchor-based design, where the predictions are obtained from refining predetermined boxes 1.5. Object detection 31 instead of being computed directly. In particular, instead of hand-picking the anchor dimensions like SSD or Faster R-CNN, they obtain them by running kmeans clustering on the datasets’ ground truth boxes; this way the distribution of anchors is closer to that of the ground truth boxes, which simpifies optimization and leads to improved performance. Finally, they increase the robustness of the system to different object scales by adopting multi-scale training: every 10 epochs, a new random scale is selected and the model is trained on images resized to that scale. YOLOv3 further boosts performance by adopting a much deeper model with residual connections and by performing prediction using feature maps at multiple scales. All previous approaches share a common problem: on average, the number of overall background grid cells or anchors is far greater than those actually containing objects. This discrepancy often leads to suboptimal optimization of the classifier, since the gradient value is overwhelmed by background examples. Some approaches, such as the RCNN family and SSD, deal with this problem by forcing the loss function to be computed on a balanced set of positive and negative samples. Even in this case, however, most of the training signal is dominated by easily classified background examples. A technique often adopted to deal with this limiatation is OHEM [62] (Online Hard Example Mining): here, the loss function is applied to all the samples, but the gradient is computed only for the nsamples for which the network performs the worst, that is the nsamples for which the loss value is highest. This has the effect of forcing the training to focus only on the more difficult cases, ignoring the easier ones. Another solution is proposed in [63]. In this work, the authors introduce RetinaNet, a single-stage object detector. This model adopts ResNet with FPN as backbone, and performs object detection at multiple scales by classifying and refining anchors, similarly to SSD. To deal with the inbalance between positive and negative classes, they introduce the focal loss: in this formulation, instead of computing the loss value as the average of cross-entropy losses of each sample, the loss value is computed as the sum of the cross-entropy losses 32 Chapter 1. Prior Art for each element, but each one is downweighted depending on how well the model performs on it. Hard examples will receive high weights, whereas simple examples will be assigned progressively lower weights. As a result, during optimization, training automatically concentrates on the hard examples; this is contrary to the original cross-entropy formulation, in which the contribution to the training of the few hard examples is decimated by the averaging operation over all samples. The focal loss, moreover, carries an advantage over OHEM: in OHEM easy samples are ignored; instead, when using focal loss all samples contribute to the optimization, albeit with reduced impact if easy. 1.5.2 Tradeoff between One-stage and Two-stage techniques The previous paragraph introduced some of the main state-of-the-art object detectors, categorizing them in two-stage and single-stage pipelines. Two-stage detectors perform detection in two distinct phases: first, they generate a set of proposals, that is a set of intermediate detections having high-recall (most required entities are detected) but low precision (most proposals are wrong or inaccurate). Then, a second part of the model analyzes each proposal by pooling its features and outputs a classification decision as well as a correction to its state. Proposal generation can either be performed using existing algorithms, such as Selective Search, or by employing a Region Proposal Network. On the other hand, single-stage methods perform estimation directly, without making use of an intermediate set of proposals. There is no clear winner between the two classes of methods; which paradigm to use depends on the specific use-case and situation. Single-stage methods tend to be faster than their two-stage counterpart while also being simpler from a logical standpoint, requiring less complex code and being more straightforward to implement and debug. Two-stage methods, on the other hand, have an edge performance-wise. Where the greatest advantage of two-stage methods lies, however, is their far greater flexibility: by adopting region proposals and by pooling their features, it is considerably easier to integrate additional tasks into the original pipeline. 1.6. 3D Object Detection 33 The most notorious work in this direction is represented by Mask R-CNN [64]. Mask R-CNN expands Faster R-CNN to include a segmentation mask prediction for each proposal, on top of the classification and position refinement. This is achieved by adding an extra convolutional module which processes the pooled feature map of each object and returns an upsampled binary mask for that object. To further improve performance and better align the pooled features for the semantic task, the authors propose RoIAlign, an upgrade to the RoIPooling operator, which avoids the quantization effects of the latter by adopting adopting bilinear interpolation. Another interesting line of research that directly takes advantage of the two-stage structure is Tracking without Bells and Whistles [65]. In this work, the authors propose a small modification to Faster R-CNN to directly perform tracking, without the need for additional training or tracking specific data. In particular, the detections of each frame are added to the proposal pool for the successive frame and corrected by the refinement network to directly create trajectories, while the original set of proposals is used to identify new objects and initialize new tracks. Given this extra flexibility and performance, two-stages methods, and especially Faster R-CNN, constitute the foundation for two of the solutions proposed in this thesis: the parking slot detection network and the monocular 3D object detection network. 1.6 3D Object Detection While being useful for applications such as monitoring systems or security cameras, in autonomous driving detecting objects at image level is often insufficient, as it gives no information on where each instance actually is in the world. As a result, new systems and algorithms are continuously being developed that perform detection in 3D. Formally, 3D object detection consists of locating in the input all instances of objects of interest and return, for each one: 34 Chapter 1. Prior Art •its class c; •its 3D bounding box, expressed with respect to some frame of reference. This box is identified by its center point (x, y, z), its dimensions (h, w, l) and its orientation expressed as roll, pitch and yaw angles (φ, ψ, θ). In other words, the state of the object iis now represented by the vector Si= (ci, xi, yi, zi, hi, wi, li, φi, ψi, θi).(1.20) When operating in driving scenarios, a common assumption is that all objects of interest lie on a plane, and can therefore be subject only to rotations around one axis, simplifying the state to Si= (ci, xi, yi, zi, hi, wi, li, θi). Input data used for 3D object detection systems often include images, either coming from monocular or stereo camera setups, and point clouds, obtained by disparity map triangulation or LiDAR sensors. 1.6.1 LiDAR-based 3D Object Detection LiDAR (Light Detection And Ranging) sensors are commonly adopted choices for autonomous driving as they are able to provide highly accurate depth information about the environment. They operate on the time-of-flight principle: distance from an object is calculated from the time required by a light impulse, emitted by the sensor itself, to reach the object and be reflected back to the sensor receiver. The resulting raw data returned by this class of sensors is a point cloud, that is a set of 3D points in space. Some sensors might also return additional information for each of these points, such as the reflectivity of the corresponding objects. The most typically adopted choice in automotive is represented by Velodyne sensors, due to their ability to rapidly perform 360◦scans while also having reasonable vertical Field-of-View and point density through the use of multiple emitter-receiver pairs (verify). The resulting scans usually contain from several tens of thousands to a few hundred thousand points, depending on the sensor model and the number of scan planes. 1.6. 3D Object Detection 35 Point clouds are considerably different from images: the latter are dense grids of elements, organized in a specific structure where position matters. On the other hand, point clouds are sparse and are not characterized by a specific ordering: a permutation of a point cloud is equivalent to the original point cloud. As a result, convolutional models, which are based on the assumption that the input is dense and regular, cannot be directly applied to this data type. An early work in deep learning-based 3D object detection [66] attempts at bridging the gap between image and point cloud data modalities by projecting the LiDAR point cloud on a Bird’s Eye View (BEV) plane, which is then used as input to a convolutional model that directly performs 3D bounding box prediction and classification. Another similar work is represented by MV3D [67]: here, both LiDAR and image data are fused together to perform detection. In particular, three separate convolutional backbones are used to extract feature maps: one from the image, one from a Bird’s Eye View projection of the LiDAR point cloud and one from a frontal view projection of the same cloud. Then, the Bird’s Eye view features are used to generate a set of initial 3D proposal boxes via an RPN. Finally, these proposals are projected back on all three feature maps, whose features are pooled accordingly, fused and used to estimate the final boxes. AVOD [12] adopts a similar strategy for fusing image and LiDAR information, but only uses the Bird’s Eye View projection of the point cloud, generated similarly to MV3D. In this method, however, the proposal boxes are generated using both image and Bird’s eye view LiDAR features, resulting in improved recall for small instances. Despite enabling all previous methods to harness the representational power of convolutional models and pre-existing architectures, point cloud projection inevitably leads to information loss, limiting the ability of such models to reason in 3D and ultimately compromising performance. Currently, the state of the art approaches for 3D Object Detection can be classified in two macro categories: voxel-based methods and point-based methods. In voxel-based methods, the input point cloud is first converted into a voxel 42 Chapter 1. Prior Art proaches to 3D detection: stereo-based methods, which use pairs of rectified images as input, and monocular-based methods, which aim at detecting 3D objects from a single image. Stereo-based Detection A recent work on deep learning-based stereo 3D detection can be identified in Triangulation Learning Network [76]. This approach first proposes a monocular baseline that performs 3D detection from a single frame, and then extends it to the stereoscopic case. The baseline operates similarly to Faster R-CNN, with the difference that all estimations revolve around 3D boxes: in particular, 3D anchors are displaced in the 3D space and then projected on the image to determine their 2D counterparts. Features from the 2D anchors are then pooled and processed by the RPN, which determines 3D proposals. Finally, the same process is repeated using the 3D proposals, whose features are passed to the detection portion of the model to determine the final predictions. To extend this framework to stereoscopic data, both left and right frames are processed in parallel by the network, the 3D anchors and proposals are projected on both frames and the respective features are fused using an ad-hoc scheme that accounts for potential mismatches due to different depths. The addition of the second image and the fusion scheme leads to moderate improvements in performance; despite this, however, this method performs considerably worse than other contemporary stereo approaches and is even outperformed by some monocular pipelines. Another similar approach is Stereo R-CNN [17], which also builds upon Faster R-CNN. Instead of relying on 3D anchors or proposals, however, this method fuses the left and right features computed by the same backbone to predict matching left-right 2D proposals from 2D anchors. These proposals are then used to pool the corresponding left-right features, which are finally employed to estimate the left-right 2D boxes, a set of image keypoints, as well as the corresponding 3D object dimensions and orientation. Using this information, the final 3D box is obtained via triangulation. To further improve 1.6. 3D Object Detection 43 performance, a final alignment phase is performed, in which the depth of the detections is corrected by minimizing the photometric error of objects in the left and right images. This correction, in particular, proves to be central to the method, accounting for most of the performance gain. A seminal work towards accurate stereoscopic (and monocular) 3D object detection is represented by Pseudo-LiDAR [77]. The idea behind this research is surprisingly simple: the performance gap that exists between LiDAR-based and Stereo-based detectors is not to be attributed solely to the technological differences between the two types of sensors, but also to the data representation that is used to train the models. In fact, what the authors observed is that the point clouds resulting from disparity triangulation are indeed not inaccurate enough to justify such a wide difference in result quality. To validate this hypothesis, they introduced a two-step pipeline: first, state-of-the-art approaches are used to extract a disparity map from the stereo pair, which is then converted into a point cloud; second, state-of-the-art LiDAR-based 3D detectors, such as AVOD and Frustum-Pointnets, are applied to the point cloud to perform detection. The resulting system decisively outperforms all purely image-based stereo systems, validating the claim. One of the main reasons as to why a point cloud representation allows for increased performance compared to an image-based one is as follows: processing point-cloud data, either by using 3D convolutions, 2D convolutions on BEV representations or PointNets, ensures that the elements that are operated upon together are physically close in space. Convolutions on images, on the other hand, operate identically on patches corresponding to objects at different scales (far away objects are smaller on images compared to nearby objects) or patches in-between objects and background (and therefore involving entities very far away in physical space), making them less suitable for reasoning in 3D. Monocular Detection Contrary to LiDAR-based or Stereo-based 3D detection, monocular detection is an ill-posed problem, as a single image does not provide enough information to 44 Chapter 1. Prior Art recover the scale of the portrayed scene, and therefore the depth of the objects. One way to recover a good enough approximation of the position of interesting objects is to use a priori knowledge about them, such as their dimension. For instance, if the height of a specific traffic sign is known then it would be possible, given the camera intrinsic parameters, to infer the approximate distance from the sensor. Another possible way is by learning the relationship between the way objects appear in the image plane and their corresponding state in the world using accurate ground truth and deep learning models. This would be similar to how humans with one eye covered would still be able to estimate the 3D structure of the world, despite not having, theoretically, enough information to do so. Due to the challenging nature of the problem, to ease learning and improve performance most monocular 3D detection pipelines embed some form of a priori knowledge or 3D reasoning mechanism directly within their models. In Mono3D [15], the authors leverage the assumption that all objects should lie on the ground plane in order to generate object proposals. In particular, they use camera calibration information to determine a fixed plane, they generate 3D candidate proposals on this plane and they project them on the image in order to score them and keep only the most promising ones. The overall score for each candidate is determined using semantic, instance, shape, location and context cues, which are computed using external methods. The best candidate proposals are then processed further by a second stage similarly to Fast-RCNN: a VGG16 model is used to extract features from the input image, RoIPooling is used to obtain the features for each proposal and fully-connected layers are used to estimate the class, the position of the object as well as its orientation. OFTNet [78] adopts a ResNet backbone to extract multi-scale feature maps from the input image. Then, an orthographic feature transform is introduced to map the image-level features to a BEV representation, in order to allow further computation to reason in 3D without perspective effects. To obtain this representation, a voxel grid fixed to the ground plane is generated, then each voxel of the grid is projected onto the feature maps and all features 1.6. 3D Object Detection 45 within the projection are accumulated into the voxel. Finally, the voxel grid is collapsed into a 2D representation by accumulating features along the height direction. As last step, each location of the BEV feature map is processed in order to classify whether there is an object as well as to determine the center, dimensions and orientation of the object. In MonoGRNet [79] the 3D bounding box estimation process is split into four sequential subtasks performed by a single, unified neural network. In the first step, image features are computed using a VGG16 backbone and 2D bounding boxes are extracted by adopting a two-stage object detection pipeline. Given the 2D boxes, RoIAlign is used to pool the respective set of features which are used for the remaining three steps. First, the depth of each instance is estimated. Then, given the depth, the true coordinates of each instance center are regressed. Finally, the positions of the eight vertices for each box are computed with respect to each box local frame of reference placed in each estimated center. MultiFusion [80] leverages a pretrained model for monocular depth prediction to compute a depth map for each image. This map is then concatenated to the image itself and fed to a VGG16 backbone followed by an RPN to extract region proposals. Given the 2D proposals locations, RoI Max Pooling is used to extract the corresponding features from the feature map and RoI Mean Pooling is used to extract a feature representation from the point cloud generated from the depth map. These two sets of features are then concatenated and used to classify each proposal and determine their corresponding 3D box. Deep MANTA[81] detects vehicles via part estimation followed by template matching. First, a standard VGG16 backbone followed by an RPN are applied to the input image to obtain region proposals. Then, these proposals go through two cascaded refinement stages, involving RoIAlign on the proposal coordinates followed by box correction. Besides the correction, the second refinement stage also outputs, for each box, its classification score, a vector of 2D image coordinates corresponding to the object parts (i.e. tyres, headlights etc...) as well as an estimation of the similarity of its 3D dimensions with re- 46 Chapter 1. Prior Art spect to a set of fixed templates. Follows a final 2D/3D matching phase in which the predicted similarities as well as the 2D object parts locations are used to recover the 3D pose of the object as well as the 3D coordinates of its parts by matching them against a database of templates using the PnP algorithm [82]. MonoPSR [16], builds upon pre-existing high performance 2D detectors and uses LiDAR data as additional information during the training of the model to improve performance. First, a pretrained 2D object detector is applied to the input image in order to compute the image-level boxes. For each detection, a set of features is then extracted by fusing together the full-image features pooled at the box location and a second set of features obtained by applying a ResNet backbone to an image-level crop of the object. These features are then fed to a proposal generation module, which estimates the orientation, dimensions and position of the object. Follows a proposal refinement module, which further corrects the location of the detection. Finally, this information as well as the available LiDAR data during training are used to guide an additional instance reconstruction module to estimate a point cloud representation for each object, which is then used to setup additional auxiliary loss functions. Chapter 2 Object Detection for Parking Slot Detection In this chapter I present the proposed deep learning method for parking slot detection from surround view images. I first provide motivation for the research, highlighting the importance of the problem as well as detailing other methodologies adopted in literature and their weaknesses. I then briefly review Faster R-CNN, as it constitutes the baseline object detection model adopted as starting point for this work. I follow up by illustrating the approach and the dataset used for optimization, including the data preparation process used to create it. Finally, I describe the experiments performed to validate the effectiveness and robustness of the system and I discuss the obtained results. 2.1 Prior Art and Motivation Advanced Driver-Assistance Systems (ADAS) are experiencing a spike in interest from the research community and are currently one the most researched technologies. Among these systems are, for instance, Adaptive Cruise Control, which allows the vehicle to automatically maintain a specific distance from the vehicle in front, or Lane Keeping, which automatically keeps the vehicle 48 Chapter 2. Object Detection for Parking Slot Detection centered in its lane. Another technology frequently available on modern cars is the Parking Assistant which monitors the surrounding space and, once it locates an available parking spot, it assists the driver throughout the maneuver. The localization of free parking space is often performed using sonar sensors by detecting enough unoccupied space to allow the maneuver to be completed successfully. While such a system might work well under the supervision of a human driver, however, it is unsuitable in the context of a fully automated vehicle: sonars are only capable of detecting vacant space, not parking slots directly, and therefore are only useful under the assumption that the vehicle is in proximity of a parking lot. On the other hand, cameras are able to percieve horizontal traffic signs and therefore can enable fully automated parking slot detection. A lot of research has focused on vision for parking slot localization and occupancy classification. Many algorithms have been developed that exploit static cameras to monitor occupancy in order to manage parking lots [83, 84, 85]. From an autonomous driving system point of view, however, these approaches are unsuitable as they all rely on the a priori knowledge about the location of each slot, information that is unavailable when the cameras are dispaced on a moving vehicle. Research in the direction of automatic parking detection during navigation started with [86], where color was used as cue to segment parking slot markings directly in the image. More recent approaches perform detection on a Bird’s Eye View representation of the image instead, in which horizontal road markings are mostly free of perspective effects: for instance, in BEV rectangular slots always appear rectangular, and line thickness does not depend on the vicinity of the slot to the sensor. Moreover, in order to obtain a complete 360◦perception of the surrounding environment, most setups involve multiple calibrated cameras, whose BEVs are then stitched together to obtain surround view images. These representations are used in approaches such as [87, 88], in which parking slots are identified using low level visual features, such as corners and lines. [89] uses boosting [90] in order to classify cross-points between parking-line segments, 2.2. Faster R-CNN 49 and then determines the entry point to each slot from those. The classifier is trained on a dataset comprised of 8600 images, in which the position of each cross-point and the orientation of each slot is annotated. In [91], sobel filters followed by a probabilistic Hough transform are used to extract lines from the surround view images. Then, the available parking slots are detected by exploiting relations between parallel lines. All the above approaches, however, suffer from a common limitation: they all rely on hand-designed visual features, and therefore they tend to perform well only for the specific and controlled environments for which they are designed. [89], for example, is only able to detect horizontal and vertical slots, so it tends to fail in presence of slanted parking slots on during the execution of the parking maneuver. [91] is relatively robust to different observation conditions, but is unable to detect slanted slots and requires a computationally expensive post processing phase. Color-based approaches such as [86] might fail in presence of occlusions, noise or variations in illumination conditions. To deal with these weaknesses, the method proposed in this thesis performs parking slot detection and occupancy classification from surround-view images directly using a deep convolutional neural network. The core idea is that, by allowing the model to automatically learn from data what features are useful for detecting parking slots, the resulting system should, given enough heterogeneous training data, show higher robustness and adaptability to different observation conditions and slot types. Before going into detail about the approach, I briefly review the 2D object detector Faster R-CNN [9], as it constitutes the foundation for the proposed pipeline. 2.2 Faster R-CNN As already stated in Section 1.5.1, Faster R-CNN is a two-stage neural network for 2D object detection from images. In particular, for each detected object, it returns: •its class ci; 50 Chapter 2. Object Detection for Parking Slot Detection •its axis-aligned 2D bounding box, described by its center coordinates (ui, vi)and its dimensions (hi, wi). More specifically, this systems is composed of three main modules: •aBackbone Network, which is used to extract generic feature representations from the input; •aRegion Proposal Network (RPN), which estimates a high-recall initial set of potential bounding boxes, called proposals; •aDetection Head, which analyzes each proposal in order to determine whether it contains an object, classify the object and compute a correction to the proposal box to make it better fit the object. Backbone Network As first step, the image is processed by the backbone, which is responsible for extracting a high-level feature representation of the input. The most commonly adopted choice for this network consists of a ResNet model whose weights are initialized via a training on the ImageNet dataset for the classification task. Recently, the ResNet model is often enhanced with a Feature Pyramid following FPN [11], which generates a pyramid of feature maps at different resolutions by progressively upsampling the last feature map produced by ResNet and by fusing it with early layers maps. Region Proposal Network Given the generated feature map, the RPN estimates an initial set of bounding boxes potentially enclosing regions of interest. To do so, it exploits a set of predefined boxes, called anchors, as well as the relationship between each pixel of the feature map and the center of its receptive field in the input image. The chosen anchors usually span multiple sizes and aspect ratio, as to be able to cover the vast majority of potential objects. More specifically, the RPN operates as follows: •each pixel of the feature map is assigned the same set of kanchors, 2.2. Faster R-CNN 51 except that the anchors are shifted at the center of the corresponding pixel receptive field in the image; •the RPN maps the feature map, using a 3×3convolutional layer followed by two parallel 1×1convolutional layers, into two outputs: a classification map and a regression map. Both maps have the same spatial size as the feature map and 2·kand 4·kchannels respectively. •the classification map contains the classification decision for each set of anchors at each location, expressed as a discrete probability vector p= (p0, p1), where p0represents the probability that the anchor contains background and p1the probability that it contains an object; likewise, the regression map contains the shift and scale corrections to apply to each anchor (ua, va, ha, wa)in order to generate the corresponding proposal box (u, v, h, w), encoded as follows: tu=u−ua wa tv=v−va ha tw= log w wa th= log h ha (2.1) In case a feature pyramid is used instead of a single feature map, anchors at different scales are assigned to different pyramid levels: bigger anchors are assigned to low-resolution feature maps, as these maps tend to focus more on the global context of the input; conversely, smaller anchors are assigned to high-resolution feature maps, as these embed more local information. Using bigger feature maps for smaller anchors also means that more of these anchors are present, which improves recall for smaller objects. This architectural choice ensures that objects can be effectively detected even if they appear at considerably different scales in the input. Note that the same RPN model is used to process all feature maps in the pyramid. Detection Head Given the generated proposals and the feature maps from the backbone, the detection head is tasked to predict the final set of objects. In particular, this second stage operates the following way: 58 Chapter 2. Object Detection for Parking Slot Detection (a) (b) (c) (d) (e) Figure 2.1: Visualization of the intermediate steps of the network. (a): Locations of the reference points on the image. (b) Region proposals estimated from the reference points. The green color indicates proposals having objectness score above 0.5. (c): Pruning of the negative region proposals. (d) Remaining positive region proposals after NMS. (e): result of the detection head on the remaining region proposals. 2.3. Parking Slot Detector 59 enough positive samples are present, enough negative samples are selected to reach 256 elements total. Since no anchors are used at this stage, positivity is determined simply by whether each reference point is contained in a ground truth parking slot or not: if it is, it is labelled as positive and associated to that slot; otherwise, it is labelled as negative. The loss function for training the RPN is given by: LRP N =1 N·Lcls (pin, c∗) + λ Npos [c∗= 1] Lreg (t, t∗),(2.7) where Lcls is a standard binary cross-entropy loss: Lcls(pin, c∗) = −c∗·log pin −(1 −c∗)·log (1 −pin)(2.8) and Lreg follows Eqs. 2.3 and 2.4. pin and trepresent, respectively, the predicted confidence probabilites and vertices residuals (parametrized as in Eq. 2.6), while t∗contains the corresponding ground truth residuals, and c∗is set equal to 1 for positive reference points and 0 otherwise. The balancing hyperparameter λis empirically set to 3. The training of the detection head follows Faster-RCNN, with the only difference that, in this case, a set of 128 proposals per image are used, with a ratio of 1:1 between positive and negative proposals. In this stage positivity is determined by measuring the intersection over union between the minimum enclosing rectangle of each proposal and the minimum enclosing rectangle of each ground truth quadrilateral. In particular, for each proposal, the ground truth box having maximum IoU with it is determined. Then, if the IoU is greater than 0.5, it is labelled as positive and associated with that ground truth. Otherwise it is marked as negative. The loss function LDET for the detection head is equal to Eq. 2.2 with λ= 1, and the total loss for the system is given by the sum of the RPN loss function and the detection head loss function: L=LRP N +LDET . The model is trained jointly using Stochastic Gradient Descent with momentum 0.9, batch size 16 and learning rate 10−3for 10.000 iterations. During training, data augmentation is employed in order to enrich the data samples 60 Chapter 2. Object Detection for Parking Slot Detection and reduce overfitting. In particular, each input image is subject to both horizontal and vertical flipping, applied independently each with probability 0.5, and is perturbed in brightness, saturation and contrast in the range ±20%, in order to simulate additional observation conditions. 2.3.2 Dataset Construction and Data Preparation In order to train the model, a small training dataset composed of 467 surroundview images was created and manually annotated with the image coordinates of the corners of each parking slot, as well as the information regarding the occupancy of the slots. The vertices of each slot have been preprocessed in order to ensure a consistent ordering among different examples. More specifically, the vertices are sorted clockwise with respect to the parking slot centroid, identified as the mean of its four corners. Ensuring the consistency of the vertices order is crucial to successful optimization, as the model must be able to attribute a semantic meaning to each one. If ground truth vertices were not ordered, very similar instances would correspond to different training objectives, which would lead to instability and inability to reach convergence. Examples of annotation are visible in Fig. 2.2. (a) (b) Figure 2.2: Examples of annotated images. Colors are used to highlight both vertex ordering and occupancy. 2.4. Experimental Results 61 To obtain the training images, a vehicle setup with four calibrated and synchronized fish-eye cameras was used to collect several sequences, both of road scenes and parking lot areas. The images from each camera were then cast to a Bird’s Eye View representation using the cameras projection model and stitched together using the calibration to create the surround-view images. Meaningful frames depicting different scenarios, parking slot types and observation conditions were then manually selected for annotation. To avoid biasing the network towards situations in which parking slots are always present, 167 of the 467 chosen frames contain no parking slots. The generated images have shape 1100 ×900, which is downsampled to 544 ×384 before being given as input to the model. The reason for downsampling the input is twofold: on the one hand, reducing the spatial size reduces the computational and memory costs of the system considerably. On the other hand, the chosen resolution is divisible by 32 on both dimensions, which simplifies the computation of the reference points positions. 2.4 Experimental Results To validate the effectiveness of the presented approach, the trained model was tested on new sequences acquired under different observation conditions and containing parking lots unobserved during training. Some qualitative results are visible in Fig. 2.3. As can be observed, despite the extremely limited training set, the model exhibits a remarkable capability to generalize to new, unseen scenes and it is able to correctly identify parking slots that are similar to those observed during training. In particular, the network is currently capable of handling parking slots displaying different patterns (Fig. 2.3b) as well as rotated slots (Figs. 2.3c, 2.3d). Moreover, despite the fact that the training set contains only as few as 20 examples of slanted parking slots, the network is able to detect them with acceptable accuracy, as shown in Fig. 2.3e. The system is also able to withstand noise in the data: in Figs. 2.3c, 2.3d and 2.3e some dirt is visible in the lenses. Also, the cameras are not perfectly calibrated, 62 Chapter 2. Object Detection for Parking Slot Detection (a) (b) (c) (d) (e) (f) Figure 2.3: Results of the network on different types of parking slots under different observation conditions: (a) the most common scenario. (b) Different pattern. (c-d) Different rotations. (e) Slanted parking slots. (f) Failure case. 2.4. Experimental Results 63 which leads to misalignments between the four bird’s eye view images. Nevertheless, the network is able to predict the observable slots well enough. The scarsity of training data, however, might lead to incorrect predictions, where unobserved horizontal traffic signs and patterns are erroneously interpreted as parking slots, such as in Fig. 2.3f. The proposed system is able to run at over 13 frames per second (fps) on a NVIDIA Geforce GTX 1080 GPU. 2.4.1 Semantic Shift Problem In the data preparation section (2.3.2) I mentioned the importance of ensuring a consistent ordering among the four vertices of each ground truth bounding box, as the network must be able to associate a semantic meaning to each point (e.g. the first prediction is the top-left point, the second is the top right and so on) in order to be able to converge. In order to guarantee this consistency, the vertices were sorted in a clockwise order with respect to the centroid of each bounding box. There are, however, some specific observation angles for which the result of the sorting rule changes abruptly, causing what I call a semantic shift (see Fig. 2.4a for a schematic representation of the problem). As a consequence, the network exhibits erratic behavior when asked to perform predictions for instances that are very close to this critical angle (see Figs. 2.4b, 2.4c, 2.4d). This is due to the fact that the model is unable to identify which is the correct order of the points and, as a result, tends to predict coordinates that are in between the right ones. Note that this problem is not due to the specific ordering rule chosen, but rather to any ordering rule. Different ordering strategies might correspond to different configurations at which the semantic shift occurs, but the overall problem remains. The hope is that, by increasing the number of training examples close to the semantic shift, the network can learn to handle these cases more effectively and narrow down the range of angles for which the confusion happens. Of course, the ideal way to handle this problem would be to adopt a representation for the output that does not depend on any specific ordering, bypassing the semantic shift altogether. 64 Chapter 2. Object Detection for Parking Slot Detection 1 2 34 1 2 3 4 2 3 1 4 (a) (b) (c) (d) Figure 2.4: Semantic shift problem. (a) Schematic illustration of the problem. (b) Output before the shift angle. (c) Output around the shift angle. (d) Output after the shift angle. 2.4. Experimental Results 65 One such representation might take ispiration from the very recent work PolyYOLO [94], in which the authors extend the latest YOLOv3 model to also predict the semantic mask for each detected object by interpreting each mask as a set of vertices that are estimated using a polar grid. 2.4.2 Ablation Study To validate the choice of adopting a two-stage detection approach for added localization accuracy, I carried out an ablation study that is made up of two distinct experiments: •first, I removed the second stage, entrusting the RPN to perform detection directly. To achieve this, the classification branch of the RPN was modified so that it predicts the class of each reference point (e.g. background, occupied, vacant) instead of just determining whether each reference point is contained in a parking slot. The cost function for the classification was changed accordingly to a standard multi-class crossentropy formulation; •second, the sampling procedure for picking a balanced set of foreground and background examples at each iteration was removed. Instead, all reference points are used to compute the loss function LRP N . To evaluate each experiment, a small test set of 107 unobserved parking spaces was annotated. As evaluation metric I chose the Average Precision (AP), utilized to evaluate Object Detection performance in the PascalVOC benchmark [95]. To compute this metric, the outputs of the system are ranked according to their confidence and are then used to determine the precision/recall curve. The AP value is then obtained by computing the mean precision value at a set of 11 equally spaced recall intervals. To determine whether it is a true or false positive, each prediction is checked against the ground truth according to a rule. Commonly, in traditional Object Detection this rule consists in the Intersection-over-Union 66 Chapter 2. Object Detection for Parking Slot Detection between the detection and the ground truth boxes. More specifically, in order to be considered a correct detection, an output of the model must have an IoU above a certain threshold with at least one ground truth element of the same class as the prediction. Duplicate detections of the same ground truth instance are handled as false positives (e.g. if the same ground truth object is detected 3 times, one is considered as true positive and the rest are false positives). Differently from the training procedure, where the IoU computation has been approximated using the minimum enclosing rectangle for efficiency, here the computation is exact, as performance is not a concern. The results of this study are shown in Tab. 2.1. The displayed mAP (mean AP) scores are obtained by averaging the AP values for the vacant and occupied classes. The subscript indicates the IoU threshold used to determine the positivity or negativity of each detection, while no subscript means that the correspoding values are obtained by averaging the mAP values over multiple thresholds, from 50% to 95% in intervals of 5%. Point sampling Second stage mAP mAP50 mAP70 X X 44.9 60.2 54.8 X20.1 36.5 25.4 14.1 29.6 17.4 Table 2.1: Results over the test set. mAP50 and mAP70 represent the mean Average Precision using IoU threshold of 50% and 70%, respectively. mAP represents the mean Average Precision averaged over multiple thresholds (from 50% to 95%, in intervals of 5%). As can be observed, the introduction of the second stage to the pipeline accounts for most of the performance gain of the system. This difference in performance can be justified by the fact that, when the second stage is not used, the predictions are no longer based on ad-hoc set of features extracted by RoIAlign, but rather on generic patches of the feature map. These patches 2.4. Experimental Results 67 are not as accurately localized as feature crops, are less informative and have a smaller receptive field, which hinders the ability of the model to produce accurate vertex predictions. Removing the sampling strategy for the reference points during training further degrades the quality of the results. The reason for this is twofold: on the one hand, using all reference points at each training iteration inflates the classification loss with background samples, slowing down progress for positive reference points; on the other hand, the sampling process provides stochasticity during optimization, meaning that the same input provides different feedback every time it is presented to the model. This has a regularizing effect, favoring generalization especially when the training data is limited. Moreover, the improvement in performance induced by the sampling strategy as well as the second stage is more pronounced at higher IoU thresholds (i.e. mAP70), which is representative of a better localization accuracy of the full model. An example of the different test-time behavior of the model in the three cases can be observed in Fig. 2.5. (a) (b) (c) Figure 2.5: Results of the ablation study. (a): Network without the detection head and using all reference points. (b): Network without the detection head using the usual sampling strategy for the RPN. (c): Full pipeline It is worth noting that the blurring at the edges of the images caused by the bird’s eye view transformation, might lead to detection instability in those regions, resulting in mAP scores for the full model that are not representative of its true performance. Moreover, some false positives might still be detected 74 Chapter 2. Object Detection for Parking Slot Detection data acquired by fish-eye lenses on a hemisphere using a spherical projection model. To obtain the ground truth parking slots in this new representation, I directly reused the previously generated annotations on the bird’s eye view images, inverting the transformation to obtain their corresponding coordinates in the spherical images. The generated bird’s eye view images, however, depict only a limited portion of the observed space, as far away regions are cropped out due to being too noisy. As a result, since the annotations are reused from the bird’s eye view case, there exist parking slots that are clearly visible in the spherical images which are not labelled; also, parking slots that are cut off in the bird’s eye view but are fully visible in the spherical images would have their annotations also cut off (see Fig. 2.7 for an example). If such data were to be used directly to train the system, it would lead to optimization instability and suboptimal convergence, as the model would receive contraddictory training signal. (a) (b) Figure 2.7: Projections of the annotations generated on surround view images onto the corresponding spherical images. The yellow polygon is used to highlight the portion of image visible from its bird’s eye view. To overcome this limitation, each input image is appended a fourth channel which is a binary mask highlighting the portion of the image that is visible in its corresponding bird’s eye view representation. This solution allows the 2.4. Experimental Results 75 model to have knowledge about regions that are valid for prediction while being considerably cheaper to implement than simply annotating all the missing slots. An alternative solution to the binary mask could consist in simply zeroing out all pixels that are not visible in bird’s eye view. This approach, however, is suboptimal as it would deprive the model from being aware of contextual information that might prove useful for the final task. Overall, the resulting training set consists of 1868 images total, as each original surround-view image is generated from a total of 4 spherical images. Each spherical image has an original resolution of 1024×992, which is cropped to 1024 ×704 by removing the rows corresponding to the sky, before using it as input. The adopted model is identical to the one used on surround view, with the exception that the first layer of the backbone is modified to accept 4-channel inputs. Qualitative results on unobserved parkings scenes are visible in Fig. 2.8. It can be seen that the model is capable of handling relatively well parking slots observed from very different point of views, and therefore characterized by very different shapes, sizes and appearances due to the perspective projection. Moreover, the network appears to have properly learned the semantic meaning of the input binary mask, as it only returns detections within its area. Note that the experiments on spherical images were conducted exclusively to evaluate the performance and robustness of the proposed system to different conditions and points of observations. Indeed, operating on spherical images (or even pinhole ones) instead of surround view representations is suboptimal for the task of parking slot detection, for several reasons. Parking slot markings, and road markings as a whole, are generally well-behaved in bird’s eye view as they are mostly free of perspective effects and preserve their shape and size independently from their distance from the observation point. Also, the bird’s eye view projection naturally eliminates all information that lies above the chosen plane, which is mostly useless for the task. For both of these reasons, surround view images represent a much more suitable domain for the task, which leads to better model optimization and increased performance. Another 76 Chapter 2. Object Detection for Parking Slot Detection (a) (b) (c) (d) (e) (f) Figure 2.8: Results of the model on spherical images. It can be noted that the network has learned to utilize correctly the information about the field of view. 2.5. Discussion 77 non negligible advantage of surround view is that it allows to cover a field of view of 360◦with a single input, while at least four inputs would be required in cases spherical images are used. This translates into higher computational requirements, as every image would need to be processed by the model to obtain a 360◦aware detection. 2.5 Discussion In this chapter I presented an end-to-end deep learning-based approach to parking slot detection and occupancy classification on surround-view images. More specifically, I built upon the existing 2D object detection framework Faster R-CNN, redesigning it to allow for generic quadrilateral prediction instead of axis-aligned bounding box estimation. To train and evaluate the system, two small datasets, containing 467 and 107 surround-view images respectively, were collected and manually annotated with the location and the occupancy of visible parking spaces. The system displayed promising results, exhibiting a remarkable capability to adapt to unseen scenarios containing parking slots of the same type as those observed in the training set while being robust to noise and misalignments between the stitched images that constitute the surround-view representations. It also showed the ability to function properly on an entirely different and more difficult domain, as proven by its effectiveness when applied on the native spherical images. Model simplification and sparsification experiments highlighted the fact that the proposed model is capable of preserving comparable performance by actively using only 1/14 of its total number of trainable parameters, which leaves plenty of room for efficiency improvements. Chapter 3 Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization 3.1 Prior Art and Motivation A key challenge in ADAS and autonomous driving systems consists of perceiving the surrounding environment and locating obstacles in it, such that planning and control can be performed accurately. Particularly critical is the detection of moving entities such as cars, pedestrians and cyclists, as they pose a major challenge for safe navigation. Amongst the most researched solutions to 3D object detection are the LiDAR-based ones, as these kinds of sensors are capable of providing a very accurate, albeit sparse, reconstruction of the surrounding environment. These sensors, however, are generally quite expensive, often making up a big fraction of the total cost of the vehicle. As a result, cameras are usually adopted as a cheaper, more consumer-friendly, alternative to perception. Cameras have the 80 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization advantage of perceiving richer information compared to LiDAR, as it is denser and contains color, and can be used to reconstruct the geometry of the environment by adopting stereo setups in conjunction with disparity computation methods [97, 98, 99], albeit not as accurately as LiDARs, especially at long ranges. This information can then be used either implicitly [100, 76, 17] or explicitely [77] to estimate the 3D locations of the objects of interest. Perhaps the most interesting variant of 3D object detection is, however, the monocular one. Here, the task is to determine a 3D bounding box for every object of interest using as input only a single image. This problem is evidently underconstrained, as a single image alone does not provide enough information to determine the overall scale of the scene and, therefore, the distance of the objects from the sensor. To determine the depth, additional information about the scene is required, such as the a priori knowledge about the dimension of the observed objects. Most contemporary state of the art monocular systems leverage the availability of this knowledge, which often comes in the form of ground truth 3D bounding boxes, to train deep models, with the objective of implicitly encoding the relationship between object appearance on the image plane and the corresponding position in the world within its parameters, such that the model can be used to perform detections in new scenarios. Obviously, for such a system to be able to function accurately, it requires a considerable amount of training data, and the new environments that it is exposed to must belong to a domain that is similar to the one it is trained on. For instance, if the camera intrinsic parameters change from the training set, the system is unlikely to produce accurate localizations, as the learned underlying mapping between appearance and position is no longer valid for the new camera model. Similar problems arise if the camera is positioned differently or if the portrayed objects are visually very different. Due to the difficulty of this task, most current 3D object detection frameworks opt for dedicated models that integrate 3D reasoning mechanisms directly into their architectures [15, 78, 79, 81, 16], in the hope that the resulting system learns features that are more suitable for 3D tasks and generalize bet- 3.1. Prior Art and Motivation 81 ter to new situations. Please refer to section 1.6.2 for a more in-depth review of such approaches. Conversely, in the proposed method, I argue that explicit 3D reasoning directly encoded into the network structure is not mandatory for good monocular 3D detection performance, as long as the underlying model has sufficient capacity. To this end, I propose an extention to the 2D detector Faster R-CNN in which a small subnetwork is added to the detection head to perform 3D box estimation. This subnetwork is simple, does not contain any kind of explicit 3D reasoning in its structure and is trained jointly with the rest of the model. See Fig. 3.1 for a schematic representation of the proposed system. Faster R-CNN 3D Head Figure 3.1: Overview of the proposed 3D detection pipeline: I extend Faster R-CNN with an additional module responsible for estimating 3D bounding boxes given the 2D detections. This extra module is trained end-to-end with the rest of the network using a novel loss based on the Generalized Intersectionover-Union. Commonly, 3D estimators are trained via a loss function that minimizes the error between the predicted box parameters (i.e. center, dimensions, orientation) and their corresponding ground truths directly. Instead, I propose a novel objective function that allows to reason in terms of boxes as a whole via the minimization of an approximation of their Generalized Intersection-over- 82 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization Union [20]. The experiments show that this formulation leads to considerably better results, likely due to better feature representations induced by a more suitable choice of the loss function. 3.2 Baseline Model As already stated in Sec. 3.1, the proposed method consists of an extention to the vanilla Faster R-CNN model for 2D detection (refer to Sec. 2.2 for an illustration of this model). More specifically, as backbone of choice I adopt the standard FPN built upon an ImageNet-pretrained ResNet-50 model, and I extend it, following [63], with an additional downsampling stage consisting of a 3×3convolution having stride 2 applied to the last feature map returned by ResNet. Formally, the resulting feature extractor generates a feature pyramid comprised of feature maps at 5 different resolutions. These maps are usually labelled as P2to P6, where Plidentifies the feature map having resolution 1/2lof that of the input. The Region Proposal Network follows the original implementation. To handle objects having different sizes, the RPN comprises 5 different anchor scales having areas 162,322,642,1282,2562which are assigned to the levels P2to P6of the feature pyramid. Each scale is made up of three different anchors having aspect ratios {0.5,1,2}, for a total of 15 anchors over the entire pyramid. Likewise, the detection head follows standard procedure. As pooling method I adopt RoIAlign, which extracts a 7×7fixed-size feature map from the pyramid for each of the top 300 scoring proposals (post NMS) returned by the RPN. These features are then propagated to the detection head for object classification and 2D box refinement. Again, in order to handle objects having different sizes, each proposal is assigned to the proper level of the feature pyramid before performing RoIAlign, according to the following rule: l=$l0+ log2 √w·h 224 !%.(3.1) 3.3. 3D Detection Module 83 Here, wand hrepresent the width and height of the region proposal and l0represents the level in the feature pyramid that a proposal having area w·h= 2242should be mapped to. Following the original implementation of FPN, I set l0= 4. 3.3 3D Detection Module To allow the system to perform detection of 3D bounding boxes, I propose to extend Faster R-CNN with an additional 3D module. In particular, given the final 2D detections produced by the detection head, a second RoIAlign step is performed to extract their specific sets of features, following the same assignment rule illustrated in Eq. 3.1. Then, given each 2D detection and its corresponding set of features, the 3D module is responsible for estimating the 3D bounding box B= (x, y, z, h, w, l, θ)corresponding to that object, where (x, y, z)are its center coordinates with respect to the camera frame of reference, (h, w, l)are its height, width and length respectively and θis its orientation, expressed as a rotation angle around the camera y-axis. See Fig. 3.2 for a bird’s eye view illustration of the targets to be estimated. In order to simplify and stabilize training, these values are not estimated directly by the 3D module, but are rather encoded as follows. Object Dimensions To estimate the object dimensions, the detection head outputs the following quantities: log h ¯ h,log w ¯w,log l ¯ l,(3.2) where ¯ h,¯w,¯ lrepresent class-specific prior values obtained by averaging the dimensions of each ground truth object across the entire training set. This formulation allows to frame the estimation of the dimensions in terms of a relative correction, where negative values correspond to a reduction in size with respect to the prior and positive values to an increase in size. 90 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization the 3D module operates on the same set of positive proposals also used by the detection head. Generally, the commonly adopted approach for training 3D box predictors consists of minimizing directly the adopted parametrization (in this case Eqs. 3.2, 3.3, 3.4, 3.7) against the ground truth using some sort of distance function (e.g. Eq. 2.4). Instead, I propose to extend the GIoU formulation to the 3D case. As already stated in Sec. 3.4.1, the IoU between two axisaligned bounding boxes has closed form solution, and this is due to the fact that their intersection is still an axis-aligned box. Moreover, by approximating the minimum enclosing box as another axis-aligned box, the GIoU can also be calculated analytically and has a well-behaved gradient. 3D boxes, however, are subject to rotations and their intersection is a cuboid only if they have a relative orientation which is a multiple of π/2. In all other cases, their intersection would be a generic, irregular, polihedron. To ensure the existance of an analytical solution, I disentangle the estimation of the angle from the rest of the dimensions, which are optimized via the GIoU by considering their respective boxes at a canonical orientation. The loss function for this module is therefore comprised of two distinct components: L3D=Lang +L3IoU ,(3.11) where Lang is responsible for optimizing the orientation and L3IoU is tasked to optimize position and dimentions through the GIoU. Formally, let B= (x, y, z, h, w, l, θ)be the 3D box predicted by the module and ˆ B=ˆx, ˆy, ˆz, ˆ h, ˆw, ˆ l, ˆ θits assigned ground truth box. Let ˆα=ˆ θ− atan2 (−ˆx, ˆz)be the ground truth observation angle. Lang is defined as the smooth-L1 loss between the estimated and the target observation angles: Lang =smoothL1(sin ˆα−sin α) + smoothL1(cos ˆα−cos α).(3.12) In order to optimize the position and dimensions of the boxes through the GIoU, I first prerotate them such that their orientation angle is equal to 0, yielding: 3.4. Model Optimization 91 B0= (x, y, z, h, w, l, 0),(3.13) ˆ B0= (ˆx, ˆy, ˆz, ˆ h, ˆw, ˆ l, 0).(3.14) Under this assumption, the boxes can be directly defined in terms of their opposing corners: x1,2=x±l/2y1,2=y±h/2z1,2=z±w/2,(3.15) ˆx1,2= ˆx±ˆ l/2 ˆy1,2= ˆy±ˆ h/2 ˆz1,2= ˆz±ˆw/2.(3.16) Given these values, and by approximating the minimum enclosing box as another cuboid having θ= 0, computing the intersection area I, the union area Uand the minimum enclosing area Acis a trivial extension of the 2D case (see Alg. ?? for the complete formulation). Finally, given the GIoU value, the loss function is obtained using Eq. 3.10. More specifically, instead of minimizing the GIoU loss function between B0and ˆ B0directly, inspired by the disentangling transformation introduced in [102] I split the optimization into six separate contributions, each responsible for a single degree of freedom: L3IoU =1 6X i∈{x,y,z,h,w,l}1−GIoU ˆ B0,Bi 0.(3.17) Here, Bi 0is used to represent the box obtained from B0by replacing all values except for iwith the ground truth (e.g. Bz 0= (ˆx, ˆy, z, ˆ h, ˆw, ˆ l, 0))). This formulation leads to a considerable optimization speedup, especially early in the training where most predictions are disjoint from their corresponding ground truth boxes. Further analysis will be presented in Sec. 3.5.3. Training Details The model is trained end-to-end on full resolution images for 90k iterations, using Stochastic Gradient Descent with batch size 4, weight 92 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization decay 5e-4 and momentum 0.9. The learning rate is initially set to 10−2and is reduced by a factor of 10 every 30k iterations. The ResNet backbone is initialized with ImageNet pretraining values and its Batch Normalization layers as well as its first two convolutional blocks are kept fixed during training. To enrich the training data, each sample is independently augmented by random horizontal flipping with probability 0.5 as well as by jittering its saturation, brightness and contrast by ±30%. 3.5 Experimental Results In this section I introduce KITTI [22], the autonomous driving dataset used to train and evaluate the proposed approach. Then, I perform a quantitative comparison against current state of the art monocular 3D object detectors. Finally, I analyze the loss function used for optimizing the 3D module and compare its effectiveness against other alternatives. 3.5.1 The KITTI Dataset The proposed system is trained and evaluated on the KITTI [22] dataset, which currently constitutes the de facto choice of the autonomous driving community for research. This dataset provides both image, LiDAR and odometry data, as well as ground truth annotations for a wide variety of tasks including semantic segmentation, instance segmentation, visual odometry/SLAM, 2D object detection and tracking, 3D object detection, depth estimation and optical flow. Image data is acquired using two stereo camera setups, one for greyscale and one for color, both displaced at the front of the vehicle and providing images at a resolution of 1382 ×512. Due to rectification, the images provided for training are smaller and have an approximate resolution of 1240×375. LiDAR scans are recorded using a 64-planes Velodyne spinning at 10 frames per second and capturing approximately 100k points per revolution. The cameras are 3.5. Experimental Results 93 synchronized with the Velodyne and capture images at the beginning of each revolution, also at 10Hz. The 2D/3D Object Detection dataset is gathered by annotating dissimilar frames from several recorded sequences with the corresponding observable 2D and 3D bounding boxes, for a total of 7481 training samples and 7518 test samples. The test set annotations are not publicly available and are used exclusively by the online evaluation server for performance evaluation. More specifically, KITTI provides box annotations for 7 different classes, that is Car, Pedestrian, Cyclist, Van, Truck, Sitting Person and Tram, but only the first three are considered for evaluation by the official benchmark, as the others are too scarce in number for proper model training. Still, like many current methods I only consider the Car class for prediction, as it is considerably more frequent and evenly distributed within the dataset compared to Pedestrians and Cyclists. Also, each annotated box is attributed one of three categories, easy,moderate or hard, depending on its size on the image plane and on how much it is occluded and truncated. Following previous work [67, 17, 76], I split the available 7481 annotated images into a training and a validation set, comprised of 3712 ad 3769 samples respectively. It is important to note that, in order to ensure proper performance evaluation, these two splits are originated from two disjoint sets of sequences, such that no similar scenes are shared between training and validation. 3.5.2 Comparison with the State of the Art I evaluate the 3D localization and detection performance of the system using the KITTI Average Precision metric for bird’s eye view (APBEV) and 3D detection (AP3D). For a exhaustive comparison, I consider both the official 0.7 IoU threshold and the more permissive 0.5 IoU threshold. The results for the two tasks are shown in Tab. 3.1 and Tab. 3.2 respectively. The proposed method exhibits state-of-the-art performance on the validation set, surpassing all other monocular approaches by a good margin on the official 0.7 IoU threshold, while also being competitive with the stereo-based 94 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization (a) (b) (c) (d) (e) (f) Figure 3.4: Results of the proposed method on the validation set. Red bounding boxes correspond to the ground truth, while green boxes represent the detections. The LiDAR point clouds are used exclusively for visualization. Best viewed in color. 3.5. Experimental Results 95 (a) (b) (c) (d) (e) (f) Figure 3.5: (cont.) Results of our method on the validation set. The red bounding boxes correspond to the ground truth, while the green boxes are our detections. The LiDAR point clouds are used exclusively for visualization. Best viewed in color. 96 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization Method Data APBEV @ 0.5 IoU APBEV @ 0.7 IoU Easy Moderate Hard Easy Moderate Hard TLNet (S) [76] Stereo 62.46 45.99 41.92 29.22 21.88 18.83 Mono3D [15] Mono 30.50 22.39 19.16 5.22 5.19 4.13 OFNet [78] Mono - - - 11.06 8.79 8.91 MultiFusion [80] Mono 55.02 36.73 31.27 22.03 13.63 11.60 TLNet (M) [76] Mono 52.72 37.22 32.16 21.91 15.72 14.32 MonoGRNet [79] Mono 54.21 39.69 33.06 24.97 19.44 16.30 MonoPSR [16] Mono 56.97 43.39 36.00 20.63 18.67 14.45 MonoDIS [102] Mono - - - 24.26 18.43 16.95 Ours Mono 60.17 43.45 36.53 29.70 21.86 18.13 Table 3.1: Results for localization on the KITTI validation set for the Car class. 3.5. Experimental Results 97 Method Data AP3D @ 0.5 IoU AP3D @ 0.7 IoU Easy Moderate Hard Easy Moderate Hard TLNet (S) [76] Stereo 59.51 43.71 37.99 18.15 14.26 13.72 Mono3D [15] Mono 25.19 18.20 15.22 2.53 2.31 2.31 OFNet [78] Mono - - - 4.07 3.27 3.29 MultiFusion [80] Mono 47.88 29.48 26.44 10.53 5.69 5.39 TLNet (M) [76] Mono 48.34 33.98 28.67 13.77 9.72 9.29 MonoGRNet [79] Mono 50.51 36.97 30.82 13.88 10.19 7.62 MonoPSR [16] Mono 49.65 41.71 29.95 12.75 11.48 8.59 MonoDIS [102] Mono - - - 18.05 14.98 13.42 Ours Mono 55.70 40.34 34.40 22.48 16.67 15.08 Table 3.2: Results for detection on the KITTI validation set for the Car class. 98 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization approach TLNet [76]. Likewise, at the 0.5 IoU threshold, the presented approach outperforms all other methods on both tasks except for the moderate samples on the detection task, where the results are slightly lower than those of MonoPSR [16]. The relatively larger improvement in performance for the 0.7 IoU case compared to the 0.5 IoU one suggests that the proposed approach returns, on average, detections having higher localization accuracy. On a NVIDIA Tesla V100 GPU, the inference time for the 3D detection module is approximately 5 ms, with slight variations depending on the number of 2D detections that must be processed by the 3D module. The total inference time of the entire pipeline is about 50ms on full resolution KITTI images. 3.5.3 Comparison with other Loss Formulations To validate the choice of utilizing an approximation of the GIoU loss formulation, I conducted several experiments in which different loss formulations are used. Is learning each dimension disjointly important? As first experiment, I investigated whether utilizing a separate loss component for each degree of freedom of position and dimensions is beneficial to the performance of the resulting system. In particular, I carried out an experiment where all six parameters are optimized together in a single loss function. What I noticed was a tendency of the system to reduce the GIoU loss value in case of non-overlapping boxes by increasing their size rather than by trying to match their positions in space. This behavior led to a significant number of spurious detections early in the training, which in turn caused a plateau of the loss function around the value of 1, ultimately slowed down the learning process and leading to worse accuracy. A visualization of this phenomenon on validation images is visible in Fig. 3.6: as it can be seen, anomalous detections resulting from this behavior are very prominent at the early stages of optimization, and still persist, although less extremely, once the system is fully trained. 3.5. Experimental Results 99 (a) (b) Figure 3.6: Anomalous behavior of the 3D detection module when trained with the joint GIoU. Fig. 3.6a shows a sample result early in the training, while 3.6b shows the same sample at the end of optimization results for the trained system. I speculate that this phenomenon is induced by how the Generalized Intersection over-Union is formulated: two very similar boxes, close together but disjoint, always leads to a higher loss value than two very different boxes such that one is completely inside the other. The reason for this behavior is illustrated in Fig. 3.7. This also holds true for very different boxes that share a low enough overlap such that E > IoU (see Eq. 3.8). As a result, the tendency of the system, especially during the first half of the training procedure where most predicted 3D boxes tend not to overlap their associated ground truth, is to reduce the loss value by increasing the size, such that Eis driven to zero. This behavior was probably not noticed when the GIoU was applied to 2D detection, as most 2D detectors optimize the refinement branch of the detection head only on anchors or proposals that share a significant overlap with the ground truth boxes to begin with. As a result, since the final predictions are obtained as a correction to these prior boxes, it is unlikely that a large portion of them are disjoint from their corresponding ground truth. Conversely, in the 106 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization 3.6.1 3D detector: standard optimization All three modules involved in the estimation of the 3D box from the sampled point cloud are optimized jointly, using a multi-task loss function: L=Lseg +λ(Lc1-reg +Lc2-reg +Lh-cls +Lh-reg +Ls-cls +Ls-reg +γLcorner). (3.19) The segmentation submodule returns a scalar value for each input point, encoding the probability that said point belongs to the object of interest. As such, the loss Lseg used to train this submodule takes the form of a standard binary cross-entropy loss (see Eq. 2.8), averaged over all input points. The T-Net for center estimation is trained to directly estimate the position of the object center given the set of foreground points shifted around their centroid. It is trained via the loss Lc1-reg, which takes the form of a smooth-L1 distance (Eq. 2.4) between the estimated and the ground truth center. The second PointNet for 3D box estimation is tasked to return the 3D box given the set of foreground points shifted at the estimated center coordinates. More specifically, this submodule returns a total of 3 + 4 ×NS + 2 ×NH outputs, where: •the first 3 outputs represents the estimated position of the object center with respect to the input foreground point cloud. These are trained via the loss Lc2-reg which is a smooth-L1 distance between the predicted and the ground truth center; •the subsequent 4×NS outputs encode the dimensions estimation, which is performed relative to a set of NS predetermined dimension templates. More specifically, the first NS outputs represent a discrete probability distribution that determines which template is used, and the following 3×NS values encode height, width and length corrections with respect to each template. The classification is trained via Ls-cls, which is a standard multi-class cross-entropy loss (see Sec. 2.2.1). The correction is trained 3.6. A case study: 3D GIoU applied to Frustum-PointNets 107 via Ls-reg, which is a smooth-L1 distance between the estimated correction and the ground truth one: note that in this case the loss function is only computed for the right template, determined by ground truth; •the last 2×NH outputs encode the orientation estimation which, like for the dimensions, is split into a classification component and a correction component. In particular, the 360◦are divided into NH bins: the first NH outputs represent a discrete probability distribution that the orientation angle lies in that bin and the remaining NH values encode corrections with respect to each bin center. Training follows the one used for the dimensions: Lh-cls is a multi-class cross-entropy loss for bin classification and Lh-reg is a smooth-L1 distance computed on the corrected bin, which is determined by ground truth. Additionally, to regularize the estimation of the 3D box and improve the accuracy, an additional loss term, the corner loss Lcorner is used, which consists of minimizing the smooth-L1 distance between the estimated and the ground truth box corners. In order to compute this loss term, the 3D box originated using the estimated corrections on the ground truth bins is considered. 3.6.2 3D detector: proposed optimization I propose to modify the multi-task loss (Eq. 3.19) such that the estimation of the 3D box center and dimensions is optimized through the proposed disjoint 3D GIoU loss. More specifically, I remove the losses previously tasked to estimate center (Lc2-reg) and dimensions (Ls-reg) as well as the corner loss (Lcorner), and I subtitute them with the formulation introduced in Eq. 3.17: L=Lseg +λ(Lc1-reg +Lh-cls +Lh-reg +Ls-cls +γL3IoU)(3.20) As originally done for the corner loss computation, in order to compute the GIoU loss value I consider the box obtained from the predicted correction on the ground truth dimension template. No other modification is done to 108 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization APBEV / AP3D @ 0.7 IoU Car Easy Moderate Hard Original v1 87.82 / 83.26 82.44 / 69.28 74.77 / 62.56 Baseline v1 87.64 / 82.97 83.09 / 71.11 75.75 / 63.43 GIoU v1 87.59 / 85.36 83.89 /73.47 76.16 /65.29 Original v2 88.16 / 83.76 84.02 / 70.92 76.44 / 63.65 Baseline v2 88.19 / 83.33 85.38 / 71.74 77.15 / 64.01 GIoU v2 88.41 /86.08 85.99 /74.47 77.64 /66.14 Table 3.4: Results of the study on applying the GIoU as cost function for optimizing Frustum-PointNets. The Car class is considered for evaluation. Original and Original v2 indicate the results reported by the original paper [23], using the PointNet and PointNet++ based architectures respectively. Baseline and Baseline v2 represent the results obtained by my reimplementation of the system. GIoU indicates that the reimplemented system makes use of the GIoU loss formulation in Eq. 3.17 to optimize the estimation of center and dimensions. the system: the 3D box orientation and dimensions are still estimated using a hybrid classification-correction approach, and the model architecture is exactly the same. Also, I adopt the exact same training routine as the original work. 3.6.3 Experimental Results The results of the experiments for the Car class are displayed in Table 3.4. More specifically, I report both the original results shown in the paper [23], as well as those obtained by my reimplementation of the system in PyTorch. I experimented woth both versions of Frustum-PointNets: •version 1 adopts simple PointNet networks for all three modules; •version 2 utilizes PointNet++ models both for the segmentation module 3.6. A case study: 3D GIoU applied to Frustum-PointNets 109 and the box estimation module, resulting in a more powerful and contextaware, albeit quite slower, network. In order to avoid ambiguities and isolate the analysis to the different training strategy used to optimize the 3D detection network, at evaluation time I adopt the exact same set of 2D detections used to evaluate the original system, which are made publicly available by the authors: this way, no performance change can be attributed to a difference in behavior of the underlying 2D object detector. The models trained with the GIoU loss formulation strongly outperform their corresponding original versions, especially for the 3D task which is the one that the loss explicitly aims to optimize. Particularly notable is the fact that the v1 model, when trained using the proposed loss, distinctly outperforms the v2 version optimized with the original loss function, despite being a considerably simpler model. This is indicative of the fact that an appropriate optimization strategy might be more important for the final result than the model architecture. To further ablate the proposed formulation, in Table 3.5 I also show the results of the trained models for the Cyclist class. Even in this case, the v1 system trained with the GIoU loss vastly outperform its v2 counterpart with the standard training procedure. Most surprising, however, is the fact that the GIoU-optimized v2 model, while still outperforming the original model, performs worse than the GIoU-optimized v1 on the 3D metric. This behavior might be the result of overfitting: as already stated in Sec. 3.5.1, available Cyclist data is very limited compared to Car data. As a result, the risk of overfitting such data is higher, especially when adopting more complex models like the version 2 of Frustum-PointNets. Conversely, the much simpler architecture of version 1 leads to simpler features being learned, which in turn improves generalization on unseen samples. On the BEV localization task, on the other hand, version 2 firmly outperforms version 1: this is possibly due to the fact that the task is simpler, and therefore the noisier predictions induced by overfitting do not affect performance as much. 110 Chapter 3. Monocular 3D Object Detection via Generalized Intersection-over-Union Minimization APBEV / AP3D @ 0.7 IoU Cyclist Easy Moderate Hard Original v2 81.82 / 77.15 60.03 / 56.49 56.32 / 53.37 GIoU v1 83.44 / 81.04 62.09 / 59.88 58.60 / 55.76 GIoU v2 84.77 / 78.53 63.75 / 57.42 59.44 / 53.96 Table 3.5: Results of the proposed system on the Cyclist class, compared against the original results. 3.7 Discussion In this chapter, I proposed an extention to the 2D object detector Faster RCNN consisting of a simple module responsible for monocular 3D detection which is trained using a disjoint formulation of the Generalized Intersectionover-Union (GIoU) loss function. To ensure the existence of an analytical solution, I disentangled the estimation of the orientation from that of position and dimensions, rotating the boxes to a canonical angle before computing their GIoU. Moreover, to avoid an anomalous behavior of the GIoU early during training, I opted for optimizing each degree of freedom separately by adopting a dedicated loss function for each. The resulting system exhibited remarkable performance, surpassing more complex and model-driven pipelines on the autonomous driving KITTI dataset. The approach is also simple, as the 3D detection module only consists in a handful of fully-connected layers, and thus could be incorporated straightforwardly into other existing 2D detection methods. Most of the performance gain achieved by the proposed system is to be attributed to the way that the 3D detection module is optimized, as shown by the study conducted by utilizing more traditional optimization functions. To further validate this claim, I adopted the GIoU loss function for optimizing the LiDAR-based 3D detector Frustum-PointNets, and showed that the proposed formulation can lead to significant improvements even for entirely different 3.7. Discussion 111 neural architectures. Chapter 4 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms 4.1 Prior Art and Motivation Camera-based perception solutions, whilst being currently considerably cheaper than LiDAR-based ones, suffer from inferior performance, mostly due to the fact that images do not encode depth information directly. Stereo setups can be used to estimate depth through disparity calculation; such depth, however, tends to be considerably noisier, especially at high distances, and tends to fail in presence of specific patterns or lack of textures. Similarly, networks for 3D detection trained on monocular images require an enormous amount of data in order to generalize properly to unobserved scenes and, even then, they still strongly underperform stereo and LiDAR-based pipelines. LiDAR sensors, on the other hand, provide extremely accurate, albeit sparser, depth information in the form of a point cloud of the surrounding environment. While currently being considerably more expensive than cameras, advancements in sensor technology are consistently reducing the costs of 114 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms LiDAR solutions, and prices are quickly approaching those of stereo-camera setups. This makes LiDAR based perception techniques worth investigating, as consumer-friendly LiDAR options are likely to emerge in the near future. As already mentioned in Sec. 1.6.1, state-of-the-art neural networks operating on point cloud data are generally divided into two macro categories: voxelbased methods and point-based methods. Voxel-based methods [70, 71, 13, 23] first transform the point cloud into a regular grid of voxels, and then process the transformed data using three-dimensional convolutional operators. Pointbased methods [14, 72, 26, 73], on the other hand, operate directly on the raw point cloud by leveraging PointNet-like [24, 25] models. Voxel-based methods tend to be faster than point-based approaches, mostly due to the fact that a regular grid of voxels is easier to process compared to points and there exist libraries [68, 69] that allow for efficient convolution computation by ignoring empty voxels. Conversely, raw point clouds are more challenging to process directly, as they are sets with no intrinsic internal structure, which makes point-based methods usually less efficient. However, processing points directly avoids the quantization introduced by the voxelization process, allowing for more accurate localization. In this chapter I investigate self-attantion [27] as a way of strengthening intermediate feature representations of point-based methods, which all rely on PointNet-like structures to extract features. More specifically, I build upon the detection pipeline Votenet, introduced in [26] for performing 3D object detection on dense point clouds of controlled scenarios, such as room scenes [103, 104]. As this method does not adapt well to noisier scenarios, such as autonomous driving scenes obtained from LiDAR scans, I introduce some simple modifications in order to boost performance. Then, I introduce self-attention as an integral part of the Set Abstraction (SA) layers, with the aim of making points aware of each other when computing features, which should lead to stronger representations and, ultimately, better detection performance. 4.2. Votenet for 3D Object Detection on Driving Scenarios 115 4.2 Votenet for 3D Object Detection on Driving Scenarios Before delving into the details of the model, I briefly review PointNet models, as they represent a central component in point processing pipelines. 4.2.1 PointNets for point cloud processing Differently from images, point clouds are sets, that is they are not characterized by any explicit internal structure. As sets, one of the properties they have is that they are invariant to permutations of their elements, meaning that shuffling the order of the points does not affect the cloud itself. Due to this property, particular care must be taken when processing this kind of data using neural networks, as the resulting model must approximate a function that is symmetric by construction, meaning that its output must be unaffected by the order of the input elements. Currently, the dominant approach in literature for dealing with set inputs is represented by PointNets [24]. The idea behind this class of models is very simple: in order for the model to approximate a symmetric function, first each element of the set is processed individually in order to extract features, and then a global descriptor of the set is obtained by aggregating the information about each individual element via a symmetric function: f({x1, . . . , xn}) = g(h(x1), . . . , h (xn)) .(4.1) Here, frepresents the model, hcan be any function and gis a symmetric function. Commonly, in PointNets the function his approximated by a stack of fully-connected layers, also called in literature as a Multi-Layer Perceptron (MLP), while gby max pooling, as the maximum operation is symmetric. Therefore, the function fas a whole is symmetric, and returns as output a single element, often called signature, that encodes a global representation of the input set. Such signature can then be used for further processing: for 122 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms For simplicity and efficiency, I modify this formulation slightly, outputting 3+4×NS + 2 ×NH values instead: •first, I notice that in most situations there exists exactly one size template per class, meaning that NS =NC. When this is the case, the predictor for the size template and the predictor for the object class carry out exactly the same task, rendering one of the two redundant. As a result, I opt for removing the size template predictor, and at test time I choose the right template according to the predicted object class; •second, instead of predicting the cluster objectness (2) as well as a discrete probability distribution over the classes (NS), I predict NS individual per-class objectness values instead. This alternative formulation performs on-par with the original, whilst being more compact and requiring 2 + NS less outputs. Optimization of box centers, dimensions and orientations follows closely that of Frustum-PointNets, with the difference that in this case these losses are computed for each positive cluster (i.e. clusters whose center is contained in a ground truth box), and averaged over their number. To train the objectness predictor, NS individual binary cross-entropy losses (Eq. 2.8) are applied to each cluster, one for each class, where the ground truth probability is equal to 1 if the cluster center is inside an object of that class, 0 otherwise. Again, the final classification loss is obtained by averaging all loss values for all clusters. At inference time, duplicate detections are handled by using Non-Maximum Suppression, favoring those having higher objectness score in case of high overlap with other detections. 4.2.3 Modifications to the baseline for Autonomous Driving Scenarios The Votenet baseline presented above was originally thought to perform 3D object detection on controlled scenarios, such as the indoor scenes depicted in 4.2. Votenet for 3D Object Detection on Driving Scenarios 123 the ScanNet [104] and SUN RGB-D [103] datasets. Here, point clouds tend to be quite dense, and most points usually belong to objects, with a very small percentage of them that are background (i.e. points on the room floor or walls). Driving scenarios captured by LiDAR sensors, on the other hand, are substantially different: objects of interest (e.g. cars, pedestrians, cyclists etc...) are very few and far between, which leaves most of the point cloud to be background. Moreover, LiDAR clouds are generally noisier and sparses. For these reasons, the original formulation of Votenet does not adapt well to this new domain, losing performance and often missing detections. I identify the root cause of this performance loss in the way that cluster center selection is performed. To recall from the previous section, cluster creation is accomplished through the use of a SA layer, which first determines a set of centers via the FPS algorithm and then aggregates information near each center. In case of controlled scenes, where most points belong to objects, it is likely that the clusters formed by the votes are far apart from each other, with relatively few isolated points in between. In driving scenarios, on the other hand, most points are likely to be background, which leads the clusters that form in correspondence of objects to be surrounded by numerous isolated votes. Since FPS determines the centers by starting from a random point and iteratively selecting the next ones such that they the farthest apart from the already sampled set, if some noise points near a cluster happen to be selected by FPS before any of the points belonging to the cluster, then all points of the cluster might be ignored. This behavior is displayed in Fig. 4.2. When this happens, no features are extracted by the SA layer for that cluster, which is subsequently ignored by the detection module. This has two major implications: due to some objects being missed, less positive samples are propagated to the detector during training, reducing the amount of feedback and slowing down optimization. Moreover, the same phenomenon might happen at test time, leading to false negatives that are not caused by misclassifications, but rather by objects being skipped when sampling. Both of these aspects severely undermine the overall performance of the model. To limit the performance 124 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms loss induced by missed clusters, I test two different solutions to cluster center sampling: seed-based sampling and feature distance-based sampling [73]. (a) 1 2 ? (b) (c) Figure 4.2: Examples of false negatives originated by the sampling strategy. (a) Vote points, displayed in red, sampled cluster centers, shown in blue and ground truth boxes for a car and a cyclist. (b) Schematic representation of what is happening: if noise points around the object are sampled first, the entire cluster might be skipped due to vicinity. (c) Resulting predicted boxes: no object is detected due to no cluster centers being sampled. Seed-based sampling Mentioned in the original work [26], the idea behind seed-based sampling is rather simple: when determining the set of cluster centers from the votes using FPS, instead of using the coordinates of the votes, the coordinates of their corresponding seeds are used instead. Since seeds are evenly spread out in space, as they are the result of applying FPS multiple times on the original point cloud, it is unlikely that all seeds that correspond to a cluster of votes are skipped when sampling the cluster centers. Whilst contributing to almost no performance gain when the system is applied on the ScanNet and SUN RGB-D datasets, this alternative sampling strategy accounts for most of the performance gain when operating on the noisier KITTI driving scenarios. An example of result is displayed in Fig. 4.3. 4.2. Votenet for 3D Object Detection on Driving Scenarios 125 (a) (b) (c) Figure 4.3: Detection results from the seed-based sampling strategy. (a) Locations of the seed points used for sampling, displayed in green along with the ground truth boxes. (b) Vote points, in red, and sampled cluster centers, in blue, along with the ground truth boxes. It can be observed that multiple centers per cluster are sampled. (c) Resulting predicted boxes. Feature distance-based sampling In the recent work 3DSSD [73], the authors propose F-FPS, a variant of the FPS algorithm in which the metric is given by the sum of euclidean and feature distance between the set elements, as opposed to the traditional formulation which only considers euclidean distance and is therefore labelled as D-FPS. They show that using a combination of F-FPS and D-FPS, which they call FS (Fusion Sampling), inside the SA layers of the backbone leads to more object points being kept, which results in higher recall and overall better detection performance. This improvement is caused by the fact that F-FPS allows for points that are close to each other to be sampled, provided that these points encode different entities in space, such as different object parts. Conversely, background points such as points on the road are less likely to be chosen, as they tend to have similar feature embeddings. As alternative solution to seed sampling, I propose to use F-FPS instead of D-FPS in the SA layer of the detection module: this avoids missed clusters, as points belonging to said clusters encode different information from the surrounding background points, and thus are likely to be selected even if the latter are 126 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms chosen first. An example of result is displayed in Fig. 4.4. (a) (b) Figure 4.4: Detection results from the feature distance-based sampling strategy. (a) Vote points, in red, and sampled cluster centers, in blue, along with the ground truth boxes. This method, on average, leads to more objects points to be selected when compared to seed-based sampling. (b) Resulting predicted boxes. Given the increase in recall shown in 3DSSD, I also opt for adopting FS in the backbone module, following the original implementation. Moreover, instead of predicting objectness like in the original system, I choose to predict centerness instead, due to its synergy with the sampling strategies above. Centerness estimation Whilst sampling cluster centers via D-FPS results in at most one center per cluster, when using either seed or F-FPS based sampling it is likely that multiple centers per cluster are sampled. As a result, each object is likely to be detected multiple times by the detection module. While duplicate detections are handled by NMS, where the estimated confidence/objectness is the determining factor in choosing which of the multiple detections is kept, there is no real correlation between said confidence and the true quality of the predicted box. Cluster centers that are closer to their corresponding object centers, however, often result in higher quality predictions. As a result, following [73], instead of predicting a per-class objectness for each 4.3. Enhancing SA layers via Self-Attention 127 cluster center like in Sec. 4.2.2, I predict its per-class centerness values instead. More specifically, if a cluster center is inside a ground truth box of a specific class, it is trained to predict the following value pctr =3 smin (f, b) max (f, b)·min (l, r) max (l, r)·min (t, d) max (t, d)(4.3) and 0 otherwise. In the above equation, {f, b, l, r, t, d}stand to indicate the distances of the cluster center from the front, back, left, right, top and bottom faces of the assigned ground truth box respectively. By predicting centerness instead of objectness, the subsequent NMS prioritizes boxes originated from cluster centers that are closer to the object center, and thus having on average higher quality, which leads to overall better performance of the system. To train for centerness prediction, I adopt the standard binary cross-entropy loss function using as target, instead of a binary 0/1 value like for the objectness, the ground truth centerness value. 4.3 Enhancing SA layers via Self-Attention While PointNet++ represents a strong model for extracting features from point clouds that are local and hierarchical, its structure suffers from a limitation. In SA layers, the feature aggregation of each group is performed through the use of a shared PointNet. To achieve permutation invariance, PointNet processes each point in each group independently using a MLP and then aggregates their information via a max pooling operation. As a result, the points in each group are unaware of each other while being processed by the MLP, which likely leads to suboptimal feature representations. A possible solution for circumventing this problem has been proposed in PointWeb [105], where a feature adjustment module is introduced before the MLP to recalibrate the features of each point according to its relationship with all other points. Instead, I propose to exploit the attention mechanism to explicitly model inter-point relationships within each group. 128 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms 4.3.1 The attention mechanism Attention can be defined as a function mapping three sets of elements, called queries,keys and values into a set of output elements. More specifically, each query element corresponds to an output, which is given by a weighted sum of the values where the weight associated to each value depends on the compatibility of the query with the key corresponding to that value. Formally, let Q∈Rnq×dkbe the set of queries, K∈Rnkv×dkthe set of keys and V∈Rnkv ×dv the set of values, where nqindicates the number of query elements, nkv indicates the number of key/value pairs, dkindicates the dimensionality of queries and keys and dvthe dimensionality of the values. Then, the most commonly adopted version of attention [27] is as follows: Att (Q, K, V )=ΩQ·KT·V. (4.4) Here, the pairwise dot product Q·KT∈Rnq×nkv measures how compatible each query is with each key, and Ω (·)is a function mapping the compatibility values into weights. Usually, Ω (·)takes the form of a Softmax function, applied to each row of Q·KTindependently. The sets of queries,keys and values can be used to model any quantity, so long as they can be encoded in the form of vectors. Commonly, in deep learning these quantities are obtained by projecting feature representations extracted by one or more neural networks: Q=PQ(xq), K =PK(xk), V =PV(xv).(4.5) Here, xq,xk,xvcould be, for instance, feature maps returned by CNNs or points extracted by a PointNet++ backbone. In the former case, each pixel represents an element of the set; in the latter, each point represent an element of the set. PQ(·),PK(·),PV(·)represent the projection functions used to map each of these representations into the sets of queries,keys and values. Such functions are usually optimized together with the rest of the models, and are commonly implemented using an MLP applied concurrently to each element 4.3. Enhancing SA layers via Self-Attention 129 or, in the simplest case, as a matrix multiplication, where the parameters of the matrices are trainable. In most cases, keys and values are obtained from the same set of features, that is xkv =xk=xv. Also, by imposing the dimentionality of the values to be the same as that of the keys and queries (i.e. d=dk=dv), the outputs returned by the attention function are often used to update the original set of query features: y=Att(PQ(xq), PK(xkv), PV(xkv)) + xq.(4.6) The resulting set of features ycan therefore be interpreted as an updated version of the original set of features xq, in which the update depends on the relationship between each element in the set xqand all the elements in the set xkv. Note that this operation is invariant to the permutation of the elements in the set xkv and equivariant to the permutation of the elements in the set xq, meaning that permuting the elements in xqinduces the same permutation on the set y, but the individual values do not change. Self-attention is a variant of attention in which queries,keys and values are all derived from the same set of features, that is x=xq=xk=xv. Self-attention can therefore be interpreted as an operation that updates each element in xdepending on its relationship with the rest of the elements in same set. Arguably the first successful application of self-attention is represented by the seminal work in [27], in which the authors propose an entirely novel neural architecture to perform the task of machine translation revolving entirely around the attention mechanism. More specifically, they introduce the Transformer, an Encoder-Decoder structure: in the Encoder, self-attention is used to model the relationship between the different words in the input sentence. In the Decoder, self-attention is first used to extract information from the translated sentence generated so far; then, by using these features as queries, and the features generated by the Encoder as keys/values, additional attention blocks are used to generate the set of features used for next word prediction. Perhaps one of the most significant innovations introduced in the Transformer model is, however, Multi-Headed Attention. The idea behind this concept is as follows: 130 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms instead of projecting the feature sets xqand xkv into a single set of queries, keys and values, they are projected into hof these sets instead: PQ(xq) = Q=hQ(1), . . . , Q(h)i∈Rnq×(h·dh), PK(xkv) = K=hK(1), . . . , K(h)i∈Rnk×(h·dh),(4.7) PV(xkv) = V=hV(1), . . . , V (h)i∈Rnk×(h·dh). Each of these sets is then processed independently and in parallel using attention, and their outputs are fused together via an additional projection function PO(·), yielding the output of the Multi-Headed Attention: MHA (Q, K, V ) = PO([Att1,...,Atth]) ∈Rnq×d.(4.8) Here, [Att1,...,Atth]∈Rnq×(h·dh)and Atti=Att Q(i), K(i), V (i). Again, the projection function PO(·)is implemented either as an MLP or, more commonly, as a matrix multiplication with learnable matrix parameters. Like before, the result of the Multi-Headed Attention is finally used to update the input set of query features: y=MHA(PQ(xq), PK(xkv), PV(xkv)) + xq.(4.9) The authors argue that using multiple, independent attention heads allows the model to concentrate on different aspects of the input at different positions, improving the overall performance of the system. Note that by choosing dh=d/h and parallelizing the computation of the several attention components, MultiHeaded Attention introduces no computational overhead compared to the traditional formulation. To better fuse the input features with those computed by the Multi-Headed Attention and improve training, each Multi-Headed Attention Block (MAB) in the Transformer encoder adopts Layer Normalization [106] for feature normalization, as well as an additional MLP for performing additional feature fusion, leading to the following final formulation: MAB (xq,xkv) = LN (e y+MLP (e y)) ,(4.10) 4.3. Enhancing SA layers via Self-Attention 131 where e y=LN (y),yis obtained using Eq. 4.9, MLP indicates the MLP used for feature fusion and LN represent the Layer Normalization operator. A schematic representation of the MAB block is depicted in Fig. 4.5. Attention Block Attention Block Attention Block Attention Mask LN MLP LN Figure 4.5: Schematic representation of a MAB block. 4.3.2 Self-attention applied to SA layers I propose to replace both the MLP for feature extraction and the max-pooling for feature aggregation inside SA layers with attention-based processing, in order to enhance the extracted feature representations by explicitly modeling the relationships between the points within each group. In order to be effective, such attention-based mechanisms should preserve the property of invariance to permutations of the original formulation. More specifically, I introduce two different formulations to replace the MLP. Full Attention Here, I propose to replace each layer of the SA MLP h(x) (see Eq. 4.1) with a self-attentive MAB block MAB (x,x), where x∈RK×C represents any single group of points returned by the sampling and grouping stages of the SA layer, Kis the number of points in the group and Cis the dimensionality of each point. This can be done directly, as MAB (x,x)is permutation equivariant. Commonly, most models progressively increase the dimensionality Cof the features they extract as their depth increases. To allow for the same flexibility when using attention, I introduce an additional 138 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms 4.4.3 Training Routine All tested models are trained on the KITTI training split for a total of 80700 iterations, using ADAM as optimizer and batch size 4. The learning rate is initially set to 5·10−4and reduced by a factor of 10 after 64560 iterations. Given the relatively limited dataset, I employ heavy data augmentation in order to favor generalization and improve performance. First of all, I adopt the so-called mixup augmentation [71], an augmentation technique often adopted in recent literature when training point-based and voxel-based detectors which consists in inserting into the current scene objects from other scenes in order to provide richer training examples to the model. Practically, a database containing all objects in the training split is generated. Then, at training time, every time a new training sample is loaded, a certain amount of randomly sampled objects from this database is pasted into it, taking care that no collisions between objects are generated in the process. An example of a mixup-augmented image is shown in Fig. 4.9. (a) (b) Figure 4.9: Example of mixup augmentation. (a) original sample with the associated ground truth boxes. (b) Sample after mixup augmentation. After mixup augmentation is performed, additional data augmentation is carried out to further increase the variability of the training data: •first, the point cloud is flipped with respect to the camera xz-plane with 4.5. Experimental Results 139 probability 0.5; •second, the point cloud is scaled by a factor randomly sampled within the interval [0.9,1.1]; •third, the point cloud is rotated about the camera y-axis by an angle randomly sampled within the interval [−π/4, π/4]; •fourth, each object, and its associated points, is independently rotated about its y-axis by an angle randomly sampled within the interval [−π/3, π/3]; •fifth, each object, and its associated points, is shifted along the camera x and z directions by two quantities randomly sampled within the interval [−1,1]. 4.5 Experimental Results In this section, I present the results obtained from the experiments performed on the proposed system. First, I perform an ablation study on the architectural design choices presented in sections 4.2.3 and 4.3.2, validating them. Then, I carry out a more in-depth study on attention, comparing different formulations against the vanilla model with no attention mechanisms. Finally, I perform a comparison between the proposed system and other state-of-the-art LiDARbased 3D detection models. I note that, despite the fact that the system is trained on all three main KITTI classes (i.e. car, pedestrian and cyclist), most comparisons are performed considering only the car class since, given the limited size of the training and evaluation splits (3712 and 3769 samples respectively), it is the only class that provides numerous enough ground truth samples to enable reliable analyses. 140 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms 4.5.1 Ablation study on the design choice The improvements in performance brought about by the modifications to the original Votenet architecture (Sec. 4.2.3) and by the integration of the attention mechanism (Sec. 4.3.2) are displayed in Tab. 4.1. It can be seen that, indeed, the main cause for Votenet poor performance on noisier autonomous driving scenarios is its inadequate cluster center sampling strategy: just switching to seed-based center sampling leads to a relative improvement of about ∼14%. Note that this performance is not the result of a different neural architecture, but rather is due to the fact that seed-based sampling mostly avoids missing clusters, increasing test-time recall. Moreover, this sampling strategy often leads to the same cluster being processed multiple times by the box predictior, as it is likely that multiple points inside it end up being sampled; as a result, the network benefits from increased feedback during training, which leads to faster and better convergence and therefore higher quality detections. Mixup augmentation contributes to a further boost in performance, despite it being quite limited for the car class and mostly isolated to the hard examples. This is likely due to the fact that cars are by far the most represented class in the dataset, while also being the most spread out among different samples. Conversely, pedestrians and cyclists are far less in number (about 1/5 and 1/10 compared to cars, respectively) and tend to concentrate on a few select samples [22]. Therefore, forcing each input sample to contain multiple instances of those classes stabilizes training, which is no longer dominated by cars. This leads to a considerable gain in performance for pedestrian and cyclist detection, as displayed in Tab. 4.2. Estimating centerness over objectness further increases detection accuracy, since in this case the score assigned by the model to each predicted box better correlates with the quality of the box itself. This has two implications: on the one hand, noisy detections and outright false positives are more likely to have low scores, which causes the Precision-Recall curve to have greater area, improving the AP metric. On the other hand, it is less likely that NMS discards 4.5. Experimental Results 141 Car AP 3D Seed Sampling Feature Sampling Mixup Augmentation Centerness Attention Easy Moderate Hard Votenet [26] 76.85 65.99 64.69 X87.81 76.91 72.81 X X 87,37 76.96 74,73 X X X 87.86 78.25 77.14 Proposed X X X 88.49 78.55 77.38 Proposed X X X Full 89.20 79.23 78.35 Proposed X X X Induced (8) 89.37 79.40 78.51 Table 4.1: Ablation study on different variants of the proposed system over the KITTI evaluation set for the Car class. 142 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms better boxes in favor of more inaccurate ones. Unsurprisingly, centerness estimation leads to better improvements for harder samples: since these samples mostly consist of truncated or far away objects, they tend to be represented by very few points and therefore are more likely to result in noisier cluster center predictions. Experimentally, using F-FPS on votes to sample cluster centers performs better than seed-based sampling, and thus it constitutes the default choice for the proposed vanilla model. Extending such model with attention-based blocks leads to further boosts in performance. A more detailed study related to attention follows in the next section. AP3D) Model class Easy Moderate Hard Vanilla Ped 36.89 35.03 32.03 SS Ped 53.28 49.96 45.72 SS+MIX Ped 61.84 56.31 51.46 Vanilla Cyc 39.29 27.50 26.79 SS Cyc 67.13 52.52 50.52 SS+MIX Cyc 83.33 64.63 60.50 Table 4.2: Comparison between the vanilla system, the system with seed-based sampling (SS), and the system with seed-based sampling and mixup augmentation (SS+MIX) on the KITTI evaluation set for the Pedestrian (Ped) and Cyclist (Cyc) classes. 4.5.2 Ablation study on attention type Point-based 3D detection methods (i.e. models based off of PointNet++) are, by construction, non-deterministic: the result of each FPS operator, in fact, depends on the first selected point in the cloud, which is chosen randomly. While this phenomenon was shown to have a negligible effect in terms of test 4.5. Experimental Results 143 results stability [25], when coupled with the heavy data augmentation involved during training and the fact that the training set is relatively limited, it might lead to non-trivial differences during optimization. Consequently, in order to perform a more accurate comparison between the vanilla model and the various attention-based formulations and to reduce the impact of such randomness on the results, I train 5 identical models of each type, showing the best model out of the 5 as well as their mean performance and their standard deviation. The results of this study are shown in Tab. 4.3. As can be observed, all attention-based models perform, on average, better than their vanilla counterpart, despite being characterized by higher training instability as indicated by their higher standard deviation values. Surprisingly, the induced-attention based models outperform their fullattention counterparts. The reason for this might be twofold: on the one hand, induced-attention results in a higher number of trainable parameters, and therefore in a model having higher representational capacity, compared to full-attention, as each induced-attention layer is comprised of two multi-headed attention blocks instead of one (see Fig. 4.7). On the other hand, the use of inducing points might allow the model to better handle potential outlying elements within each group, reducing their contribution to the output. Performing feature aggregation via attention instead of max-pooling also leads to relevant improvements, as it allows the model to specifically modulate the contribution to the output of each component of the group, allowing the system to represent a much wider family of functions compared to simply performing a channel-wise max-pooling. This stronger representational power, however, also leads to further training instability, as displayed by the increase in the standard deviation value of the validation AP. I also test both full-attention and induced-attention applied to the SA layer of the detection module (SA-C), which is responsible for cluster creation and grouping. In both cases, despite the higher representational capacity, the results are inferior to their counterparts using a standard MLP followed by maxpooling. While induced-attention performs only slightly worse, full-attention 144 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms leads to a more pronounced drop in performance, resulting to a solution that on average performs worse than the vanilla model. This solution is also considerably more unstable during training, as shown by the standard deviation of its performance, which is almost four times that of the model with no attention in SA-C. This degradation in performance is likely to be attributed to the fact that SA-C utilizes BQS with low radius as grouping algorithm, which leads to many clusters having repeated points due to padding. This is especially true for votes that do not belong to objects, which are likely to be isolated in space. Repeated points lead to instabilities when used in conjuction with attention, as each one actively contributes to the end result. The obvious solution for avoiding this problem would be to replace BQS with kNN; this, however, leads to additional noise in the clusters, as each one is far more likely to include background noise points or even points from external objects. Considering that the resulting system performs worse than its BQS-based counterpart (see Tab. 4.4) while being less efficient, I opt against this kind of solution. 4.5.3 Comparison with State-of-the-Art Systems Tab. 4.5 shows a comparison between the proposed solution and other LiDAR and fusion-based state-of-the-art 3D detection systems on the KITTI validation set. The vanilla system performs considerably better than all other approaches barring PointRCNN, which slightly outperforms it at all difficulties. PointRCNN, however, is overall slower, exhibiting an inference time of about 100ms against the 70ms of the vanilla model. Integrating attention in the backbone network considerably increases performance, allowing the resulting models to convincingly surpass PointRCNN while only requiring an extra 15ms for full attention and 19ms for inducedattention. This improvement is particularly relevant for harder objects, which advocates for the effectiveness of attention mechanisms to deal with harder and noisier data. 4.5. Experimental Results 145 Car AP3D: mean ±st.d. (max) Model Easy Moderate Hard Vanilla 88.45 ±0.17 (88.49) 78.43 ±0.17 (78.55) 77.29 ±0.11 (77.38) F-23-MP 88.35 ±0.23 (88.65) 78.59 ±0.22 (78.90) 77.67 ±0.19 (77.99) F-23-AP 88.83 ±0.24 (89.20) 78.93 ±0.23 (79.23) 78.03 ±0.25 (78.35) I(8)-23-MP 88.61 ±0.23 (88.72) 78.77 ±0.15 (78.93) 77.79 ±0.20 (77.89) I(8)-23-AP 89.01 ±0.32 (89.37) 79.00 ±0.26 (79.40) 78.10 ±0.24 (78.51) F-23C-AP 88.29 ±0.72 (89.07) 78.37 ±0.82 (79.09) 77.34 ±0.98 (78.15) I(8)-23C-AP 88.92 ±0.15 (89.15) 78.93 ±0.20 (79.25) 78.03 ±0.26 (78.43) Table 4.3: Ablation study on the attention mechanism. Attention-based models are shown in the format A-B-C:Arepresents the attention type, where F indicates full-attention and I(n) indicates induced attention with n inducing points; Brepresents the SA layers the attention is applied to, where 2, 3 and C indicates the layers SA-2, SA-3 and SA-C repsectively (see Fig. 4.8); Crepresents the technique adopted for performing feature aggregation: MP is the standard channel-wise max-pooling, AP is the attention-based aggregation. The results are shown in the format mean ±st.d. (max), over a total of 5 experiments. Car AP3D: mean ±st.d. (max) SA-C Easy Moderate Hard BQS 88.45 ±0.17 (88.49) 78.43 ±0.17 (78.55) 77.29 ±0.11 (77.38) kNN 88.19 ±0.23 (88.48) 78.11 ±0.19 (78.33) 76.82 ±0.30 (77.19) Table 4.4: Performance comparison between using BQS and kNN in the SA-C layer. Both models are without attention. 146 Chapter 4. 3D Object Detection on LiDAR scans via Voting and Self-Attention Mechanisms Method APBEV @ 0.7 IoU AP3D @ 0.7 IoU Easy Moderate Hard Easy Moderate Hard MV3D [67] 86.55 78.10 76.67 71.29 62.68 56.56 F-PointNet [23] 88.16 84.02 76.44 83.76 70.92 63.65 AVOD [12] - - - 84.41 74.44 68.65 VoxelNet [70] 89.60 84.81 78.57 81.97 65.46 62.85 SECOND [71] 89.96 87.07 79.66 87.43 76.48 69.10 PointPillars [13] - - - - 77.98 - PointRCNN [13] - - - 88.88 78.63 77.38 Vanilla (mean) 90.08 87.95 84.73 88.45 78.43 77.29 I(8)-23-AP (mean) 90.26 88.34 87.36 89.01 79.00 78.10 F-23-AP (mean) 90.22 88.26 87.30 88.83 78.93 78.03 Vanilla (max) 90.17 88.18 86.03 88.49 78.55 77.38 I(8)-23-AP (max) 90.46 88.58 87.53 89.37 79.40 78.51 F-23-AP (max) 90.36 88.56 87.52 89.20 79.23 78.35 Table 4.5: Performance comparison between the proposed models and state-ofthe-art 3D detection systems on the KITTI validation set for the Car class. I highlight the best performing models, considering both mean performance and best performance. 4.5. Experimental Results 147 4.5.4 Qualitative results In Figs. 4.10 and 4.11 I show the results of the best performing induced attention-based model on some KITTI validation samples, displaying all three main classes: Cars, Pedestrians and Cyclists. As can be observed, the model is capable of detecting objects with high positional accuracy, while being robust to occlusions and uncommon orientations (Figs. 4.10a, 4.10b, 4.10c, 4.11a). A common cause for detection inaccuracies is heavy object truncation, such as in Fig. 4.10d, in which case the predicted orientation might be noisy. High distances might also be problematic, as they could lead to false negatives (Fig. 4.10e) especially for the smaller classes, as the number of points might be insufficient for successful object identification. Of all the classes, pedestrians are easily the most challenging for a LiDARbased system due to the fact that they are small, non-rigid and have no standard structure like cars or bicycles. As a result, they are difficult to estimate, especially in terms of orientation (Fig. 4.11a), and might give rise to false positives as they are easily confused with other entities, such as traffic sign, poles or small trees. An example of this phenomenon is displayed if Fig. 4.10f, where a traffic sign is mistakenly interpreted as a pedestrian. Another challenging scenario for this kind of model is represented by situations in which many small objects are close together in space, such as groups of pedestrians (Fig. 4.11d). In these situations, vote points might erroneously gather around objects that are different from the ones they belong to: when this happens, certain objects might be left with very small clusters whose features are very similar to those of neighboring clusters, and therefore run the risk of being ignored during sampling, originating a false negative. The examples shown in Fig. 4.11, in particular, highlight cases in which the model correctly detects objects that are clearly visible, but not annotated: •the pedestrian on the right in Fig. 4.11a, distinctly visible in both the image and the LiDAR scan; •the pedestrian and the cyclist on the right in Fig. 4.11b;