Machine Learning algorithm evaluation on advanced driver assistance

EVALUACIÓN DE ALGORITMOS DE MACHINE LEARNING PARA CONDUCCIÓN MACHINE LEARNING ALGORITHM EVALUATION ON ADVANCED DRIVER ASSISTANCE TRABAJO FIN DE GRADO CURSO 2020-2021 BY WENBO SUN DIRECTOR CARLOS GARCÍA SÁNCHEZ GUILLERMO BOTELLA JUAN GRADO EN INGENIERÍA INFORMÁTICA FACULTAD DE INFORMÁTICA UNIVERSIDAD COMPLUTENSE DE MADRID THANKS I am very grateful to the two teachers for their help in this tfg. From the beginning, I didn't know and didn't know how to do it, not only gave me the general direction of the project, but also gave me some necessary information. In the later period, Carlos pointed me to the shortcomings of the project and encouraged me to complete the project. For this I must say that the teachers are very good. I am very grateful and grateful to them. At the same time, I am very grateful to the school for the help I learned during the epidemic. ABSTRACT Machine learning algorithm evaluation on advanced driver assistance In this research and development project, our main purpose is to study four deep learning architectures for real-time object detection of people and bicycles encountered in front of driving. We use 4 different algorithms for the same data set, and compare the mAPs obtained after training. And discuss which method is the most accurate, but also consider the time it takes to get what is suitable for what kind of scene. The project I came up with would like to be used in a driving assistance system. The system uses camera sensors to get input, and then uses algorithms to assist, so that the safety of the car is guaranteed when driving. At the same time, it can run on a lowperformance version of the machine and compare the fps of different algorithms. Keywords Computer vision, autonomous driving, convolutional neural network, object detection, deep learning, machine learning, feature selection and extraction, driving assistance INDEX OF CONTENTS Chapter1 -!Introduction ............................................................................................... 1! Chapter1.1 -!Motivation ........................................................................................ 2! Chapter1.2 -!Objective ......................................................................................... 4! Chapter2 -!Contribution of student ............................................................................ 5! Chapter2.1 -!Plan of Works .................................................................................... 5! Chapter2.2 -!Flow Chart ........................................................................................ 6! Chapter3 -!Advanced and architecture ................................................................... 7! Chapter3.1 -!Convolutional Neural Network, CNN ............................................ 7! Chapter3.2 -!Target detection network ............................................................ 10! Chapter3.2.1 -!Two stage ............................................................................. 10! 1.!Faster-RCNN[4] ................................................................................... 10! 2.!Fpn[5] ................................................................................................... 13! Chapter3.2.2 -!One stage ............................................................................ 17! 1.!SSD[6] ................................................................................................... 17! 2.!VGG16[8] ............................................................................................ 18! 3.!MobileNetV3[9] ................................................................................... 20! 4.!YOLOv4[7] ........................................................................................... 23! Chapter4 -!Data Set ................................................................................................... 30! Chapter4.1 -!Tshingua-Daimler[13] .................................................................... 30! Chapter4.2 -!INRIA Person Dataset[14] .............................................................. 30! Chapter4.3 -!Created data set .......................................................................... 30! Chapter5 -!Hardware ................................................................................................. 31! Chapter5.1 -!Intel(R) Core(TM) i7-9700F CPU .................................................... 31! Chapter5.2 -!GEFORCE RTX 2080 Ti .................................................................... 32! Chapter5.3 -!Jetson Nano 2GB Developer Kit[28] ........................................... 32! Chapter6 -!Target detection evaluation ................................................................. 33! Chapter6.1 -!Intersection Over Union (IOU) ...................................................... 33! Chapter6.2 -!True Positive, False Positive, False Negative and True Negative 33! Chapter6.3 -!Precision ......................................................................................... 34! Chapter6.4 -!Recall .............................................................................................. 34! Chapter6.5 -!AP calculation ............................................................................... 35! Chapter7 -!Frameworks .............................................................................................. 35! Chapter7.1 -!Cuda ............................................................................................... 35! Chapter7.2 -!Pytorch ........................................................................................... 36! Chapter7.3 -!MMCV ............................................................................................. 37! Chapter8 -!Results of the experiment ....................................................................... 37! Chapter8.1 -!Results of VGG16 ........................................................................... 38! Chapter8.2 -!Results of SSD MobileNet .............................................................. 40! Chapter8.3 -!Results of YOLOv4 .......................................................................... 41! Chapter8.4 -!Result of FASTER RCNN+FPN ......................................................... 42! Chapter9 -!Comparison of experimental results ..................................................... 43! Chapter9.1 -!Comparison of results of high-performance machines ............ 43! Chapter9.2 -!Comparison of results of low-performance machines ............. 45! Chapter10 -!Conclusions and future work ............................................................... 46! Chapter10.1 -!Conclusions .................................................................................. 46! Chapter10.2 -!Future Work .................................................................................. 47! BIBLIOGRAPHY ................................................................................................................ 49! Appendix ........................................................................................................................ 51! 1 Chapter1 - Introduction Pedestrian Detection has always been a hot and difficult point in computer vision research. The problem to be solved by pedestrian detection is to find all pedestrians in an image or video frame, including their position and size, which are generally represented by rectangular boxes, similar to face detection, which is also a typical target detection problem. Pedestrian detection technology has a strong use value. It can be combined with pedestrian tracking, pedestrian re-identification and other technologies. It can be used in automotive unmanned driving systems (ADAS), intelligent robots, intelligent video surveillance, human behavior analysis, passenger flow statistics systems, and intelligence Transportation and other fields. Since the human body is quite flexible, there will be various postures and shapes, and its appearance is greatly affected by wearing, posture, viewing angle, etc., and it also faces the influence of factors such as occlusion and illumination. This makes pedestrian detection a computer vision A very challenging subject. The main problems to be solved in pedestrian detection are: The appearance is very different. Including viewing angle, posture, clothing and attachments, lighting, imaging distance, etc. Looking at the past from different angles, the appearance of pedestrians is very different. Pedestrians in different postures have very different appearances. Due to the different clothes that people wear, as well as the influence of umbrellas, hats, scarves, luggage and other attachments, the appearance is very different. The difference in lighting also caused some difficulties. The human body at a distance and the human body at a close distance are also very different in appearance. 2 Occlusion problem. In many application scenarios, pedestrians are very dense and there are serious occlusions. We can only see a part of the human body, which brings serious challenges to the detection algorithm. The background is complicated. Whether indoor or outdoor, pedestrian detection generally faces very complicated backgrounds. The appearance and shape, color, and texture of some objects are very similar to human bodies, which makes the algorithm unable to distinguish accurately. Detection speed. Pedestrian detection generally uses a complex model with a large amount of calculations. It is very difficult to achieve real-time and generally requires a lot of optimization. Chapter1.1 - Motivation One day in the future, new energy vehicles will eventually replace fuel vehicles, and autonomous driving will also replace the driver one day in the future. The current level of autonomous driving is formulated by the SAE International[27], which is divided into 6 levels (L0-L5). -Level L0: The driver is in full control of the vehicle; -Level L1: The automatic system can sometimes assist the driver to complete certain driving tasks; -Level L2 assisted driving: The automatic system can complete certain driving tasks, but the driver needs to monitor the driving environment and complete the rest, while ensuring that problems occur and take over at any time. At this level, the wrong perception and judgment of the automatic system can be corrected by the driver at any time, and most car companies can provide this system. L2 can be divided into different usage 3 scenarios based on speed and environment, such as low-speed traffic jams on the loop, fast driving on highways, and automatic parking by the driver in the car; -Level L3 semi-autonomous driving: The automatic system can not only complete certain driving tasks, but also monitor the driving environment under certain conditions, but the driver must be ready to regain driving control (when the automatic system requests it). Therefore, at this level, the driver still cannot sleep or take a deep rest. After the completion of L2, the research field of car companies is extended from here. Due to the particularity of L3, the most meaningful deployment currently seen is to upgrade on the high-speed L2; the difference between L3 and L2 is that the vehicle is responsible for peripheral monitoring, and the human driver only needs to maintain attention for emergencies. -Level L4 highly automated driving: Automated systems can complete driving tasks and monitor the driving environment under certain environments and specific conditions; currently, the deployment of L4 is mostly based on city use, which can be fully automated valet parking. It can also be directly combined with taxi services. At this stage, within the scope of autonomous driving, all tasks related to driving have nothing to do with the driver and passengers. The perception of external responsibility lies in the autonomous driving system, and there are different design and deployment ideas here; -Level L5 fully automated driving: all driving tasks that the automated system can complete under all conditions. 10 Figure[3.1.3] Fully connected layer output result Chapter3.2 - Target detection network -Commonly used detection can be roughly divided into two categories: two stage and one stage. Two stage means that the detection network is divided into two steps. The first step is to frame the object, and the second step is to determine the classification of the object. One stage is to directly use the regression network to get its classification and detection frame based on the extracted features. Among them, Faster R-CNN belongs to two stage, SSD and YOLO belong to one stage. -The advantage of one stage is fast, and the advantage of two stage is accuracy. After getting the detection frame, it is much easier to classify this task than directly returning to the network to get the classification. Chapter3.2.1 - Two stage 1. Faster-RCNN[4] The Faster-RCNN model introduces RPN (Region Proposal Network) to directly generate candidate regions. Faster-RCNN can be seen as a combination of RPN and Fast RCNN models, that is, Faster-RCNN = RPN + Fast-RCNN. 11 For the RPN network, a CNN model (generally called a feature extractor) is first used to receive the entire picture and extract the feature map. Then an N×N (3×3 in the text) sliding window is used on this feature map, and a low-dimensional feature (such as 256-d) is mapped to each sliding window position. Then this feature is sent to two fully connected layers, one for classification prediction and the other for regression. Generally, k a priori boxes (anchors, default bounding boxes) of different sizes or proportions are set for each window position, which means that k candidate regions (region proposals) are predicted for each position. For the classification layer, its output size is 2k, which means that each candidate area contains the object or background probability value, while the regression layer outputs 4k coordinate values, which indicate the position of each candidate area (relative to each a priori box). For each sliding window position, these two fully connected layers are shared. Therefore, RPN can be implemented using convolutional layers: first, an n×n convolution to obtain low-dimensional features, and then two 1×1 convolutions, which are used for classification and regression respectively. Figure[3.2.1.1] CNN model (generally called a feature extractor) RPN uses two classifications, only distinguishing background and objects, but does not predict the category of the object, that is, class-agnostic. Since the coordinate values need to be predicted at the same time, during training, the a priori box must be matched with the ground-truth box. The principle is: (1) the a priori box with the highest 12 IoU of a certain ground-truth box; (2) and A ground-truth box with an IoU value greater than 0.7 a priori box, as long as one is satisfied, the a priori box can match a ground-truth, so that the a priori box is a positive sample (belonging to the object), and the groundtruth For the return goal. For those a priori boxes whose IoU value with any ground-truth box is lower than 0.3, they are considered as negative samples. The RPN network can be trained separately, and the separately trained RPN model gives many region proposals. Due to the large number of a priori boxes, many of the candidate regions predicted by RPN overlap. We must first perform NMS (non-maximum suppression, IoU threshold is set to 0.7) to reduce the number of candidate regions, and then arrange them in descending order of confidence, select top -N region proposals are used to train the Fast R-CNN model. The role of RPN is to replace the role of Selective search, but it is faster, so Faster R-CNN can accelerate both training and prediction. Figure[3.2.1.2] RPN structure 13 Faster-RCNN follows the following training process: Step 1: Use the pre-trained model on ImageNe to initialize the feature extraction network and train the RPN network; Step 2: Use the model pre-trained on ImageNet to initialize the Fast-RCNN feature extraction network, and use the candidate box generated by the trained RPN network in step one as input to train a Fast-RCNN network. So far, each of the two networks The parameters of the first layer are not shared at all; Step 3: Use the Fast-RCNN network parameters of step two to initialize a new RPN network, but set the learning rate of the feature extraction network parameters shared by RPN and Fast-RCNN to 0, even if the unique parameters of the RPN network are learned, it is fixed Feature extraction network. At this point, the two networks have shared all common convolutional layers; Step 4: For those network layers that are still shared, add Fast-RCNN-specific network layers, continue training, and fine-tune Fast-RCNN-specific network layers. Up to this point, RPN and Fast-RCNN networks completely share parameters. Use Fast-RCNN to complete candidate frame extraction and target detection functions at the same time. 2. Fpn[5] In the past faster rcnn for target detection, no matter it is rpn or fast rcnn, roi acts on the last layer. This is no problem for the detection of large targets, but there are some problems for detection of small targets. Because for small targets, when convolutional pooling is performed to the last layer, the semantic information is actually gone, because we all know that the method for mapping a roi to a feature map is to directly divide the 14 underlying coordinates by stride, Obviously, the later, the smaller the map will be, and it may even disappear. Therefore, in order to solve the problem of multi-scale detection, a feature pyramid network is introduced. Figure[3.2.1.3] Multi-scale pyramid Figure[3.2.1.3] (a) is a fairly common multi-scale method called a feature image pyramid. This method was widely used in the earlier artificial design features (DPM), and it was also used in CNN. It is to multi-scale the input image by setting different zoom ratios. This can solve multiple scales, but it is equivalent to training multiple models (assuming a fixed input size is required). Even if the input size is not allowed, it also increases the memory space for storing images of different scales. Figure[3.2.1.3] (b) is CNN. Compared with artificially designed features, cnn can learn more advanced semantic features by itself. At the same time, CNN is robust to scale changes. Therefore, as shown in the figure, the features calculated from the input of a single scale can also be used to identify , But when encountering obvious multi-scale target detection, the pyramid structure is still needed to further improve the accuracy. Judging from some of the leading methods on the imageNet and COCO data sets, the feature image pyramid method is used in the test, that is, the combination of Figure[3.2.1.3] (a) and Figure[3.2.1.3] (b). Explains that the advantage of each level of the characterized image pyramid is that it produces a multi-scale feature representation, 15 and the features of each level have strong semantics (because all features generated by cnn), including the high-resolution level (The largest scale input image). However, this mode has obvious drawbacks. Compared with the original method, the time has increased by 4 times, and it is difficult to use in real-time applications. Similarly, it also increases the storage cost, which is why the image pyramid is only used in the testing phase. But if it is only used in the testing phase, then training and testing will be inconsistent in inference. Therefore, some recent methods have simply abandoned the image pyramid. Figure[3.2.1.3] (c), SSD tried to use CNN pyramid-shaped hierarchical features earlier. Ideally, the SSD-style pyramid reuses the multi-scale feature maps from multiple layers calculated by the forward process, so this form does not consume additional resources. However, in order to avoid the use of low-level features, SSD abandoned the shallow feature map, but started building pyramids from conv4_3, and added some new layers. Therefore, SSD has given up on reusing higher-resolution feature maps, but these feature maps are very important for detecting small targets. This is the difference between SSD and FPN. Figure[3.2.1.3] (4) is the structure of FPN. FPN is a pyramid form for natural use of CNN hierarchical features, while generating feature pyramids with strong semantic information at all scales. Therefore, the structure of FPN is designed with a top-down structure and a horizontal connection to integrate a shallow layer with high resolution and a deep layer with rich semantic information. In this way, it is possible to quickly build a feature pyramid with strong semantic information on all scales from a single input image at a single scale, without incurring significant costs. FPN is not a complete target detection network, but a feature pyramid network. The FPN mentioned is actually a feature pyramid extraction feature. Therefore, the idea of Faster RCNN plus FPN essentially changes the feature extraction part because there are more feature layers. So ROIpooling has also increased. 16 Figure[3.2.1.4] faster_rcnn+fpn structure ① The backbone network generates four-scale feature maps, and then sequentially passes through their respective Lateral_conv to make their channel numbers consistent. ②The deep feature maps are down-sampled to adjacent layer scales step by step, and then the two are added and fused. ③Output the feature map after fusion The low-level feature semantic information is relatively small, but the target location is accurate; the high-level feature semantic information is richer, but the target location is relatively rough. Fpn is independently predicted in different feature maps. 17 Figure[3.2.1.5] Independent prediction in different feature maps Chapter3.2.2 - One stage 1. SSD[6] SSD uses the idea of meshing, and unlike Faster RCNN, it integrates all operations into a convolutional network. In order to detect targets of different scales, SSD performs sliding window scanning on the feature images of different convolutional layers; small targets are detected in the feature images output by the previous convolutional layer, and large targets are detected in the feature images output by the subsequent convolutional layer. The goal. Its main features are: 1.1. Detection based on multi-scale feature images: Prediction on multi-scale convolution feature maps to detect targets of different sizes, which improves the detection accuracy of small target objects to a certain extent. 1.2. Drawing on the idea of Anchor boxes in Faster R-CNN, sampling candidate regions on feature maps of different scales, which improves the recall rate of detection and the detection effect of small targets to a certain extent. The following figure shows the principle of SSD: 18 Figure[3.2.2.1] Schematic of SSD 2. VGG16[8] VGG was proposed in 2014 by the Visual Geometry Group, Department of Science and Engineering, Oxford University. The main work is to prove that increasing the depth of the network can affect the final performance of the network to a certain extent. VGG has two structures, namely VGG16 and VGG19. Except for the difference in network depth, there is no difference in essence between the two. Compared with AlexNet in 2012, a high advance of VGG is to use continuous 3x3 small convolution kernels to replace the larger ones in AlexNet (AlexNet uses 11x11, 7x7 and 5x5 convolution kernels). The superposition of two 3x3 convolution kernels with a step length of 1, its receptive field is equivalent to a 5x5 convolution kernel. However, the use of stacked small convolution kernels is due to the large convolution kernel, because the increase in the number of layers 19 increases the nonlinearity of the network, which allows the network to learn more complex models, and the small convolution kernel has fewer parameters. Figure[3.2.2.2] VGG16 model VGG16 contains: 13 convolutional layers (Convolutional Layer), respectively represented by conv3XXX 3 fully connected layers (Fully connected Layer), respectively represented by FCXXXX 5 pooling layers (Pool layer), respectively represented by maxpool The outstanding feature of VGG16 is simplicity, which is reflected in: 3. The convolutional layers all use the same convolution kernel parameters The convolutional layers are all expressed as conv3-XXX, where conv3 indicates that the kernel size of the convolutional layer used by the convolutional layer is 3, that is, the width and height are 3, and 3*3 is very Small convolution kernel size, combined with other parameters (stride=1, padding=same), so that each convolutional layer (tensor) can maintain the same width and width as the 26 a)Combine the original Darknet53 with CSPNet. YOLOv3 is composed of a series of residual structures. After the combination, the main job of CSPnet is to split the stack of the original residual block, and split it into two parts: the main part continues to stack the original residual block, and the branch part is equivalent to a residual edge. After a small amount of processing, it is directly connected to the end. b)Use the MIsh activation function to replace the original Leaky ReLU. In YOLOv3, each convolutional layer includes a batch normalization layer and a Leaky ReLU. In the backbone network CSPDarknet53 of YOLOv4, Mish is used instead of the original Leaky ReLU. (2)SPP[11] The original design purpose of SPP is to make the convolutional neural network not restricted by the fixed input size. In YOLOv4, the author introduced SPP because it significantly increases the receptive field, isolates the most important context features, and hardly reduces the running speed of YOLOv4. As shown in the figure below, it is the classic spatial pyramid pooling layer in SPP. Figure[3.2.4.3] spp space pyramid pooling layer In YOLOv4, the specific method is to use the maximum pooling of four different scales to process the feature map output by the upper layer. The maximum pooling core size is 13x13, 9x9, 5x5, 1x1, and 1x1 is equivalent to not processing. 27 (3)PANet[12] PANet as a whole can be seen as a number of improvements on Mask R-CNN, making full use of feature fusion, such as introducing Bottom-up path augmentation structure, making full use of network shallow features for segmentation; introducing Adaptive feature pooling to make the extracted ROI Features are more abundant; FullyconneFcted fusion is introduced, and a more accurate segmentation result is obtained by fusing the output of a front-background binary classification branch. Figure[3.2.4.4] PANet schematic (a)FPN backbone,(b)Bottom-up path augmentation,(c) Adaptive feature pooling,(d)Box branch,(e)Fully-connected fusion In YOLOv4, the author uses PANet instead of FPN in YOLOv3 as the method of parameter aggregation, and performs parameter aggregation from different backbone layers for different detector levels. And modified the original PANet method, using tensor connection (concat) instead of the original shortcut connection (shortcut connection). (4) YOLOv3 Head In YOLOv4, the Head that inherits YOLOv3 performs multi-scale prediction, which improves the detection performance of targets of different sizes.YOLOv4 learns the 28 YOLOv3 method, uses three different levels of feature maps for fusion, and inherits the Head of YOLOv3. (5)Tricks To obtain better accuracy without increasing the inference cost, but only change the training strategy or only increase the training cost method, the author calls it "Bag of freebies"; it only increases a small amount of inference cost but can significantly improve the accuracy of target detection Plug-in modules and post-processing methods, called "Bag of specials" - Bag of freebies Random zoom Flip, rotate Image disturbance, noise, occlusion Change brightness, contrast, saturation, random erase(Cutout MixUp CutMix) Common regularization methods are: DropOut DropConnect DropBlock The methods to balance positive and negative samples are: Focal loss OHEM (online hard-to-separate sample mining) In addition, there are improvements in return loss: GIOU DIOU CIOU - Bag of specials Increase the receptive field skills: SPP ASPP RFB Attention mechanism: Squeeze-and-Excitation (SE) Spatial Attention Module (SAM) Feature fusion integration: FPN SFAM ASFF BiFPN (from the famous EfficientDet) Better activation function: ReLU LReLU PReLU ReLU6 SELU Swish hard-Swish Post-processing non-maximum suppression algorithm: soft-NMS DIoU NMS (6)Ways to improve In addition to the various Tricks mentioned above, in order to make the target detector easier to train on a single GPU, the author also proposes 5 improved methods: 29 - Mosaic This is a new data enhancement method proposed by the author, which draws on the idea of CutMix data enhancement method. CutMix data enhancement method uses two pictures for stitching, but Mosaic uses four pictures for stitching. - SAT SAT is a self-adversarial training data enhancement method, which is a new adversarial training method. In the first stage, the neural network changes the original image without changing the network weights. In this way, the neural network conducts an adversarial attack on itself, changing the original image to create a deception that there is no desired object on the image. In the second stage, the normal method is used to train the neural network to detect the target. - CmBN The full name of CmBN is Cross mini-Batch Normalization, which is defined as Cross MiniBatch Normalization (CmBN). CmBN is an improved version of CBN, which is used to collect statistical data in multiple mini-batches in a batch. - SAM The author modified the original SAM (Spatial Attention Module) method, changing SAM from spatial attention to point attention. As shown in the figure below, for the conventional SAM, the maximum pooling layer and the average pooling layer act on the input feature maps respectively to obtain two sets of feature maps with the same shape, and then input the results into a convolutional layer, followed by Sigmoid function to create spatial attention. - PAN The author modified the original PAN (Path Aggregation Network) method, using tensor connection (concat) instead of the original shortcut connection (shortcut connection) 30 Chapter4 - Data Set Chapter4.1 - Tshingua-Daimler[13] Tsinghua-Daimler Cyclist Detection contains 9741 images with annotations only for "cyclist". Only cyclists which are fully visible (occlusion<10%) and higher than 60 pixels have been labeled here. Chapter4.2 - INRIA Person Dataset[14] This dataset has cropped the image of pedestrians so that the person can be more prominent, and the posture is not particularly strange. Only a few pictures are obtained from Google. Chapter4.3 - Created data set I did not select pedestrians from Tsinghua's data set, because the size of pedestrian images in Tsinghua's data set is 64x64, which is too small for experiments, so I did not choose. For the KITTI data set, when I got their data set, I found that it was all cars, so I judged that it did not meet my topic. Finally. Cyclist data set adopts Tsinghua-Daimler Cyclist Detection part of the data set ,1507 image resolution 2048*1024. The Person data set uses the 614 training set and 288 test set of the INRIAPerson pedestrian data set. A total of 2088 sheets of all data sets are divided into training set (1671) and test set (417) at a ratio of 8:2. 31 All the pictures used need to use the .jpg format. The created data set uses codes to distinguish the training set and the validation set, and then uses the code to convert the VOC format. They are 1.Annotations (.xml, each picture corresponds to a file, which stores the name of the picture, the long section, the upper left and lower right coordinates of the target frame, the category name of the target frame) 2.ImagesSet(.txt, save The name of the image that needs to be trained) 3.JPEGImage (the original image) Chapter5 - Hardware The hardware I used to complete this project. Knowing this is necessary, because different hardware will affect the training process, especially the training time, or efficiency. Chapter5.1 - Intel(R) Core(TM) i7-9700F CPU l Cores: 8 l Threads: 8 l Processor Base Frequency: 3.00GHz l Max Turbo Frequency: 4.7GHz l TDP: 65W 32 Chapter5.2 - GEFORCE RTX 2080 Ti l NVIDIA CUDA ® Cores: 4352 l Boost Clock (MHz): 1545 l Base Clock (MHz): 1350 l Standard Memory Config: 11 GB GDDR6 l Graphics Card Power (W): 250W Chapter5.3 - Jetson Nano 2GB Developer Kit[28] l CPU Quad-core ARM® A57 @ 1.43 GHz l GPU 128-core NVIDIA Maxwell™ l MEMORY 2 GB 64-bit LPDDR4 25.6 GB/s Since its release in 2019, NVIDIA Jetson Nano has created a frenzy in the field of AIOT edge computing applications worldwide, He was also named "Best AI Processor" in the 2020 Best Visual Products award list by Edge AI and Vision Alliance. This is mainly because the NVIDIA Jetson Nano chip fully meets the six challenges of AIOT chip: high performance, small size, low power consumption, full interface, ecological integrity, and excellent cost. The reason for using this edge device was to test various models on a low-power machine. For comparison, test the FPS on a low-performance machine. 33 Chapter6 - Target detection evaluation The classification task will be judged by the precision/recall curve. The principal quantitative measure used will be the average precision (AP).[15] There are some important things to know about calculating AP.[16] Chapter6.1 - Intersection Over Union (IOU) Figure[6.1.1] IOU formula To put it simply, we require that the IOU of the two boxes is divided by the overlapping part of the two boxes by the union of the two boxes. It is clear from the second formula above. Pay attention to the two boxes when calculating. The area of the union is equal to the sum of the areas of the two minus the area of the intersection of the two. Chapter6.2 - True Positive, False Positive, False Negative and True Negative True Positive (TP): A correct positioning result is that the IOU between your predicted box and our groundtruth can be greater than our specified threshold, we generally take this threshold to be 0.5 34 False Positive (FP): It is a wrong result, that is, the IOU of the box and groundtruth you predicted is less than the threshold False Negative (FN): We originally had an object here, so there should be a box in our place, but you did not predict it, then this groundtruth is a FN for your model True Negative (TN): This is that our groundtruth has found a prediction box whose IOU is greater than the threshold, then the ground is considered to be successfully detected Chapter6.3 - Precision Precision is the ability of a model to identify only the relevant objects. It is the percentage of correct positive predictions and is given by: Chapter6.4 - Recall Recall is the ability of a model to find all the relevant cases (all ground truth bounding boxes). It is the percentage of true positive detected among all relevant ground truths and is given by: 35 Chapter6.5 - AP calculation (1) Before VOC2010, you only need to select the maximum Precision when Recall >= 0, 0.1, 0.2, ..., 1 total 11 points, and then AP is the average of these 11 Precision. Figure[6.5.1] AP calculation Calculating the total area, we have the AP. (2) In VOC2010 and later, for each different Recall value (including 0 and 1), select the maximum Precision when it is greater than or equal to these Recall values, and then calculate the area under the PR curve as the AP value. Finally the average AP value of all classes is mAP. Chapter7 - Frameworks Chapter7.1 - Cuda CUDA (an acronym for Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) model created by Nvidia.[17]It allows software developers and software engineers to use a CUDA-enabled graphics 42 Figure[8.3.2] loss line chart of YOLOv4 The same is a single-layer target detection, yolov4's performance is significantly better than before. Chapter8.4 - Result of FASTER RCNN+FPN 43 Figure[8.4.1] mAP line chart of FASTER RCNN+FPN Figure[8.4.2] loss line chart of FASTER RCNN+FPN I am very satisfied with the performance of faster rcnn+fpn, because the same data set has a very big difference through the algorithm. Below I will compare all the above results together. Chapter9 - Comparison of experimental results Chapter9.1 - Comparison of results of high-performance machines Model Input Size FPS mAP Cyclist Person VGG16 512*512 47.3 0.81 0.85 0.77 MobileNetv3 320*320 43.1 0.67 0.54 0.80 YOLOv4 608*608 19.6 0.78 0.75 0.81 FASTER RCNN+FPN 1333*800 12 0.9 0.899 0.902 Table[9.1] Comparison of algorithm results-high 44 The table shows us the various data of each algorithm in the high-performance computer. From the experimental results, we can see that FASTER RCNN+FPN performs best in terms of detection accuracy, but the shortcomings are also very obvious. The fps only reached 12. This is due to its two-stage network. It has a region proposal network RPN, this part uses CNN, and the region proposal network and classification network share convolutional features. And there is FPN, independent prediction of different feature layers. These can effectively target detection, but the speed is slower. We can easily find this from the table. The above SSD-MobileNet input resolution is relatively small, and the recognition effect of the person class with the original image size of about 800 is better. The input resolution of SSD-VGG16 is relatively large, and it can better recognize the cyclist class with the original image size of 2048*1024. The reason why mobileNet's mAP is too low is because of overfitting in the person part. It is because the data set of the person part is too small. The way to solve this problem and get the performance it should be is to increase the amount of data, as I explained in the results section. YOLOv4 is already very good as a non-lightweight performance. It shows a very average value, and there are some aspects that must be explained. Compared with its previous version, it has added many improvements, such as adding feature pyramids (SPP, PAN) to the last convolutional layer, and modifying the activation function to Mish, learning rate annealing algorithm. From the structural point of view, YOLOv4 also has an FPN structure. Compared with FASTER RCNN+FPN, YOLOv4 has a higher fps performance. Although mAP did not reach the two stage, I got a better result in the one stage. 45 Chapter9.2 - Comparison of results of low-performance machines Model FPS VGG16 0.98 MobileNetv3 7.61 YOLOv4 -- FASTER RCNN+FPN -- Table[9.2] Comparison of algorithm results-low On low-performance computers, I focus on the performance of fps. Because the machine I used is the jetson nano 2gb version of NVIDIA. It means that in the case of supporting a series of environments, the theoretical video memory of the machine is only 2gb. In actual use, the video memory is only about 1.3gb. This has actually reached a limit value. From the table, we can see that in the case of very low video memory, only lightweight algorithms can actually run normally and perform target recognition. The normal version of YOLOv4 cannot run on a machine with only 2gb of video memory. The two-stage FASTER RCNN+FPN is even more impossible. Therefore, for a low-performance machine, it is very important to choose a correct algorithm. From the table, as a simple convolutional neural model, VGG16 can operate correctly and recognize targets normally. But I think its fps is not good enough. As a lightweight neural network, MobileNet can run normally and has a fps of about 7 I think it is understandable. There may be doubts that this fps is not very high, why is it still a lightweight neural network? This is because we are using the Large version, which has 3 46 bottleneck layers and a point-wise convolutional layer more than the Low version. Its Depthwise Separable Convolution is very suitable for low-performance versions. Chapter10 - Conclusions and future work Chapter10.1 - Conclusions For these four algorithms and the selected data set, from the results, if the machine performance is not considered, then FASTER RCNN+FPN is undoubtedly the best auxiliary system. Because it has higher accuracy. It is undeniable that two stage is better than one stage in terms of accuracy. But the shortcomings are also very obvious. The training time of the two stage is much longer than that of the one stage. And in actual use, the fps of the two stage application on high-performance machines is not high enough, and it cannot run smoothly on low-performance machines. The speed of yolov4 is much faster than faster-rcnn on high-performance machines. And because the feature extraction layer of yolov4 uses a feature pyramid structure, it can also get very good accuracy. SSD-VGG16 uses feature maps of different scales for detection. This may also be the reason why in this data set, for small targets such as people and bicycles, and there may be obstructions on the road, it has a higher accuracy rate than the YOLO algorithm. For this time, MobileNetv3, which performed best in low-performance machines. I think the lightweight neural network can be used as a solution in solving business problems, or in vehicle safety systems. Because the speed of the car is fast and the neural network needs to be fast, then a lightweight neural network is a very good choice. Video memory, memory, and cpu will be relatively small. Of course, this is just a structural point of view of these four algorithms. 47 I think the choice of the training data set is very important. From the final result, the choice of the picture size is actually very important, because there are actually restrictions on the pictures on the Internet or the pictures taken by yourself. Today's camera equipment is getting better and better, for example, the pixels of mobile phones are getting better and better. The requirements for the training set are also very high. Maybe a certain target detection algorithm is excellent, but due to the inappropriate choice of the data set, the final result is bad. So in the end, because of the different models of different models and the different camera options in the actual use, the final results will appear to be different. I finally used all the algorithms to make actual detections for assisted driving. Using the video of learning to drive in a driving school, a novice may not notice pedestrians and bicycles on the road. 4 algorithms are used for the screenshots of the video in the appendix. I am satisfied with the training results and can achieve target detection for most of the scenes. Chapter10.2 - Future Work For future work, from the target detection framework. From the Faster R-CNN related series focusing on accuracy to the YOLO series focusing on speed, the future target detection research direction will focus more on the balance of accuracy and speed. Therefore, many network structures are generated on the SSD framework. I think that if target detection is to be used as an in-vehicle safety system, various lightweight neural networks are the most preferred in recent years. There are reports that YOLOv4tiny performs very well, we can also try it in low-performance machine. The new YOLOv5 has appeared, we can try to use it in high-performance machine. 48 I have seen some research papers that show that the rotation invariance of convolutional neural networks is obtained through data enhancement and large sample learning, and it does not have this feature itself. Maybe continue to study this will change the basic framework of the target detection algorithm. The use of 3D lidar will be more refined than the image detection. Better sensors will make target detection more refined, and now more autonomous driving applications on the market are 3D lidar. Of course, some conventional ideas are to improve the loss function, activation function, and feature fusion, which can also be considered. 49 BIBLIOGRAPHY [1] Goodfellow, I., Bengio, Y., Courville, A.．Deep learning (Vol. 1)．Cambridge：MIT press，2016 [2] Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, L., Wang, G. and Cai, J., 2015. Recent advances in convolutional neural networks. arXiv preprint arXiv:1512.07108. [3] Zhang, W., 1988. Shift-invariant pattern recognition neural network and its optical architecture. In Proceedings of annual conference of the Japan Society of Applied Physics. [4] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks [5] Tsung-Yi Lin1,2, Piotr Dollar´ 1 , Ross Girshick1 , Kaiming He1 , Bharath Hariharan1 , and Serge Belongie.Feature Pyramid Networks for Object Detection [6] Wei Liu1 , Dragomir Anguelov2 , Dumitru Erhan3 , Christian Szegedy3 , Scott Reed4 , Cheng-Yang Fu1 , Alexander C. Berg.SSD: Single Shot MultiBox Detector [7] Alexey Bochkovskiy,Chien-Yao Wang,Hong-Yuan Mark Liao.YOLOv4: Optimal Speed and Accuracy of Object Detection [8] Simonyan,Zisserman.Very Deep Convolutional Networks for Large Scale Image Recognition [9] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen,Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, Hartwig Adam,Searching for MobileNetV3 [10] Chien-Yao Wang,Hong-Yuan Mark Liao,I-Hau Yeh,Yueh-Hua Wu,Ping-Yang Chen,Jun-Wei Hsieh,CSPNET: A NEW BACKBONE THAT CAN ENHANCE LEARNING CAPABILITY OF CNN [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition [12] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, Jiaya Jia,Path Aggregation Network for Instance Segmentation [13] Y. Yang H. Xiong M. Braun S. Pan K. Li X. Li, F. Flohr and D. M. Gavrila. A new benchmark for vision-based cyclist detection, June 2016. [14] Navneet DALAL,Finding People in Images and Videos [15] Mark Everingham, John Winn, The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Development Kit [16] https://github.com/rafaelpadilla/Object-Detection-Metrics [17] https://developer.nvidia.com/cuda-zone [18] Abi-Chahla, Fedy (June 18, 2008). "Nvidia's CUDA: The End of the CPU?". Tom's Hardware. Retrieved May 17, 2015. [19] Yegulalp, Serdar (19 January 2017). "Facebook brings GPU-powered machine learning to Python". InfoWorld. Retrieved 11 December 2017 [20] Lorica, Ben (3 August 2017). "Why AI and machine learning researchers are beginning to embrace PyTorch". O'Reilly Media. Retrieved 11 December 2017 50 [21] Ketkar, Nikhil (2017). "Introduction to PyTorch". Deep Learning with Python. Apress, Berkeley, CA. pp. 195–208. doi:10.1007/978-1-4842-2766-4_12. ISBN 9781484227657. [22] Patel, Mo (2017-12-07). "When two trends fuse: PyTorch and recommender systems". O'Reilly Media. Retrieved 2017-12-18. [23] Mannes, John. "Facebook and Microsoft collaborate to simplify conversions from PyTorch to Caffe2". TechCrunch. Retrieved 2017-12-18. FAIR is accustomed to working with PyTorch – a deep learning framework optimized for achieving state of the art results in research, regardless of resource constraints. Unfortunately in the real world, most of us are limited by the computational capabilities of our smartphones and computers.” [24] Arakelyan, Sophia (2017-11-29). "Tech giants are using open source frameworks to dominate the AI community". VentureBeat. Retrieved 2017-12-18. [25] "The C++ Frontend". PyTorch Master Documentation. Retrieved 2019-07-29. [26] https://github.com/open-mmlab/mmcv/blob/master/README.md [27] SAE. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. https://www.sae.org/misc/pdfs/automated_driving.pdf,2018. [28] https://developer.nvidia.com/embedded/jetson-nano-2gb-developer-kit 51 Appendix Table[1] SSD-MobileNet mAP Epoch mAP Cyclist person 1 0.612 0.425 0.798 2 0.649 0.491 0.807 3 0.658 0.504 0.811 4 0.669 0.540 0.799 5 0.657 0.499 0.814 6 0.652 0.498 0.805 7 0.663 0.519 0.808 8 0.648 0.494 0.802 9 0.668 0.534 0.803 10 0.665 0.533 0.797 11 0.670 0.541 0.798 12 0.667 0.537 0.797