Repositorio Institucional de Documentos

2021 89 José María Fácil Ledesma Deep Learning for 3D Visual Perception Director/es Civera Sancho, Javier Montesano del Campo, Luis © Universidad de Zaragoza Servicio de Publicaciones ISSN 2254-7606 José María Fácil Ledesma DEEP LEARNING FOR 3D VISUAL PERCEPTION Director/es Civera Sancho, Javier Montesano del Campo, Luis Tesis Doctoral Autor 2021 UNIVERSIDAD DE ZARAGOZA Escuela de Doctorado Programa de Doctorado en Ingeniería de Sistemas e Informática Repositorio de la Universidad de Zaragoza – Zaguan http://zaguan.unizar.es Deep Learning for 3D Visual Perception José M. Fácil Ledesma Advisors: Javier Civera & Luis Montesano Departmento de Informática e Ingeniería de Sistemas Universidad de Zaragoza This dissertation is submitted for the degree of Doctor of Philosophy October 2020 A mi hermana, mis padres, mis abuelas y Mª Luisa. Acknowledgements I started my thesis four years ago but I made the decisions that brought me here around a decade ago. Naturally, during all these years there has been many people influencing my life, supporting and helping me even if they or I did not notice. It is going to be impossible to mention them all; they simply are too many. I thank you all in advance. There are some major players I would like to mention, for whom I am deeply thankful and in debt. First, I want to save a special mention to those people to who I owe such amazing opportunities and for whom I have great respect. To my advisors Javier and Luis for motivating and advising me through all these years. To Thomas Brox for letting me join his lab for a few months. To Alejo for always teaching and helping me since my first days; even if he did not have to. To Lina, for letting me be part of her team. To my high school teacher Ramon for opening this door for me. Thanks to the external reviewers Alberto and Manuel and to the PhD committee Ruben, Jesus, Gabriel, Ana Cris and Taihú. I would not like to forget all the PhD students and Professors at Unizar and specially those that I have directly work with. I would like to specially mention all my co-authors Alejo, Clara, Berta, Alejandro, Josechu, Jose Neira and my two super-advisors. There are some that I would like to have included in the previous list – I still want to – and that I want to thank for there discussions and opinions: Jose, Iñigo, Jesus, Seong and Carlos. My former lab mates Jason, Edu and the two Carlos and also my three outstanding Bachellor and Master students I have co-supervised: Dani, Mikel and Juan for who I hope the best in the next professional steps. The day which comes to mind, while writing these lines, as one of my best days of work was with Jose, Clara and Alejandro. This was the best wine-ending conference deadline, for CVPR 2019. Exhausting experience, that finished around 5AM, but one of the best moments of team work I have ever experienced. If I have half as good teammates and friends as you in xii por ejemplo 50 pixeles, esta a cierta distancia de la cámara. Para ello nosotros presentamos el primer trabajo capaz de estimar profundidad a partir de una sola vista que es capaz de obtener un funcionamiento razonable con múltiples tipos de cámara; como un teléfono o una cámara de video. También presentamos como estimar, utilizando una sola imagen, la estructura de una habitación o el plan de la habitación. Para este segundo trabajo, aprovechamos imágenes esféricas tomadas por una cámara panorámica utilizando una representación equirectangular. Utilizando estas imágenes recuperamos el plan de la habitación, nuestro objetivo es reconocer las pistas en la imagen que definen la estructura de una habitación. Nos centramos en recuperar la versión más simple, que son las lineas que separan suelo, paredes y techo. Localización y mapeo a largo plazo requiere dar solución a los cambios de apariencia en el entorno; el efecto que puede tener en una imagen tomarla en invierno o verano puede ser muy grande. Introducimos un modelo multivista invariante a cambios de apariencia que resuelve el problema de reconocimiento de lugares de forma robusta. El reconocimiento de lugares visual trata de identificar un lugar que ya hemos visitado asociando pistas visuales que se ven en las imágenes; la tomada en el pasado y la tomada en el presente. Lo preferible es ser invariante a cambios en punto de vista, iluminación, objetos dinámicos y cambios de apariencia a largo plazo como el día y la noche, las estaciones o el clima. Para tener funcionalidad a largo plazo también presentamos DynaSLAM, un sistema de SLAM que distingue las partes estáticas y dinámicas de la escena. Se asegura de estimar su posición unicamente basándose en las partes estáticas y solo reconstruye el mapa de las partes estáticas. De forma que si visitamos una escena de nuevo, nuestro mapa no se ve afectado por la presencia de nuevos objetos dinámicos o la desaparición de los anteriores. En resumen, en esta tesis contribuimos a diferentes problemas de percepción 3D; todos ellos resuelven problemas del SLAM Visual. Table of contents 1 Introduction 1 1.1 3D Visual Perception and Visual SLAM . . . . . . . . . . . . . . . . . . . 1 1.2 How Deep Learning is Improving Visual Perception . . . . . . . . . . . . . 3 1.3 Our contributions in 3D Visual Perception . . . . . . . . . . . . . . . . . . 5 1.3.1 Visual Mapping without Motion . . . . . . . . . . . . . . . . . . . 5 1.3.2 Visual Mapping with Little Motion . . . . . . . . . . . . . . . . . 6 1.3.3 Place Recognition under Appearance Changes . . . . . . . . . . . 9 1.3.4 Visual Reconstruction of High-Level Structures . . . . . . . . . . . 10 1.3.5 Visual SLAM on Dynamic Environments . . . . . . . . . . . . . . 11 1.4 ListofPublications.............................. 12 1.5 CodeReleased ................................ 14 1.6 Manuscript Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2 Camera-Aware Multi-Scale Convolutions for Single-View Depth 17 2.1 Introduction.................................. 17 2.2 RelatedWork ................................. 19 2.3 Camera-Aware Multi-scale Convolutions . . . . . . . . . . . . . . . . . . . 20 2.3.1 Focal Length Normalization . . . . . . . . . . . . . . . . . . . . . 23 2.4 ModelandTraining.............................. 23 2.4.1 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.4.2 Losses................................. 27 2.4.3 TrainingSchedule .......................... 28 2.5 Multi-Camera Experiments and Results . . . . . . . . . . . . . . . . . . . 29 2.5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.5.2 Influence of context . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.5.3 Overfitting of standard networks . . . . . . . . . . . . . . . . . . . 30 2.5.4 Robust Generalization with CAM-Convs . . . . . . . . . . . . . . 34 2.5.5 Experiments on Multiple Datasets . . . . . . . . . . . . . . . . . . 36 xiv Table of contents 2.6 Conclusions.................................. 38 3 Combining Single-View Deep Learning Depth with Multi-View Depth 41 3.1 Introduction.................................. 42 3.2 RelatedWork ................................. 43 3.2.1 Multi-ViewDepth .......................... 43 3.2.2 Single-ViewDepth.......................... 44 3.3 Single and Multi-View Depth Fusion . . . . . . . . . . . . . . . . . . . . . 45 3.3.1 Multi-viewDepth........................... 46 3.3.2 Single-viewDepth .......................... 47 3.3.3 DepthFusion............................. 48 3.3.4 Multi-view Low-Error Point Selection . . . . . . . . . . . . . . . . 50 3.4 ExperimentalResults............................. 52 3.5 Conclusions.................................. 55 4 Condition-Invariant Place Recognition 59 4.1 Introduction.................................. 59 4.2 The Partitioned Nordland Dataset . . . . . . . . . . . . . . . . . . . . . . 61 4.2.1 Data Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.2.2 DatasetPartitions........................... 63 4.2.3 Placelabels.............................. 63 4.3 RelatedWork ................................. 63 4.3.1 Descriptors.............................. 64 4.3.2 Visual Place Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 66 4.3.3 Multi-View Place Recognition using Multi-View Descriptor . . . . 67 4.4 NetworkArchitectures ............................ 67 4.4.1 Single-View ResNet-50 . . . . . . . . . . . . . . . . . . . . . . . . 67 4.4.2 Descriptor Grouping . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.4.3 DescriptorFusion........................... 68 4.4.4 Recurrent Descriptors . . . . . . . . . . . . . . . . . . . . . . . . 68 4.5 Training.................................... 69 4.5.1 Convention for Same Place . . . . . . . . . . . . . . . . . . . . . . 69 4.5.2 Modeltraining ............................ 70 4.6 ExperimentalResults............................. 71 4.6.1 Partitioned Nordland Dataset . . . . . . . . . . . . . . . . . . . . . 72 4.6.2 Alderley................................ 75 4.6.3 Multi-View Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 76 Table of contents xv 4.6.4 Executiontime ............................ 79 4.7 Conclusions.................................. 81 5 Corner Prediction for Layout Reconstruction 83 5.1 Introduction.................................. 84 5.2 RelatedWork ................................. 85 5.3 CornersforLayout .............................. 87 5.3.1 Network architecture . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.3.2 Training................................ 89 5.3.3 From Corner Maps to 3D Layout . . . . . . . . . . . . . . . . . . . 92 5.4 Equirectangular Convolutions . . . . . . . . . . . . . . . . . . . . . . . . 94 5.4.1 EquiConvsDetails .......................... 95 5.5 Experiments.................................. 98 5.5.1 Datasets................................ 98 5.5.2 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . 98 5.5.3 Network’s output evaluation . . . . . . . . . . . . . . . . . . . . . 99 5.5.4 Robustness analysis . . . . . . . . . . . . . . . . . . . . . . . . . 101 5.5.5 3D Layout comparison . . . . . . . . . . . . . . . . . . . . . . . . 103 5.5.6 Extra Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . 103 5.6 Conclusions.................................. 106 6 Monocular and RGB-D SLAM on Dynamic Environments 111 6.1 Introduction.................................. 111 6.2 RelatedWork ................................. 114 6.3 DynaSLAM System Description . . . . . . . . . . . . . . . . . . . . . . . 115 6.3.1 Segmentation of Potentially Dynamic Content using a CNN . . . . 116 6.3.2 Low-Cost Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . 116 6.3.3 Segmentation of Dynamic Content using Mask R-CNN and MultiviewGeometry............................ 117 6.3.4 Tracking and Mapping . . . . . . . . . . . . . . . . . . . . . . . . 119 6.3.5 Background Inpainting . . . . . . . . . . . . . . . . . . . . . . . . 119 6.4 ExperimentalResults............................. 121 6.4.1 TUMDataset............................. 121 6.4.2 KITTIDataset ............................ 125 6.4.3 TimingAnalysis ........................... 126 6.5 Conclusions.................................. 127 xvi Table of contents 7 Conclusions 129 7.1 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 131 References 133 Appendix A Detailed Experiments for Camera-Aware Convolutions 149 A.1 Experiments on Stanford Dataset . . . . . . . . . . . . . . . . . . . . . . . 149 A.1.1 2D-3D Semantics Stanford Dataset . . . . . . . . . . . . . . . . . 149 A.1.2 Notation ............................... 149 A.1.3 Influence of Context in Single View Depth . . . . . . . . . . . . . 151 A.1.4 Focal Length Overfitting . . . . . . . . . . . . . . . . . . . . . . . 153 A.1.5 Sensor Size Overfitting . . . . . . . . . . . . . . . . . . . . . . . . 153 A.1.6 Generalization with CAM-Convs . . . . . . . . . . . . . . . . . . 158 A.2 NYUExperiment............................... 161 A.2.1 Training................................ 162 A.2.2 Testing ................................ 163 Chapter 1 Introduction Computer vision has experienced a huge growth during the last decades. It can be broadly defined as extracting information from images with similar goals to humans, making computers see as we do. In order to meet this ambitious goal, computer vision addresses a vast number of different perception problems like 3D reconstruction, object detection or object tracking. And it does that by using many different models, strategies, algorithms and techniques, such as epipolar geometry, probabilistic models or machine learning to name a few examples. Computer vision is every time more present in our lives and we have grown to naturalize it, benefiting from its advantages and getting used to its ubiquitous presence in many modern technologies like face recognition in phones, lane detection in cars or camera tracking in virtual and augmented reality (VR/AR) headsets. 1.1 3D Visual Perception and Visual SLAM 3D visual perception consists on recovering information of the 3D structure behind the images taken by cameras. There is a wide range of research problems grouped under the general term perception, that goes from recognizing the objects around the camera to estimating the ego-motion of the camera itself. Sometimes we may dispose of multiple views of a scene taken at the same time (e.g. stereo cameras) or taken sequentially at different times; and sometimes we will perceive our environment from just one single image. Many different applications may benefit from 3D perception. Some include, cars with pedestrian detection as extra safety feature, medical image for precise surgery or people tracking in video vigilance among many others. In the last years, many of these perception solutions have been transferred to commercial products and they are every time more present. However, there are still many points to improve both in resolution, like complete scene 2Introduction understanding, and precision, like object classification or dense depth estimation. Some of these problems have not yet been addressed successfully or still have a poor performance. There are two particular perception problems that humans resolve very well, which are the estimation of our trajectory – tracking our motion – and the 3D perception of our environment and elements around us – mapping the scene. In the robotics community this problem is usually referred to as SLAM (which stands for Simultaneous Localization And Mapping). Another acronym is used by the computer vision community for a similar problem, SfM (which stands for Structure from Motion). The difference lies in that SLAM, differently to SfM, assumes that the data will appear sequentially and aims to complete the computation within the sampling time of the sensor(s) (real time). SLAM is a very relevant problem that is essential for a wide range of applications such as virtual, augmented or mixed reality (Newcombe et al., 2011, Klein and Murray, 2007), Micro Aerial Vehicles (MAVs) navigation (Shen et al., 2011), autonomous driving (McManus et al., 2013) and assisted surgery (Lamarca et al., 2020). Visual SLAM addresses the localization and mapping tasks using only visual sensors, e.g., RGB cameras. Visual SLAM is currently a mature problem, but still a relevant and challenging research area (Cadena et al., 2016). Literature has proposed many different approaches to SLAM, the two main families being the feature-based models – using salient points and matching them across views – (Klein and Murray, 2007, Mur-Artal et al., 2015) and direct approaches – that rely directly on the pixel colors – (Newcombe et al., 2011, Engel et al., 2014). Although there are excellent approaches to the general SLAM problem from visual clues, the problem is not fully solved yet. There are many open challenges that can actually appear very often: surfaces with no texture and therefore difficult to match between different views (Concha and Civera, 2015a), small camera motions that do not create enough parallax to triangulate points, revisiting places after their appearances have changed (e.g. day and night) (Gomez-Ojeda et al., 2015), dynamic elements of the scene (e.g., people walking on the street) that violate the usual rigidity constraints of the algorithm (Alcantarilla et al., 2012) or adding semantic information to certain parts of the scene (e.g. road lanes or room layout) (Fernandez-Labrador et al., 2018b). As the technology for SLAM has become more known and present in applications like VR/AR, its challenges have become more evident. Despite tackling the general problem with great performance, large errors due to cases like day/night or small motions limit the potential of the technology in critical applications, for example autonomous driving. In order to have a reliable perception system, these cases must be addressed as well. Concurrently with the growth in use of SLAM systems, the technology for recording and storing data has 1.2 How Deep Learning is Improving Visual Perception 3 become better and more accessible. As result of this, more data with better ground truth is available. This has allowed the community to create better benchmarks and evaluation tools. The large amount of data available, in this as in many other problems, has made machine learning algorithms gain relevance. A machine learning algorithm is an algorithm that learns from data (Goodfellow et al., 2016). Specifically, it learns to improve its performance (measured by a certain metric) on a task, based on experience. An example of a task can be object classification where previous experience are images that have been labeled (or not) and the measurement of performance might be the classification accuracy. The machine learning technique that has benefited more from having a vast amount of data, as well as more computational power, is deep learning. Deep learning is a family of algorithms that forms part of the machine learning and artificial intelligence field. Deep learning is based on artificial neural networks, being deep feed-forward networks or multilayer perceptron the quintessential model for deep learning (Goodfellow et al., 2016). Deep neural networks (DNNs) are a set of interconnected processing units able to approximate complex functions based on a performance measurement normally referred to as loss. It is in this context of deep learning growing in popularity and being used to tackle perception problems that this thesis takes place. In particular, we have contributed to several challenges in different 3D perception problems: dense monocular depth estimation, place recognition under drastic appearance changes, semantic reconstruction and ego-motion estimation in dynamic environments. All of them are connected to visual SLAM to a greater or lesser extent. 1.2 How Deep Learning is Improving Visual Perception The increase of data and computational power has made DNNs very popular in many different applications. Some of the most popular examples are found in the natural language processing (NLP) domain where we can differentiate several tasks: speech recognition (Amodei et al., 2016, Wang et al., 2019), sentiment analysis (Zhang et al., 2018) or hate speech detection (Gomez et al., 2020) among many others. Deep neural networks have also been widely used in computer vision for many different tasks, using mainly the convolutional layers introduced in LeCun et al. (1989). Convolutional neural networks (CNNs) are a specialized type of neural network for processing data that has a fixed-size grid-like topology. They are only locally connected, sharing the weight on each local connection that is normally referred as kernel and allow the same pattern to be learned in any part of the grid. Other network architectures were later used with a CNN backbone, for example Generative Adversarial Networks (GANs) introduced by Goodfellow et al. (2014). 4Introduction GANs are a proposal for training generative models that consist in two networks competing in a game where the success of one implies the other one’s loss. These models have been used in many applications with different purposes, such as training context encoders as unsupervised pretraining for object classification and semantic segmentation (Pathak et al., 2016). In parallel, Ronneberger et al. (2015) introduced the U-Net architecture, adding extra connections at different levels between encoder and decoder, consequently better propagating high-level high-resolution features. Another relevant architectural design are the Siamese networks, used for tasks such as image retrieval (Gordo et al., 2016). A Siamese neural network consists of two DNNs that share their weights during training, while being fed with two different input images to compute comparable output vectors. When working with data that has a temporal dimension, like language or video (Donahue et al., 2014), recurrent neural networks (RNN) have demonstrated a good performance. Differently from a normal DNN, a RNN defines a directed graph with temporal connections along a sequence of inputs (Hochreiter and Schmidhuber, 1997). These connections allow the network to capture temporal patterns from sequential data, and they have been used in tasks like speech recognition, video captioning and handwriting recognition among others. Visual perception has greatly benefited from the use of deep learning models. Some popular problems that have made use of deep networks are object classification (Simonyan and Zisserman, 2014, He et al., 2016), object detection (Liu et al., 2020), image and video semantic segmentation (Garcia-Garcia et al., 2018), and face recognition (Parkhi et al., 2015). More aligned with the purpose of this thesis, in 3D visual perception many tasks have also been addressed using deep learning, starting from pure monocular depth estimation (Eigen et al., 2014), camera pose estimation (Kendall et al., 2015), object pose tracking (Tompson et al., 2014) and flow estimation (Dosovitskiy et al., 2015). Geometric models began to be added to the network architectures and losses, for self-supervision (Godard et al., 2017) or to be learned (Ummenhofer et al., 2017). In this thesis we have identified several challenges that needed to be addressed inside 3D perception. We have proposed solutions and contributed to different areas, one of them being monocular depth estimation. Along this manuscript we will present works that discuss some of these architectural models mentioned before. For instance, RNN have been used for generating distinctive descriptors for sequence of images to perform visual place recognition. Siamese networks have been used to compare these descriptors for several images and also to train single-view depth estimation with images of different sizes. In the next section we briefly explain each one of these problems and our contributions. 1.2 How Deep Learning is Improving Visual Perception 5 1.3 Our contributions in 3D Visual Perception “I suppose it is tempting, if all you have is a hammer, everything looks like a nail” — Abraham Maslow, The Psychology of Science (1966) We have tackled the following significant challenges on 3D visual perception: dense monocular depth estimation, place recognition under drastic appearance changes, semantic reconstruction and ego-motion estimation in dynamic environments. We contribute in all of these challenges using deep learning. We make use of convolutional networks to work with images and address these challenges by extracting and processing visual clues present in the input images. We have also contributed to make deep learning and specifically CNNs more adaptable to different cameras or image representations by introducing CAM-Convs and EquiConvs, two types of convolutions that will be further detailed in Chapters 2 and 5. In the following sections we introduce the research problems, the related literature and our contributions for all the different 3D visual perception challenges we have addressed. 1.3.1 Visual Mapping without Motion Visual SLAM is mainly based on matching points across different views and jointly estimate the relative movement of the camera and the triangulation of these points (Triggs et al., 1999). However, that procedure’s accuracy depends on the parallax angle generated by the camera motion. When the camera does not move, the trajectory does not need to be calculated but the estimation of the 3D of the scene becomes challenging as it is an ill-posed problem. This problem is referred to in the computer vision community as single-view depth estimation or monocular depth estimation. Depth perception from one single image has usually been addressed using machine learning algorithms (Saxena et al., 2009). One of the most impactful advances in single-view depth estimation was the use of deep learning by Eigen et al. (2014), that trained in a supervised manner a deep neural network to predict depth from single RGB images. Their proposal is a convolutional neural network (CNN) trained with RGB-D examples to predict a depth value for each RGB pixel. Following works farther improved predictions by using deeper models and networks pretrained in other tasks, mainly classification (Eigen and Fergus, 2015, Laina et al., 2016). Different training procedures allowed to train with different datasets despite not being supervised – meaning that they do not have depth values for all the pixels. Godard et al. (2017) proposed a self-supervised scheme by forcing left-right stereo camera consistency 12 Introduction if we use them to estimate our pose. As humans, we experiment a similar effect when we sit inside a train and through the window we see another train in the station. Sometimes one of the two trains starts to move and we do not really understand which one of the two is moving. Sometimes we wrongly perceive as if we were moving because we use a moving image to locate ourselves. For visual tracking we locate the camera with respect to the parts of the scene we are viewing, but if we are not careful of using only parts of the scene that are static as an anchor to estimate the pose of the device, the recovered pose can be erroneous. Riazuelo et al. (2017) proposed a RGB-D SLAM system with an embedded real-time human tacker module. Li and Lee (2017) also proposed a similar RGB-D SLAM system, but differently to previous approaches, they use depth edge points as an indicator of the probability that parts of the image belong to a dynamic object. We propose, in Chapter 6, a combination of traditional geometry and deep learning techniques to detect and consider dynamic objects during tracking and mapping for monocular, stereo and RGB-D SLAM. This system combines the use of a semantic segmentation CNN with a classical feature-based SLAM system. More recent works have advanced more in the topic of SLAM with dynamic objects. Yang and Scherer (2019) build an RGB-D SLAM system where by retrieving 3D bounding boxes of objects they jointly optimize the poses of the camera, objects and points among different views. Xu et al. (2019) are able to maintain a map tracking all the instances of objects by building an object-level octree-based volumetric representation, providing a robust RGB-D camera tracking as well. 1.4 List of Publications The work developed during this PhD thesis has produced the following publications. • Facil, J. M., Concha, A., Montesano, L., & Civera, J. (2017). Single-view and Multiview Depth fusion. IEEE Robotics and Automation Letters, 2(4), (pp. 1994-2001). With Oral Presentation at International Conference on Intelligent Robots and Systems (IROS) 2017. • Facil, J. M., Ummenhofer, B., Zhou, H., Montesano, L., Brox, T., & Civera, J. (2019). CAM-Convs: CAM-Convs: Camera-Aware Multi-scale Convolutions for SingleView Depth. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019 (pp. 11826-11835). • Olid, D., Fácil, J. M., & Civera, J. (2018). Single-view place recognition under seasonal changes. Planning, Perception and Navigation for Intelligent Vehicles Workshop at International Conference on Intelligent Robots and Systems (IROS) 2018. 1.4 List of Publications 13 Camera Trajectory Dynamic Elements Fig. 1.6 This image shows an example of how dynamic objects are detected over time and taken into account when tracking the camera position. At the same time, the map could be built, not considering the dynamic objects because they should not be part of it. • Facil, J. M., Olid, D., Montesano, L., & Civera, J. (2019). Condition-Invariant MultiView Place Recognition. Technical Report 2019 – arXiv preprint arXiv:1902.09516. • Fernandez-Labrador, C., Facil, J. M., Perez-Yus, A., Demonceaux, C., & Guerrero, J. J. (2018). PanoRoom: From the Sphere to the 3D layout . 3D Meets Semantics at European Conference in Computer Vision (ECCV) 2018. • Fernandez-Labrador, C.*, Facil, J. M.*, Perez-Yus, A., Demonceaux, C., Civera, J., & Guerrero, J. J. (2020). ThreeSixty End-to-End Layout Recovery . Woman in Computer Vision at IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2019. * - Equal Contribution • Fernandez-Labrador, C.*, Facil, J. M.*, Perez-Yus, A., Demonceaux, C., Civera, J., & Guerrero, J. J. (2020). Corners for layout: End-to-end layout recovery from 360 images . IEEE Robotics and Automation Letters, 5(2), (pp. 1255-1262). * - Equal Contribution 14 Introduction • Bescos, B., Facil J.M., Civera, J. & Neira, J., Detecting, Tracking and Eliminating Dynamic Objects in 3D Mapping using Deep Learning and Inpainting , Oral and Poster Presentation within the Workshop at ICRA 2018: Representing a Complex World: Perception, Inference, and Learning for Joint Semantic, Geometric, and Physical Understanding • Bescos, B., Facil J.M., Civera, J. & Neira, J., Robust and Accurate 3D Mapping by combining Geometry and Machine Learning to deal with Dynamic Objects , Poster Presentation within the Workshop at IROS 2017: Learning for Localization and Mapping • Bescos, B., Facil J.M., Civera, J. & Neira, J., DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes , IEEE Robotics and Automation Letters 3 (4), (pp. 4076 - 4083). With Oral Spotlight and Poster at International Conference on Intelligent Robots and Systems (IROS) 2018. 1.5 Code Released During the realization of this thesis we have released the following code repositories. • Source code for CAM-Convs: Camera-Aware Multi-Scale Convolutions for SingleView Depth in TensorFlow 1.4 and 2.0. https://github.com/jmfacil/ camconvs • Source code for Single-View Place Recognition in Caffe. https://github.com/ jmfacil/single-view-place-recognition • Source code of Corners-for-Layout and EquiConvs (EquiRectangular Convolutions) for TensorFlow 1.4. https://github.com/cfernandezlab/CFL • Source code of DynaSLAM, build upon ORB-SLAM2, Tracking, Mapping and Inpainting in Dynamic Scenes for Monocular, Stereo and RGB-D Cameras. https: //github.com/BertaBescos/DynaSLAM 1.6 Manuscript Organization The structure of this Ph.D. thesis is as follows. In the Chapter 2 we present CAM-Convs for monocular single-image depth estimation with different cameras. In Chapter 3 we propose a 1.6 Manuscript Organization 15 fusion between single and multi-view depth to improve mapping with very little motion and texture for Visual SLAM. Chapter 4 presents several deep neural network architectures for robust visual place recognition considering appearance changes. In Chapter 5 we introduce Corners for Layout (CFL), an end-to-end network to extract corners of indoor rooms, and the Equirectangular Convolutions (EquiConvs), a type of convolution that allows CNNs to adapt to equirectangular distortions. Finally, Chapter 6 presents DynaSLAM, a monocular, stereo and RGB-D SLAM system that is aware of dynamic objects on the scene and ignores them when tracking the camera pose and mapping the scene. Chapter 2 Camera-Aware Multi-Scale Convolutions for Single-View Depth As we have mentioned in the introduction chapter, structure without motion is an ill-posed problem that is not solvable in general with traditional geometric techniques. For that reason many learning approaches have been presented over the last years trying to tackle this problem. Deep learning, as in many other problems, have proven to show the best results. However, single-view depth estimation suffers from the problem that a network trained on images from one camera does not generalize to images taken with a different camera model. Thus, changing the camera model requires collecting an entirely new training dataset. In this chapter, we propose a new type of convolution that can take the camera parameters into account, thus allowing neural networks to learn calibration-aware patterns. Our experiments confirm that this improves the generalization capabilities of depth prediction networks considerably, and clearly outperforms the state of the art when the train and test images are acquired with different cameras. 2.1 Introduction Recovering 3D information from 2D images is one of the fundamental problems in computer vision that, due to recent advances and applications, is receiving nowadays a renewed attention. Among others, there has been recent relevant results on problems such as 6D object pose detection (Kehl et al., 2017, Shrivastava et al., 2017, Sundermeyer et al., 2018), 3D model reconstruction Fan et al. (2017), Tatarchenko et al. (2017), depth estimation from single (Laina et al., 2016, Fu et al., 2018, Li et al., 2018) and multiple views (Ummenhofer et al., 2017, Huang et al., 2018), 6D camera pose recovery (Kendall et al., 2015, 2017) or camera tracking and mapping (Zhou et al., 2018, Bloesch et al., 2018, Tang and Tan, 2018, 18 Camera-Aware Multi-Scale Convolutions for Single-View Depth * CAM-Conv Encoder Decoder Camera Model Fig. 2.1 CAM-Convs allows efficient specialization of a camera-generic network for various camera models by feeding camera-specific parameters into the network. Tateno et al., 2017). While traditional multi-view methods (Schonberger and Frahm, 2016) are mostly based on geometry and optimization and, thus, are largely independent of the data, these recent deep learning approaches depend on training data that demonstrates the mapping from images to depth. The common strategy to collect such data is by using an RGBD sensor, like the Kinect camera, which conveniently provides both the RGB image and what can be considered ground truth depth. It is implicitly assumed that training on this type of data will generalize to other RGB sensors that do not provide depth. However, the evaluation of recent learningbased methods relies largely on public benchmarks where images have been recorded with the same RGBD camera as the training data. Thus, evaluation on these benchmarks does not reveal whether a depth estimation method generalizes to RGB images from another camera. Overfitting to a benchmark is a common problem in computer vision research. Torralba and Efros (2011) have shown that datasets may have strong biases that make researchers over-confident regarding the performance of their method. In particular, train-test divisions of the same kind of data are not enough to prove generalization. In this work we show that, indeed, state-of-the-art single-view depth prediction networks do not generalize when the camera parameters of the test images are different from the training ones. Moreover, we show that for single-view depth prediction the problem of missing generalization to images from different cameras is even more severe: it cannot be solved by training on images from a diverse set of cameras with different parameters. For present methods to adapt to a different camera model, they require changes in the architecture. We present a deep neural network for single-view depth prediction that, for the first time, addresses the variability on the camera’s internal parameters. We show that this allows to 2.2 Related Work 19 use images from different cameras at train and test time without a performance degradation. This is of particular interest, as it enables the exploitation of images from any camera for training the data-hungry deep networks. Specifically, within our proposed network, our main contribution is a novel type of convolution, that we name as CAM-Convs (CameraAware Multi-scale Convolutions), that concatenates the camera internal parameters to the feature maps, and hence allows the network to learn the dependence of the depth from these parameters. Figure 2.1 shows an illustration of how CAM-Convs act in the typical encoderdecoder depth estimation pipeline. The network can be trained with a mixture of images from different cameras without overfitting to specific intrinsics. We show that the network generalizes also to images from cameras it has not been trained on. A comparison with the state of the art in single-image depth estimation demonstrates that the better generalization properties do not reduce the accuracy of the depth estimates. 2.2 Related Work Estimating 3D structure and 6 degrees-of-freedom motion using deep learning has been addressed recently from several angles: Supervised (Laina et al., 2016) and unsupervised (Zhou et al., 2017), from single (Eigen and Fergus, 2015) and multiple views (Tang and Tan, 2018), using end-to-end networks (Laina et al., 2016) or fusing with multi-view geometry (Fácil et al., 2017), completing depth maps (Zhang et al., 2018, Weerasekera et al., 2018), and estimating geolocation (Weyand et al., 2016, Kendall et al., 2015), relative motion (Ummenhofer et al., 2017), visual odometry (Wang et al., 2017, 2018), and simultaneous localization and mapping (SLAM) (Tateno et al., 2017, Bloesch et al., 2018, Zhou et al., 2018). In this work we deal with single-view supervised depth learning, so we will focus our literature review in this case. Among the pioneering work we can reference Hoiem et al. (2005), that similarly to pop-up illustrations, cut and fold a 2D image based on a segmentation into geometric classes and some geometric assumptions. Saxena et al. (2009) is another seminal work that, with minimal assumptions on the scene, learned a model based on a MRF. Eigen et al. (2014) was the first paper that used deep learning for single-view depth prediction, proposing a multi-scale depth network. Its results were improved later by Eigen and Fergus (2015), Liu et al. (2015b), Laina et al. (2016), Chakrabarti et al. (2016) and He et al. (2018). Many methods focus on specific datasets which enable to train learning-based methods for specific tasks. For instance, Eigen and Fergus (2015) extend the multi-scale architecture in Eigen et al. (2014) to the prediction of surface normals and semantic labels on the NYU 20 Camera-Aware Multi-Scale Convolutions for Single-View Depth dataset (Silberman et al., 2012). Similarly, Wang et al. (2015) train a network that jointly predicts depth and segmentation on the same dataset. For depth, Laina et al. (2016), Liu et al. (2015b) and Eigen and Fergus (2015) show that their methods can be adapted to other datasets like Make3D (Saxena et al., 2009) or KITTI (Geiger et al., 2012). However, they treat datasets like different tasks and require retraining for each dataset to achieve state-of-the-art performance. Chen et al. (2016), inspired by Zoran et al. (2015), introduce the Depth in the Wild dataset and train a CNN using ordinal relations between point pairs. While the images stem from internet photo collections taken with many different cameras, they do not make use of the camera parameters during training. Li and Snavely (2018) use a structure from motion pipeline to extract depth from internet photo collections and use this to train a CNN predicting depth up to a scale factor. Again, information about camera parameters is not exploited and generalization is solely driven by large diverse datasets. Extrinsic parameters have been considered for other tasks such as stereo estimates (Ummenhofer et al., 2017) or synthesis of view point changes (Zhou et al., 2016). Intrinsic parameters are usually left out in deep learning pipelines, with the exception of He et al. (2018). They embed focal length information in a fully-connected approach, making it impossible to train and test in different image sizes, while our proposal is flexible and can deal with different image sizes. Posterior to the publication of this work, López-Antequera et al. (2020) have proposed a canonical camera model, they show that converting every image to this model it is a simpler method to train than the one we propose, despite of having some drawbacks like forcing the images to have same size and optical center can force to drop parts of them. In the next section we describe how to explicitly implement the internal camera parameters into the network and thereby improve generalization by CAM-Convs. 2.3 Camera-Aware Multi-scale Convolutions CAM-Convs (standing for Camera-Aware Multi-scale Convolutions), is the variant of the convolution operation that we present in this thesis. CAM-Convs include the camera intrinsics in the convolutions, allowing the network to learn and predict depth patterns that depend on the camera calibration. Specifically, we add CAM-Convs in the mapping from RGB features to 3D information–e.g. depth, normals–, that is, between the encoder and the decoder. As shown in Figure 2.2, we add them at every level, such that we include CAM-Convs on every skip-connection too. Notice that all the CAM-Convs are added after the encoder, allowing the use of pretrained models. 2.3 Camera-Aware Multi-scale Convolutions 21 * * * * * *CAM-Conv Fig. 2.2 Adding CAM-Convs to an Encoder-Decoder U-Net architecture. The basics of CAM-Convs are as follows: We pre-compute pixel-wise coordinates and field-of-view maps and feed them along with the input features to the convolution operation. CAM-Convs use the idea behind Coord-Convs presented in Liu et al. (2018), on adding normalized coordinates per pixel, but incorporating information on the camera calibration. An illustrative scheme of how CAM-Convs extra channels work is shown in Figure 2.3. The different maps included are computed using the camera intrinsic parameters (focal length f and principal point coordinates (cx,cy)) and the sensor size (width wand height h): Centered Coordinates (cc): To add the information of the principal point location to the convolutions, we include ccx and ccy coordinate channels centered at the principal point–i.e. the principal point has coordinates (0,0). Specifically, the channels are ccx=     0−cx 1−cx . . . w−cx     w×1 ·     1 1 . . . 1      ⊺ h×1 =   −cx··· w−cx . . ..... . . −cx··· w−cx   (2.1) ccy=     1 1 . . . 1     w×1 ·     0−cy 1−cy . . . h−cy      ⊺ h×1 =   −cy··· −cy . . ..... . . h−cy··· h−cy   .(2.2) We resize these maps to the input feature size using bilinear interpolation and concatenate them as new input channels. These channels are sensitive to the sensor size and resolution (pixel size) of the camera, as their values depend on it. We assume the sensor size is measured in pixels. In Figure 2.3 we represent cc with a color gradient from red (for negative coordinates) to blue (for positive coordinates), white for 0. Notice in the figure how cc values change when camera sensor size, principal point or pixel size change. 28 Camera-Aware Multi-Scale Convolutions for Single-View Depth Normal Loss: For the normal loss, we use the L2 norm. The ground truth for the normals (ˆ n) is derived from the ground truth depth image. The loss for the normals is as follows: Ln=∑ i,jn(i,j)−ˆ n(i,j)|2.(2.10) Total Loss: The individual losses are weighted by factors obtained empirically, so the total loss Lis L=λ1Ld+λ2Lg+λ3Lc+λ4Ln,(2.11) where λ1,λ2,λ3and λ4are 150, 100, 50 and 25 respectively. 2.4.3 Training Schedule We train all our networks using the TensorFlow framework (Abadi et al., 2016). We start from ResNet-50 (pre-trained) and a randomly initialized decoder. For optimization we use Adam Optimizer (Kingma and Ba, 2014) with a momentum of 0.9 . The complete training of the network is composed by three different stages, each adding more layers and predictions to the decoder (SR,MR and HR respectively in Table 2.1). 1st stage. We train until the first two resolutions of the decoder (LR-1 in Table 2.1). This stage is the shortest one, trained only for 10k iterations with batch size 16 . We only train the encoder layers and the decoder until the smallest prediction. We do not apply the scale-invariant loss for this resolution. 2nd stage. We train until the next two predictions (LR-1, MR-1 and MR-2 in Table 2.1). As in the previous stage we train only the layers that affect the outputs. This stage is trained for 50k iterations with batch size 16 . We apply a scale-invariant loss to prediction MR-2 after 25kiterations. 3rd stage. We train the whole network, for 200k iterations with batch size 16 . We apply a scale-invariant loss to predictions MR-2, HR-1 and HR-2 after 25kiterations. Learning rate. The learning rate policy for the three stages is shown in Figure 2.5. The base learning rates for the three stages are 1e−3 , 5e−4 and 1e−4 . The learning rate drops along the iterations, never being less than a minimum of 1e−6 . As we train multiple models we use a fixed automatic learning rate decay. Weighting losses. For all stages we minimize the losses for several resolutions. We scale the losses according to the resolution level. Specifically, we multiply the losses by a factor 1 k , where k denotes the resolution level. Starting with the finest resolution of the active stage. I.e in the 1 st stage LR-1 prediction loss would be multiplied by 1. While in the 3 rd stage LR-1 2.5 Multi-Camera Experiments and Results 29 10000 60000 260000 Ite ra tion s 10 6 10 5 10 4 10 3 Le a rning Ra te 1st stage 2nd stage 3rd stage Fig. 2.5 Learning rate policy for the three-stages training. would be multiplied by 1 5 , MR-1 would be multiplied by 1 4 and so on until HR-2 that would be multiplied by 1. 2.5 Multi-Camera Experiments and Results Most of the single-view depth prediction networks have been trained and tested using the same or very similar camera models. Generalizing to different camera models has several implications that are not straightforward. For this reason, we first present a thorough analysis on the generalization capabilities of current approaches. To this end we apply naïve generalization techniques (focal normalization and image resizing) during training on a network without our special convolutions (as Figure 2.4 but without CAM-Convs) and examine the limitations. Finally, we train and evaluate our network with CAM-Convs (as Figure 2.4) and show the improved generalization performance with respect to different camera parameters. 2.5.1 Experimental Setup The major part of our experiments are done on the 2D-3D Semantics Dataset (Armeni et al., 2017), that contains RGB-D equi-rectangular images. This dataset allows us to generate images with different camera intrinsics but the same content. We have observed that depth estimation networks overfit to the camera parameters and the image content distribution (the latter being different in indoors and outdoors datasets, for example). In this manner we eliminate the content distribution factor and isolate the effect of the camera parameters. All the experiments were done using the 3-fold cross-validation suggested by Armeni et al. (2017). In this section we present median values for the most relevant experiments. To see the complete results, more details on the dataset and image generation process and additional experiments we refer the reader to the Appendix A. 30 Camera-Aware Multi-Scale Convolutions for Single-View Depth Name s1s2s3 Sensor 256×192 192×256 224×224 Name s4s5sSsK Sensor 128×96 320×320 256×192 384×128 Name f72 f128 f64 fn Focal 72 128 64 100 Table 2.3 Notation for different sensor sizes and focal lengths. The notation for sensor sizes and focal lengths used during the evaluation is in Table 2.3. As an example, if a network has been trained with sensor sizes 192×256 and 224×224 , and focal length 72, we will denote this model as s2s3f72 . In some experiments we use a random distribution for the focal length. As an example, if the synthesized focal lengths are uniformly distributed between 72 and 128, the model will be denoted as Uf72 f128. We evaluate the performance on both depth and inverse depth. All the error metrics we used in our experiments are standard from the literature. In addition we use relative metrics and the scale-invariant metric presented by Eigen et al. (2014), which are widely used in depth estimation. 2.5.2 Influence of context Modifying the camera parameters affects the field of view, and hence the amount of context the image is capturing. We evaluate the influence of the context in the depth prediction of a standard U-Net encoder-decoder architecture (network in Figure 2.4 without CAM-Convs) with two different experiments. First, we compare two networks trained with images with sensor size s1 and two different focal lengths f128 and f64 (Table 2.4). Second, we compare two networks with images with the same focal length but different sensor sizes: s1 and s4 (Table 2.5). As expected, context helps. The performance is better for the smallest focal f64 , which results in a wider FOV and hence more context. Also the performance is better for the bigger sensor size s1 , which also provides more context. To remove the context dependency in our analysis, for some of the experiments in next subsections we will generate images with uniformly distributed focal lengths. 2.5.3 Overfitting of standard networks In this experiment we evaluate the performance of a standard U-Net architecture for variations of the camera parameters on the training and test sets. We will focus the study on two 2.5 Multi-Camera Experiments and Results 31 Test Train abs.rel rmse sc.inv sq.rel : 1 m lg(m): 1 s1f64 s1f64 0.17 0.378 0.0347 0.048 s1f128 s1f128 0.195 0.51 0.0387 0.0606 smaller is better Table 2.4 Influence of context, different focal lengths. Test Train abs.rel rmse sc.inv sq.rel : 1 m lg(m): 1 s1f64 s1f64 0.17 0.378 0.0347 0.048 s4f64 s4f64 0.204 0.54 0.0384 0.0637 smaller is better Table 2.5 Influence of context, different sensor sizes. parameters: (a) focal length and (b) sensor size. First we will fix the sensor size to s1 and we will test on images with focal lengths f64 , f72 and f128 (first three test sets in Table 2.6). Second we will sample random focal lengths from a uniform distribution between f72 and f128 and we will evaluate on images with sensor sizes s1 and s2 (last two test sets in Table 2.6). For every test set there are 4 to 5 different train sets (referred in the 2 nd column of the table). For every test set we will refer to the case where the cameras from the training and test set are the same as the same-camera baseline. Training sets where we did not use focal length normalization are denoted with a ’*’. Networks trained on train sets with two sensor sizes have been trained either as Siamese networks with weight sharing or with image resizing to size s1(denoted with a ’†’). It is important to remark that, for all the experiments, the test and training data was generated from the exact same images and the networks have the same architecture and were trained for the same number of iterations. Any performance variation, then, should be attributed to the variations in the camera intrinsics and the naïve solutions we analyze. Notice in Table 2.6 that, in general, the same-camera baseline outperforms the rest, demonstrating the overfit to the camera parameters. The conclusions of these experiments are as follows. (a) Single-focal training overfits. The performance of a depth network degrades when trained on images from a particular camera and tested on images from different cameras. See, for example, the drop in performance between the 1 st row (test: s1f64 , train: s1f64∗ ) and the 2nd (test: s1f64, train: s1f72) and 3rd (test: s1f64, train: s1f128) rows in all metrics. Multi-focal training with normalization helps. The results improve when the training set contains images with different focal lengths and is done with focal normalization. See, for example, that the results on test set s1f64 with training set s1f72 f128 is close to the same-camera baseline. Notice, however, that the multi-focal train set does not reach the 32 Camera-Aware Multi-Scale Convolutions for Single-View Depth Test set Train set l1.inv rmse sc.inv pixels pixels 1/m m lg(m) s1f64 s1f64*0.184 0.378 0.0347 s1f72 0.193 0.395 0.0354 s1f128 0.318 0.572 0.0483 s1f72 f128*0.659 0.864 0.0614 s1f72 f128 0.189 0.387 0.0361 s1f72 s1f72*0.17 0.4 0.0354 s1f128 0.272 0.564 0.0459 s1f72 f128*0.552 0.888 0.0609 s1f72 f128 0.175 0.404 0.0364 s1f128 s1f128*0.141 0.51 0.0387 s1f72 0.133 0.524 0.0411 s1f72 f128*0.208 0.813 0.063 s1f72 f128 0.132 0.504 0.038 s1Uf72 f128 s1Uf72 f128 0.15 0.46 0.037 s2Uf72 f128 0.175 0.51 0.0422 s1s2Uf72 f128 0.153 0.484 0.0401 s1s2s3Uf72 f128†0.179 0.742 0.064 s2Uf72 f128 s1Uf72 f128 0.151 0.44 0.038 s2Uf72 f128 0.133 0.412 0.0323 s1s2Uf72 f128 0.139 0.436 0.0352 s1s2s3Uf72 f128†0.16 0.622 0.0514 smaller is better *trained without focal length normalization. †images resized to s1during training. Table 2.6 Overfit to camera parameters of standard encoder-decoder architectures. Networks trained from images with variations in their intrinsics perform worse than the same-camera baseline. performance of the same-camera baseline. In section 2.5.4 we will show how CAM-Convs are able to outperform the same-camera baseline even when the training data does not contain the test focal length. The performance degrades without focal normalization. Compare, for example, the error metrics of the train sets s1f72 f128∗ and s1f72 f128 . Networks trained on f72 f128∗ , in fact, did not converge easily. Limitations of focal normalization. Two things should be noticed regarding focal normalization: First, it does not model the changes on the sensor size and the resolution, and we will see now how changes on them degrade the performance. And second, Equation 2.4 only holds if the pixel size is the same for every camera in the training and test sets, which in general is not the case. 2.5 Multi-Camera Experiments and Results 33 Test Train abs.rel rmse.inv sc.inv sq.rel % 1/km lg(m)100 % sK sK9.16 10.54 13.3 2.33 sSsK24.58 36.82 26.51 9.28 sSsK†9.08 10.55 13.98 2.56 abs.rel l1.inv rmse.inv sq.rel : 1 1/m1/m: 1 sS sS0.12 0.09 0.12 0.03 sSsK0.26 0.13 0.16 0.18 sSsK†0.12 0.09 0.12 0.03 smaller is better † sensor size has been resized to the first one in the list. Table 2.7 Naïve train and test on KITTI Uhrig et al. (2017) and ScanNet Dai et al. (2017a). See that training FCN in multiple image sizes ( sKsS ) does not generalize. Resizing works, but only in this particular case, because of the small overlap of visual features. (b) Single-sensor size training overfits. Networks trained on a sensor size and tested on other sensor sizes do not perform as well as the same-camera baseline. This can be seen in Table 2.6 in the last two test sets s1Uf72 f128 and s2Uf72 f128 . Single-view depth estimation is a context-dependent task, and the network overfits to the amount of context in the training sensor size. Multi-sensor size training with weight sharing does not generalize. Training with multiple sensor sizes works better than training with the wrong sensor size but cannot reach the same performance as same-camera baselines. Further, training a stack of weight sharing networks also does not scale to large numbers of different sensor sizes. Resizing does not work. As a naïve approach, which scales to multiple sensor sizes, we use resizing (denoted with ’ † ’ in Table 2.6), which converts all the images to size ( s1 ) during training. Notice that resizing changes the aspect ratio. It also implies the recalculation of a new average focal length fr=frx+ry 2 for normalization. The performance degradation introduced by resizing is noticeable. Resizing creates inconsistent data in train and testing, which leads to learning and convergence difficulties. Resizing helps only in a particular case (non-overlapping distributions of visual features). Table 2.7 shows an experiment, similar to the previous one, on two public datasets: KITTI (Uhrig et al., 2017), with sensor size sK , and ScanNet (Dai et al., 2017a), with sensor size sS . In this case, training with both sensor sizes (by weight sharing) decreased the performance. However, resizing reduced the error to the level of the same-camera baselines. The reason for this is the completely different distribution of the two datasets, with null intersection of 34 Camera-Aware Multi-Scale Convolutions for Single-View Depth Test Train abs.rel l1.inv rmse sc.inv : 1 1/m m lg(m) s1Uf72 f128 s1Uf72 f128 0.189 0.15 0.46 0.037 CAM-C‡0.175 0.144 0.433 0.0312 s2Uf72 f128 s2Uf72 f128 0.166 0.133 0.412 0.0323 CAM-C‡0.158 0.131 0.39 0.0265 s3Uf72 f128 s3Uf72 f128 0.174 0.14 0.425 0.0336 s1Uf72 f128 0.184 0.143 0.44 0.0357 s2Uf72 f128 0.177 0.145 0.435 0.0356 s1s2Uf72 f128 0.178 0.143 0.451 0.0365 CAM-C‡0.164 0.134 0.402 0.0283 s5f64 s5f64 0.163 0.227 0.309 0.0356 s1f64 0.245 0.292 0.337 0.0598 s1s2Uf72 f128 0.369 0.369 0.44 0.0427 CAM-C‡0.177 0.236 0.289 0.0362 smaller is better ‡ Trained with weight sharing in sensor sizes s1 , s2 and Uf72 f128. Table 2.8 Camera parameter generalization with EquiConvs. Results on training and testing on different cameras. 1st column: camera parameters for test set. 2nd column: camera parameters seen during training. This is a continuation of Table 2.6. Notice how the network with EquiConvs is the only model that generalize getting better performance than the same-camera baseline on most test sets. visual features (e.g. there are no chairs on KITTI and no cars on ScanNet). This is, however, a very particular case, resizing degrades significantly the accuracy in general. 2.5.4 Robust Generalization with CAM-Convs In this experiment we show that CAM-Convs generalize to different camera models. In order to evaluate the influence of CAM-Convs we trained our model with two different sensor sizes ( s1 and s2 ) and weight sharing. Focal length during training is sampled randomly from a uniform distribution Uf72 f128 . We evaluated the trained model in four different test sets, see Table 2.8. The first two include the camera model the network was trained with, the third has a sensor size unseen during training, and the last ( s5f64 ) was generated from a camera completely different from the training ones with bigger sensor size and smaller focal length. This case augments considerably the context–e.g field of view–which proved to be the hardest case in previous experiments (see network trained with s1f128 in Table 2.6). CAM-Convs generalize over camera intrinsics, outperforming the same-camera baseline. Results on the test sets s1Uf72 f128 and s2Uf72 f128 in Table 2.8 show that the network 2.5 Multi-Camera Experiments and Results 35 CAM-CONVS NO CAM-CONVS Ground-Truth Input Fig. 2.6 Qualitative results for the test set s5f64 . 1st column : RGB input. 2nd column: Ground truth depth. 3rd column: Prediction with our network using CAM-Convs trained on s1s2Uf72 f128 . 4th column: Prediction of a network trained without CAM-Convs. Notice that the test camera parameters are significantly different from the training set and images have a much wider field of view. Despite the large difference in the camera parameters the network with CAM-Convs produces sharp depth maps on which room corners are clearly visible. with CAM-Convs trained on images of two sizes clearly outperforms the baselines, which was trained on the exact test size. The addition of CAM-Convs allowed the network to learn the dependence of the image features from the calibration parameters. CAM-Convs generalize to sensor sizes unseen during training. Remarkably, the network with CAM-Convs also outperforms the same-camera baseline on the test set with sensor size s3 (third test set in Table 2.8), which is not included in the training data. Further, it generalizes better than a network trained on the exact same conditions but without CAM-Convs (see s1s2Uf72 f128 in the table). CAM-Convs generalize to cameras unseen during training. With the last test set ( s5f64 ) in Table 2.8 we evaluate our network on an extreme case of camera parameters with a very wide field of view and very different sensor size from the training ones. Table 2.8 shows that CAM-Convs improve considerably the generalization to new unseen cameras over the naïve approaches. Figure 2.6 shows a qualitative comparison between our network with CAM-Convs and the network without CAM-Convs (s1s2Uf72 f128) in the test set s5f64. 36 Camera-Aware Multi-Scale Convolutions for Single-View Depth 0.3 0.25 0.2 abs relative 0.2 0.15 0.1 l1_inverse 0.9 0.8 0.7 0.6 rmse Laina ours 0.4 0.3 1.3 Fig. 2.7 Error distribution on the test set of NYUv2 with 6 different camera parameters. In orange, our network with CAM-Convs, trained on several datasets not including NYUv2. In blue, Laina et al. (2016), trained on NYUv2. 2.5.5 Experiments on Multiple Datasets In our last experiment we demonstrate how CAM-Convs can generalize across datasests by training on four datasets with different cameras (KITTI (Uhrig et al., 2017), ScanNet(Dai et al., 2017a), MegaDepth (Li and Snavely, 2018) and Sun3D(Xiao et al., 2013) and testing on a different one (NYUv2 (Silberman et al., 2012)). Training: We trained our network for three different sensor sizes ( 320×320 , 256×256 and 224×224 ) using weight sharing. We augmented the training data by scaling the images and shifting the principal point to increase the variation of the camera parameters and then crop to image to one of the target sensor sizes. We did not use focal length normalization in this experiment, as we cannot ensure constant pixel size across datasets. As MegaDepth has only up-to-scale ground truth, we applied only scale-invariant losses and added the scaleinvariant cost function of Eigen et al. (2014). The same network without CAM-Convs, and hence with no camera information, did not converge during training. The lack of calibration information creates inconsistencies (e.g. same-size objects may have different depths due to different focal lengths). Testing: We evaluated our network on the official test set of NYUv2 and compared against the state of the art (Laina et al., 2016) (similar network without CAM-Convs) . Note that the network of Laina et al. (2016) was trained exclusively on NYUv2, while our network was trained on a set of datasets excluding NYUv2 with different cameras and data distributions (some of the datasets are outdoors, see Figure 2.9 and Figure 2.10). This is important since our model cannot benefit from the dataset bias (Torralba and Efros, 2011). We predicted depths for images from 6 different cameras: the original camera of the NYUv2 dataset and 5 simulated ones by cropping (to shift principal point and reduce sensor size) and resizing (to change focal length). 2.5 Multi-Camera Experiments and Results 37 128x320 640x480 128x320 256x256 128x320 320x224 640x480 Input GT Ours Laina Fig. 2.8 Qualitative results, NYUv2 test set with intrinsics variations. 1st column : Input RGB images. Each row shows the original one and scaled and cropped versions 2nd column : Depth groundtruth. 3rd column : Prediction from our network with CAM-Convs, trained on several datasets NOT including NYUv2. Our network produces consistent depth close to the ground truth for all images. 4th column : Laina et al. (2016), trained exclusively on NYUv2. Its errors are low on the training resolution but does not generalize to new intrinsics. Figure 2.7 shows the distribution of the mean error of the usual metrics obtained for the 6 different cameras. Since Laina et al. (2016) was trained on the NYUv2 dataset, it works slightly better when it predicts the images from the camera it was trained on (the point with the smallest error). However, performance degrades when the camera changes and CAM-Convs have always smaller error and variance. Figure 2.8 illustrate how CAM-Convs depth predictions are stable for different cameras, while predictions of Laina et al. (2016) vary significantly. Recall that CAM-Convs were not trained on NYUv2, which indicates that they are able to generalize over different camera models and outperform Laina et al. (2016) although they trained on the same dataset. Figures 2.8, 2.9 and 2.10 show depth predictions for images (and cropped/resized versions) from the NYUv2, KITTI and MegaDepth test sets. Again, note the excellent performance across datasets with different data distributions and camera intrinsics. All predictions 44 Combining Single-View Deep Learning Depth with Multi-View Depth and the regularization of the multi-view estimation by adding the total variation (TV) norm to the cost function. TV regularization has low accuracy for large textureless areas, as shown by Concha et al. (2014), Pinies et al. (2015), Piniés et al. (2015) among others. In order to overcome this Concha et al. (2014) propose a piecewise-planar regularization; the plane parameters coming from multi-view superpixel triangulation (Concha and Civera, 2014) or layout estimation (Hedau et al., 2009b). Pinies et al. (2015) propose higher-order regularization terms that enforce piecewise affine constraints even in separated pixels. Piniés et al. (2015) selects the best regularization function among a set using sparse laser data. Building upon Concha et al. (2014), Concha et al. (2015) adds the sparse data-driven 3D primitives of Fouhey et al. (2013) as a regularization prior. Compared to these works, our fusion is the first one where the information added to the multi-view depth is fully dense, data-driven and single-view; and hence it does not rely on additional sensors, parallax or Manhattan and piecewise-planar assumptions. It only relies on the network capabilities for the current domain, assuming that the test data follows the same distribution that the data used for training. Due to the difficulty of estimating an accurate and fully dense map from monocular views there are several approaches that estimate only the depth for the highest-gradient pixels (Engel et al., 2014). While this approach produces maps of higher density than the more traditional feature-based ones (Mur-Artal et al., 2015), they are still incomplete models of the scene and hence their applicability might be more limited. 3.2.2 Single-View Depth For a more detailed and updated revision of the state of the art on single-view depth estimation we refer the reader to the Chapter 2 of this thesis that specifically addresses this problem. In this Chapter we discuss the original literature consider for this research. Depth can be estimated from a single view using different image cues, for example focus (Ens and Lawrence, 1993) or perspective (Sturm and Maybank, 1999). Learning-based approaches, as the one we use, basically discover RGB patterns that are relevant for accurate depth regression. The pioneering work of Saxena et al. (2009) trained a MRF to model depth from a set of global and local image features. Before that, Saxena et al. (2007) presented an early approach to depth prediction from monocular and stereo cues. Eigen et al. (2014) presented a two deep convolutional neural network (CNN) stacked, one to predict global depth an the second one that refines it locally. Build upon this method, Eigen and Fergus (2015) recently presented a three scale convolutional network to estimate depth, surface normals and semantic labeling. 3.3 Single and Multi-View Depth Fusion 45 High-Gradient Low-Gradient Multi-View 0.18 1.02 Single-View 0.36 0.42 Table 3.1 Median depth error [m] for single and multi-view depth estimation, and high and low-gradient pixels. This evaluation has been done in the sequence living_room_0030a from the NYUv2 dataset (one of the sequences with higher parallax). The normalized threshold between high and low-gradient pixels is 0.35 (gray scale). Liu et al. (2015b) use a unified continuous CRF-and-CNN framework to estimate depth. The CNN is used to learn the unary and pairwise potentials that the CRF uses for depth prediction. Based on Eigen and Fergus (2015), Li et al. (2016) incorporates mid-level features in its prediction using skip-layers. It shows competitive results and a small batch-size training strategy that makes their network faster to train. Chakrabarti et al. (2016) introduces a different method to predict depth from single-view using deep neural networks, showing that training the network with a much richer output improves the accuracy. Cao et al. (2016) formulates the depth prediction as a classification problem and the net output is a pixelwise distribution over a discrete depth range. Finally, Godard et al. (2017) presents an unsupervised network for depth prediction using stereo images. 3.3 Single and Multi-View Depth Fusion State-of-the-art multi-view techniques have a strong dependency on high-parallax motion and heterogeneous-texture scenes. Only a reduced set of salient pixels that hold both constraints has a small error, and the error for the majority of the points is large and uncorrelated. In contrast, single-view methods based on CNN networks achieve reasonable errors in all the image but they are locally correlated. Our proposal exploits the best properties of these two methods. Specifically, it uses a deep convolutional network (CNN) to produce rough depth maps and fuses their structure with the results of a semi-dense multi-view depth method (Fig. 3.1). Before delving into the technical aspects, we will motivate our proposal with some illustrative results. Table 3.1 shows the median depth error of the high-gradient and lowgradient pixels for a multi-view and single view reconstruction using a medium/high-parallax sequence of the NYUv2 dataset. For the multi-view reconstruction, the error for the lowgradient pixels increases by a factor of 2. Notice that the opposite happens for the single-view reconstruction: the error of high-gradient pixels is the one increasing by a factor of 2. For 46 Combining Single-View Deep Learning Depth with Multi-View Depth 2000 1000 00 1 23 4 Error (m) 0 1 23 4 0 1 23 4 Fig. 3.2 Histogram of single-view depth error [m] for three sample sequences. Notice the multiple modes, each one corresponding to a local image structure, this can be seen in the error images in the top row of the figure. this experiment, the threshold used to distinguish between high and low-gradient pixels is 0.35 in gray scale (where the maximum gradient would be 1). Furthermore, the single-view depth error usually has a structure that indicates the presence of local correlations. For instance, Fig. 3.2 shows the histogram of the single-view depth estimation error for three different sequences (two of the NYUv2 dataset and one of the TUM dataset). Notice that the error distribution is grouped in different modes, each one corresponding to an image segment. This effect is caused by the use of the high-level image features of the latest layers of the CNN network, that extend over dozens of pixels in the original image and hence over homogeneous texture areas. The different nature of the errors can be exploited to outperform both individual estimations. This fusion, however, cannot be naïvely implemented with a simple global model as it requires content-based deformations. In the next subsections we detail the specific multi and single-view methods that we use in this work and our fusion algorithm. 3.3.1 Multi-view Depth For the estimation of the multi-view depth we adopt a direct approach (Engel et al., 2014), that allows us to estimate a dense or semi-dense map in contrast to the more sparse maps of the feature-based approaches. In order to estimate the depth of a keyframe Ik we first select a set of n overlapping frames {I1,...,Io,...,In} from the monocular sequence. After that, every pixel xk l of the reference image Ik is first backprojected at an inverse depth ρ and 3.3 Single and Multi-View Depth Fusion 47 projected again in every overlapping image Io. xo l=Tko(xk l,ρl) = KR⊤ ko    K−1xk l ||K−1xk l|| ρl  −tko ,(3.1) where Tko , Rko and tko are respectively the relative transformation, rotation and translation between the keyframe Ik and every overlapping frame Io . K is the camera internal calibration matrix. We define the total photo-metric error C(ρ) as the summation of every photo-metric error εl between every pixel (or every high-gradient pixel if we want a semi-dense map) xk l in the reference image Ik and its corresponding one xo l in every other overlapping image Io at an hypothesized inverse depth ρl, C(ρ) = 1 n n ∑ o=1,o=k t ∑ l=1 εl(Ik,Io,xk l,ρl).(3.2) The error εl(Ik,Io,xk l,ρl) for each individual pixel xk l is the difference between the photometric values of the pixel and its corresponding one εl(Ik,Io,xk l,ρl) = Ik(xk l)−Io(xo l).(3.3) The estimated depth for every pixel ˆ ρ= ( ˆ ρ1... ˆ ρl... ˆ ρt)⊤ is obtained by the minimization of the total photometric error C(ρ): ˆ ρ=argmin ρ C(ρ)(3.4) 3.3.2 Single-view Depth For single-view depth estimation we use the Deep Convolutional Neural Network presented by Eigen and Fergus (2015). This network uses three stacked CNN to process the images in three different scales. The input to the network is the RGB keyframe Ik . As we use the network structure and parameters released by the authors without further training, our input image size is 320×240 . The output of the network is the predicted depth, that we will denote as s . The size of the output is 147×109 , that we upsample in our pipeline in order to fuse it with the multi-view depth. The first scale CNN extract high-level features tuned for depth estimation. This CNN produces 64 feature maps of size 19×14 that are the input, along with the RGB image, of the second scale CNN. This second stacked CNN refines the output of the first one with 48 Combining Single-View Deep Learning Depth with Multi-View Depth mid-level features to produce a first coarse depth map of size 74×55 . This depth map is upsampled and feeds a third stacked CNN that does a local refinement of the depth. This final step is necessary, as the convolution and pooling steps of the previous layers filter out the high-frequency details. The first scale was initialized with two different pre-trained networks: the AlexNet (Krizhevsky et al., 2012) and the Oxford VGG (Simonyan and Zisserman, 2014). We use the VGG version, the most accurate one as reported by the authors. This network has been trained in indoor scenes with the NYUDepth v2 dataset (Nathan Silberman and Fergus, 2012). As they used the official train/test splits of the dataset, so do we. We decided to use this neural network because it was the best-performing dense single-view method at the moment we started this work and still it is the one that keeps better trade off between quality and efficiency. We refer the reader to the original work by Eigen and Fergus (2015) for more details on this part of our pipeline. 3.3.3 Depth Fusion As we mentioned before, the objective is to fuse the output of each previous method while keeping the best properties of each of them: the single-view reliable local structure and the accurate, but semi-dense multi-view depth estimation. Let denote s and m to the single-view depth and the multi-view semi-dense depth estimation, respectively. s is predicted as detailed in section 3.3.2 and m=1 ρis the inverse of the inverse depth estimated in section 3.3.1. The fused depth estimation fi j for each pixel (i,j) of a keyframe Ik is computed as a weighted interpolation of depths over the set of pixels in the multi-view depth image fi j =∑ (u,v)∈Ω Wmuv si j (muv +(si j −suv)),(3.5) where Ω is the semi-dense set of pixels estimated by the multi-view algorithm (e.g. in a highparallax sequence, they usually correspond with the high-gradient pixels). The interpolation weights Wmuv si j model the likelihood for each pixel (u,v)∈Ω belonging to the same local structure as pixel (i,j) . The interpolation can be interpreted in two ways. First, the depth gradient (si j −suv) is added to each multi-view depth muv , i.e. we create depth map for each muv with the structure of sand then weigh them with pixel based weights. Second, for each depth si j we modify it according to the weighted discrepancy between (muv −suv). The key ingredient of this interpolation are the weights Wmuv si j that model a deformation based on the local image structures. Each weight is computed as the product of four different 3.3 Single and Multi-View Depth Fusion 49 factors. The first factor ˜ W1muv si j =e−√(i−u)2+( j−v)2)) σ1,(3.6) simply measures proximity based on the distance of the pixels (i,j) and (u,v) . The parameter σ1 controls the radius of proximity for each point. The remainder three factors depend on the structure of the single-view prediction s. The second factor ˜ W2muv si j =1 |∇xsuv −∇xsi j|+σ2 ·1 |∇ysuv −∇ysi j|+σ2 (3.7) measures the similarity of depth gradients and assigns larger weights to similar ones. ∇xsi j and ∇ysi j represent the depth gradient in the x and y direction respectively at the pixel (i,j) . σ2 limits the influence of a point to avoid extremely high weights for very similar or identical gradients. We set it to 0.1 in the experiments. Finally, the factors ˜ W3muv si j and ˜ W4muv si j strengthen the influence between the points lying in the same plane and are defined as ˜ W3muv si j =e−|(si j+∇xsi j·(u−i))))−suv|+σ3(3.8) and ˜ W4muv si j =e−|(si j+∇ysi j·(v−j))))−suv|+σ3,(3.9) where σ3 sets a minimum weight to any point in Ω . This is required to avoid vanishing weights when they are combined with ˜ W1muv si j and ˜ W2muv si j . The product of this four factor makes a non-normalized weight for each pixel in Ω ˜ Wsi j muv = 4 ∏ n=1 ˜ Wn si j muv (3.10) and represents its area of influence. The parameters σ1 , σ2 and σ3 shape the area of influence and have to be selected to balance proximity, gradient and planarity and to avoid discontinuities in the result of the fusion. This was done empirically on a small set of three images. The values of the parameters are 15 , 0.1 and 1e−3 , respectively, and we kept them fixed for all our experiments. Fig. 3.3 shows this area for a point on an image and how it is computed. Notice how the influence expands around the point but is kept inside the same local structure (the table). Once all the factors has been computed, since all the pixels (i,j) are influenced by all the 50 Combining Single-View Deep Learning Depth with Multi-View Depth RGB image with the point Weigth factors of the point Non-normalized influence Fig. 3.3 Non-normalized influence of the highlighted red point in the image. First column: RGB input image with a red point over the table, this point represent one pixel estimated by the multi-view algorithm. Second column: each one of the weights calculated separately, the third and fourth weights are shown as a product for a more intuitive view. Third column: Nonnormalized influence of the highlighted point in the RGB image. Notice how its influence is cut on the edge of the table. Figure best viewed in electronic format. pixels in Ω (see Eq. 3.5), we normalize the weights for each single-view pixel so all the weights over a pixel (i,j)sum 1. Wmuv si j = ˜ Wmuv si j −min(g,h)∈Ω˜ Wmgh si j ∑(p,k)∈Ω˜ Wmpk si j −min(g,h)∈Ω˜ Wmgh si j (3.11) The normalized weights expand the local influence to the whole image (see Fig. 3.4 and Fig. 3.5 for a more detailed view). Notice how the influence expands along planes even if the points in Ω do not reach the end of the plane; and is sharply reduced when the local structure changes. Once these influence weights have been calculated and normalized, the fusion depth estimation, f , for each point (i,j) is a combination of all the selected points in Ω, as presented in Eq. 3.5. 3.3.4 Multi-view Low-Error Point Selection Up to now we have assumed that all the points in the multi-view semi-dense depth map Ω have low error. This is easily achievable in high-parallax sequences by using robust estimators –robust cost functions or RANSAC. However, it is problematic for the degenerate or quasi-degenerate low-parallax geometries that we also target in this work. In this case, multi-view depths may contain large errors that will propagate to the fused depth map and it is necessary to filter them out. Unexpectedly, selecting high gradient pixels was not robust enough to remove points with large depth errors and we have developed a two step 3.3 Single and Multi-View Depth Fusion 51 Fig. 3.4 Normalized influence area of the points. Notice how it expands around local structure areas given a set of points in Ω .First column: RGB image with the points of Ω labeled with different colors. Second column: influence areas computed by our method. Notice how this influence expands in areas with the same local structure but can be misled in areas where there is a lack of points or where the estimation from the neural net is not accurate enough. Figure best viewed in color. algorithm that takes into account photometric and geometric information in the first step and the single-view depth map in the second one. The first step selects a fixed percentage of the best correspondence candidates –the best 25% in our experiments– based on the product of a photometric and a geometric scores. On one hand, the photometric criterion focuses on the quality of the correspondences using image information. We apply a modified version of the second best ratio.We first extract the two closest matches for a pixel (smallest photometric errors according to Eq. 3.3). We then compute the score as a function of the ratio between the distance of the two descriptors (a high ratio suggesting a good match) and the gradient of the distance function along the epipolar line (i.e., the error function presenting a distinct V-shape around this match and 52 Combining Single-View Deep Learning Depth with Multi-View Depth Fig. 3.5 Detail of the influence area. Notice how it expands mainly in the areas with same local structure. Figure best viewed in color. suggesting spatial accuracy). On the other hand, the geometric score simply backpropagates the image correspondence error to the depth estimation, resulting in low scores for lowparallax correspondences. In a second stage we also use the structure of the single-view reconstruction and apply RANSAC to estimate a spurious-free linear transformation between the multi and single-view points using only the points pre-filtered in the first stage. We apply this linear model along the entire image, consensus with outliers is found if small patches are used. This reduces further the number of spurious depth values from the multi-view algorithm. The result is a small set of low-error points that we use for the interpolation of the previous section. As mentioned before, in our experiments this algorithm behaves better than a geometric-only compatibility test, especially in the low-parallax sequences of the NYUv2 dataset. 3.4 Experimental Results In this section we evaluate the algorithm and compare its performance against two state-ofthe-art methods: multi-view direct mapping using TV regularization (implemented following Newcombe et al. (2011), Handa et al. (2011)) and the single-view depth estimation using the network of Eigen and Fergus (2015). We have selected two datasets with different properties. The first one is the NYUv2 Depth Dataset (Nathan Silberman and Fergus, 2012), a general dataset aimed at image segmentation evaluation and hence likely to contain low-parallax and low-texture sequences. We analyze results in six sequences from the test set (i.e. the single-view net had not been trained on these sequences) selected just to include different 3.4 Experimental Results 53 RMSE SCALE INVARIANT MEAN ERROR (m) MEAN ERROR (m) Sequence TV Eigen Ours(a) TV Eigen Ours(a) TV Eigen Ours(a) Ours(m) NYUDepth v2 bath_0018 1.458 0.852 0.793 0.405 0.150 0.145 1.174 0.692 0.612 0.263 bed_0013 1.004 0.550 0.482 0.212 0.139 0.136 0.690 0.441 0.344 0.163 dr_0032 2.212 0.710 0.694 0.416 0.209 0.204 1.797 0.581 0.554 0.318 kit_0032 3.599 1.621 1.572 0.812 0.592 0.583 2.920 1.222 1.183 0.805 lr_0025 1.073 0.620 0.597 0.289 0.236 0.219 0.798 0.471 0.435 0.289 lr_0030a 1.031 0.818 0.792 0.411 0.228 0.219 0.849 0.532 0.440 0.329 TUM fr1_desk 1.581 0.433 0.410 0.255 0.121 0.103 1.211 0.317 0.294 0.154 fr1_room 1.467 0.323 0.301 0.167 0.092 0.081 1.163 0.231 0.207 0.102 Table 3.2 Left table: Error metrics for the NYUv2 and TUM datasets. For each sequence and metric we compare the TV-regularized multi-view depth, the single-view depth Eigen and Fergus (2015) and our fused depth. Ours(a) represent our proposal with the automaic selection of points. Right table: Mean error for the fused depth with manual multi-view point selection (Ours(m)); selected points under certain threshold. (The evaluation has been performed in the first 100 frames of each sequence) types of rooms. The second one is the TUM RGB-D SLAM Dataset (Sturm et al., 2012a), a dataset oriented to visual SLAM and then likely to present a bias benefiting multi-view depth. In this case, we evaluated two sequences selected randomly. We run our algorithm in a 320×240 subsampled version of the images, as this is the size of the single-view neural network given by the authors. We also run our multi-view depth estimation at this image size, and upsample the fused depth to 640×480 in order to compare it against the ground truth D channel from the kinect camera. As our aim is to evaluate the accuracy of the depth estimation, we will assume that camera poses are known for the multi-view estimation. In the TUM RGB-D SLAM Dataset (Sturm et al., 2012a) we use the ground truth camera poses. In the NYUv2 Depth Dataset sequences we estimate them using the RGB-D Dense Visual Odometry by Gutiérrez-Gómez et al. (2015). These camera poses will remain fixed and used to create the multi-view depth maps. As mentioned before, the parameters of the fusion algorithm were experimentally set prior to the evaluation on a small separate set of images. To evaluate the methods, we computed three different metrics, the RMSE, the Mean Absolute Error in meters and the scale invariant error proposed in Eigen et al. (2014) 1 n∑id2 i−1 n2(∑idi)2 where d is (log(y)−log(y∗)) , y and y∗ are the ground truth depth and the estimated depth respectively. The results are summarized in Table 3.2. Our method outperforms the TV regularization in both datasets obtaining an average improvement over 50% with respect to the mean of the error in meters. As expected, the TV regularization performs better in the TUM sequences and achieves lower errors, but in terms of improvement there seems not to be big differences between both datasets. Our fusion of depths also outperforms the single-view depth reconstruction, the improvement being 10% on average. 60 Condition-Invariant Place Recognition multi-view descriptor t t-(n-1) t-n 1 descriptor database CNN CNN CNN Place Recognition by descriptor matching Input Sequence Query Nearest Neighbor Retrieved Place: Visited Places Fig. 4.1 Overview of our proposal. We extract descriptors (using deep networks) for small sequences of n frames. We use such descriptors to find the closest match in a database of already visited places. challenging and result in lower performance. Using multiple frames in a sequence can improve the robustness of place recognition against such changes. But the sequence models proposed by the state of the art (Milford and Wyeth, 2012, Naseer et al., 2018) are handcrafted for a certain set of assumptions (e.g. overlapping trajectories, similar velocity patterns), and their performance suffers if these are not hold. Also, typically, they require a high number of frames. Descriptors directly extracted from CNNs have shown good generalization properties (Gomez-Ojeda et al., 2015), but they usually do not exploit multi-view information. Improvements usually come at the cost of large descriptors, with dimensionality in the order of thousands or hundreds of thousands. The complexity of all place recognition algorithms depends on the size of the descriptor and the number of images in the database, the latest being typically high. This limits the applicability of these techniques in robotics and AR/VR scenarios, in which the computational budget is limited due to real-time constrained loops and limited on-board computational power. In this thesis we target place recognition in the presence of challenging changes in the condition of an environment, that eventually happen in most of the scenes as time passes. 4.2 The Partitioned Nordland Dataset 61 For example day/night illumination, seasonal and weather changes, or objects that are moved (cars, persons or furniture). We address this problem by generating a global descriptor of the visual input, i.e., every input (one or several images) is encoded as a descriptor and matched versus the rest of the descriptors retrieving the closest place. Our contribution is the proposal and evaluation of three different deep network architectures that exploit multi-view and temporal information for place recognition: 1) naïve descriptor grouping, 2) learning the fusion of single-view descriptors, and 3) recurrent networks using LSTM (Long Short Term Memory) layers Hochreiter and Schmidhuber (1997). Fig. 4.1 shows an overview of our proposal. Up to our knowledge, ours are the first models that use deep learning to combine multiple views to generate descriptors for place recognition. Encoding temporal information allows us to model short-time relations between images (e.g. short smartphone videos or live photos) without the need of keeping a global map or long image sequences to achieve a high accuracy. We compare our models to state-of-the-art single-view deep baselines and a non-deep sequential one using two standard datasets: the Partitioned Nordland (Olid et al., 2018) and Alderley (Milford and Wyeth, 2012). The experimental results show that the performance of our three multi-view descriptors outperforms single-view ones. We also outperform SeqSLAM, a state-of-the-art baseline for place recognition from image sequences. Furthermore, our learned descriptors are at least one order of magnitude smaller than those of the state of the art showing that multi-view learning is able to extract relevant information for place recognition. The rest of the chapter is organized as follows. Section 4.2 presents a partition of the Norland dataset that we have used in this chapter that was published in it is available in our project website 1 . Section 4.3 refers the related work. Section 4.4 details our network architectures, and section 4.5 details how they are trained. Finally, section 4.6 presents the experimental results and section 4.7 the conclusions and lines for future work. Our code and avideo showing our results can be found in our project website 2. 4.2 The Partitioned Nordland Dataset For this work, we have used the Nordland railroad videos. In 2012, the Norway broadcasting company (NRK) made a documentary about the Nordland Railway, a railway line between the cities of Trondheim and Bodø. They filmed the 729 km journey with a camera in the front 1https://webdiis.unizar.es/ jmfacil/pr-nordland/ 2http://webdiis.unizar.es/~jmfacil/cimvpr/ 62 Condition-Invariant Place Recognition training set test set discarded data Fig. 4.2 Proposed dataset partition for the Nordland dataset. Top: Geographical representation of the training (red) and test (yellow) sets. Bottom: Index representation of the distribution, w.r.t. frame index in the videos. part of the train in winter, spring, fall and summer. The length of each video is about 10 hours and each frame is timestamped with the GPS coordinates. This dataset has been used by other research groups in place recognition, for example Gomez-Ojeda et al. (2015) and Lowry and Milford (2016). Each group uses different partitions for training and test, making difficult to reproduce the results. In this work we 4.3 Related Work 63 propose a specific partition of the dataset and a baseline, to guarantee a fair comparison between algorithms. We released this dataset partition with the publication of Olid et al. (2018). 4.2.1 Data Pre-processing The first step, creating the dataset, was to extract the maximum number of images from each video. Moreover, GPS data corruption was fixed and we also eliminated tunnels and stations. After these steps, grabbing one frame per second, we obtained 28,865 images per video. We used speed information from the GPS data to filter stations and a darkness threshold to filter tunnels. 4.2.2 Dataset Partitions Fig. 4.2 illustrates the partition of the whole image set in the Nordland dataset. We decided to create the test set with three different sequences of 1,150 images (a total of 3,450 , yellow in the figure). The rest of the images were used for training ( 24,569 , red in the figure). By using multiple sections, the variety of places and appearance changes contained in the test set increases. We also left a separation of a few kilometers between each test and train section by discarding some images in order to guarantee the difference between test and train data. 4.2.3 Place labels Given the similarity between consecutive images, in this work we propose to consider that two images are of the same place if temporally they are separated by 3 images or less. We applied a sliding window of 5 images over the whole dataset in order to group images taken from five consecutive seconds. This process can be seen in Fig. 4.3. 4.3 Related Work There have been many works addressing visual place recognition and related problems. For a general overview, we refer the reader to two surveys, Garcia-Fidalgo and Ortiz (2015) on topological mapping and Lowry et al. (2016) exclusively for visual place recognition. In this section we will focus on the works that are most relevant to our proposal. We will review first the literature on place recognition descriptors, and later refer to full place recognition pipelines. Notice that our contribution lies mainly on the former, that is, the proposal of novel multi-view descriptors. 64 Condition-Invariant Place Recognition place 1 place 2 place 3 Fig. 4.3 A sliding window of five images is considered in this work as the same place. Notice the similarity of consecutive images. The figure is best viewed in electronic format. Descriptor Grouping Descriptor Fusion Recurrent Descriptors CNN t t-(n-1) t-n Input Sequence CNN CNN + multi-view descriptor (a) t t-(n-1) t-n Input Sequence CNN CNN CNN fc + multi-view descriptor (b) t t-(n-1) t-n Input Sequence CNN CNN CNN LSTM LSTM LSTM multi-view descriptor (c) Fig. 4.4 Multi-view descriptors proposed in this thesis. From left to right: (a) Descriptor Grouping , where the descriptor of a sequence is the concatenation of all the single image descriptors. (b) Descriptor Fusion , the output of the CNNs serves as input to a fullyconnected layer that combines the information into a single descriptor. (c) Recurrent Descriptors , the output of the CNNs serves as input to an LSTM network that integrates over time the single-image features to create a multi-image descriptor. 4.3.1 Descriptors Local Descriptors Techniques based on local descriptors address place recognition by detecting a set of salient keypoints in an image and generating descriptors for each one of them. These descriptors are used to find correspondences in other images, that would potentially allow us to perform visual place recognition or even 6DoF camera pose recovery. Detection and description are usually decoupled applying a method (Harris et al., 1988, Lowe, 2004, Mikolajczyk et al., 4.3 Related Work 65 2005) that detects salient points and then generate every a descriptor for every point using (Bay et al., 2006, Lowe, 2004, Rublee et al., 2011). Recently, some deep-learning approaches have addressed descriptor generation (Luo et al., 2018, 2019), keypoint detection (Savinov et al., 2017, Ono et al., 2018) or both in an end-to-end manner (Revaud et al., 2019, Dusmanu et al., 2019). Global Descriptors Despite the advantages of local descriptors, global descriptors come handy when the tasks requires larger context. One example of this is in the presence of large appearance changes like weather or illumination conditions. On those occasions, when local patterns might change substantially, a global view of the scene captured by high level features (e.g. the skyline of the city) may be more helpful. Traditional global codes include handcrafted holistic image descriptors, like low-resolution thumbnails (Milford and Wyeth, 2012) or GIST (Murillo et al., 2013). None of these are robust to appearance changes due to scene dynamics, seasonal and weather changes, or extreme viewpoint or lighting variations. To address such cases, Lowry and Milford (2016) used PCA to reduce the dimensionality of descriptors eliminating the dimensions that are influenced by condition changes. Chen et al. (2018) incorporates attention in order to focus on the most relevant image features for place recognition. Descriptors based on CNNs have shown a high degree of robustness against appearance changes. Sünderhauf et al. (2015a) and Sünderhauf et al. (2015b) showed that CNNs outperform other models, especially for drastic appearance changes. They used AlexNet (Krizhevsky et al., 2012), pretrained on ImageNet (Russakovsky et al., 2015). The features of AlexNet contain semantic information about the whole scene, which improves the invariance to certain appearance changes. Thereafter, may other works have studied CNNs as condition-invariant feature extractors (Gomez-Ojeda et al., 2015, Arandjelovic et al., 2016, Arroyo et al., 2016, Chen et al., 2017, Lopez-Antequera et al., 2017, Olid et al., 2018). Gomez-Ojeda et al. (2015) were the first that trained a network as single-image feature extractor for visual place recognition under appearance changes. In NetVLAD (Arandjelovic et al., 2016), they proposed a new type of layer inspired in VLAD, an image representation commonly used in image retrieval. Chen et al. (2017) proposed a network trained to classify the place the image was taken. Olid et al. (2018) proposed a model based on pre-trained VGG-16 Simonyan and Zisserman (2014) and fine-tuned it for the place recognition task in a Triplet-Siamese architecture. 66 Condition-Invariant Place Recognition 4.3.2 Visual Place Retrieval This groups the approaches used to find retrieve the right place for every image. That comes down to the matching algorithms uses to retrieve the right image from the database of visited places. Single-View Place Recognition These are all those approaches which goal is to retrieve a single image from a single image; ignoring the time or pose relation between different frames in the query or in the database. The literature addresses the nearest neighbor problem in place recognition in many different ways: brute force (Olid et al., 2018, Murillo et al., 2013), KD-tree (Murillo et al., 2013) and bag of words (Gálvez-López and Tardos, 2012). Lowry and Andreasson (2018) presented a model using SURF detector and HOG features and studies the use of Bag of Words and Vectors of Locally Aggregated Descriptors (VLAD) for place matching. Multi-View Place Recognition using Single-View Descriptors For place recognition, it can be assumed that images in the database are independent and all that can be done is 1-to-1 matching, or else that you know the links between the images (in the query sequence and/or in the database) an use that as a prior knowledge. Although there are only a few works that consider temporal and multi-view information for place recognition, they all have shown that sequences provide useful information for place recognition. For instance, DBoW (Gálvez-López and Tardos, 2012) and Bampis et al. (2016) incorporate a temporal consistency constraint. SeqSLAM (Milford and Wyeth, 2012) and following works (Pepperell et al., 2014) use sequence matching, similarly to Newman et al. (2006). Differently to our approach, they assume linear temporal correlation for sequence matching. Also, we use data-driven high-level features, while they use downsampled images. More recently, some works have extended SeqSLAM in several aspects. On the one hand, Chen et al. (2014) refines the temporal filtering. On the other hand, Naseer et al. (2018) proposes a graph of single-view descriptors (based on HOG and AlexNet) to model and match image sequences. Their approach is similar to SeqSLAM, with two main differences. The most straightforward one is that they use different descriptors. The second one, more subtle, is that their search of the best-matching sequence does not assume a constant speed variation between the sequences. SeqSLAM looks for straight lines in the similarity matrix, while Naseer et al. (2018) uses a more sophisticated model. In any case, none of them model changes in the sequence direction. Also, they typically rely on long-term sequence matching (i.e., query and database sequences having many consecutive matching frames), 4.4 Network Architectures 67 Method Descriptor Size Accuracy avg % ours (single-view) 128 80.19% ours (single-view) 256 80.62% ours (single-view) 512 80.65% ours (single-view) 1024 81.46% the bigger the better Table 4.1 Evaluation of different descriptor sizes on the Partitioned Nordland Dataset (Olid et al., 2018) introduced previously in this chapter. 1st column: Method. 2nd column: Descriptor size in 32-bits floating point numbers. 3rd column: Average accuracy. which limits their applicability to such case. Assuming that consecutive frames have similar appearance, Neubert et al. (2015) combines CNN single-view descriptors with a directed search. Continuing their previous work Vysotska and Stachniss (2016) and Vysotska and Stachniss (2017) propose a combination of their lazy data association with a hash reduction of single-view CNN features. 4.3.3 Multi-View Place Recognition using Multi-View Descriptor All the multi-view models described so far are handcrafted. Up to our knowledge, ours are the first ones that learn to combine multiple single-view features maps into a multi-view descriptor. We compare three different approaches to generate a descriptor based on multiple images, simple concatenation as baseline, learning to fuse descriptors and using a recurrent neural network to accumulate the knowledge over time. The traditional approach to using multiple views in place recognition is adding extra constraints when looking for the nearest neighbor. In our case, we do not add any constraints but generate descriptors that already include temporal and/or spatial information. 4.4 Network Architectures In this section, we discuss four different models for place recognition: A single-view one, based on ResNet-50, and the three multi-view ones proposed in this chapter. 4.4.1 Single-View ResNet-50 Our first network is based on the model presented in Olid et al. (2018). The main difference is that we start from ResNet-50 (He et al., 2016) pretrained on ImageNet (Russakovsky et al., 2015) as our backbone, instead of VGG-16 (Simonyan and Zisserman, 2014). Although it 68 Condition-Invariant Place Recognition is common to directly use the descriptors of different layers (see Section 4.6.1 for results on this), in our case we added and trained a fully connected layer after ResNet-50 to learn a 128 -dimensional descriptor especially designated to the task of visual place recognition. We chose a size of 128 experimentally (see Table 4.1) , as a reasonable compromise between performance and compacity. 4.4.2 Descriptor Grouping In order to include temporal information into the descriptors, our first approach is the naïve concatenation of the descriptors of consecutive frames, see Fig. 4.4a. Thus, starting from our previous single-view model, we first choose a temporal window of frames ( n ), we then generate a 128 -dimensional descriptor per frame, and we finally concatenate them. The descriptor size is then 128 ×n . Notice that this model is trained only from single-view samples. Hence, the relation between consecutive frames is not learned and this model only provides a filtering effect. 4.4.3 Descriptor Fusion Descriptor Grouping, as the simplest strategy to consider several frames, is limited in its capability to weight differently certain features (i.e. features of some of the frames may be more representative of the place than others). It is also limited to cases where the sequences (map/query) are aligned – meaning that both sequences follow the same trajectory. For that reason, we designed a model that learns how to fuse the information of our n -frames window into a more discriminant –as well as smaller– 128 -dimensional descriptor. With this Descriptor Fusion strategy, we add an extra fully connected layer that learns how to combine the outputs of n ResNet-50 into a single compact descriptor. See Fig. 4.4b for an illustration of this approach. As this network is able to learn how to weight the features from different frames, it can model more complex cases. For example, when sequences are recorded in reverse order Descriptor Grouping is limited, while Descriptor Fusion has the capability of learning a suitable fusion. 4.4.4 Recurrent Descriptors Descriptor Fusion does not explicitly exploits the sequential nature of the data. With Recurrent Descriptors, we update in an online manner the sequence coding as new frames come, keeping the most relevant previous information. With that intention, we propose a Recurrent Neural Network (see Recurrent Descriptors in Fig. 4.4c). In this model, every 4.5 Training 69 query-sequence 1 query-sequence 2 query-sequence M place 1 place 2 place N Fig. 4.5 Same place convention, illustrated with an example where the query-sequence has a length of 3 frames. A place represents a set of frames that are considered to be on the same place. Notice that a frame can be in more that one place. A query-sequence is an input sequence for our model. We want to recover the corresponding place of a query-sequence. frame is the input to a ResNet-50, and the top layers serve as the input of a LSTM network (Hochreiter and Schmidhuber, 1997), that generates a 128 -dimensional descriptor. LSTMs keep an inner state, that is updated with each input frame, and the output depends on the state and the input. Differently to previous models, keeping a recurrent inner state allows this network to produce a descriptor from the first frame, and update it sequentially as more frames arrive. 4.5 Training 4.5.1 Convention for Same Place Since our descriptor is generated from a sequence of images (query-sequence) instead of a single image, we must define when two query-sequence of n frames are considered to be at the same place (the definition of a place being dataset-dependent). To illustrate this definition we will make use of Fig. 4.5. The figure shows a sequence of frames and several examples of query-sequence, and also shows the set of frames that we consider as the same place. Therefore, during training, we consider two query-sequence to be on the same place if they 76 Condition-Invariant Place Recognition Method Trained on Number of frames Descriptor Size D vs N # % Olid et al. (2018) Norland 1 128 0.15% Olid et al. (2018) Alderley 1 128 6.84% ours (single-view) Norland 1 128 1.65% ours (single-view) Alderley 1 128 6.8% ours (grouping) Alderley 3 384 9.05% ours (grouping) Alderley 6 768 11.48% ours (fusion) Alderley 3 128 10.18% ours (recurrent) Alderley 3 128 5.73% biggest the best Table 4.3 Results on the Alderley Dataset Milford and Wyeth (2012), 1st column: Method. 2nd column: Dataset in which the model it has been trained with. 3rd column: Number of frames used for recognition (e.g. 1 would imply to be single-view). 4th column: Descriptor size in 32b floating point numbers. 5th column: Day vs Night (D vs N), query with daylight image while reference database composed by nighttime images. Qualitative results . Fig. 4.9 shows several test samples. Notice the increased challenge with respect to the Nordland dataset, with the presence of severe illumination changes plus inclusion of artificial illumination and dynamic objects. 4.6.3 Multi-View Evaluation Sequence Speed Changes: Inspecting the previous results (Table 4.2), Descriptor Grouping ( ours (grouping) ) trained only on single-view and then applied on multi-view by concatenation is the best performing. This is surprising at first sight, as the other two models were trained on multi-view data. We designed two extra experiments ( Reverse Gear and Random Speed ) to illustrate why this is happening. Both experiments are performed during test time, which means none of the networks has been retrained. The Reverse Gear experiment consist on changing the direction of the train motion on one of the sequences at test time (e.g. when testing Winter vs Fall, the sequence of Fall is played in reverse order, see Fig 4.10a). This experiment will help to discern how much the model exploits the multi-view information rather than just the sequence consistency. Table 4.4 shows that, as we expected, models trained with multi-view examples ( ours (fusion) and ours (recurrent) ) have learned to exploit multiple views: Its performance only degrades by 6% and 4% respectively. On the other side, ours (grouping) drops severely its performance, by 18%. 4.6 Experimental Results 77 Reverse Gear 3385 3390 3395 3400 3405 3410 3415 3415 3410 3405 3400 3395 3390 3385 Summer Fall (a) Random Speed 130 131 133 136 140 145 151 130 135 140 144 148 151 153 Winter Spring (b) Fig. 4.10 Experiment setup details. (a) Reverse Gear , in which the sequence is played in reverse order for one of the seasons (Fall in the figure). (b) Random Speed , in which the vehicle speed is modified for both seasons, reference (Winter) and query (Spring). In both cases, we mark with a dashed green box the same-place three-frames sequences. In the Random Speed experiment we synthetically modified the speed of the train motion on one of the sequences at test time. Specifically, we modified the frame rate along the sequence simulating changes on the train velocity, see Fig. 4.10b (in our experiments the velocity was randomly multiplied by ×1 , ×2 or ×3 at every moment of the sequence). The “speed” is modified for the whole sequence, implying that the one-to-one correspondence in plain Nordland does not hold. Table 4.4 proves that Random Speed is the most challenging setup for the ours (grouping) approach, dropping its accuracy to 36% . ours (fusion) and 78 Condition-Invariant Place Recognition Method Number of frames Normal Test Reverse Gear Random Speed Mean± Std # % % % % SeqSLAM 3 33% 0.08% 9% 14.0±13.9 ours (grouping) 392% 74% 36% 67.3±23.3 ours (fusion) 3 86% 80% 78% 81.33±3.4 ours (recurrent) 3 86% 82% 84%84.0±1.6 biggest the best Table 4.4 Experimental results for Speed Changes in the Partitioned Norland Dataset Olid et al. (2018). Comparison of all our multi-view methods and SeqSLAM presented by Milford and Wyeth (2012). 1st column: Method. 2nd column: Number of frames used for recognition. 3rd-5th column: Summer vs Winter experiments: 3 rd column: Normal Test corresponds to the one showed on Table 4.2. 4 th column: Reverse Gear experiment where the query frames are all in reversed order, i.e. simulating the train has used a reverse gear. 5 th column: Random Speed experiment where the speed of the train is simulated to be random, which means some of the frames are lost). The speed variations are independent for the query and the reference databases, and this implies no more 1 to 1 correspondence. These speed changes are perform only on the test sequences, this implies that for a network, e.g. ours (grouping) , results on the experiments are achieved using the same set parameters than in Table 4.2 with no retraining. ours (recurrent) keep its performance at a very similar level than the standard Nordland setup ( 78% and 84% respectively). Notice that the goal of this experiment is not creating a very realistic setup, but misaligning the query sequence and the database ones. The aim is to evaluate the generalization over a plain frame-to-frame filtering effect. We run the state-of-the-art multi-view baseline SeqSLAM (Milford and Wyeth, 2012) in both experiments, Reverse Gear and Random Speed , observing that its performance drops in both. This should be expected, as SeqSLAM assumes a linear relation between the velocities of the query and the reference sequences (sequence consistency). The last column of Table 4.4 ( Mean/Std ) summarizes the conclusions of both experiments, reporting the mean (biggest the best) and standard deviation (smallest the best) for all experiments ( Normal Test , Reverse Gear and Random Speed ). Observe that ours (recurrent) is the best performing, presenting both the highest average accuracy and smallest variations. This confirms our hypothesis: The sequence descriptors that use learning ( ours (fusion) and ( ours (recurrent) ) are more resilient than those based on plain concatenation (ours (grouping)) or handcrafted relations (SeqSLAM). 4.6 Experimental Results 79 Method Num of frames Descriptor Size Accuracy W vs S Norland Accuracy S vs W Norland Accuracy D vs N Alderley # % % SeqSLAM 3 6144 31% 33% 3.91% SeqSLAM 10 20480 71% 70% 9.90% SeqSLAM 100 204800 95% 94% - ours (grouping) 3 384 92% 92% 9.05% ours (grouping) 6 768 97% 97% 11.48% ours (fusion) 3128 87% 86% 10.18% ours (recurrent) 3128 85% 86% 5.73% ours (recurrent) 6 128 87% 88% - the bigger the better Table 4.5 Results on the Partitioned Nordland Dataset (Olid et al., 2018) and Alderley adding more frames in the for the search. Comparison of all our multi-view methods and SeqSLAM presented by Milford and Wyeth (2012). 1st column: Method. 2nd column: Number of frames used for recognition (e.g., 1 stands for single-view). 3rd column: Descriptor size in 32-bits floating point numbers. 4th column: Winter vs Summer, query taken from winter and matched to summer database. 5th column: Summer vs Winter, query taken from summer and matched to winter database. 6th column: Day vs Night, query taken from night and matched to day database. Last, in Table 4.5 we show a comparison between our methods and SeqSLAM (Milford and Wyeth, 2012) on both Norland and Alderley. We also evaluate the effect of considering more frames in the different methods. Our methods outperform SeqSLAM (Milford and Wyeth, 2012), a state-of-the-art baseline able to model information from several frames, when both use the same number of frames (specifically, 3 ). As all multi-view approaches improve their performance when increasing the number of frames, we increased the number of frames used by SeqSLAM. Notice that, in order to outperform our approach in Norland, the number of frames has to be increased up to 100 and 10 in Alderley. Similarly occurs when increasing the number of frames used by ours. 4.6.4 Execution time We compared the execution time of our models in the upper part of Table 4.6. The fourth column ( Descriptor Extraction ) shows the time needed to extract the descriptor of a query 3-frames sequence on a NVIDIA TITAN Xp. In this part of our pipeline Descriptor Grouping is the fastest method, as is uses the simplest network. 80 Condition-Invariant Place Recognition Method Descriptor Size Descriptor Extraction Search 1 vs 10K ms ms ours (fusion) 128 17 3.86 ours (recurrent) 128 22 3.86 ours (grouping) 384 15 10.70 - 1860 - 49.21 - 4096 - 111.44 - 6144 - 166.13 - 20480 - 688.24 - 204800 - 9279.62 smallest the best Table 4.6 Execution Time of all our models. 1st column: Method. 2nd column: Descriptor size. 3rd column: Time in milliseconds needed to extract 1 descriptor. 4th column: Given a descriptor and a reference data base of 10K descriptors, time in milliseconds needed to find the best match. Last column ( Search ) shows the time needed to find the best match (Nearest Neighbor (NN)) given a query and a database of 10K descriptors. Notice that our methods Descriptor Fusion and Recurrent Descriptors are faster. This was expected, as their descriptor sizes are n (query-sequence size) times smaller (3 times in our experiments). Our NN algorithm consists on an exhaustive search through the database. We iterate over all the visited places, compute the distance between their descriptors and the query and keep the minimum-distance one. For the distance function we use the Squared Euclidean Distance ( d2(dq,di) ). The computational complexity of this search is O(N) where N is the number of elements in the database and for the distance function O(k)where kis the descriptor size. Additionally, we computed the search time corresponding to the sizes of some of the other descriptors used in Tables 4.2 and 4.3 (bottom part of the table). As expected, the time increases with the descriptor size. Notice that high dimensional descriptors rule out the use of more efficient data structures, such as KD-trees, to speed up the search techniques, since it is not possible to reject candidates by using the difference of a single coordinate (Marimont and Shapiro, 1979). A directed search using the sequentiality of data (Neubert et al., 2015, Vysotska and Stachniss, 2016) would reduce the number of comparisons. Hashing (Vysotska and Stachniss, 2017) can also reduce the computational cost. In any case, reducing the dimensionality of the descriptor has a direct influence in the cost of all the approaches mentioned. 4.7 Conclusions 81 4.7 Conclusions In this chapter we have introduced three deep learning-based multi-view global descriptor models, that outperform existing baselines both in accuracy and compactness. We analyzed different approaches to combine the information of the features from multiple views (Grouping, Fusion and Recurrent), and we evaluated them on different experimental setups in two public datasets: Partitioned Norland and Alderley. Each model we propose has its own strengths and weaknesses. On the one side, Descriptor Grouping ensures the sequential consistency of the frame in a sequence, achieving the best performance in the standard Nordland/Alderley benchmarks, where the inter-frame motion is similar in different runs. On the other side, Descriptor Fusion and Recurrent Descriptors are able to learn more complex relations between frames and hence proved to be better in cases where the velocities differ or the frames ordering is different. We also show the low computational cost of all the approaches, demonstrating its potential for robotic applications. We believe that recurrent models, in spite of challenges associated to training and generalizing to different vehicle dynamics, are promising. They are robust to speed changes and adapt to different sequence lengths without increasing the descriptor and network sizes. We have observed such challenges in preliminary results, along with a slight performance improvement with an increase of the sequence length (up to 6 frames). Encouraged by this, our future work will study carefully the use of recurrent models with large image sequences. However that presents the challenge of the beginning and the end of the sequence being two place too far apart. Chapter 5 Corner Prediction for Layout Reconstruction Another relevant cue for visual 3D reconstruction is the extraction of high-level semantic information corresponding to the structure of the scene. For example, detecting and estimating depth and orientation of planar structures (Concha and Civera, 2015a). In this chapter, we focus on detecting the main structure for indoor scenes. Also refereed as layout recovery, that is detecting the walls, ceiling and floor. We have not applied our advances directly to visual localization or mapping for sequences but we refer the reader to Salas et al. (2015) as an example of how this advances could be use for Visual SLAM. The problem of 3D layout recovery in indoor scenes has been a core research topic for over a decade. However, there are still several major challenges that remain unsolved. Among the most relevant ones, a major part of the state-of-the-art methods make implicit or explicit assumptions on the scenes –e.g. box-shaped or Manhattan layouts. Also, current methods are computationally expensive and not suitable for real-time applications like robot navigation and AR/VR. In this work we present CFL (Corners for Layout), the first end-to-end model that predicts layout corners for 3D layout recovery on 360◦ images. Our experimental results show that we outperform the state of the art, making less assumptions on the scene than other works, and with lower cost. We also show that our model generalizes better to camera position variations than conventional approaches by using EquiConvs, a convolution applied directly on the spherical projection and hence invariant to the equirectangular distortions. 84 Corner Prediction for Layout Reconstruction CFL: End-to-End Layout Recovery Fig. 5.1 Corners for Layout : The model end-to-end predicts the layout corners from the spherical image. Connecting the corners and assuming ceiling-floor parallelism, we can directly obtain the 3D layout in a very short time. 5.1 Introduction Recovering the 3D layout of an indoor scene from a single view has attracted the attention of computer vision and graphics researchers in the last decade. The idea is going beyond pure geometrical reconstructions and provide higher-level contextual information about the scene, even in the presence of clutter. Layout estimation is a key technology in several emerging application markets, such as augmented and virtual reality and robot navigation (Salas et al., 2015). But also for more traditional ones, like real estate (Liu et al., 2015a). Layout estimation, however, is not a trivial task and there are several major problems that still remain unsolved. For example, most existing methods are based on strong assumptions on the geometry (e.g. Manhattan scenes) or the over-simplification of the room types (e.g. box-shaped layouts), often underfitting the richness of real indoor spaces. The limited field of view of conventional cameras leads to ambiguities, which could be solved by considering a wider context. For this reason it is advantageous to use wide fields of view, like 360 ◦ panoramas. In these cases, however, the methods for conventional cameras are not suitable due to the image distortions and new ones have to be developed (Pais et al., 2019). In the last years, the main improvements in layout recovery from panoramas have come from the application of deep learning. The high-level features learned by deep networks have proven to be as useful for this problem as for many others. Nevertheless, these techniques entail other problems such as the lack of data or overfitting. State-of-the-art methods require additional preand/or post-processing. As a consequence they are very slow, and this is a major drawback considering the aforementioned applications for real-time layout recovery. In this work, we present Corners for Layout (CFL), the first end-to-end neural network that predicts a map of the corners of the room to directly obtain the 3D layout from a single 360◦ image (Figure 5.1). This makes CFL more than 100 times faster than the state of the art, while still outperforming the accuracy of current approaches . Furthermore, our 5.2 Related Work 85 proposal is not limited by typical scene assumptions, meaning that it can predict complex geometries, such as rooms with more than four walls or non strict Manhattan structures. Additionally, we propose a novel implementation of the convolution for 360◦ images (Tateno et al., 2018, Cohen et al., 2018) in the equirectangular projection. We deform the kernel,using the advances presented by Dai et al. (2017b), to compensate the distortion and make CFL more robust to camera rotation and pose variations , generalizing to unseen configurations. Hence, it is equivalent to applying directly a convolution operation to the spherical image, which is geometrically more coherent than applying a standard convolution on the equirectangular panorama. We have extensively evaluated our network in two public datasets with several training configurations, including data augmentation techniques to address occlusions by enforcing the network to learn from the context. We also propose a robustness analysis to see the effect of extrinsic variations in panoramas and dataset bias. Our code and labeled dataset can be found here: CFL webpage. 5.2 Related Work The layout of a room provides a strong prior for other visual tasks like single-view (Eigen and Fergus, 2015) and multi-view depth recovery (Concha et al., 2014), realistic insertions of virtual objects into indoor images (Karsch et al., 2011), indoor object recognition (Bao et al., 2011, Song and Xiao, 2016), indoor place recognition (Hussain et al., 2016) or human pose estimation (Fouhey et al., 2014). A large variety of methods have been developed for this purpose using multiple input images (Tsai et al., 2011, Flint et al., 2011) or depth sensors (Zhang et al., 2013), which deliver high-quality reconstruction results. For the common case when a single RGB image is available, the problem becomes considerably more challenging and researchers need very often to rely on strong assumptions. The seminal approaches to layout prediction from a single view were (Delage et al., 2006, Lee et al., 2009), followed by (Hedau et al., 2009a, Schwing et al., 2013). They basically model the layout of the room with a vanishing-point-aligned 3D box, being hence constrained to this particular room geometry and unable to generalize to others appearing frequently in real applications. Most recent approaches exploit CNNs and their excellent performance in a wide range of applications such as image classification, segmentation and detection. (Mallya and Lazebnik, 2015, Ren et al., 2016, Zhang et al., 2017, Zhao et al., 2017), for example, focus on predicting the informative edges separating the geometric classes (walls, floor and ceiling). Alternatively, Dasgupta et al. (2016) proposed a FCN to predict labels for each of the surfaces of the room. All these methods require extra computation added to the forward propagation of the network to retrieve the actual layout. In Lee et al. (2017), for example, an 92 Corner Prediction for Layout Reconstruction the network output ( k=4 ) and 3 intermediate layers ( k={1,...,3} ). The total loss is then the sum over all pixels, the 4 resolutions and both the edge and corner maps L=∑ k={1,...,4} ∑ m={e,c} ∑ i Lm i[k].(5.2) 5.3.3 From Corner Maps to 3D Layout Current methods (Zou et al., 2018, Fernandez-Labrador et al., 2018b, Zhang et al., 2014) use pre-computed vanishing points and posterior optimizations, being constrained to produce strict Manhattan 3D layouts. Aiming to a fast end-to-end simple model, CFL avoids extra computation and adopt a representation usually referred as Soft/Weak Manhattan (Furlan et al, 2013) or Atlanta World (Joo et al, 2018). Following this, horizontal directions are not necessarily orthogonal to each other, thus relaxing the model assumptions. To this end, we simply follow a natural transformation from corners coordinates to 2D and 3D layout. The 2D corners coordinates are the maximum activations in the probability map. Assuming that the corner set is consistent, they are directly joined, from left to right, in the unit sphere space and re-projected to the equirectangular image plane. The 3D layout is inferred by only assuming ceiling-floor parallelism, leaving the wall structure unconstrained –i.e., we assume that the floor corners are on the same plane and the top corners are directly above the floor ones, but we do not force the usual Manhattan perpendicularity between walls. Corners are projected to floor and ceiling planes given a unitary camera height (trivial as results are up to scale). See Figure 5.3. Here we provide a further explanation of how the process to go from 2D to 3D works. From the predicted 2D corner positions, we can directly recover the 3D layout by doing the following assumptions: 1. Soft Manhattan or Atlanta world. This is a relaxation of the Manhattan World assumption whereby horizontal directions are not necessarily orthogonal to each other. That is, walls can intersect with each other in any direction. 2. Ceiling-floor parallelism. Corners can be classified depending on their position along the vertical direction (above or below the horizon line, which in central panoramas is at the middle row) between ceiling and floor corners respectively. Floor corners are on the same floor plane and ceiling corners are directly above the floor ones. The vertical direction is the normal direction of both floor and ceiling planes. 3. Unitary camera height. This is trivial as results are up to scale but needed to predict the total height of the room. 5.3 Corners for Layout 93 Taking all of this into account, we can define a plane as the set of all points P= (x,y,z) such that P·N+d=0 , where the normal N= (nx,ny,nz) is a normalized vector perpendicular to its surface and d is the distance that separates it from the origin of coordinates in the direction of the normal. Due to assumptions b) and c), N of both the floor and ceiling planes is equal and corresponds to the vertical direction, and the distance d from the floor to the camera is known. The distance to the ceiling is yet unknown. Additionally, thanks to the nature of spherical images, we can easily obtain the 3D ray R(t) = O+ V·t (parametric representation) going from the center of the sphere O= (ox,oy,oz) through the corner position, with normalized direction vector  V= (vx,vy,vz) . To obtain the normalized direction vector  V , we need the corner position in the sphere, thus we transform the image coordinates of the corners (u,v) into spherical coordinates and then to the Euclidean 3D space. Equations for this can be found in Section 4.1. of this chapter. In the first place, Eq (5.3) give us the angles that define the point (u,v)in the sphere. φ= (u−W 2)2π W;θ=−(v−H 2)π H(5.3) Where W and H are the width and height of the equirectangular image. Second, once these rotations are known we can compute the direction of the ray. Therefore, using Eq (5.4) we can calculate  V.  V=   −cos(θ)sin(φ) sin(θ) cos(θ)cos(φ)   (5.4) The intersection between the corner ray and the corresponding floor or ceiling plane will give us the actual 3D corner point P= (x,y,z) (up to scale), ie. the intersection represents that point P on the surface of the plane that verifies the ray equation: (ox+vx·t)nx+(oy+ vy·t)ny+(oz+vz·t)nz+d=0 . The point P of intersection would simply be the result of evaluating the calculated t, Eq (5.5), in the ray equation R(t). t=−oxnx+oyny+oznz+d vxnx+vyny+vznz (5.5) Let’s consider we have performed the operations to compute one corner point on the floor plane, PF= (xF,yF,zF) . The corresponding point on the ceiling plane ( PC ) will be on top of it (ie. xF=xC and yF=yC ). Therefore, we can use this to compute tC , Eq (5.6), and thus the ceiling point: tC=(xF−ox) vC x (5.6) 94 Corner Prediction for Layout Reconstruction Fig. 5.4 Spherical parametrization of EquiConvs . The spherical kernel, defined by its angular size ( αw×αh ) and resolution ( rw×rh ), is convolved around the sphere with angles φand θ. where  VC= (vC x,vC y,vC z) is computed as in (5.4) with the corresponding ceiling point in the image. Notice that with PCwe have the information we were missing to recover the ceiling plane. Limitations of CFL: We directly join corners from left to right, meaning that our model would not work if any wall is occluded because of the convexity of the scene. In those particular cases, the joining process should follow a different order. Fernandez-Labrador et al. (2018b) proposes a geometry-based post-processing that could alleviate this problem, but its cost is high and it needs the Manhattan World assumption. The addition of this post-processing into our work, in any case, could be done similarly to Fernandez-Labrador et al. (2018a). 5.4 Equirectangular Convolutions Spherical images are receiving an increasing attention due to the growing number of omnidirectional sensors in drones, robots and autonomous cars. A naïve application of convolutional networks to a equirectangular projection, is not, in principle, a good choice due to the spacevarying distortions introduced by such projection. 5.4 Equirectangular Convolutions 95 In this section we present a convolution that we name EquiConv, which is defined in the spherical domain instead of the image domain and it is implicitly invariant to equirectangular representation distortions. The kernel in EquiConvs is defined as a spherical surface patch –see Figure 5.4. We parametrize its receptive field by the angles αw and αh . Thus, we directly define a convolution over the field of view. The kernel is rotated and applied along the sphere and its position is defined by the spherical coordinates ( φ and θ in the figure) of its center. Unlike standard kernels, that are parameterized by their size kw×kh , with EquiConvs we define the angular size ( αw×αh ) and resolution ( rw×rh ). In practice, we keep the aspect ratio, αw rw=αh rh , and we use square kernels, so we will refer the field of view as α ( αw=αh ) and the resolution as r ( rw=rh ) respectively from now on. In this work, we choose values of resolution and field of view to be the same as the image. 5.4.1 EquiConvs Details In Dai et al. (2017b), they introduce deformable convolutions by learning additional offsets from the preceding feature maps. Offsets are added to the regular kernel locations in the Standard Convolution enabling free form deformation of the kernel. Inspired by this work, we deform the shape of the kernels according to the geometrical priors of the equirectangular image projection. To do that, we generate offsets that are not learned but fixed given the spherical distortion model and constant over the same horizontal locations. Here, we describe how to obtain the distorted pixel locations from the original ones. Let us define (u0,0,v0,0) as the pixel location on the equirectangular image where we apply the convolution operation (i.e. the image coordinate where the center of the kernel is located). First, we define the coordinates for every element in the kernel and afterwards we rotate them to the point of the sphere where the kernel is being applied. We define each point of the kernel as ˆpi j =   ˆxi j ˆyi j ˆzi j   =   i j d   ,(5.7) where i and j are integers in the range [−r−1 2,r−1 2] and d is the distance from the center of the sphere to the kernel grid. In order to cover the field of view α, d=r 2tan(α 2).(5.8) 96 Corner Prediction for Layout Reconstruction Standard Deformable Equirectangular Fig. 5.5 Effect of offsets on a 3×3kernel . Left: Regular kernel in Standard Convolution. Center: Deformable kernel in Dai et al. (2017b). Right: Spherical surface patch in EquiConvs. We project each point into the sphere surface by normalizing the vectors, and rotate them to align the kernel center to the point where the kernel is applied. pi j =   xi j yi j zi j   =Ry(φ0,0)Rx(θ0,0)ˆpi j |ˆpi j|,(5.9) where Ra(β) stands for a rotation matrix of an angle β around the a axis. φ0,0 and θ0,0 are the spherical angles of the center of the kernel –see Figure 5.4, and are defined as φ0,0= (u0,0−W 2)2π W;θ0,0=−(v0,0−H 2)π H,(5.10) where W and H are, respectively, the width and height of the equirectangular image in pixels. Finally, the rest of elements are back-projected to the equirectangular image domain. First, we convert the unit sphere coordinates to latitude and longitude angles: φi j =arctan(xi j zi j );θi j =arcsin(yi j).(5.11) And then, to the original 2D equirectangular image domain: ui j = (φi j 2π+1 2)W;vi j = (−θi j π+1 2)H.(5.12) In Figure 5.5 we show how these offsets are applied to a regular kernel; and in Figure 5.6 three kernel samples on the spherical and on the equirectangular images. 5.4 Equirectangular Convolutions 97 Fig. 5.6 EquiConvs on spherical images. We show three kernel positions to highlight the differences between the offsets. As we approach to the poles (larger θ angles) the deformation of the kernel on the equirectangular image is bigger, in order to reproduce a regular kernel on the sphere surface. Additionally, with EquiConvs, we do not use padding when the kernel is on the border of the image since offsets take the points to their correct position on the other side of the 360◦image. 98 Corner Prediction for Layout Reconstruction 5.5 Experiments We present a set of experiments to evaluate CFL using both Standard Convolutions (StdConvs) and the proposed Equirectangular Convolutions (EquiConvs). We do not only analyze the corner maps predicted by our model, but also the impact of each algorithmic component through ablation studies. We report the performance of our proposal in two different datasets, and show qualitative 2D and 3D models of different indoor scenes. 5.5.1 Datasets We use two public datasets that comprise several indoor scenes, SUN360 (Xiao et al., 2012) and Stanford (2D-3D-S) Armeni et al. (2017) in equirectangular projection (360 ◦ ). The former is used for ablation studies, and both are used for comparison against several state-of-the-art baselines. SUN360 (Xiao et al., 2012) : We use ∼ 500 bedroom and livingroom panoramas from this dataset labeled by Zhang et al. (2014). We use these labels but, since all panoramas were labeled as box-type rooms, we hand-label and substitute 35 panoramas representing more faithfully the actual shapes of the rooms. We split the raw dataset in 85 % training scenes and 15 % test scenes randomly by making sure that there were rooms of more than 4 walls in both partitions. Stanford 2D-3D-S (Armeni et al., 2017) : This dataset contains more challenging scenarios like cluttered laboratories or corridors. In Zou et al. (2018), they use areas 1, 2, 4, 6 for training, and area 5 for testing. For our experiments we use same partitions and the ground truth provided by them. 5.5.2 Implementation details The input to the network is a single panoramic RGB image of resolution 256×128 . The outputs are, on the one hand, the room layout edge map and on the other hand, the corner map, both of them at resolution 128×64 . A widely used strategy to improve generalization of neural networks is data augmentation. We apply random erasing, horizontal mirroring as well as horizontal rotation from 0◦ to 360◦ of input images during training. The weights are all initialized using ResNet-50 (He et al., 2016) trained on ImageNet (Russakovsky et al., 2015). For CFL EquiConvs we use the same kernel resolutions and field of views as in ResNet-50. This means that for a standard 3 × 3 kernel applied to a W × H feature map, r=3 and α=rf ov W , where f ov =360◦ for panoramas. We minimize the cross-entropy loss using Adam (Kingma and Ba, 2014), regularized by penalizing the loss with the sum of the L2 5.5 Experiments 99 Corners Conv. IP EM IoU Acc P R F1 : 1 : 1 : 1 : 1 : 1 StdConvs - - 0.519 0.978 0.611 0.763 0.675 StdConvs -✓0.531 0.979 0.639 0.749 0.685 StdConvs ✓ ✓ 0.569 0.982 0.684 0.761 0.718 EquiConvs - - 0.485 0.972 0.551 0.786 0.642 EquiConvs -✓0.536 0.980 0.649 0.744 0.690 EquiConvs ✓ ✓ 0.580 0.983 0.697 0.762 0.726 bigger is better Table 5.3 Ablation study on SUN360 dataset. We show results for both Standard Convolutions (StdConvs) and our proposed Equirectangular Convolutions (EquiConvs) with some modifications: Using or not intermediate predictions (IP) in the decoder and edge map predictions (EM). of all weights. The initial learning rate is 2.5e−4 and is exponentially decayed by a rate of 0.995 every epoch. We apply a dropout rate of 0.3. The network is implemented using TensorFlow (Abadi et al., 2016) and trained and tested in a NVIDIA Titan X. The training time for StdConvs is around 1 hour and the test time is 0.31 seconds per image. For EquiConvs, training takes 3 hours and test around 3.32 seconds per image. 5.5.3 Network’s output evaluation We measure the quality of our predicted probability corner maps using five standard metrics: intersection over union IoU, precision P, recall R, F1 Score F1 and accuracy Acc. Table 5.3 summarizes our results and allows us to answer the following questions: What are the effects of different convolutions? As one would expect, EquiConvs, aware of the distortion model, learn in a non-distorted generic feature space achieving accurate predictions, like StdConvs on conventional images (Lee et al., 2017). Distortion understanding, additionally, gives the network other advantages. While StdConvs learn strong bias correlation between features and distortion patterns (e.g. ceiling line on the top of the image or clutter in the mid-bottom), EquiConvs are invariant to that. For this reason, the performance of EquiConvs does not degrade when varying the camera DOF pose – see Section 5.5.4. Additionally, EquiConvs allow to directly leverage networks pre-trained on conventional images. Specifically, this translates into a faster convergence, which is desirable as, to date, 360◦ datasets contain far less images than datasets with conventional images. In 100 Corner Prediction for Layout Reconstruction Fig. 5.7 EquiConvs show more consistent qualitative results whereas StdConvs simply do not understand that the image wraps around the sphere, losing the continuous context that these images provide. omnidirectional images, the right and the left edge are the same spot in reality so, another strength of EquiConvs lie in the fact that we can avoid padding when the kernel reaches the border of the image since offsets take the points to their correct position on the other side of the 360◦ image. This allows the model to understand the continuity of the scene. StdConvs, instead, simply do not understand that the image wraps around the sphere. As a consequence, in most cases when corners approach the borders, StdConvs predict these corners twice, i.e. at both ends, or the edges at one side would not coincide with the edges at the other side. This effect is highlighted in Figure 5.7 and further demonstrated in the supplementary video. How can we refine predictions? There are some techniques that we can use in order to obtain more accurate and refined predictions. Here, we make pyramid preliminary predictions in the decoder and iteratively refine them, by feeding them back to the network, until the final prediction. Also, although we only use the corner map to recover the layout of the room, we train the network to additionally predict edge maps as an auxiliary task. This is another representation of the same task that ensures that the network learns to exploit the relationship between both outputs, i.e., the network learns how edges intersect between them generating the corners. The improvement is shown in the Table 5.3. How can we deal with occlusions? We do Random Erasing Data Augmentation. This operation randomly selects rectangles in the training images and removes its content, generating various levels of virtual occlusion. In this manner we simulate real situations where objects in the scene occlude the corners of the room layout, and force the network to learn context-aware features to overcome this challenging situation. Figure 5.8 illustrates this strategy with an example. Is it possible to relax the scene assumptions while keeping a good performance? By avoiding constrained Manhattan 3D layout predictions we not only achieve better results 5.5 Experiments 101 Input Panorama Without random erasing With random erasing Erasing example Fig. 5.8 Augmenting the data with virtual occlusions. Left: Image with erased pixels. Right: Input panorama and predictions without and with pixel erasing. Notice the improvement by random erasing. compared with current arts, but also we save in computation. Additionally, our model overcomes the classic box-room simplification (four-walls room setups), even if we still have a largely unbalanced dataset after labeling some panoramas more accurately to their actual shape. We address this problem by choosing a batch size of 16 and forcing it to always include one non-box sample. This favors the learning of more complex rooms despite having few examples. F1Acc IoU Trans StdConvs 55.32±8.23 95.46±1.3 39.135±7.82 EquiConvs 59.55 ±8.95 96.21 ±1.14 43.47 ±8.83 Rot x StdConvs 45.89±14.72 93.44±3.18 31.26±12.83 EquiConvs 46.2 ±15.1 94.43 ±2.18 31.625 ±13.41 Rot y StdConvs 72.28±2.7 98.21±0.21 57.54±3.25 EquiConvs 72.96 ±2.02 98.29 ±0.14 58.44 ±2.44 Table 5.4 Robustness analysis . Values represent the mean value (bigger is better) ± standard deviation (smaller is better) in %. We apply three types of transformations to the panoramas: translations in y dependant on the room height from −0.3h to 0.3h , rotations in x from −30◦ to +30◦ and rotations in y from 0◦ to 360◦ . We do not use these images for training but just for testing in order to show the generalization capabilities of both models. 5.5.4 Robustness analysis We test our model with previously unseen images where the camera viewpoint is different from that in the training set. The distortion in equirectangular projection is location dependent, 108 Corner Prediction for Layout Reconstruction Fig. 5.12 Layout predictions (light magenta) and ground truth (dark magenta) for complex room geometries on the SUN360 annotation dataset (Xiao et al., 2012). Best viewed in color. 5.6 Conclusions 109 Fig. 5.13 Layout predictions (light magenta) and ground truth (dark magenta) on the Stanford 2D-3D annotation dataset (Armeni et al., 2017). Best viewed in color. Chapter 6 Monocular and RGB-D SLAM on Dynamic Environments The assumption of scene rigidity is typical in SLAM algorithms. Such a strong assumption limits the use of most visual SLAM systems in populated real-world environments, which are the target of several relevant applications like service robotics or autonomous vehicles. In this chapter we present DynaSLAM, a visual SLAM system that, building on ORBSLAM2 (Mur-Artal and Tardós, 2017), adds the capabilities of dynamic object detection and background inpainting. DynaSLAM is robust in dynamic scenarios for monocular, stereo and RGB-D configurations. We are capable of detecting the moving objects either by multi-view geometry, deep learning or both. Having a static map of the scene allows inpainting the frame background that has been occluded by such dynamic objects. We evaluate our system in public monocular, stereo and RGB-D datasets. We study the impact of several accuracy/speed trade-offs to assess the limits of the proposed methodology. DynaSLAM outperforms the accuracy of standard visual SLAM baselines in highly dynamic scenarios. And it also estimates a map of the static parts of the scene, which is a must for long-term applications in real-world environments. 6.1 Introduction SLAM is a prerequisite for many robotic applications, for example collision-less navigation. SLAM techniques estimate jointly a map of an unknown environment and the robot pose within such map, only from the data streams of its on-board sensors. The map allows the robot to continually localize within the same environment without accumulating drift. This 112 Monocular and RGB-D SLAM on Dynamic Environments is in contrast to odometry approaches that integrate the incremental motion estimated within a local window and are unable to correct the drift when revisiting places. Visual SLAM, where the main sensor is a camera, has received a high degree of attention and research efforts over the last years. The minimalistic solution of a monocular camera has practical advantages with respect to size, power and cost, but also several challenges such as the unobservability of the scale or state initialization. By using more complex setups, like stereo or RGB-D cameras, these issues are solved and the robustness of visual SLAM systems can be greatly improved. The research community has addressed SLAM from many different angles. However, the vast majority of the approaches and datasets assume a static environment. As a consequence, they can only manage small fractions of dynamic content by classifying them as outliers to such static model. Although the static assumption holds for some robotic applications, it limits the applicability of visual SLAM in many relevant cases, such as intelligent autonomous systems operating in populated real-world environments over long periods of time. Visual SLAM can be classified into feature-based methods (Klein and Murray, 2007, Mur-Artal et al., 2015), that rely on salient points matching and can only estimate a sparse reconstruction; and direct methods (Stühmer et al., 2010, Newcombe et al., 2011, Graber et al., 2011), which are able to estimate in principle a completely dense reconstruction by the direct minimization of the photometric error and TV regularization. Some direct methods focus on the high-gradient areas estimating semi-dense maps (Engel et al., 2014, 2017). None of the above methods, considered the state of the art, address the very common problem of dynamic objects in the scene, e.g., people walking, bicycles or cars. Detecting and dealing with dynamic objects in visual SLAM reveals several challenges for both mapping and tracking, including: 1. How to detect such dynamic objects in the images to: (a) Prevent the tracking algorithm from using matches that belong to dynamic objects. (b) Prevent the mapping algorithm from including moving objects as part of the 3D map. 2. How to complete the part of the 3D map that is temporally occluded by a moving object. Many applications would greatly benefit from progress along these lines. Among others, augmented reality, autonomous vehicles, and medical imaging. All of them could for instance safely reuse maps from previous runs. Detecting and dealing with dynamic objects is a requisite to estimate stable maps, useful for long-term applications. If the dynamic 6.1 Introduction 113 (a) Input RGB-D frames with dynamic content. (b) Output RGB-D frames. Dynamic content has been removed. Occluded background has been reconstructed with information from previous views. (c) Map of the static part of the scene, after removal of the dynamic objects. Fig. 6.1 Overview of DynaSLAM results for the RGB-D case. content is not detected, it becomes part of the 3D map, complicating its usability for tracking or relocation purposes. In this work we propose an on-line algorithm to deal with dynamic objects in RGB-D , stereo and monocular SLAM. This is done by adding a front-end stage to the state-of-the-art ORB-SLAM2 system (Mur-Artal and Tardós, 2017), with the purpose of having a more accurate tracking and a reusable map of the scene. In the monocular and stereo cases our proposal is to use a CNN to pixel-wise segment the a priori dynamic objects in the frames (e.g., people and cars), so that the SLAM algorithm does not extract features on them. In the 114 Monocular and RGB-D SLAM on Dynamic Environments RGB-D case we propose to combine multi-view geometry models and deep-learning-based algorithms for detecting dynamic objects and, after having removed them from the images, inpaint the occluded background with the correct information of the scene (Fig. 6.1). The rest of the chapter is structured as follows: section 6.2 discusses related work, section 6.3 gives the details of our proposal, section 6.4 details the experimental results, and section 6.5 presents the conclusions and lines for future work. 6.2 Related Work Dynamic objects are, in most SLAM systems, classified as spurious data and therefore neither included in the map nor used for camera tracking. The most typical outlier rejection algorithms are RANSAC (e.g., in ORB-SLAM (Mur-Artal et al., 2015, Mur-Artal and Tardós, 2017)) and robust cost functions (e.g., in PTAM by Klein and Murray (2007)). There are several SLAM systems that address more specifically the dynamic scene content. Within feature-based SLAM methods, some of the most relevant on dealing with dynamic scenes are the following. Tan et al. (2013) that detect changes that take place in the scene by projecting the map features into the current frame for appearance and structure validation. Wangsiripitak and Murray (2009) track known 3D dynamic objects in the scene. Similarly, Riazuelo et al. (2017) deal with human activity by detecting and tracking people. More recently, the work of Li and Lee (2017) uses depth edges points, which have an associated weight indicating its probability of belonging to a dynamic object. Direct methods are, in general, more sensitive to dynamic objects in the scene. The most relevant works specifically designed for dynamic scenes are mentioned bellow. Alcantarilla et al. (2012) detect moving objects by means of a scene flow representation with stereo cameras. Wang and Huang (2014) segment the dynamic objects in the scene using RGB optical flow. Kim and Kim (2016) propose to obtain the static parts of the scene by computing the difference between consecutive depth images projected over the same plane. Sun et al. (2017) calculate the difference in intensity between consecutive RGB images. Pixel classification is done with the segmentation of the quantized depth image. All the methods –both feature-based and direct ones– that map the static scene parts only from the information contained in the sequence (Mur-Artal and Tardós, 2017, Mur-Artal et al., 2015, Tan et al., 2013, Li and Lee, 2017, Alcantarilla et al., 2012, Wang and Huang, 2014, Kim and Kim, 2016, Sun et al., 2017, Concha and Civera, 2015a), fail to estimate lifelong models when an a priori dynamic object remains static, e.g., parked cars or people sitting. On the other hand, Wangsiripitak and Murray (2009), and Riazuelo et al. (2017) would detect those a priori dynamic objects, but would fail to detect changes produced by 6.3 DynaSLAM System Description 115 Fig. 6.2 Block diagram of our proposal. In the stereo and monocular pipeline (black continuous line) the images pass through a Convolutional Neural Network ( Mask R-CNN ) for computing the pixel-wise semantic segmentation of the a priori dynamic objects before being used for the mapping and tracking. In the RGB-D case (black dashed line) a second approach based on multi-view geometry is added for a more accurate motion segmentation, for which we need a low-cost tracking algorithm. Once the position of the camera is known (Tracking and Mapping output), we can inpaint the background occluded by dynamic objects. The red dotted line represents the data flow of the stored sparse map. static objects, e.g., a chair a person is pushing, or a ball that someone has thrown. That is, the former approach succeeds in detecting moving objects, and the second one in detecting several movable objects. Our proposal, DynaSLAM, combines multi-view geometry and deep learning in order to address both situations. Similarly, Ambrus et al. (2016) segment dynamic objects by combining a dynamic classifier and multi-view geometry. 6.3 DynaSLAM System Description Fig. 6.2 shows an overview of our system. First of all, the RGB channels pass through a CNN that segments out pixel-wise all the a priori dynamic content, e.g., people or vehicles. In the RGB-D case, we use multi-view geometry to improve the dynamic content segmentation in two ways. First, we refine the segmentation of the dynamic objects previously obtained by the CNN. Second, we label as dynamic new object instances that are static most of the time (i.e., detect moving objects that were not set to movable in the CNN stage). For that purpose, it is necessary to know the camera pose, for which a low-cost tracking module has been implemented to localize the camera within the already created scene map. These segmented frames are the ones which are used to obtain the camera trajectory and the map of the scene. Notice that if the moving objects in the scene are not within the CNN classes, the multi-view geometry stage would still detect the dynamic content, but the accuracy might decrease. Once this full dynamic object detection and localization of the camera have been done, we aim to reconstruct the occluded background of the current frame with static information 116 Monocular and RGB-D SLAM on Dynamic Environments from previous views. These synthetic frames are relevant for applications like augmented and virtual reality, and place recognition in lifelong mapping. In the monocular and stereo cases, the images are segmented by the CNN so that keypoints belonging to the a priori dynamic objects are neither tracked nor mapped. All the different stages are described in depth in the next subsections (6.3.1 to 6.3.5). 6.3.1 Segmentation of Potentially Dynamic Content using a CNN For detecting dynamic objects we propose to use a CNN that obtains a pixel-wise semantic segmentation of the images. In our experiments we use Mask R-CNN (He et al., 2017), which is the state of the art for object instance segmentation. Mask R-CNN can obtain both pixel-wise semantic segmentation and the instance labels. For this work we use the pixel-wise semantic segmentation information, but the instance labels could be useful in future work for the tracking of the different moving objects. We use the TensorFlow implementation by Matterport1. The input of Mask R-CNN is the RGB original image. The idea is to segment those classes that are potentially dynamic or movable (person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra and giraffe). We consider that, for most environments, the dynamic objects likely to appear are included within this list. If other classes were needed, the network, trained on MS COCO (Lin et al., 2014), could be fine-tuned with new training data. The output of the network, assuming that the input is an RGB image of size m×n×3 , is a matrix of size m×n×l , where l is the number of objects in the image. For each output channel i∈l a binary mask is obtained. By combining all the channels into one, we can obtain the segmentation of all dynamic objects appearing in one image of the scene. 6.3.2 Low-Cost Tracking After the potentially dynamic content has been segmented, the pose of the camera is tracked using the static part of the image. Because the segment contours usually become highgradient areas, salient point features tend to appear. We do not consider the features in such contour areas. The tracking implemented at this stage of the algorithm is a simpler and therefore computationally lighter version of the one in ORB-SLAM2 (Mur-Artal and Tardós, 2017). It projects the map features in the image frame, searches for the correspondences in the static areas of the image, and minimizes the reprojection error to optimize the camera pose. 1https://github.com/matterport/Mask_RCNN 6.3 DynaSLAM System Description 117 (a) Keypoint x′ belongs to a static object ( z′= zpro j). (b) Keypoint x′ belongs to a dynamic object (z′≪zpro j). Fig. 6.3 Keypoint x from the Key Frame (KF) is projected into the Current Frame (CF) using its depth and camera pose, resulting in point x′ with depth z′ . The projected depth zpro j is then computed. A pixel is labeled as dynamic if the difference ∆z=zpro j −z′ is greater than a threshold τz. 6.3.3 Segmentation of Dynamic Content using Mask R-CNN and Multiview Geometry By using Mask R-CNN , most of the dynamic objects can be segmented and not used for tracking and mapping. However, there are objects that cannot be detected by this approach because they are not a priori dynamic, but movable. Examples of the latest are a book carried by someone, a chair that someone is moving, or even furniture changes in long-term mapping. The approach utilized for dealing with these cases is detailed in this section. For each input frame, we select the previous keyframes that have the highest overlaps. This is done by taking into account both the distance and the rotation between the new frame and each of the keyframes, similarly to Tan et al. (2013). The number of overlapping keyframes has been set to 5 in our experiments, as a compromise between computational cost and accuracy in the detection of dynamic objects. We then compute the projection of each keypoint x from the previous keyframes into the current frame, obtaining the keypoints x′ , as well as their projected depth zpro j , computed from the camera motion. Notice that the keypoints x come from the features extractor algorithm used in ORB-SLAM2 . For each keypoint, whose corresponding 3D point is X , we 124 Monocular and RGB-D SLAM on Dynamic Environments Sequence Depth Edge Motion Segmentation DSLAM Motion Removal DVO-SLAM DynaSLAM (N+G) (RGB-D) SLAM w/o Motion Detection w/ Motion Detection Improv. w/ MD w/o Motion Detection w/ Motion Detection Improv. w/ MD w/o Motion Detection w/ Motion Detection Improv. w/ MD [m] [m] [m] [%] [m] [m] [%] [m] [m] [%] w_hal f 0.049 0.116 0.055 52.59% 0.529 0.125 76.32% 0.351 0.025 92.88% w_xyz 0.060 0.202 0.040 80.20% 0.597 0.093 84.38% 0.459 0.015 96.73% w_rpy 0.179 0.515 0.076 85.24% 0.730 0.133 81.75% 0.662 0.035 94.71% w_stat 0.026 0.470 0.024 94.89% 0.212 0.066 69.06% 0.090 0.006 93.33% s_hal f 0.043 - - - 0.062 0.047 23.70% 0.020 0.017 15.00% s_xyz 0.040 - - - 0.051 0.048 4.55% 0.009 0.015 X Depth Edge SLAM by Li and Lee (2017). Motion Segmentation DSLAM by Wang and Huang (2014) Motion Removal DVO-SLAM by Sun et al. (2017) DynaSLAM w/o Motion Detection is ORB-SLAM2 by Mur-Artal and Tardós (2017) Table 6.3 Absolute trajectory RMSE [m] of DynaSLAM against state-of-the-art RGB-D SLAM systems in dynamic scenes. To evaluate the effectiveness of the specific module addressing dynamic content, we report the improvement with respect to the original SLAM systems (w/o Motion Detection). Our results are estimated using Mask R-CNN and multiview geometry. used in every case. DynaSLAM significantly outperforms all of them in all sequences (both high and low dynamic ones). The error is, in general, around 1-2 cm, similar to that of the state of the art in static scenes. Our motion detection approach also outperforms the other methods. ORB-SLAM, the monocular version of ORB-SLAM2, is generally more accurate than the RGB-D one in dynamic scenes, due to their different initialization algorithms. RGB-D ORB-SLAM2 is initialized and starts the tracking from the very first frame, and hence dynamic objects can introduce errors. ORB-SLAM delays the initialization until there is parallax and consensus using the staticity assumption. Hence, it does not track the camera for the full sequence, sometimes missing a substantial part of it, or even not initializing. Table 6.4 shows the tracking results and percentage of the tracked trajectory for ORBSLAM and DynaSLAM (monocular) in the TUM dataset. The initialization in DynaSLAM is always quicker than that of ORB-SLAM. In fact, in highly dynamic sequences, ORB-SLAM initialization only occurs when the moving objects disappear from the scene. In conclusion, although the accuracy of DynaSLAM is slightly lower, it succeeds in bootstrapping the system with dynamic content and producing a map without such content (see Fig. 6.1 ), to be re-used for long-term applications. The reason why DynaSLAM is slightly less accurate is that the estimated trajectory is longer, and there is therefore room for accumulating errors. 6.4 Experimental Results 125 Sequence ORB-SLAM DynaSLAM Mur-Artal and Tardós (2017) (Monocular) ATE [m] % Traj ATE [m] % Traj f r3/walking_hal f sphere 0.017 87.16 0.021 97.84 f r3/walking_xyz 0.012 57.63 0.014 87.37 f r2/desk_with_person 0.006 95.30 0.008 97.07 f r3/sitting_xyz 0.007 91.44 0.013 100.00 Table 6.4 Absolute trajectory RMSE [m] and percentage of successfully tracked trajectory for both ORB-SLAM and DynaSLAM (monocular). Sequence ORB-SLAM2 (Stereo) DynaSLAM (Stereo) Mur-Artal and Tardós (2017) RPE RRE ATE RPE RRE ATE [%] [◦/100m] [m] [%] [◦/100m] [m] KITTI 00 0.70 0.25 1.3 0.74 0.26 1.4 KITTI 01 1.39 0.21 10.4 1.57 0.22 9.4 KITTI 02 0.76 0.23 5.7 0.80 0.24 6.7 KITTI 03 0.71 0.18 0.6 0.69 0.18 0.6 KITTI 04 0.48 0.13 0.2 0.45 0.09 0.2 KITTI 05 0.40 0.16 0.8 0.40 0.16 0.8 KITTI 06 0.51 0.15 0.8 0.50 0.17 0.8 KITTI 07 0.50 0.28 0.5 0.52 0.29 0.5 KITTI 08 1.05 0.32 3.6 1.05 0.32 3.5 KITTI 09 0.87 0.27 3.2 0.93 0.29 1.6 KITTI 10 0.60 0.27 1.0 0.67 0.32 1.2 Table 6.5 Comparison of the RMSE of the ATE [m], the average of the RPE [ % ] and the RRE [◦/100m] of DynaSLAM against ORB-SLAM2 system for stereo cameras. 6.4.2 KITTI Dataset The KITTI Dataset (Geiger et al., 2013) contains stereo sequences recorded from a car in urban and highway environments. Table 6.5 shows our results in the eleven training sequences, compared against stereo ORB-SLAM2. We use two different metrics, the absolute trajectory RMSE proposed in Sturm et al. (2012b), and the average relative translation and rotation errors, proposed in Geiger et al. (2013). Table 6.6 shows the results in the same sequences for the monocular variants of ORB-SLAM and DynaSLAM. 126 Monocular and RGB-D SLAM on Dynamic Environments Sequence ORB-SLAM DynaSLAM (Monocular) Mur-Artal and Tardós (2017) KITTI 00 5.33 7.55 KITTI 02 21.28 26.29 KITTI 03 1.51 1.81 KITTI 04 1.62 0.97 KITTI 05 4.85 4.60 KITTI 06 12.34 14.74 KITTI 07 2.26 2.36 KITTI 08 46.68 40.28 KITTI 09 6.62 3.32 KITTI 10 8.80 6.78 Table 6.6 Absolute trajectory RMSE [m] for ORB-SLAM and DynaSLAM (monocular). Note that the results are similar in both the monocular and stereo cases, but the former is more sensitive to dynamic objects and therefore to the additions in DynaSLAM. In some sequences the accuracy of the tracking is improved when not using features belonging to a priori dynamic objects, i.e., cars, bicycles, etc. An example of this would be the sequences KITTI 01 and KITTI 04, in which all vehicles that appear are moving. In the sequences in which most of the recorded cars and vehicles are parked (hence static), the absolute trajectory RMSE is usually bigger since the keypoints used for tracking are more distant and usually belong to low-texture areas (KITTI 00, KITTI 02, KITTI 06). However, the loop closure and relocalization algorithms work more robustly since the resulting map only contains structural objects, i.e., the map can be re-used and work in long-term applications. As future work, it is interesting to make a distinction between those movable and moving objects, by using only RGB information. If a car is detected by the CNN (movable) but is not currently moving, its corresponding keypoints should be used for the local tracking, but should not be in the map. 6.4.3 Timing Analysis To complete the evaluation of our proposal, Table 6.7 shows the average computational time for its different stages. Note that DynaSLAM is not optimized for real-time operation. However, its capability for creating life-long maps of the static scene content are also relevant for running on offline mode. 6.5 Conclusions 127 Sequence Low-Cost Tracking [ms] Multi-view Geometry [ms] Background Inpainting [ms] w_hal f sphere 1.69 333.68 208.09 w_rpy 1.59 235.98 183.56 Table 6.7 DynaSLAM average computational time [ms]. Mur et al. show real-time results for and ORB-SLAM2 (Mur-Artal and Tardós, 2017). He et al. (2017) report that Mask R-CNN runs at 195 ms per image on a Nvidia Tesla M40 GPU. The addition of the multi-view geometry stage is an additional slowdown, due mainly to the region growth algorithm. The background inpainting also introduces a delay, which is another reason why it should be done after the tracking and mapping stage, as it has been shown in Fig. 6.2. 6.5 Conclusions We have presented a visual SLAM system that, building on ORB-SLAM, adds a motion segmentation approach that makes it robust in dynamic environments for monocular, stereo and RGB-D cameras, offering a solution to a very well known Visual SLAM problem. Our system accurately tracks the camera and creates a static and therefore reusable map of the scene. In the RGB-D case, DynaSLAM is capable of obtaining the synthetic RGB frames with no dynamic content and with the occluded background inpainted, as well as their corresponding synthesized depth frames, which might be together very useful for virtual reality applications. We include a video showing the potential of DynaSLAM 2. The comparison against the state of the art shows that DynaSLAM achieves in most cases the highest accuracy. In the videos of the TUM dataset that include Dynamic Objects dataset, at the moment of publishing this work DynaSLAM was the best RGB-D SLAM solution. Currently it is still the most accurate, although works like the one presented in Dai et al. (2020) offer a model that does not require GPU sacrificing on performance, the authors also claim that their proposal could be combined to some of the ideas presented in this chapter. Similar conclusions are found in Vincent et al. (2020). Being the computation time one of the biggest drawback of DynaSLAM according to these recent publication we think is interesting to share the work by Alonso et al. (2020) introducing MiniNet, a real-time semantic segmentation CNN; our proposal in this chapter is not dependent on the CNN we are currently using but it can use a different less time-consuming network as MiniNet. In the monocular case, our 2https://youtu.be/EabI_goFmQs 128 Monocular and RGB-D SLAM on Dynamic Environments accuracy is similar to that of ORB-SLAM, obtaining however a static map of the scene with an earlier initialization. In the KITTI dataset DynaSLAM is slightly less accurate than monocular and stereo ORB-SLAM, except for those cases in which dynamic objects represent an important part of the scene. However, our estimated map only contains structural objects and can therefore be re-used in long-term applications. Future works in this line of research have looked, real-time performance Dai et al. (2020), Vincent et al. (2020), Alonso et al. (2020), an RGB-based motion detector, or a more realistic appearance of the synthesized RGB frames by using a more elaborate inpainting technique, e.g., the one used by Pathak et al. (2016) by the use of Generative Adversarial Networks (GANs). This last idea was carried out, posterior to this work, by Bescos et al. (2019) where they use GANs to inpaint already detected dynamic objects. At the moment it has only been tested in simulation with ground-truth segmentation. Chapter 7 Conclusions In general, 3D visual perception is far from being fully solved. There are significant research challenges ahead, in particular related to scene understanding. In this thesis we have advanced the state of the art in several areas of this exciting and relevant topic. The first contribution described in this thesis is on single-view depth estimation. In CAMConvs (Facil et al., 2019) we have proposed a new type of convolution and demonstrated the advantages of accounting for the camera intrinsic parameters in depth estimation tasks. We have shown that our CAM-Convs allow us to train and test with different cameras; something that has been explored further in López-Antequera et al. (2020). Future work along this line should explore models that leverage the camera intrinsics and, unlike CAM-Convs, do not require to learn how to use them. A major drawback of CAM-Convs is that, as any learning procedure, they are strongly data-dependent. Therefore, a sufficiently well sampled dataset of images taken by different cameras is needed for a reasonable performance, and 1 or 2 cameras might not be enough. On this line, López-Antequera et al. (2020) has started making progress on their proposal of a canonical camera model, similar to the focal length normalization we use in our work. We have demonstrated in Chapter 3 that traditional multi-view geometry and deep learning can benefit from each other, achieving toghether an accuracy that outperforms both of them separately. It is worth remarking that our work was one of the first addressing this idea. After us, many novel approaches have been proposed. In particular, I would highlight two of them that couple deep learning and multi-view geometry quite tightly, instead of them being two different procedures subsequently merged. On the one side, CodeSLAM (Bloesch et al., 2018) and its following work DeepFactors (Czarnowski et al., 2020) successfully propose a deep neural network that defines a manifold for each depth map, in which traditional multi-view geometry optimization finds the best depth maps according to geometric and photo-metric errors. On the other hand, Zhou et al. (2018) presented DeepTAM, continuing their work 130 Conclusions DeMoN in Ummenhofer et al. (2017), proposing an deep neural network that iteratively refines its predictions using multi-view geometry, achieving an impressive accuracy. On visual place recognition, we have presented three novel approaches for multi-view global descriptors that are robust to changes in the appearance created by different conditions. We have tested it for multiple seasons and for different light conditions. It is also worth remarking that our work is the first one proposing multi-view descriptors based on deep learning, and that our descriptors were further explored in the dataset compiled in Warburg et al. (2020) with similar conclusions as in our work. For future research, it would be interesting to explore condition-invariant local descriptors that would allow to recover a metric pose and not only a topological one (Revaud et al., 2019). In Chapter 5 we presented CFL and EquiConvs. CFL is a network that achieves state-ofthe-art results in indoor layout recovery, and EquiConvs are a special type of convolutions that adapt is shape to the equirectangular distortion. Both contributions can have several applications in Visual SLAM (Salas et al., 2015). EquiConvs is also a general model, from which any deep network using panoramic images can benefit from. Regarding SLAM in dynamic environments, we have proposed a pipeline to avoid dynamic or movable objects to perturb mapping and tracking algorithms assuming a rigid world. Our main contribution is the design of the DynaSLAM pipeline and the inclusion of a segmentation CNN embedded in a visual SLAM system. We demonstrate that our results are very competitive, and that our proposal outperforms state-of-the-art SLAM systems. A reasonable line for future work would be to focus on tracking the dynamic objects and possibly use it also into its advantage. Preliminary results on this direction can already be seen in Ballester et al. (2020). A potential advantage could be, for example: if an object is being tracked and at some point the camera is occluded by it, the object motion estimation would allow a reasonable estimation of the camera motion for some time. We can draw a general conclusion for this thesis by writing that, on the one hand, we have proposed several novel methods to use deep networks for 3D perception challenges. And, on the other hand, we have also made contributions within deep learning for this particular domain. On the first set of proposals, we have developed novel methods to fuse multi-view and single-view depths and to detect and remove dynamic objects in visual SLAM. On the second set of proposals, we have developed novel multi-view embeddings for place recognition and two novel convolution types, CAM-Convs and Equiconvs, explicitly including the camera intrinsics and demonstrating better performances for single-view depth learning with multiple cameras and layout estimation from equirectangular images. 7.1 Limitations and Future Work 131 7.1 Limitations and Future Work Deep learning has supposed a great advance in 3D visual perception and it is making its way into visual SLAM. The biggest limitation we found while working on this thesis is the dependency on data, and more specifically on good-quality and diverse, sufficiently well sampled data. To exemplify this, look at the dense depth estimation problem. The progress achieved by using deep learning has no precedents. However, it is very easy to fall into small segments of the problem by evaluating the models in a subset of the real cases, e.g. a relatively small dataset on a very specific and biased domain. A common case is the training on different domains separately or, as we pointed in Chapter 2, commonly used datasets only provide images taken by one type of camera. This is different to traditional 3D vision algorithms that explicitly consider the camera model and do not make any assumption (or the smallest possible number of them) in the type of data or domain a priory. In this thesis we always kept this in mind. CAM-Convs introduce the camera model into convolutions for the first time. We also made use of unbiased depth estimation from a traditional geometrybased triangulation to complement the learned depth prediction. We have adapted standard convolutions to equirectangular distortion in EquiConvs, again taking into account the camera model. Lastly, we have combined deep learning and traditional methods for dynamic object detection in DynaSLAM. We agree with the general thought that deep learning has a great potential for 3D perception. However, future research needs to address the data dependency, creating more complete and general benchmarks (as López-Antequera et al. (2020)) and also models that account for the 3D-to-2D projection and the data noise. We can cite (Czarnowski et al., 2020, Zhou et al., 2020) as examples of the former, and Bayesian deep learning (for example, Gustafsson et al. (2020)) as a promising line of work for the latter. Learning from data might be the key for a complete scene understanding, but, in our believe, only those proposals that complement machine learning with uncertainty, geometric and physical models will achieve the best performance. References Abadi, Martín, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (2016), “Tensorflow: a system for large-scale machine learning.” In OSDI, volume 16, 265–283. Alcantarilla, Pablo F, José J Yebes, Javier Almazán, and Luis M Bergasa (2012), “On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments.” In ICRA. Alonso, Inigo, Luis Riazuelo, and Ana C Murillo (2020), “Mininet: An efficient semantic segmentation convnet for real-time robotic applications.” IEEE Transactions on Robotics. Ambrus, Rares, John Folkesson, and Patric Jensfelt (2016), “Unsupervised object segmentation through change detection in a long term autonomy scenario.” In Humanoid Robots (Humanoids), IEEE. Amodei, Dario, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. (2016), “Deep speech 2: End-to-end speech recognition in english and mandarin.” In International conference on machine learning, 173–182. Arandjelovic, Relja, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic (2016), “NetVLAD: CNN architecture for weakly supervised place recognition.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5297–5307. Arandjelovi´ c, Relja and Andrew Zisserman (2014), “Dislocation: Scalable descriptor distinctiveness for location recognition.” In Asian Conference on Computer Vision, 188–204, Springer. Armeni, I., A. Sax, A. R. Zamir, and S. Savarese (2017), “Joint 2D-3D-Semantic Data for Indoor Scene Understanding.” ArXiv. Armeni, Iro, Sasha Sax, Amir R Zamir, and Silvio Savarese (2017), “Joint 2D-3D-semantic data for indoor scene understanding.” arXiv preprint arXiv:1702.01105. Arroyo, Roberto, Pablo F Alcantarilla, Luis M Bergasa, and Eduardo Romera (2016), “Fusion and binarization of CNN features for robust topological localization across seasons.” In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, 4656–4663, IEEE. Ballester, Irene, Alejandro Fontan, Javier Civera, Klaus H Strobl, and Rudolph Triebel (2020), “Dot: Dynamic object tracking for visual slam.” arXiv preprint arXiv:2010.00052.